This is an automated email from the ASF dual-hosted git repository. csy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 02e1aa306 [KYUUBI #6387] [TRINO] Trino engine improve operation log
02e1aa306 is described below
commit 02e1aa30636f3c5bea36e54d90986935fcafa2e3
Author: senmiaoliu <[email protected]>
AuthorDate: Wed May 15 16:38:54 2024 +0800
[KYUUBI #6387] [TRINO] Trino engine improve operation log
trino engine increment mode support fetch log
# :mag: Description
## Issue References ๐
This pull request fixes #
## Describe Your Solution ๐ง
1. Trino engine supports outputting final status information, and we can
also support outputting running information.
2. In increment mode, the operation status is set to FINISHED quickly,
which prevents the client from fetching detailed operation logs. The client is
unaware of the exact execution status of the operation. It should be possible
to explicitly check if the ResultSet has more data (hasNext), and delay setting
the status to output more operation information.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
- [ ] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6387 from lsm1/branch-trino-print-info.
Closes #6387
b9b733a2f [senmiaoliu] trino engine output running info trino engine
increment mode support fetch log
Authored-by: senmiaoliu <[email protected]>
Signed-off-by: Shaoyun Chen <[email protected]>
---
docs/configuration/settings.md | 105 +++++++++++----------
.../kyuubi/engine/trino/TrinoStatement.scala | 20 +++-
.../kyuubi/engine/trino/TrinoStatusPrinter.scala | 10 +-
.../engine/trino/operation/ExecuteStatement.scala | 3 +
.../org/apache/kyuubi/config/KyuubiConf.scala | 8 ++
5 files changed, 91 insertions(+), 55 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index c3231cbb0..3d4177e86 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -449,58 +449,59 @@ You can configure the Kyuubi properties in
`$KYUUBI_HOME/conf/kyuubi-defaults.co
### Session
-| Key | Default
|
Meaning
[...]
-|---------------------------------------------------------|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
-| kyuubi.session.check.interval | PT5M
| The check interval for session timeout.
[...]
-| kyuubi.session.close.on.disconnect | true
| Session will be closed when client disconnects from kyuubi gateway. Set
this to false to have session outlive its parent connection.
[...]
-| kyuubi.session.conf.advisor | <undefined>
| A config advisor plugin for Kyuubi Server. This plugin can provide a
list of custom configs for different users or session configs and overwrite the
session configs before opening a new session. This config value should be a
subclass of `org.apache.kyuubi.plugin.SessionConfAdvisor` which has a zero-arg
constructor.
[...]
-| kyuubi.session.conf.file.reload.interval | PT10M
| When `FileSessionConfAdvisor` is used, this configuration defines the
expired time of `$KYUUBI_CONF_DIR/kyuubi-session-<profile>.conf` in the cache.
After exceeding this value, the file will be reloaded.
[...]
-| kyuubi.session.conf.ignore.list
|| A comma-separated list of ignored keys. If the client connection
contains any of them, the key and the corresponding value will be removed
silently during engine bootstrap and connection setup. Note that this rule is
for server-side protection defined via administrators to prevent some essential
configs from tampering but will not forbid users to set dynamic configurations
via SET syntax. [...]
-| kyuubi.session.conf.profile | <undefined>
| Specify a profile to load session-level configurations from
`$KYUUBI_CONF_DIR/kyuubi-session-<profile>.conf`. This configuration will be
ignored if the file does not exist. This configuration only takes effect when
`kyuubi.session.conf.advisor` is set as
`org.apache.kyuubi.session.FileSessionConfAdvisor`.
[...]
-| kyuubi.session.conf.restrict.list
|| A comma-separated list of restricted keys. If the client connection
contains any of them, the connection will be rejected explicitly during engine
bootstrap and connection setup. Note that this rule is for server-side
protection defined via administrators to prevent some essential configs from
tampering but will not forbid users to set dynamic configurations via SET
syntax. [...]
-| kyuubi.session.engine.alive.max.failures | 3
| The maximum number of failures allowed for the engine.
[...]
-| kyuubi.session.engine.alive.probe.enabled | false
| Whether to enable the engine alive probe, it true, we will create a
companion thrift client that keeps sending simple requests to check whether the
engine is alive.
[...]
-| kyuubi.session.engine.alive.probe.interval | PT10S
| The interval for engine alive probe.
[...]
-| kyuubi.session.engine.alive.timeout | PT2M
| The timeout for engine alive. If there is no alive probe success in the
last timeout window, the engine will be marked as no-alive.
[...]
-| kyuubi.session.engine.check.interval | PT1M
| The check interval for engine timeout
[...]
-| kyuubi.session.engine.flink.fetch.timeout | <undefined>
| Result fetch timeout for Flink engine. If the timeout is reached, the
result fetch would be stopped and the current fetched would be returned. If no
data are fetched, a TimeoutException would be thrown.
[...]
-| kyuubi.session.engine.flink.initialize.sql
|| The initialize sql for Flink session. It fallback to
`kyuubi.engine.session.initialize.sql`
[...]
-| kyuubi.session.engine.flink.main.resource | <undefined>
| The package used to create Flink SQL engine remote job. If it is
undefined, Kyuubi will use the default
[...]
-| kyuubi.session.engine.flink.max.rows | 1000000
| Max rows of Flink query results. For batch queries, rows exceeding the
limit would be ignored. For streaming queries, the query would be canceled if
the limit is reached.
[...]
-| kyuubi.session.engine.hive.main.resource | <undefined>
| The package used to create Hive engine remote job. If it is undefined,
Kyuubi will use the default
[...]
-| kyuubi.session.engine.idle.timeout | PT30M
| engine timeout, the engine will self-terminate when it's not accessed
for this duration. 0 or negative means not to self-terminate.
[...]
-| kyuubi.session.engine.initialize.timeout | PT3M
| Timeout for starting the background engine, e.g. SparkSQLEngine.
[...]
-| kyuubi.session.engine.launch.async | true
| When opening kyuubi session, whether to launch the backend engine
asynchronously. When true, the Kyuubi server will set up the connection with
the client without delay as the backend engine will be created asynchronously.
[...]
-| kyuubi.session.engine.log.timeout | PT24H
| If we use Spark as the engine then the session submit log is the console
output of spark-submit. We will retain the session submit log until over the
config value.
[...]
-| kyuubi.session.engine.login.timeout | PT15S
| The timeout of creating the connection to remote sql query engine
[...]
-| kyuubi.session.engine.open.max.attempts | 9
| The number of times an open engine will retry when encountering a
special error.
[...]
-| kyuubi.session.engine.open.onFailure | RETRY
| The behavior when opening engine failed: <ul> <li>RETRY: retry to open
engine for kyuubi.session.engine.open.max.attempts times.</li>
<li>DEREGISTER_IMMEDIATELY: deregister the engine immediately.</li>
<li>DEREGISTER_AFTER_RETRY: deregister the engine after retry to open engine
for kyuubi.session.engine.open.max.attempts times.</li></ul>
[...]
-| kyuubi.session.engine.open.retry.wait | PT10S
| How long to wait before retrying to open the engine after failure.
[...]
-| kyuubi.session.engine.share.level | USER
| (deprecated) - Using kyuubi.engine.share.level instead
[...]
-| kyuubi.session.engine.spark.initialize.sql
|| The initialize sql for Spark session. It fallback to
`kyuubi.engine.session.initialize.sql`
[...]
-| kyuubi.session.engine.spark.main.resource | <undefined>
| The package used to create Spark SQL engine remote application. If it is
undefined, Kyuubi will use the default
[...]
-| kyuubi.session.engine.spark.max.initial.wait | PT1M
| Max wait time for the initial connection to Spark engine. The engine
will self-terminate no new incoming connection is established within this time.
This setting only applies at the CONNECTION share level. 0 or negative means
not to self-terminate.
[...]
-| kyuubi.session.engine.spark.max.lifetime | PT0S
| Max lifetime for Spark engine, the engine will self-terminate when it
reaches the end of life. 0 or negative means not to self-terminate.
[...]
-| kyuubi.session.engine.spark.max.lifetime.gracefulPeriod | PT0S
| Graceful period for Spark engine to wait the connections disconnected
after reaching the end of life. After the graceful period, all the connections
without running operations will be forcibly disconnected. 0 or negative means
always waiting the connections disconnected.
[...]
-| kyuubi.session.engine.spark.progress.timeFormat | yyyy-MM-dd
HH:mm:ss.SSS | The time format of the progress bar
[...]
-| kyuubi.session.engine.spark.progress.update.interval | PT1S
| Update period of progress bar.
[...]
-| kyuubi.session.engine.spark.showProgress | false
| When true, show the progress bar in the Spark's engine log.
[...]
-| kyuubi.session.engine.startup.destroy.timeout | PT5S
| Engine startup process destroy wait time, if the process does not stop
after this time, force destroy instead. This configuration only takes effect
when `kyuubi.session.engine.startup.waitCompletion=false`.
[...]
-| kyuubi.session.engine.startup.error.max.size | 8192
| During engine bootstrapping, if anderror occurs, using this config to
limit the length of error message(characters).
[...]
-| kyuubi.session.engine.startup.maxLogLines | 10
| The maximum number of engine log lines when errors occur during the
engine startup phase. Note that this config effects on client-side to help
track engine startup issues.
[...]
-| kyuubi.session.engine.startup.waitCompletion | true
| Whether to wait for completion after the engine starts. If false, the
startup process will be destroyed after the engine is started. Note that only
use it when the driver is not running locally, such as in yarn-cluster mode;
Otherwise, the engine will be killed.
[...]
-| kyuubi.session.engine.trino.connection.catalog | <undefined>
| The default catalog that Trino engine will connect to
[...]
-| kyuubi.session.engine.trino.connection.url | <undefined>
| The server url that Trino engine will connect to
[...]
-| kyuubi.session.engine.trino.main.resource | <undefined>
| The package used to create Trino engine remote job. If it is undefined,
Kyuubi will use the default
[...]
-| kyuubi.session.engine.trino.showProgress | true
| When true, show the progress bar and final info in the Trino engine log.
[...]
-| kyuubi.session.engine.trino.showProgress.debug | false
| When true, show the progress debug info in the Trino engine log.
[...]
-| kyuubi.session.group.provider | hadoop
| A group provider plugin for Kyuubi Server. This plugin can provide
primary group and groups information for different users or session configs.
This config value should be a subclass of
`org.apache.kyuubi.plugin.GroupProvider` which has a zero-arg constructor.
Kyuubi provides the following built-in implementations: <li>hadoop: delegate
the user group mapping to hadoop UserGroupInformation.</li> [...]
-| kyuubi.session.idle.timeout | PT6H
| session idle timeout, it will be closed when it's not accessed for this
duration
[...]
-| kyuubi.session.local.dir.allow.list
|| The local dir list that are allowed to access by the kyuubi session
application. End-users might set some parameters such as `spark.files` and it
will upload some local files when launching the kyuubi engine, if the local
dir allow list is defined, kyuubi will check whether the path to upload is in
the allow list. Note that, if it is empty, there is no limitation for that. And
please use absolute path [...]
-| kyuubi.session.name | <undefined>
| A human readable name of the session and we use empty string by default.
This name will be recorded in the event. Note that, we only apply this value
from session conf.
[...]
-| kyuubi.session.proxy.user | <undefined>
| An alternative to hive.server2.proxy.user. The current behavior is
consistent with hive.server2.proxy.user and now only takes effect in RESTFul
API. When both parameters are set, kyuubi.session.proxy.user takes precedence.
[...]
-| kyuubi.session.timeout | PT6H
| (deprecated)session timeout, it will be closed when it's not accessed
for this duration
[...]
-| kyuubi.session.user.sign.enabled | false
| Whether to verify the integrity of session user name on the engine side,
e.g. Authz plugin in Spark.
[...]
+| Key | Default
|
Meaning
[...]
+|---------------------------------------------------------|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| kyuubi.session.check.interval | PT5M
| The check interval for session timeout.
[...]
+| kyuubi.session.close.on.disconnect | true
| Session will be closed when client disconnects from kyuubi gateway. Set
this to false to have session outlive its parent connection.
[...]
+| kyuubi.session.conf.advisor | <undefined>
| A config advisor plugin for Kyuubi Server. This plugin can provide a
list of custom configs for different users or session configs and overwrite the
session configs before opening a new session. This config value should be a
subclass of `org.apache.kyuubi.plugin.SessionConfAdvisor` which has a zero-arg
constructor.
[...]
+| kyuubi.session.conf.file.reload.interval | PT10M
| When `FileSessionConfAdvisor` is used, this configuration defines the
expired time of `$KYUUBI_CONF_DIR/kyuubi-session-<profile>.conf` in the cache.
After exceeding this value, the file will be reloaded.
[...]
+| kyuubi.session.conf.ignore.list
|| A comma-separated list of ignored keys. If the client connection
contains any of them, the key and the corresponding value will be removed
silently during engine bootstrap and connection setup. Note that this rule is
for server-side protection defined via administrators to prevent some essential
configs from tampering but will not forbid users to set dynamic configurations
via SET syntax. [...]
+| kyuubi.session.conf.profile | <undefined>
| Specify a profile to load session-level configurations from
`$KYUUBI_CONF_DIR/kyuubi-session-<profile>.conf`. This configuration will be
ignored if the file does not exist. This configuration only takes effect when
`kyuubi.session.conf.advisor` is set as
`org.apache.kyuubi.session.FileSessionConfAdvisor`.
[...]
+| kyuubi.session.conf.restrict.list
|| A comma-separated list of restricted keys. If the client connection
contains any of them, the connection will be rejected explicitly during engine
bootstrap and connection setup. Note that this rule is for server-side
protection defined via administrators to prevent some essential configs from
tampering but will not forbid users to set dynamic configurations via SET
syntax. [...]
+| kyuubi.session.engine.alive.max.failures | 3
| The maximum number of failures allowed for the engine.
[...]
+| kyuubi.session.engine.alive.probe.enabled | false
| Whether to enable the engine alive probe, it true, we will create a
companion thrift client that keeps sending simple requests to check whether the
engine is alive.
[...]
+| kyuubi.session.engine.alive.probe.interval | PT10S
| The interval for engine alive probe.
[...]
+| kyuubi.session.engine.alive.timeout | PT2M
| The timeout for engine alive. If there is no alive probe success in the
last timeout window, the engine will be marked as no-alive.
[...]
+| kyuubi.session.engine.check.interval | PT1M
| The check interval for engine timeout
[...]
+| kyuubi.session.engine.flink.fetch.timeout | <undefined>
| Result fetch timeout for Flink engine. If the timeout is reached, the
result fetch would be stopped and the current fetched would be returned. If no
data are fetched, a TimeoutException would be thrown.
[...]
+| kyuubi.session.engine.flink.initialize.sql
|| The initialize sql for Flink session. It fallback to
`kyuubi.engine.session.initialize.sql`
[...]
+| kyuubi.session.engine.flink.main.resource | <undefined>
| The package used to create Flink SQL engine remote job. If it is
undefined, Kyuubi will use the default
[...]
+| kyuubi.session.engine.flink.max.rows | 1000000
| Max rows of Flink query results. For batch queries, rows exceeding the
limit would be ignored. For streaming queries, the query would be canceled if
the limit is reached.
[...]
+| kyuubi.session.engine.hive.main.resource | <undefined>
| The package used to create Hive engine remote job. If it is undefined,
Kyuubi will use the default
[...]
+| kyuubi.session.engine.idle.timeout | PT30M
| engine timeout, the engine will self-terminate when it's not accessed
for this duration. 0 or negative means not to self-terminate.
[...]
+| kyuubi.session.engine.initialize.timeout | PT3M
| Timeout for starting the background engine, e.g. SparkSQLEngine.
[...]
+| kyuubi.session.engine.launch.async | true
| When opening kyuubi session, whether to launch the backend engine
asynchronously. When true, the Kyuubi server will set up the connection with
the client without delay as the backend engine will be created asynchronously.
[...]
+| kyuubi.session.engine.log.timeout | PT24H
| If we use Spark as the engine then the session submit log is the console
output of spark-submit. We will retain the session submit log until over the
config value.
[...]
+| kyuubi.session.engine.login.timeout | PT15S
| The timeout of creating the connection to remote sql query engine
[...]
+| kyuubi.session.engine.open.max.attempts | 9
| The number of times an open engine will retry when encountering a
special error.
[...]
+| kyuubi.session.engine.open.onFailure | RETRY
| The behavior when opening engine failed: <ul> <li>RETRY: retry to open
engine for kyuubi.session.engine.open.max.attempts times.</li>
<li>DEREGISTER_IMMEDIATELY: deregister the engine immediately.</li>
<li>DEREGISTER_AFTER_RETRY: deregister the engine after retry to open engine
for kyuubi.session.engine.open.max.attempts times.</li></ul>
[...]
+| kyuubi.session.engine.open.retry.wait | PT10S
| How long to wait before retrying to open the engine after failure.
[...]
+| kyuubi.session.engine.share.level | USER
| (deprecated) - Using kyuubi.engine.share.level instead
[...]
+| kyuubi.session.engine.spark.initialize.sql
|| The initialize sql for Spark session. It fallback to
`kyuubi.engine.session.initialize.sql`
[...]
+| kyuubi.session.engine.spark.main.resource | <undefined>
| The package used to create Spark SQL engine remote application. If it is
undefined, Kyuubi will use the default
[...]
+| kyuubi.session.engine.spark.max.initial.wait | PT1M
| Max wait time for the initial connection to Spark engine. The engine
will self-terminate no new incoming connection is established within this time.
This setting only applies at the CONNECTION share level. 0 or negative means
not to self-terminate.
[...]
+| kyuubi.session.engine.spark.max.lifetime | PT0S
| Max lifetime for Spark engine, the engine will self-terminate when it
reaches the end of life. 0 or negative means not to self-terminate.
[...]
+| kyuubi.session.engine.spark.max.lifetime.gracefulPeriod | PT0S
| Graceful period for Spark engine to wait the connections disconnected
after reaching the end of life. After the graceful period, all the connections
without running operations will be forcibly disconnected. 0 or negative means
always waiting the connections disconnected.
[...]
+| kyuubi.session.engine.spark.progress.timeFormat | yyyy-MM-dd
HH:mm:ss.SSS | The time format of the progress bar
[...]
+| kyuubi.session.engine.spark.progress.update.interval | PT1S
| Update period of progress bar.
[...]
+| kyuubi.session.engine.spark.showProgress | false
| When true, show the progress bar in the Spark's engine log.
[...]
+| kyuubi.session.engine.startup.destroy.timeout | PT5S
| Engine startup process destroy wait time, if the process does not stop
after this time, force destroy instead. This configuration only takes effect
when `kyuubi.session.engine.startup.waitCompletion=false`.
[...]
+| kyuubi.session.engine.startup.error.max.size | 8192
| During engine bootstrapping, if anderror occurs, using this config to
limit the length of error message(characters).
[...]
+| kyuubi.session.engine.startup.maxLogLines | 10
| The maximum number of engine log lines when errors occur during the
engine startup phase. Note that this config effects on client-side to help
track engine startup issues.
[...]
+| kyuubi.session.engine.startup.waitCompletion | true
| Whether to wait for completion after the engine starts. If false, the
startup process will be destroyed after the engine is started. Note that only
use it when the driver is not running locally, such as in yarn-cluster mode;
Otherwise, the engine will be killed.
[...]
+| kyuubi.session.engine.trino.connection.catalog | <undefined>
| The default catalog that Trino engine will connect to
[...]
+| kyuubi.session.engine.trino.connection.url | <undefined>
| The server url that Trino engine will connect to
[...]
+| kyuubi.session.engine.trino.main.resource | <undefined>
| The package used to create Trino engine remote job. If it is undefined,
Kyuubi will use the default
[...]
+| kyuubi.session.engine.trino.progress.update.interval | PT1S
| Update period of progress bar.
[...]
+| kyuubi.session.engine.trino.showProgress | true
| When true, show the progress bar and final info in the Trino engine log.
[...]
+| kyuubi.session.engine.trino.showProgress.debug | false
| When true, show the progress debug info in the Trino engine log.
[...]
+| kyuubi.session.group.provider | hadoop
| A group provider plugin for Kyuubi Server. This plugin can provide
primary group and groups information for different users or session configs.
This config value should be a subclass of
`org.apache.kyuubi.plugin.GroupProvider` which has a zero-arg constructor.
Kyuubi provides the following built-in implementations: <li>hadoop: delegate
the user group mapping to hadoop UserGroupInformation.</li> [...]
+| kyuubi.session.idle.timeout | PT6H
| session idle timeout, it will be closed when it's not accessed for this
duration
[...]
+| kyuubi.session.local.dir.allow.list
|| The local dir list that are allowed to access by the kyuubi session
application. End-users might set some parameters such as `spark.files` and it
will upload some local files when launching the kyuubi engine, if the local
dir allow list is defined, kyuubi will check whether the path to upload is in
the allow list. Note that, if it is empty, there is no limitation for that. And
please use absolute path [...]
+| kyuubi.session.name | <undefined>
| A human readable name of the session and we use empty string by default.
This name will be recorded in the event. Note that, we only apply this value
from session conf.
[...]
+| kyuubi.session.proxy.user | <undefined>
| An alternative to hive.server2.proxy.user. The current behavior is
consistent with hive.server2.proxy.user and now only takes effect in RESTFul
API. When both parameters are set, kyuubi.session.proxy.user takes precedence.
[...]
+| kyuubi.session.timeout | PT6H
| (deprecated)session timeout, it will be closed when it's not accessed
for this duration
[...]
+| kyuubi.session.user.sign.enabled | false
| Whether to verify the integrity of session user name on the engine side,
e.g. Authz plugin in Spark.
[...]
### Spnego
diff --git
a/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatement.scala
b/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatement.scala
index 48a6d6259..51deee84a 100644
---
a/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatement.scala
+++
b/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatement.scala
@@ -17,6 +17,7 @@
package org.apache.kyuubi.engine.trino
+import java.util.{Timer, TimerTask}
import java.util.concurrent.Executors
import scala.annotation.tailrec
@@ -57,6 +58,8 @@ class TrinoStatement(
private lazy val showProcess = kyuubiConf.get(ENGINE_TRINO_SHOW_PROGRESS)
private lazy val showDebug = kyuubiConf.get(ENGINE_TRINO_SHOW_PROGRESS_DEBUG)
+ private lazy val timer = new Timer("refresh status info", true)
+
implicit val ec: ExecutionContext =
ExecutionContext.fromExecutor(Executors.newFixedThreadPool(dataProcessingPoolSize))
@@ -102,9 +105,10 @@ class TrinoStatement(
getData()
}
} else {
+ timer.cancel()
Verify.verify(trino.isFinished)
if (operationLog.isDefined && showProcess) {
- TrinoStatusPrinter.printFinalInfo(trino, operationLog.get,
showDebug)
+ TrinoStatusPrinter.printStatusInfo(trino, operationLog.get,
showDebug)
}
val finalStatus = trino.finalStatusInfo()
if (finalStatus.getError() != null) {
@@ -147,6 +151,20 @@ class TrinoStatement(
trinoContext.clientSession.set(builder.build())
}
+ def printStatusInfo(): Unit = {
+ if (operationLog.isDefined && showProcess) {
+ timer.schedule(
+ new TimerTask {
+ override def run(): Unit = {
+ if (trino.isRunning) {
+ TrinoStatusPrinter.printStatusInfo(trino, operationLog.get,
showDebug)
+ }
+ }
+ },
+ 500L,
+ kyuubiConf.get(KyuubiConf.ENGINE_TRINO_SHOW_PROGRESS_UPDATE_INTERVAL))
+ }
+ }
}
object TrinoStatement {
diff --git
a/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatusPrinter.scala
b/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatusPrinter.scala
index c6bc6732e..2654f5413 100644
---
a/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatusPrinter.scala
+++
b/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/TrinoStatusPrinter.scala
@@ -32,12 +32,18 @@ import org.apache.kyuubi.operation.log.OperationLog
*/
object TrinoStatusPrinter {
- def printFinalInfo(
+ def printStatusInfo(
client: StatementClient,
operationLog: OperationLog,
debug: Boolean = false): Unit = {
val out = new TrinoConsoleProgressBar(operationLog)
- val results = client.finalStatusInfo()
+ val results =
+ if (client.isRunning) {
+ client.currentStatusInfo()
+ } else {
+ client.finalStatusInfo()
+ }
+
val stats = results.getStats
val wallTime = Duration.succinctDuration(stats.getElapsedTimeMillis(),
MILLISECONDS)
diff --git
a/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/operation/ExecuteStatement.scala
b/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/operation/ExecuteStatement.scala
index 3de2ae59f..0ba2297c3 100644
---
a/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/operation/ExecuteStatement.scala
+++
b/externals/kyuubi-trino-engine/src/main/scala/org/apache/kyuubi/engine/trino/operation/ExecuteStatement.scala
@@ -108,10 +108,13 @@ class ExecuteStatement(
private def executeStatement(trinoStatement: TrinoStatement): Unit = {
setState(OperationState.RUNNING)
try {
+ trinoStatement.printStatusInfo()
schema = trinoStatement.getColumns
val resultSet = trinoStatement.execute()
iter =
if (incrementalCollect) {
+ val hasResult = resultSet.hasNext
+ debug(s"Increment collect mode hasResult: $hasResult")
info("Execute in incremental collect mode")
FetchIterator.fromIterator(resultSet)
} else {
diff --git
a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index d8fec5833..344806dc9 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -1520,6 +1520,14 @@ object KyuubiConf {
.booleanConf
.createWithDefault(false)
+ val ENGINE_TRINO_SHOW_PROGRESS_UPDATE_INTERVAL: ConfigEntry[Long] =
+ buildConf("kyuubi.session.engine.trino.progress.update.interval")
+ .doc("Update period of progress bar.")
+ .version("1.10.0")
+ .timeConf
+ .checkValue(_ >= 200, "Minimum 200 milliseconds")
+ .createWithDefault(1000)
+
val ENGINE_HIVE_MAIN_RESOURCE: OptionalConfigEntry[String] =
buildConf("kyuubi.session.engine.hive.main.resource")
.doc("The package used to create Hive engine remote job. If it is
undefined," +
