This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 3167692732 [KYUUBI #6829] Add metrics for batch pending max elapse time
3167692732 is described below
commit 3167692732b7fe6ba3928db3ec89ca8419ed030d
Author: Wang, Fei <[email protected]>
AuthorDate: Thu Dec 5 18:12:39 2024 +0800
[KYUUBI #6829] Add metrics for batch pending max elapse time
### Why are the changes needed?
1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch
pending max elapse time, which is helpful for batch health monitoring, and we
can send alert if the batch pending elapse time too long
2. For `GET /api/v1/batches` api, limit the max time window for listing
batches, which is helpful that, we want to reserve more metadata in kyuubi
server end, for example: 90 days, but for list batches, we just want to allow
user to search the last 7 days. It is optional. And if `create_time` is
specified, order by `create_time` instead of `key_id`.
https://github.com/apache/kyuubi/blob/68a6f48da53dd0ad2e20b450a41ca600b8c1e1d2/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql#L32
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #6829 from turboFei/batch_pending_time.
Closes #6829
ee4f93125 [Wang, Fei] docs
bf8169ad4 [Wang, Fei] comments
f493a2af8 [Wang, Fei] new config
ab7b6db65 [Wang, Fei] ut
168017587 [Wang, Fei] in memory session
510a30b6a [Wang, Fei] batchSearchWindow opt
1e93dd276 [Wang, Fei] save
Authored-by: Wang, Fei <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
docs/configuration/settings.md | 37 ++++----
docs/monitor/metrics.md | 101 +++++++++++----------
.../org/apache/kyuubi/config/KyuubiConf.scala | 13 +++
.../apache/kyuubi/metrics/MetricsConstants.scala | 1 +
.../kyuubi/operation/BatchJobSubmission.scala | 8 ++
.../kyuubi/server/KyuubiRestFrontendService.scala | 15 ++-
.../kyuubi/server/api/v1/BatchesResource.scala | 5 +-
.../kyuubi/server/metadata/MetadataManager.scala | 9 +-
.../kyuubi/server/metadata/MetadataStore.scala | 5 +-
.../server/metadata/jdbc/JDBCMetadataStore.scala | 6 +-
.../server/api/v1/BatchesResourceSuite.scala | 1 +
11 files changed, 125 insertions(+), 76 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index fede369a84..acbc36197f 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -377,24 +377,25 @@ You can configure the Kyuubi properties in
`$KYUUBI_HOME/conf/kyuubi-defaults.co
### Metadata
-| Key |
Default |
Meaning
[...]
-|-------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
-| kyuubi.metadata.cleaner.enabled | true
| Whether to clean the metadata periodically. If
it is enabled, Kyuubi will clean the metadata that is in the terminate state
with max age limitation.
[...]
-| kyuubi.metadata.cleaner.interval | PT30M
| The interval to check and clean expired
metadata.
[...]
-| kyuubi.metadata.max.age | PT72H
| The maximum age of metadata, the metadata
exceeding the age will be cleaned.
[...]
-| kyuubi.metadata.recovery.threads | 10
| The number of threads for recovery from the
metadata store when the Kyuubi server restarts.
[...]
-| kyuubi.metadata.request.async.retry.enabled | true
| Whether to retry in async when metadata request
failed. When true, return success response immediately even the metadata
request failed, and schedule it in background until success, to tolerate
long-time metadata store outages w/o blocking the submission request.
[...]
-| kyuubi.metadata.request.async.retry.queue.size | 65536
| The maximum queue size for buffering metadata
requests in memory when the external metadata storage is down. Requests will be
dropped if the queue exceeds. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
-| kyuubi.metadata.request.async.retry.threads | 10
| Number of threads in the metadata request async
retry manager thread pool. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
-| kyuubi.metadata.request.retry.interval | PT5S
| The interval to check and trigger the metadata
request retry tasks.
[...]
-| kyuubi.metadata.store.class |
org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified
class name for server metadata store.
[...]
-| kyuubi.metadata.store.jdbc.database.schema.init | true
| Whether to init the JDBC metadata store
database schema.
[...]
-| kyuubi.metadata.store.jdbc.database.type | SQLITE
| The database type for server jdbc metadata
store.<ul> <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li> <li>MYSQL:
MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` (fallback
`com.mysql.jdbc.Driver`).</li> <li>POSTGRESQL: PostgreSQL, JDBC driver
`org.postgresql.Driver`.</li> <li>CUSTOM: User-defined database type, need to
specify corresponding JDBC driver.</li> Note that: The [...]
-| kyuubi.metadata.store.jdbc.driver | <undefined>
| JDBC driver class name for server jdbc metadata
store.
[...]
-| kyuubi.metadata.store.jdbc.password
|| The password for server JDBC metadata store.
[...]
-| kyuubi.metadata.store.jdbc.priority.enabled | false
| Whether to enable the priority scheduling for
batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO
ordering strategy for batch job scheduling. Note: this feature may cause
significant performance issues when using MySQL 5.7 as the metastore backend
due to the lack of support for mixed order index. See more details at KYUUBI
#5329. [...]
-| kyuubi.metadata.store.jdbc.url |
jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for
server JDBC metadata store. By default, it is a SQLite database url, and the
state information is not shared across Kyuubi instances. To enable high
availability for multiple kyuubi instances, please specify a production JDBC
url. Note: this value support the variables substitution: `<KYUUBI_HOME>`.
[...]
-| kyuubi.metadata.store.jdbc.user
|| The username for server JDBC metadata store.
[...]
+| Key |
Default |
Meaning
[...]
+|-------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| kyuubi.metadata.cleaner.enabled | true
| Whether to clean the metadata periodically. If
it is enabled, Kyuubi will clean the metadata that is in the terminate state
with max age limitation.
[...]
+| kyuubi.metadata.cleaner.interval | PT30M
| The interval to check and clean expired
metadata.
[...]
+| kyuubi.metadata.max.age | PT72H
| The maximum age of metadata, the metadata
exceeding the age will be cleaned.
[...]
+| kyuubi.metadata.recovery.threads | 10
| The number of threads for recovery from the
metadata store when the Kyuubi server restarts.
[...]
+| kyuubi.metadata.request.async.retry.enabled | true
| Whether to retry in async when metadata request
failed. When true, return success response immediately even the metadata
request failed, and schedule it in background until success, to tolerate
long-time metadata store outages w/o blocking the submission request.
[...]
+| kyuubi.metadata.request.async.retry.queue.size | 65536
| The maximum queue size for buffering metadata
requests in memory when the external metadata storage is down. Requests will be
dropped if the queue exceeds. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
+| kyuubi.metadata.request.async.retry.threads | 10
| Number of threads in the metadata request async
retry manager thread pool. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
+| kyuubi.metadata.request.retry.interval | PT5S
| The interval to check and trigger the metadata
request retry tasks.
[...]
+| kyuubi.metadata.search.window | <undefined>
| The time window to restrict user queries to
metadata within a specific period, starting from the current time to the past.
It only affects `GET /api/v1/batches` API. You may want to set this to short
period to improve query performance and reduce load on the metadata store when
administer want to reserve the metadata for long time. The side-effects is
that, the metadata created [...]
+| kyuubi.metadata.store.class |
org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified
class name for server metadata store.
[...]
+| kyuubi.metadata.store.jdbc.database.schema.init | true
| Whether to init the JDBC metadata store
database schema.
[...]
+| kyuubi.metadata.store.jdbc.database.type | SQLITE
| The database type for server jdbc metadata
store.<ul> <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li> <li>MYSQL:
MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` (fallback
`com.mysql.jdbc.Driver`).</li> <li>POSTGRESQL: PostgreSQL, JDBC driver
`org.postgresql.Driver`.</li> <li>CUSTOM: User-defined database type, need to
specify corresponding JDBC driver.</li> Note that: The [...]
+| kyuubi.metadata.store.jdbc.driver | <undefined>
| JDBC driver class name for server jdbc metadata
store.
[...]
+| kyuubi.metadata.store.jdbc.password
|| The password for server JDBC metadata store.
[...]
+| kyuubi.metadata.store.jdbc.priority.enabled | false
| Whether to enable the priority scheduling for
batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO
ordering strategy for batch job scheduling. Note: this feature may cause
significant performance issues when using MySQL 5.7 as the metastore backend
due to the lack of support for mixed order index. See more details at KYUUBI
#5329. [...]
+| kyuubi.metadata.store.jdbc.url |
jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for
server JDBC metadata store. By default, it is a SQLite database url, and the
state information is not shared across Kyuubi instances. To enable high
availability for multiple kyuubi instances, please specify a production JDBC
url. Note: this value support the variables substitution: `<KYUUBI_HOME>`.
[...]
+| kyuubi.metadata.store.jdbc.user
|| The username for server JDBC metadata store.
[...]
### Metrics
diff --git a/docs/monitor/metrics.md b/docs/monitor/metrics.md
index 561014c370..8043fa0810 100644
--- a/docs/monitor/metrics.md
+++ b/docs/monitor/metrics.md
@@ -40,56 +40,57 @@ The metrics system is configured via
`$KYUUBI_HOME/conf/kyuubi-defaults.conf`.
These metrics include:
-| Metrics Prefix | Metrics
Suffix | Type | Since |
Description
|
-|--------------------------------------------------|----------------------------------------|-----------|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `kyuubi.exec.pool.threads.alive` |
| gauge | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> threads keepAlive in the backend executive
thread pool</div>
|
-| `kyuubi.exec.pool.threads.active` |
| gauge | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> threads active in the backend executive thread
pool</div>
|
-| `kyuubi.exec.pool.work_queue.size` |
| gauge | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> work queue size in the backend executive
thread pool</div>
|
-| `kyuubi.connection.total` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative connection count</div>
|
-| `kyuubi.connection.total` | `${sessionType}`
| counter | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative connection count with session type
`${sessionType}`</div>
|
-| `kyuubi.connection.opened` |
| gauge | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current active connection count</div>
|
-| `kyuubi.connection.opened` | `${user}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current active connections count requested by
a `${user}`</div>
|
-| `kyuubi.connection.opened` |
`${user}`</br>`${sessionType}` | counter | 1.7.0 | <div style='width:
150pt;word-wrap: break-word;white-space: normal'> current active connections
count requested by a `${user}` with session type `${sessionType}`</div>
|
-| `kyuubi.connection.opened` | `${sessionType}`
| counter | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current active connections count with session
type `${sessionType}`</div>
|
-| `kyuubi.connection.failed` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative failed connection count</div>
|
-| `kyuubi.connection.failed` | `${user}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative failed connections for a
`${user}`</div>
|
-| `kyuubi.connection.failed` | `${sessionType}`
| counter | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative failed connection count with
session type `${sessionType}`</div>
|
-| `kyuubi.operation.total` |
| counter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative opened operation count</div>
|
-| `kyuubi.operation.total` | `${operationType}`
| counter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative opened count for the operation
`${operationType}`</div>
|
-| `kyuubi.operation.opened` |
| gauge | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current opened operation count</div>
|
-| `kyuubi.operation.opened` | `${operationType}`
| counter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current opened count for the operation
`${operationType}`</div>
|
-| `kyuubi.operation.failed` |
`${operationType}`<br/>`.${errorType}` | counter | 1.5.0 | <div style='width:
150pt;word-wrap: break-word;white-space: normal'> cumulative failed count for
the operation `${operationType}` with a particular `${errorType}`, e.g.
`execute_statement.AnalysisException`</div>
|
-| `kyuubi.operation.state` | `${operationState}`
| meter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi operation state rate </div>
|
-| `kyuubi.operation.exec_time` | `${operationType}`
| histogram | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> execution time histogram for the operation
`${operationType}`, now only `ExecuteStatement` is enabled. </div>
|
-| `kyuubi.engine.total` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative created engines</div>
|
-| `kyuubi.engine.timeout` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative timeout engines</div>
|
-| `kyuubi.engine.failed` | `${user}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative explicitly failed engine count for
a `${user}`</div>
|
-| `kyuubi.engine.failed` | `${errorType}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative explicitly failed engine count for
a particular `${errorType}`, e.g. `ClassNotFoundException`</div>
|
-| `kyuubi.backend_service.open_session` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `openSession` method
execution time and rate </div>
|
-| `kyuubi.backend_service.close_session` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `closeSession` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_info` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getInfo` method
execution time and rate </div>
|
-| `kyuubi.backend_service.execute_statement` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `executeStatement`
method execution time and rate </div>
|
-| `kyuubi.backend_service.get_type_info` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getTypeInfo` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_catalogs` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getCatalogs` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_schemas` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getSchemas` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_tables` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getTables` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_table_types` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getTableTypes` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_columns` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getColumns` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_functions` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getFunctions` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_operation_status` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getOperationStatus`
method execution time and rate </div>
|
-| `kyuubi.backend_service.cancel_operation` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `cancelOperation`
method execution time and rate </div>
|
-| `kyuubi.backend_service.close_operation` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `closeOperation` method
execution time and rate </div>
|
-| `kyuubi.backend_service.get_result_set_metadata` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getResultSetMetadata`
method execution time and rate </div>
|
-| `kyuubi.backend_service.fetch_results` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `fetchResults` method
execution time and rate </div>
|
-| `kyuubi.backend_service.fetch_log_rows_rate` |
| meter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `fetchResults` method
that fetch log rows rate </div>
|
-| `kyuubi.backend_service.fetch_result_rows_rate` |
| meter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `fetchResults` method
that fetch result rows rate </div>
|
-| `kyuubi.backend_service.get_primary_keys` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `get_primary_keys`
method execution time and rate </div>
|
-| `kyuubi.backend_service.get_cross_reference` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `get_cross_reference`
method execution time and rate </div>
|
-| `kyuubi.operation.state` |
`${operationType}`<br/>`.${state}` | meter | 1.6.0 | <div style='width:
150pt;word-wrap: break-word;white-space: normal'> The `${operationType}` with
a particular `${state}` rate, e.g. `BatchJobSubmission.pending`,
`BatchJobSubmission.finished`. Note that, the terminal states are cumulative,
but the intermediate ones are not. </div> |
-| `kyuubi.metadata.request.opened` |
| counter | 1.6.1 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current opened count for the metadata requests
</div>
|
-| `kyuubi.metadata.request.total` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> metadata requests time and rate </div>
|
-| `kyuubi.metadata.request.failed` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> metadata requests failure time and rate </div>
|
-| `kyuubi.metadata.request.retrying` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> retrying metadata requests time and rate, it
is not cumulative </div>
|
+| Metrics Prefix | Metrics
Suffix | Type | Since |
Description
|
+|--------------------------------------------------|----------------------------------------|-----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `kyuubi.exec.pool.threads.alive` |
| gauge | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> threads keepAlive in the backend executive
thread pool</div>
|
+| `kyuubi.exec.pool.threads.active` |
| gauge | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> threads active in the backend executive thread
pool</div>
|
+| `kyuubi.exec.pool.work_queue.size` |
| gauge | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> work queue size in the backend executive
thread pool</div>
|
+| `kyuubi.connection.total` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative connection count</div>
|
+| `kyuubi.connection.total` | `${sessionType}`
| counter | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative connection count with session type
`${sessionType}`</div>
|
+| `kyuubi.connection.opened` |
| gauge | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current active connection count</div>
|
+| `kyuubi.connection.opened` | `${user}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current active connections count requested by
a `${user}`</div>
|
+| `kyuubi.connection.opened` |
`${user}`</br>`${sessionType}` | counter | 1.7.0 | <div
style='width: 150pt;word-wrap: break-word;white-space: normal'> current active
connections count requested by a `${user}` with session type
`${sessionType}`</div>
|
+| `kyuubi.connection.opened` | `${sessionType}`
| counter | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current active connections count with session
type `${sessionType}`</div>
|
+| `kyuubi.connection.failed` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative failed connection count</div>
|
+| `kyuubi.connection.failed` | `${user}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative failed connections for a
`${user}`</div>
|
+| `kyuubi.connection.failed` | `${sessionType}`
| counter | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative failed connection count with
session type `${sessionType}`</div>
|
+| `kyuubi.operation.total` |
| counter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative opened operation count</div>
|
+| `kyuubi.operation.total` | `${operationType}`
| counter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative opened count for the operation
`${operationType}`</div>
|
+| `kyuubi.operation.opened` |
| gauge | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current opened operation count</div>
|
+| `kyuubi.operation.opened` | `${operationType}`
| counter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current opened count for the operation
`${operationType}`</div>
|
+| `kyuubi.operation.failed` |
`${operationType}`<br/>`.${errorType}` | counter | 1.5.0 | <div
style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative
failed count for the operation `${operationType}` with a particular
`${errorType}`, e.g. `execute_statement.AnalysisException`</div>
|
+| `kyuubi.operation.state` | `${operationState}`
| meter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi operation state rate </div>
|
+| `kyuubi.operation.exec_time` | `${operationType}`
| histogram | 1.7.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> execution time histogram for the operation
`${operationType}`, now only `ExecuteStatement` is enabled. </div>
|
+| `kyuubi.engine.total` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative created engines</div>
|
+| `kyuubi.engine.timeout` |
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative timeout engines</div>
|
+| `kyuubi.engine.failed` | `${user}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative explicitly failed engine count for
a `${user}`</div>
|
+| `kyuubi.engine.failed` | `${errorType}`
| counter | 1.2.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> cumulative explicitly failed engine count for
a particular `${errorType}`, e.g. `ClassNotFoundException`</div>
|
+| `kyuubi.backend_service.open_session` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `openSession` method
execution time and rate </div>
|
+| `kyuubi.backend_service.close_session` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `closeSession` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_info` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getInfo` method
execution time and rate </div>
|
+| `kyuubi.backend_service.execute_statement` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `executeStatement`
method execution time and rate </div>
|
+| `kyuubi.backend_service.get_type_info` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getTypeInfo` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_catalogs` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getCatalogs` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_schemas` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getSchemas` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_tables` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getTables` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_table_types` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getTableTypes` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_columns` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getColumns` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_functions` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getFunctions` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_operation_status` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getOperationStatus`
method execution time and rate </div>
|
+| `kyuubi.backend_service.cancel_operation` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `cancelOperation`
method execution time and rate </div>
|
+| `kyuubi.backend_service.close_operation` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `closeOperation` method
execution time and rate </div>
|
+| `kyuubi.backend_service.get_result_set_metadata` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `getResultSetMetadata`
method execution time and rate </div>
|
+| `kyuubi.backend_service.fetch_results` |
| timer | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `fetchResults` method
execution time and rate </div>
|
+| `kyuubi.backend_service.fetch_log_rows_rate` |
| meter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `fetchResults` method
that fetch log rows rate </div>
|
+| `kyuubi.backend_service.fetch_result_rows_rate` |
| meter | 1.5.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `fetchResults` method
that fetch result rows rate </div>
|
+| `kyuubi.backend_service.get_primary_keys` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `get_primary_keys`
method execution time and rate </div>
|
+| `kyuubi.backend_service.get_cross_reference` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> kyuubi backend service `get_cross_reference`
method execution time and rate </div>
|
+| `kyuubi.operation.state` |
`${operationType}`<br/>`.${state}` | meter | 1.6.0 | <div
style='width: 150pt;word-wrap: break-word;white-space: normal'> The
`${operationType}` with a particular `${state}` rate, e.g.
`BatchJobSubmission.pending`, `BatchJobSubmission.finished`. Note that, the
terminal states are cumulative, but the intermediate ones are not. </div> |
+| `kyuubi.metadata.request.opened` |
| counter | 1.6.1 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> current opened count for the metadata requests
</div>
|
+| `kyuubi.metadata.request.total` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> metadata requests time and rate </div>
|
+| `kyuubi.metadata.request.failed` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> metadata requests failure time and rate </div>
|
+| `kyuubi.metadata.request.retrying` |
| meter | 1.6.0 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> retrying metadata requests time and rate, it
is not cumulative </div>
|
+| `kyuubi.operartion.batch_pending_max_elapse` |
| gauge | 1.10.1 | <div style='width: 150pt;word-wrap:
break-word;white-space: normal'> the batch pending max elapsed time on current
kyuubi instance </div>
|
Before v1.5.0, if you use these metrics:
- `kyuubi.statement.total`
diff --git
a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index e26033bf0f..a493d7c457 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -2023,6 +2023,19 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to restrict user queries to metadata within a
specific period, " +
+ "starting from the current time to the past. It only affects `GET
/api/v1/batches` API. " +
+ "You may want to set this to short period to improve query performance
and reduce load " +
+ "on the metadata store when administer want to reserve the metadata
for long time. " +
+ "The side-effects is that, the metadata created outside the window
will not be " +
+ "invisible to users. If it is undefined, all metadata will be visible
for users.")
+ .version("1.10.1")
+ .timeConf
+ .checkValue(_ > 0, "must be positive number")
+ .createOptional
+
val ENGINE_EXEC_WAIT_QUEUE_SIZE: ConfigEntry[Int] =
buildConf("kyuubi.backend.engine.exec.pool.wait.queue.size")
.doc("Size of the wait queue for the operation execution thread pool in
SQL engine" +
diff --git
a/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
b/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
index f615467f3f..336060e8f7 100644
---
a/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
+++
b/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
@@ -64,6 +64,7 @@ object MetricsConstants {
final val OPERATION_TOTAL: String = OPERATION + "total"
final val OPERATION_STATE: String = OPERATION + "state"
final val OPERATION_EXEC_TIME: String = OPERATION + "exec_time"
+ final val OPERATION_BATCH_PENDING_MAX_ELAPSE: String = OPERATION +
"batch_pending_max_elapse"
final private val BACKEND_SERVICE = KYUUBI + "backend_service."
final val BS_FETCH_LOG_ROWS_RATE = BACKEND_SERVICE + "fetch_log_rows_rate"
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
index 129fbc8d9a..aa17c5436a 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
@@ -424,6 +424,14 @@ class BatchJobSubmission(
Utils.deleteDirectoryRecursively(session.resourceUploadFolderPath.toFile)
}
}
+
+ def getPendingElapsedTime: Long = {
+ if (state == OperationState.PENDING) {
+ System.currentTimeMillis() - createTime
+ } else {
+ 0L
+ }
+ }
}
object BatchJobSubmission {
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
index 5b8b1686ad..b93dcf7b5b 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
@@ -31,12 +31,14 @@ import org.eclipse.jetty.servlet.{ErrorPageErrorHandler,
FilterHolder}
import org.apache.kyuubi.{KyuubiException, Utils}
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf._
+import
org.apache.kyuubi.metrics.MetricsConstants.OPERATION_BATCH_PENDING_MAX_ELAPSE
+import org.apache.kyuubi.metrics.MetricsSystem
import org.apache.kyuubi.server.api.v1.ApiRootResource
import org.apache.kyuubi.server.http.authentication.{AuthenticationFilter,
KyuubiHttpAuthenticationFactory}
import org.apache.kyuubi.server.ui.{JettyServer, JettyUtils}
import org.apache.kyuubi.service.{AbstractFrontendService, Serverable,
Service, ServiceUtils}
import org.apache.kyuubi.service.authentication.{AuthTypes, AuthUtils}
-import org.apache.kyuubi.session.{KyuubiSessionManager, SessionHandle}
+import org.apache.kyuubi.session.{KyuubiBatchSession, KyuubiSessionManager,
SessionHandle}
import org.apache.kyuubi.util.{JavaUtils, ThreadUtils}
import
org.apache.kyuubi.util.ThreadUtils.scheduleTolerableRunnableWithFixedDelay
@@ -202,6 +204,14 @@ class KyuubiRestFrontendService(override val serverable:
Serverable)
}
}
+ private def getBatchPendingMaxElapse(): Long = {
+ val batchPendingElapseTimes = sessionManager.allSessions().map {
+ case session: KyuubiBatchSession =>
session.batchJobSubmissionOp.getPendingElapsedTime
+ case _ => 0L
+ }
+ if (batchPendingElapseTimes.isEmpty) 0L else batchPendingElapseTimes.max
+ }
+
def waitForServerStarted(): Unit = {
// block until the HTTP server is started, otherwise, we may get
// the wrong HTTP server port -1
@@ -220,6 +230,9 @@ class KyuubiRestFrontendService(override val serverable:
Serverable)
isStarted.set(true)
startBatchChecker()
recoverBatchSessions()
+ MetricsSystem.tracing { ms =>
+ ms.registerGauge(OPERATION_BATCH_PENDING_MAX_ELAPSE,
getBatchPendingMaxElapse, 0)
+ }
} catch {
case e: Exception => throw new KyuubiException(s"Cannot start
$getName", e)
}
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
index e3e981abdc..499110cf68 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
@@ -67,6 +67,7 @@ private[v1] class BatchesResource extends ApiRequestContext
with Logging {
fe.getConf.get(ENGINE_SECURITY_ENABLED)
private lazy val resourceFileMaxSize =
fe.getConf.get(BATCH_RESOURCE_FILE_MAX_SIZE)
private lazy val extraResourceFileMaxSize =
fe.getConf.get(BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE)
+ private lazy val metadataSearchWindow =
fe.getConf.get(METADATA_SEARCH_WINDOW)
private def batchV2Enabled(reqConf: Map[String, String]): Boolean = {
fe.getConf.get(BATCH_SUBMITTER_ENABLED) &&
@@ -420,13 +421,15 @@ private[v1] class BatchesResource extends
ApiRequestContext with Logging {
s"The valid batch state can be one of the following:
${VALID_BATCH_STATES.mkString(",")}")
}
+ val createTimeFilter =
+ math.max(createTime, metadataSearchWindow.map(System.currentTimeMillis()
- _).getOrElse(0L))
val filter = MetadataFilter(
sessionType = SessionType.BATCH,
engineType = batchType,
username = batchUser,
state = batchState,
requestName = batchName,
- createTime = createTime,
+ createTime = createTimeFilter,
endTime = endTime)
val batches = sessionManager.getBatchesFromMetadataStore(filter, from,
size, desc)
new GetBatchesResponse(from, batches.size, batches.asJava)
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
index 8d939aefc0..9ca1948dec 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
@@ -139,8 +139,13 @@ class MetadataManager extends
AbstractService("MetadataManager") {
from: Int,
size: Int,
desc: Boolean = false): Seq[Batch] = {
- withMetadataRequestMetrics(_metadataStore.getMetadataList(filter, from,
size, desc)).map(
- buildBatch)
+ withMetadataRequestMetrics(_metadataStore.getMetadataList(
+ filter,
+ from,
+ size,
+ // if create_file field is set, order by create_time, which is faster,
otherwise by key_id
+ orderBy = if (filter.createTime > 0) Some("create_time") else
Some("key_id"),
+ direction = if (desc) "DESC" else "ASC")).map(buildBatch)
}
def countBatch(
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
index d350050f14..b9fb52f779 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
@@ -58,13 +58,16 @@ trait MetadataStore extends Closeable {
* @param from the metadata offset.
* @param size the size to get.
* @param desc the order of metadata list.
+ * @param orderBy the order by column, default is the auto increment primary
key, `key_id`.
+ * @param direction the order direction, default is `ASC`.
* @return selected metadata list.
*/
def getMetadataList(
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata]
+ orderBy: Option[String] = Some("key_id"),
+ direction: String = "ASC"): Seq[Metadata]
/**
* Count the metadata list with filter conditions.
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
index e3db01f87e..af965a220f 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
@@ -257,15 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends
MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ orderBy: Option[String] = Some("key_id"),
+ direction: String = "ASC"): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
- queryBuilder.append(if (desc) "DESC " else "ASC ")
+ orderBy.foreach(o => queryBuilder.append(s" ORDER BY $o $direction "))
queryBuilder.append(dialect.limitClause(size, from))
val query = queryBuilder.toString
JdbcUtils.withConnection { connection =>
diff --git
a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
index 1a376e93e5..4d32de7026 100644
---
a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
+++
b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
@@ -486,6 +486,7 @@ abstract class BatchesResourceSuiteBase extends
KyuubiFunSuite
.queryParam("from", "0")
.queryParam("size", "1")
.queryParam("desc", "true")
+ .queryParam("createTime", "1")
.request(MediaType.APPLICATION_JSON_TYPE)
.header(AUTHORIZATION_HEADER, basicAuthorizationHeader("anonymous"))
.get()
