This is an automated email from the ASF dual-hosted git repository.
csy pushed a commit to branch branch-1.8
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.8 by this push:
new dec8a129b [KYUUBI #5408] MetadataManager tries MySQL 8 driver class
first
dec8a129b is described below
commit dec8a129bbe295d3fca9aa6e37dee85138bcb8aa
Author: senmiaoliu <[email protected]>
AuthorDate: Wed Oct 18 12:55:00 2023 +0800
[KYUUBI #5408] MetadataManager tries MySQL 8 driver class first
### _Why are the changes needed?_
close #5408
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
NO
Closes #5433 from lsm1/branch-kyuubi-5408.
Closes #5408
8c6cc223d [senmiaoliu] fix style
20415dc44 [senmiaoliu] use com.mysql.cj.jdbc.Driver first
Authored-by: senmiaoliu <[email protected]>
Signed-off-by: Shaoyun Chen <[email protected]>
(cherry picked from commit 16752164a1e4f433241d7ee4735cf414c27f2a6d)
Signed-off-by: Shaoyun Chen <[email protected]>
---
docs/configuration/settings.md | 36 +++++++++++-----------
.../server/metadata/jdbc/JDBCMetadataStore.scala | 9 +++++-
.../metadata/jdbc/JDBCMetadataStoreConf.scala | 3 +-
3 files changed, 28 insertions(+), 20 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index 0dc4922f9..2bfee4d47 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -332,24 +332,24 @@ You can configure the Kyuubi properties in
`$KYUUBI_HOME/conf/kyuubi-defaults.co
### Metadata
-| Key |
Default |
Meaning
[...]
-|-------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
-| kyuubi.metadata.cleaner.enabled | true
| Whether to clean the metadata periodically. If
it is enabled, Kyuubi will clean the metadata that is in the terminate state
with max age limitation.
[...]
-| kyuubi.metadata.cleaner.interval | PT30M
| The interval to check and clean expired
metadata.
[...]
-| kyuubi.metadata.max.age | PT72H
| The maximum age of metadata, the metadata
exceeding the age will be cleaned.
[...]
-| kyuubi.metadata.recovery.threads | 10
| The number of threads for recovery from the
metadata store when the Kyuubi server restarts.
[...]
-| kyuubi.metadata.request.async.retry.enabled | true
| Whether to retry in async when metadata request
failed. When true, return success response immediately even the metadata
request failed, and schedule it in background until success, to tolerate
long-time metadata store outages w/o blocking the submission request.
[...]
-| kyuubi.metadata.request.async.retry.queue.size | 65536
| The maximum queue size for buffering metadata
requests in memory when the external metadata storage is down. Requests will be
dropped if the queue exceeds. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
-| kyuubi.metadata.request.async.retry.threads | 10
| Number of threads in the metadata request async
retry manager thread pool. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
-| kyuubi.metadata.request.retry.interval | PT5S
| The interval to check and trigger the metadata
request retry tasks.
[...]
-| kyuubi.metadata.store.class |
org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified
class name for server metadata store.
[...]
-| kyuubi.metadata.store.jdbc.database.schema.init | true
| Whether to init the JDBC metadata store
database schema.
[...]
-| kyuubi.metadata.store.jdbc.database.type | SQLITE
| The database type for server jdbc metadata
store.<ul> <li>(Deprecated) DERBY: Apache Derby, JDBC driver
`org.apache.derby.jdbc.AutoloadedDriver`.</li> <li>SQLITE: SQLite3, JDBC driver
`org.sqlite.JDBC`.</li> <li>MYSQL: MySQL, JDBC driver
`com.mysql.jdbc.Driver`.</li> <li>CUSTOM: User-defined database type, need to
specify corresponding JDBC driver.</li> Note that: The JDBC datas [...]
-| kyuubi.metadata.store.jdbc.driver | <undefined>
| JDBC driver class name for server jdbc metadata
store.
[...]
-| kyuubi.metadata.store.jdbc.password
|| The password for server JDBC metadata store.
[...]
-| kyuubi.metadata.store.jdbc.priority.enabled | false
| Whether to enable the priority scheduling for
batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO
ordering strategy for batch job scheduling. Note: this feature may cause
significant performance issues when using MySQL 5.7 as the metastore backend
due to the lack of support for mixed order index. See more details at KYUUBI
#5329. [...]
-| kyuubi.metadata.store.jdbc.url |
jdbc:sqlite:kyuubi_state_store.db | The JDBC url for
server JDBC metadata store. By default, it is a SQLite database url, and the
state information is not shared across kyuubi instances. To enable high
availability for multiple kyuubi instances, please specify a production JDBC
url.
[...]
-| kyuubi.metadata.store.jdbc.user
|| The username for server JDBC metadata store.
[...]
+| Key |
Default |
Meaning
[...]
+|-------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| kyuubi.metadata.cleaner.enabled | true
| Whether to clean the metadata periodically. If
it is enabled, Kyuubi will clean the metadata that is in the terminate state
with max age limitation.
[...]
+| kyuubi.metadata.cleaner.interval | PT30M
| The interval to check and clean expired
metadata.
[...]
+| kyuubi.metadata.max.age | PT72H
| The maximum age of metadata, the metadata
exceeding the age will be cleaned.
[...]
+| kyuubi.metadata.recovery.threads | 10
| The number of threads for recovery from the
metadata store when the Kyuubi server restarts.
[...]
+| kyuubi.metadata.request.async.retry.enabled | true
| Whether to retry in async when metadata request
failed. When true, return success response immediately even the metadata
request failed, and schedule it in background until success, to tolerate
long-time metadata store outages w/o blocking the submission request.
[...]
+| kyuubi.metadata.request.async.retry.queue.size | 65536
| The maximum queue size for buffering metadata
requests in memory when the external metadata storage is down. Requests will be
dropped if the queue exceeds. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
+| kyuubi.metadata.request.async.retry.threads | 10
| Number of threads in the metadata request async
retry manager thread pool. Only take affect when
kyuubi.metadata.request.async.retry.enabled is `true`.
[...]
+| kyuubi.metadata.request.retry.interval | PT5S
| The interval to check and trigger the metadata
request retry tasks.
[...]
+| kyuubi.metadata.store.class |
org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified
class name for server metadata store.
[...]
+| kyuubi.metadata.store.jdbc.database.schema.init | true
| Whether to init the JDBC metadata store
database schema.
[...]
+| kyuubi.metadata.store.jdbc.database.type | SQLITE
| The database type for server jdbc metadata
store.<ul> <li>(Deprecated) DERBY: Apache Derby, JDBC driver
`org.apache.derby.jdbc.AutoloadedDriver`.</li> <li>SQLITE: SQLite3, JDBC driver
`org.sqlite.JDBC`.</li> <li>MYSQL: MySQL, JDBC driver
`com.mysql.cj.jdbc.Driver` (fallback `com.mysql.jdbc.Driver`).</li> <li>CUSTOM:
User-defined database type, need to specify corresponding JDBC [...]
+| kyuubi.metadata.store.jdbc.driver | <undefined>
| JDBC driver class name for server jdbc metadata
store.
[...]
+| kyuubi.metadata.store.jdbc.password
|| The password for server JDBC metadata store.
[...]
+| kyuubi.metadata.store.jdbc.priority.enabled | false
| Whether to enable the priority scheduling for
batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO
ordering strategy for batch job scheduling. Note: this feature may cause
significant performance issues when using MySQL 5.7 as the metastore backend
due to the lack of support for mixed order index. See more details at KYUUBI
#5329. [...]
+| kyuubi.metadata.store.jdbc.url |
jdbc:sqlite:kyuubi_state_store.db | The JDBC url for
server JDBC metadata store. By default, it is a SQLite database url, and the
state information is not shared across kyuubi instances. To enable high
availability for multiple kyuubi instances, please specify a production JDBC
url.
[...]
+| kyuubi.metadata.store.jdbc.user
|| The username for server JDBC metadata store.
[...]
### Metrics
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
index 419fa8447..9b1c89d77 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
@@ -40,16 +40,23 @@ import org.apache.kyuubi.server.metadata.jdbc.DatabaseType._
import org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreConf._
import org.apache.kyuubi.session.SessionType
import org.apache.kyuubi.util.JdbcUtils
+import org.apache.kyuubi.util.reflect.ReflectUtils
class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
import JDBCMetadataStore._
private val dbType =
DatabaseType.withName(conf.get(METADATA_STORE_JDBC_DATABASE_TYPE))
private val driverClassOpt = conf.get(METADATA_STORE_JDBC_DRIVER)
+ private lazy val mysqlDriverClass =
+ if (ReflectUtils.isClassLoadable("com.mysql.cj.jdbc.Driver")) {
+ "com.mysql.cj.jdbc.Driver"
+ } else {
+ "com.mysql.jdbc.Driver"
+ }
private val driverClass = dbType match {
case SQLITE => driverClassOpt.getOrElse("org.sqlite.JDBC")
case DERBY =>
driverClassOpt.getOrElse("org.apache.derby.jdbc.AutoloadedDriver")
- case MYSQL => driverClassOpt.getOrElse("com.mysql.jdbc.Driver")
+ case MYSQL => driverClassOpt.getOrElse(mysqlDriverClass)
case CUSTOM => driverClassOpt.getOrElse(
throw new IllegalArgumentException("No jdbc driver defined"))
}
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
index 292cf4174..96a5539fb 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
@@ -40,7 +40,8 @@ object JDBCMetadataStoreConf {
" <li>(Deprecated) DERBY: Apache Derby, JDBC driver " +
"`org.apache.derby.jdbc.AutoloadedDriver`.</li>" +
" <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li>" +
- " <li>MYSQL: MySQL, JDBC driver `com.mysql.jdbc.Driver`.</li>" +
+ " <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` " +
+ "(fallback `com.mysql.jdbc.Driver`).</li>" +
" <li>CUSTOM: User-defined database type, need to specify
corresponding JDBC driver.</li>" +
" Note that: The JDBC datasource is powered by HiKariCP, for
datasource properties," +
" please specify them with the prefix:
kyuubi.metadata.store.jdbc.datasource." +
