This is an automated email from the ASF dual-hosted git repository.
feiwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 2aa330c85 [KYUUBI #5892] Metadata store JDBC url supports
<KYUUBI_HOME> variable
2aa330c85 is described below
commit 2aa330c8501b82cf58e78f8ac2430b8821c83d60
Author: Cheng Pan <[email protected]>
AuthorDate: Thu Dec 21 14:19:24 2023 -0800
[KYUUBI #5892] Metadata store JDBC url supports <KYUUBI_HOME> variable
# :mag: Description
## Issue References ๐
Similar to https://github.com/apache/kyuubi/pull/5412, this PR aims to
change the default SQLite file `kyuubi_state_store.db` for Metadata store is
located under `$KYUUBI_HOME` instead of `$PWD`.
## Describe Your Solution ๐ง
Make `kyuubi.metadata.store.jdbc.url` supports the variables substitution:
`<KYUUBI_HOME>`, and change the default value to
`jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db`
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
Verified locally. Make a binary distribution, and bootstrap the Kyuubi
Server process from a non-KYUUBI_HOME dir, `kyuubi_state_store.db` is created
at KYUUBI_HOME as expected.
---
# Checklists
## ๐ Author Self Checklist
- [ ] My code follows the [style
guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html)
of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
## ๐ Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
Closes #5892 from pan3793/sqlite-jdbc-url.
Closes #5892
4341f4263 [Cheng Pan] docs
7062bcdf9 [Cheng Pan] migration guide
989de0e5b [Cheng Pan] Metadata store JDBC url supports <KYUUBI_HOME>
variable
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Fei Wang <[email protected]>
---
docs/configuration/settings.md | 2 +-
docs/deployment/migration-guide.md | 1 +
.../server/metadata/jdbc/JDBCMetadataStore.scala | 2 +-
.../metadata/jdbc/JDBCMetadataStoreConf.scala | 24 +++++++++++++++++-----
4 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index 299cb7052..b42bdcf10 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -362,7 +362,7 @@ You can configure the Kyuubi properties in
`$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.metadata.store.jdbc.driver | <undefined>
| JDBC driver class name for server jdbc metadata
store.
[...]
| kyuubi.metadata.store.jdbc.password
|| The password for server JDBC metadata store.
[...]
| kyuubi.metadata.store.jdbc.priority.enabled | false
| Whether to enable the priority scheduling for
batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO
ordering strategy for batch job scheduling. Note: this feature may cause
significant performance issues when using MySQL 5.7 as the metastore backend
due to the lack of support for mixed order index. See more details at KYUUBI
#5329. [...]
-| kyuubi.metadata.store.jdbc.url |
jdbc:sqlite:kyuubi_state_store.db | The JDBC url for
server JDBC metadata store. By default, it is a SQLite database url, and the
state information is not shared across kyuubi instances. To enable high
availability for multiple kyuubi instances, please specify a production JDBC
url.
[...]
+| kyuubi.metadata.store.jdbc.url |
jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for
server JDBC metadata store. By default, it is a SQLite database url, and the
state information is not shared across Kyuubi instances. To enable high
availability for multiple kyuubi instances, please specify a production JDBC
url. Note: this value support the variables substitution: `<KYUUBI_HOME>`.
[...]
| kyuubi.metadata.store.jdbc.user
|| The username for server JDBC metadata store.
[...]
### Metrics
diff --git a/docs/deployment/migration-guide.md
b/docs/deployment/migration-guide.md
index 2468bb155..9a099d585 100644
--- a/docs/deployment/migration-guide.md
+++ b/docs/deployment/migration-guide.md
@@ -24,6 +24,7 @@
## Upgrading from Kyuubi 1.8.0 to 1.8.1
* Since Kyuubi 1.8.1, for `DELETE /batches/${batchId}`,
`hive.server2.proxy.user` is not needed in the request parameters.
+* Since Kyuubi 1.8.1, the default SQLite file `kyuubi_state_store.db` for
Metadata store is located under `$KYUUBI_HOME` instead of `$PWD`. To restore
previous behavior, set `kyuubi.metadata.store.jdbc.url` to
`jdbc:sqlite:kyuubi_state_store.db`.
## Upgrading from Kyuubi 1.7 to 1.8
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
index 9b1c89d77..0a6d40229 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
@@ -74,7 +74,7 @@ class JDBCMetadataStore(conf: KyuubiConf) extends
MetadataStore with Logging {
JDBCMetadataStoreConf.getMetadataStoreJDBCDataSourceProperties(conf)
private val hikariConfig = new HikariConfig(datasourceProperties)
hikariConfig.setDriverClassName(driverClass)
- hikariConfig.setJdbcUrl(conf.get(METADATA_STORE_JDBC_URL))
+ hikariConfig.setJdbcUrl(getMetadataStoreJdbcUrl(conf))
hikariConfig.setUsername(conf.get(METADATA_STORE_JDBC_USER))
hikariConfig.setPassword(conf.get(METADATA_STORE_JDBC_PASSWORD))
hikariConfig.setPoolName("jdbc-metadata-store-pool")
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
index 96a5539fb..e2b06541d 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStoreConf.scala
@@ -19,12 +19,26 @@ package org.apache.kyuubi.server.metadata.jdbc
import java.util.Properties
+import org.apache.kyuubi.Utils
import org.apache.kyuubi.config.{ConfigEntry, KyuubiConf, OptionalConfigEntry}
import org.apache.kyuubi.config.KyuubiConf.buildConf
object JDBCMetadataStoreConf {
final val METADATA_STORE_JDBC_DATASOURCE_PREFIX =
"kyuubi.metadata.store.jdbc.datasource"
+ def getMetadataStoreJdbcUrl(conf: KyuubiConf): String = {
+ val rawJdbcUrl = conf.get(METADATA_STORE_JDBC_URL)
+ if (rawJdbcUrl.contains("<KYUUBI_HOME>")) {
+ rawJdbcUrl.replace(
+ "<KYUUBI_HOME>",
+ sys.env.getOrElse(
+ "KYUUBI_HOME",
+ Utils.getCodeSourceLocation(getClass).split("kyuubi-server").head))
+ } else {
+ rawJdbcUrl
+ }
+ }
+
/** Get metadata store jdbc datasource properties. */
def getMetadataStoreJDBCDataSourceProperties(conf: KyuubiConf): Properties =
{
val datasourceProperties = new Properties()
@@ -70,14 +84,14 @@ object JDBCMetadataStoreConf {
val METADATA_STORE_JDBC_URL: ConfigEntry[String] =
buildConf("kyuubi.metadata.store.jdbc.url")
- .doc("The JDBC url for server JDBC metadata store. By default, it is a
SQLite" +
- " database url, and the state information is not shared across kyuubi
instances. To" +
- " enable high availability for multiple kyuubi instances," +
- " please specify a production JDBC url.")
+ .doc("The JDBC url for server JDBC metadata store. By default, it is a
SQLite database " +
+ "url, and the state information is not shared across Kyuubi instances.
To enable high " +
+ "availability for multiple kyuubi instances, please specify a
production JDBC url. " +
+ "Note: this value support the variables substitution:
`<KYUUBI_HOME>`.")
.version("1.6.0")
.serverOnly
.stringConf
- .createWithDefault("jdbc:sqlite:kyuubi_state_store.db")
+ .createWithDefault("jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db")
val METADATA_STORE_JDBC_USER: ConfigEntry[String] =
buildConf("kyuubi.metadata.store.jdbc.user")
