This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new db0dbdedf7 [#13582] improvement(core): Make JDBC idle connection
retention configurable (#13620)
db0dbdedf7 is described below
commit db0dbdedf7f7c726e698ff75294f28c90bde40b1
Author: Qi Yu <[email protected]>
AuthorDate: Fri Oct 9 11:22:56 2026 +0800
[#13582] improvement(core): Make JDBC idle connection retention
configurable (#13620)
### What changes were proposed in this pull request?
Make JDBC idle-connection retention configurable for the entity-store
pool and for JDBC catalog pools.
- Entity store: add
`gravitino.entity.store.relational.maxIdleConnections` (default `10`,
the previous hard-coded value, capped by `maxConnections`) in place of
the hard-coded `maxIdle = 10` in `SqlSessionFactoryHelper`. `minIdle` is
capped by the effective `maxIdle`. The Helm chart exposes it as
`entity.maxIdleConnections`.
- JDBC catalogs: add the catalog property `jdbc.pool.max-idle` (default
`8`, DBCP's previous default, capped by `jdbc.pool.max-size`). When it
is not set, a `maxIdle` supplied through `gravitino.bypass.` is still
honored; when both are set, `jdbc.pool.max-idle` wins.
- Document both settings, including that connections are reused only up
to the idle limit, that entity-store idle connections are released
slowly (the evictor checks at most three idle connections every ten
minutes) and that catalog pools do not evict idle connections by
default.
- Add `dev/performance/jdbc_idle_retention.py`, a script that measures
MySQL connection churn and HTTP tail latency under concurrent reads.
### Why are the changes needed?
With `maxConnections = 100` but `maxIdle = 10`, a 64-client
metalake-list workload repeatedly destroyed and recreated database
connections. The measurements in #13582 compare `maxIdle = 10` with
`maxIdle = 100`: about 3,400 physical connections per measurement window
versus 4–36, and a median p99 of 214 ms versus 90 ms. The idle limit was
not configurable.
The churn comes from the pool closing a returned connection whenever it
already holds `maxIdle` idle connections; idle-time eviction plays no
part. Churn continues while the number of connections in use swings by
more than the limit, so the limit should be at or above the steady peak
concurrency.
Measured on PR head e1e1bc9cc5 with
`dev/performance/jdbc_idle_retention.py`: single server, MySQL 8.0.35,
512 MiB heap, 73 metalakes, 64 keep-alive clients on `GET
/api/metalakes`, a fresh server per setting, 3 rounds of 15 s each, run
in the order 10, 32, 64, 100, 32, 10. No request failed. Round 1
includes server warm-up, so the p99 column uses rounds 2-3.
| `maxIdleConnections` | New MySQL connections per round | p99, rounds
2-3 (ms) | Throughput (req/s) |
|---|---|---|---|
| 10 (two runs) | 1,367-2,726 | 211-234 | 2,000-3,400 |
| 32 (two runs) | 272-439 | 185-215 | 3,080-4,540 |
| 64 | 1-2 | 163-180 | 2,080-4,100 |
| 100 | 1-16 | 151-181 | 3,430-4,460 |
The default stays at `10`. Retained idle connections are multiplied by
the number of servers (typically three or more), and they are released
slowly, so a higher default would raise the connection budget of every
deployment. Deployments with high read concurrency can raise the limit
toward their peak concurrency within their database connection budget.
Out of scope, left for follow-ups: the DBCP pools of
`JdbcAuthorizationPlugin` (fixed `maxIdle = 5`) and of JDBC partition
statistics storage (`JdbcDataSourceFactory`, DBCP's default `8`) keep
their fixed idle limits; pool created/destroyed metrics for diagnosing
churn (MySQL's `Connections` counter and the script cover it for now),
and validating catalog pool properties before datasource creation so an
invalid value is reported by name rather than as "Error creating
datasource" (an existing behavior shared with `jdbc.pool.max-size`).
Both new settings accept only positive values on purpose; a catalog that
needs DBCP's `0` or `-1` can still use `gravitino.bypass.maxIdle`.
Fix: #13582
### Does this PR introduce _any_ user-facing change?
Yes.
- New server configuration
`gravitino.entity.store.relational.maxIdleConnections`. Its default
equals the previous hard-coded value `10`, so existing deployments are
unchanged. Idle connections are released slowly, so budget for the full
value on every server when raising it.
- New JDBC catalog property `jdbc.pool.max-idle`. Its default equals the
previous behavior, so existing catalogs are unchanged.
- New Helm value `entity.maxIdleConnections` (unset by default).
### How was this patch tested?
- Unit tests for the new settings, their caps and defaults, and the
`gravitino.bypass.maxIdle` precedence (tested at the `DataSourceUtils`
level; the prefix stripping in `JdbcCatalogOperations.initialize` is
unchanged): `TestSqlSession`, `TestConfig`, `TestJdbcConfig`,
`TestJdbcCatalogOperations` (including a burst test that checks physical
connections are reused), and `TestJdbcCatalogPropertiesMetadata`.
- Every test class that stubs the entity-store config, plus the full
`catalog-jdbc-common` and `idp-basic` suites, pass locally. Reverting
the `SqlSessionFactoryHelper` change makes
`testConfiguredMaxIdleAndPoolCap` and `testMinIdleCappedByMaxIdle` fail;
reverting the `DataSourceUtils` change makes `testConfigMaxIdle` and
`testRetainsConnectionsAcrossReadBursts` fail.
- `:core:compileJmhJava`, javadoc for `core` and `catalog-jdbc-common`,
and `spotlessApply`. The Helm chart tests are covered by CI
(`chart-test`); Helm is not installed locally.
- The benchmark script's self-test: `cd dev/performance && python3 -m
unittest test_jdbc_idle_retention`. It is a developer tool and is not
run in CI on purpose.
- `minIdle` is capped by the effective `maxIdle` so the DBCP getters
report the value the pool actually uses; commons-pool2 applies the same
cap internally, so behavior is unchanged.
`TestSqlSession.testMinIdleCappedByMaxIdle` and
`TestJdbcCatalogOperations.testConfigMaxIdle` pin the cap.
---------
Signed-off-by: yuqi <[email protected]>
---
.../fileset/TestFilesetCatalogOperations.java | 2 +
.../jdbc/JdbcCatalogPropertiesMetadata.java | 9 +
.../gravitino/catalog/jdbc/config/JdbcConfig.java | 21 +++
.../catalog/jdbc/utils/DataSourceUtils.java | 12 +-
.../catalog/jdbc/TestJdbcCatalogOperations.java | 91 ++++++++++
.../jdbc/TestJdbcCatalogPropertiesMetadata.java | 8 +
.../catalog/jdbc/config/TestJdbcConfig.java | 17 ++
.../catalog/kafka/TestKafkaCatalogOperations.java | 2 +
.../generic/TestGenericCatalogOperations.java | 2 +
.../catalog/model/TestModelCatalogOperations.java | 2 +
conf/gravitino.conf.template | 3 +
.../cache/it/AbstractEntityStorageBenchmark.java | 4 +
.../main/java/org/apache/gravitino/Configs.java | 16 ++
.../session/SqlSessionFactoryHelper.java | 12 +-
.../test/java/org/apache/gravitino/TestConfig.java | 14 ++
.../authorization/TestAccessControlManager.java | 2 +
.../gravitino/authorization/TestOwnerManager.java | 2 +
.../gravitino/hook/TestFilesetHookDispatcher.java | 3 +
.../apache/gravitino/policy/TestPolicyManager.java | 2 +
.../gravitino/stats/TestStatisticManager.java | 2 +
.../storage/AbstractEntityStorageTest.java | 2 +
.../storage/relational/BackendTestExtension.java | 2 +
...TestRelationalEntityStoreHierarchicalCache.java | 2 +
.../mapper/provider/base/TestAuthMappers.java | 2 +
.../provider/base/TestEntityChangeLogMapper.java | 2 +
.../provider/base/TestUserRoleRelMapper.java | 2 +
.../storage/relational/session/TestSqlSession.java | 64 +++++++
.../org/apache/gravitino/tag/TestTagManager.java | 2 +
.../gravitino/resources/config/gravitino.conf | 3 +
dev/charts/gravitino/tests/configmap_test.yaml | 19 ++
dev/charts/gravitino/values.yaml | 5 +
dev/performance/jdbc_idle_retention.py | 195 +++++++++++++++++++++
dev/performance/test_jdbc_idle_retention.py | 77 ++++++++
docs/gravitino-server-config.md | 29 +--
docs/how-to-use-relational-backend-storage.md | 21 +++
docs/jdbc-clickhouse-catalog.md | 19 +-
docs/jdbc-doris-catalog.md | 19 +-
docs/jdbc-hologres-catalog.md | 19 +-
docs/jdbc-mysql-catalog.md | 19 +-
docs/jdbc-oceanbase-catalog.md | 19 +-
docs/jdbc-postgresql-catalog.md | 21 +--
docs/jdbc-starrocks-catalog.md | 19 +-
42 files changed, 705 insertions(+), 83 deletions(-)
diff --git
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogOperations.java
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogOperations.java
index eeee30469d..6a9c49dfdd 100644
---
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogOperations.java
+++
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogOperations.java
@@ -24,6 +24,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
@@ -267,6 +268,7 @@ public class TestFilesetCatalogOperations {
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD)).thenReturn("gravitino");
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
File f = FileUtils.getFile(STORE_PATH);
diff --git
a/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/JdbcCatalogPropertiesMetadata.java
b/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/JdbcCatalogPropertiesMetadata.java
index ebb6a1bf31..062dd43888 100644
---
a/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/JdbcCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/JdbcCatalogPropertiesMetadata.java
@@ -45,6 +45,7 @@ public class JdbcCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
JdbcConfig.PASSWORD.getKey(),
JdbcConfig.POOL_MIN_SIZE.getKey(),
JdbcConfig.POOL_MAX_SIZE.getKey(),
+ JdbcConfig.POOL_MAX_IDLE.getKey(),
JdbcConfig.TEST_ON_BORROW.getKey());
static {
@@ -104,6 +105,14 @@ public class JdbcCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
JdbcConfig.POOL_MAX_SIZE.getDefaultValue(),
false /* hidden */,
false /* reserved */),
+ integerPropertyEntry(
+ JdbcConfig.POOL_MAX_IDLE.getKey(),
+ JdbcConfig.POOL_MAX_IDLE.getDoc(),
+ false /* required */,
+ false /* immutable */,
+ JdbcConfig.POOL_MAX_IDLE.getDefaultValue(),
+ false /* hidden */,
+ false /* reserved */),
booleanPropertyEntry(
JdbcConfig.TEST_ON_BORROW.getKey(),
JdbcConfig.TEST_ON_BORROW.getDoc(),
diff --git
a/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/config/JdbcConfig.java
b/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/config/JdbcConfig.java
index e474bcda39..753a9fac84 100644
---
a/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/config/JdbcConfig.java
+++
b/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/config/JdbcConfig.java
@@ -87,6 +87,18 @@ public class JdbcConfig extends Config {
.checkValue(value -> value > 0,
ConfigConstants.POSITIVE_NUMBER_ERROR_MSG)
.createWithDefault(10);
+ /**
+ * The maximum number of idle connections a catalog pool retains. The
default matches the DBCP
+ * default that catalog pools used before this setting existed.
+ */
+ public static final ConfigEntry<Integer> POOL_MAX_IDLE =
+ new ConfigBuilder("jdbc.pool.max-idle")
+ .doc("The maximum number of idle connections retained in the pool")
+ .version(ConfigConstants.VERSION_2_0_0)
+ .intConf()
+ .checkValue(value -> value > 0,
ConfigConstants.POSITIVE_NUMBER_ERROR_MSG)
+ .createWithDefault(8);
+
public static final ConfigEntry<Boolean> TEST_ON_BORROW =
new ConfigBuilder("jdbc.pool.test-on-borrow")
.doc("Whether to test the connection on borrow")
@@ -126,6 +138,15 @@ public class JdbcConfig extends Config {
return get(POOL_MAX_SIZE);
}
+ /**
+ * Returns the effective idle-connection limit, capped by the maximum pool
size.
+ *
+ * @return the maximum number of idle connections to retain
+ */
+ public int getPoolMaxIdle() {
+ return Math.min(get(POOL_MAX_IDLE), getPoolMaxSize());
+ }
+
public String getJdbcDatabase() {
return get(JDBC_DATABASE);
}
diff --git
a/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/utils/DataSourceUtils.java
b/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/utils/DataSourceUtils.java
index 7a7847ebf5..91c4a6bb4e 100644
---
a/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/utils/DataSourceUtils.java
+++
b/catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/utils/DataSourceUtils.java
@@ -152,7 +152,17 @@ public class DataSourceUtils {
String password = jdbcConfig.getPassword();
basicDataSource.setPassword(password);
basicDataSource.setMaxTotal(jdbcConfig.getPoolMaxSize());
- basicDataSource.setMinIdle(jdbcConfig.getPoolMinSize());
+ // Preserve maxIdle supplied through the existing DBCP bypass unless the
canonical setting is
+ // explicitly configured. The canonical default applies to catalogs
without either setting.
+ if
(jdbcConfig.getAllConfig().containsKey(JdbcConfig.POOL_MAX_IDLE.getKey())
+ || !jdbcConfig.getAllConfig().containsKey("maxIdle")) {
+ basicDataSource.setMaxIdle(jdbcConfig.getPoolMaxIdle());
+ }
+ int maxIdle = basicDataSource.getMaxIdle();
+ basicDataSource.setMinIdle(
+ maxIdle >= 0
+ ? Math.min(jdbcConfig.getPoolMinSize(), maxIdle)
+ : jdbcConfig.getPoolMinSize());
// Set each time a connection is taken out from the connection pool, a
test statement will be
// executed to confirm whether the connection is valid.
basicDataSource.setTestOnBorrow(jdbcConfig.getTestOnBorrow());
diff --git
a/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogOperations.java
b/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogOperations.java
index a070f101c6..abfe47cf0a 100644
---
a/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogOperations.java
+++
b/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogOperations.java
@@ -19,11 +19,18 @@
package org.apache.gravitino.catalog.jdbc;
import com.google.common.collect.Maps;
+import java.sql.Connection;
import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashMap;
+import java.util.IdentityHashMap;
import java.util.List;
+import java.util.Set;
import javax.sql.DataSource;
import org.apache.commons.dbcp2.BasicDataSource;
+import org.apache.commons.dbcp2.DelegatingConnection;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.catalog.jdbc.config.JdbcConfig;
import
org.apache.gravitino.catalog.jdbc.converter.SqliteColumnDefaultValueConverter;
@@ -81,6 +88,90 @@ public class TestJdbcCatalogOperations {
((BasicDataSource) dataSource).close();
}
+ @Test
+ public void testConfigMaxIdle() throws SQLException {
+ HashMap<String, String> properties = Maps.newHashMap();
+ properties.put(JdbcConfig.JDBC_DRIVER.getKey(), "org.sqlite.JDBC");
+ properties.put(JdbcConfig.JDBC_URL.getKey(), "jdbc:sqlite::memory:");
+ properties.put(JdbcConfig.USERNAME.getKey(), "test");
+ properties.put(JdbcConfig.PASSWORD.getKey(), "test");
+ properties.put(JdbcConfig.POOL_MAX_SIZE.getKey(), "16");
+ properties.put(JdbcConfig.POOL_MAX_IDLE.getKey(), "12");
+
+ try (BasicDataSource dataSource =
+ (BasicDataSource) DataSourceUtils.createDataSource(properties)) {
+ Assertions.assertEquals(16, dataSource.getMaxTotal());
+ Assertions.assertEquals(12, dataSource.getMaxIdle());
+ }
+
+ properties.remove(JdbcConfig.POOL_MAX_IDLE.getKey());
+ properties.put("maxIdle", "7");
+ try (BasicDataSource dataSource =
+ (BasicDataSource) DataSourceUtils.createDataSource(properties)) {
+ Assertions.assertEquals(7, dataSource.getMaxIdle());
+ }
+
+ properties.put(JdbcConfig.POOL_MAX_IDLE.getKey(), "12");
+ try (BasicDataSource dataSource =
+ (BasicDataSource) DataSourceUtils.createDataSource(properties)) {
+ Assertions.assertEquals(12, dataSource.getMaxIdle());
+ }
+
+ properties.remove(JdbcConfig.POOL_MAX_IDLE.getKey());
+ properties.put("maxIdle", "-1");
+ try (BasicDataSource dataSource =
+ (BasicDataSource) DataSourceUtils.createDataSource(properties)) {
+ Assertions.assertEquals(-1, dataSource.getMaxIdle());
+ Assertions.assertEquals(2, dataSource.getMinIdle());
+ }
+
+ // An idle limit below the minimum pool size also caps minIdle.
+ properties.remove("maxIdle");
+ properties.put(JdbcConfig.POOL_MAX_IDLE.getKey(), "1");
+ try (BasicDataSource dataSource =
+ (BasicDataSource) DataSourceUtils.createDataSource(properties)) {
+ Assertions.assertEquals(1, dataSource.getMaxIdle());
+ Assertions.assertEquals(1, dataSource.getMinIdle());
+ }
+ }
+
+ @Test
+ public void testRetainsConnectionsAcrossReadBursts() throws SQLException {
+ HashMap<String, String> properties = Maps.newHashMap();
+ properties.put(JdbcConfig.JDBC_DRIVER.getKey(), "org.sqlite.JDBC");
+ properties.put(JdbcConfig.JDBC_URL.getKey(), "jdbc:sqlite::memory:");
+ properties.put(JdbcConfig.USERNAME.getKey(), "test");
+ properties.put(JdbcConfig.PASSWORD.getKey(), "test");
+ properties.put(JdbcConfig.POOL_MAX_SIZE.getKey(), "16");
+ properties.put(JdbcConfig.POOL_MAX_IDLE.getKey(), "12");
+
+ try (BasicDataSource dataSource =
+ (BasicDataSource) DataSourceUtils.createDataSource(properties)) {
+ dataSource.setAccessToUnderlyingConnectionAllowed(true);
+ Set<Connection> physicalConnections = Collections.newSetFromMap(new
IdentityHashMap<>());
+ for (int round = 0; round < 5; round++) {
+ List<Connection> borrowed = new ArrayList<>();
+ try {
+ for (int i = 0; i < 12; i++) {
+ Connection connection = dataSource.getConnection();
+ borrowed.add(connection);
+ physicalConnections.add(((DelegatingConnection<?>)
connection).getInnermostDelegate());
+ try (Statement statement = connection.createStatement()) {
+ statement.execute("SELECT 1");
+ }
+ }
+ } finally {
+ for (Connection connection : borrowed) {
+ connection.close();
+ }
+ }
+ Assertions.assertEquals(12, dataSource.getNumIdle());
+ }
+ Assertions.assertFalse(physicalConnections.contains(null));
+ Assertions.assertEquals(12, physicalConnections.size());
+ }
+ }
+
@Test
public void testCloseDoesNotThrow() {
JdbcCatalogOperations catalogOperations =
diff --git
a/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogPropertiesMetadata.java
b/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogPropertiesMetadata.java
index 84099abd0c..1974134071 100644
---
a/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogPropertiesMetadata.java
@@ -45,6 +45,7 @@ public class TestJdbcCatalogPropertiesMetadata {
Assertions.assertFalse(propertiesMetadata.isHiddenProperty(JdbcConfig.POOL_MIN_SIZE.getKey()));
Assertions.assertFalse(propertiesMetadata.isHiddenProperty(JdbcConfig.POOL_MAX_SIZE.getKey()));
+
Assertions.assertFalse(propertiesMetadata.isHiddenProperty(JdbcConfig.POOL_MAX_IDLE.getKey()));
}
@Test
@@ -57,6 +58,7 @@ public class TestJdbcCatalogPropertiesMetadata {
.put(JdbcConfig.PASSWORD.getKey(), "password")
.put(JdbcConfig.POOL_MIN_SIZE.getKey(), "5")
.put(JdbcConfig.POOL_MAX_SIZE.getKey(), "20")
+ .put(JdbcConfig.POOL_MAX_IDLE.getKey(), "16")
.build();
CatalogEntity catalogEntity =
@@ -80,6 +82,12 @@ public class TestJdbcCatalogPropertiesMetadata {
Map<String, String> loadedProperties = loadCatalog.properties();
Assertions.assertEquals("5",
loadedProperties.get(JdbcConfig.POOL_MIN_SIZE.getKey()));
Assertions.assertEquals("20",
loadedProperties.get(JdbcConfig.POOL_MAX_SIZE.getKey()));
+ Assertions.assertEquals("16",
loadedProperties.get(JdbcConfig.POOL_MAX_IDLE.getKey()));
+ Assertions.assertEquals(
+ "16",
+ new JdbcCatalogPropertiesMetadata()
+ .transformProperties(loadedProperties)
+ .get(JdbcConfig.POOL_MAX_IDLE.getKey()));
}
private static class TestJdbcCatalog extends JdbcCatalog {
diff --git
a/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/config/TestJdbcConfig.java
b/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/config/TestJdbcConfig.java
index 194f82a76b..cfb19f0029 100644
---
a/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/config/TestJdbcConfig.java
+++
b/catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/config/TestJdbcConfig.java
@@ -31,4 +31,21 @@ public class TestJdbcConfig {
properties.put(JdbcConfig.JDBC_URL.getKey(), "jdbc:sqlite::memory:");
Assertions.assertDoesNotThrow(() -> new JdbcConfig(properties));
}
+
+ @Test
+ public void testMaxIdleDefaultsAndCap() {
+ JdbcConfig defaultConfig = new JdbcConfig(Maps.newHashMap());
+ Assertions.assertEquals(8, defaultConfig.getPoolMaxIdle());
+
+ HashMap<String, String> properties = Maps.newHashMap();
+ properties.put(JdbcConfig.POOL_MAX_SIZE.getKey(), "6");
+ Assertions.assertEquals(6, new JdbcConfig(properties).getPoolMaxIdle());
+
+ properties.put(JdbcConfig.POOL_MAX_IDLE.getKey(), "4");
+ Assertions.assertEquals(4, new JdbcConfig(properties).getPoolMaxIdle());
+
+ properties.put(JdbcConfig.POOL_MAX_IDLE.getKey(), "0");
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () -> new
JdbcConfig(properties).getPoolMaxIdle());
+ }
}
diff --git
a/catalogs/catalog-kafka/src/test/java/org/apache/gravitino/catalog/kafka/TestKafkaCatalogOperations.java
b/catalogs/catalog-kafka/src/test/java/org/apache/gravitino/catalog/kafka/TestKafkaCatalogOperations.java
index 0e0d9df1dc..a974515386 100644
---
a/catalogs/catalog-kafka/src/test/java/org/apache/gravitino/catalog/kafka/TestKafkaCatalogOperations.java
+++
b/catalogs/catalog-kafka/src/test/java/org/apache/gravitino/catalog/kafka/TestKafkaCatalogOperations.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
@@ -156,6 +157,7 @@ public class TestKafkaCatalogOperations extends
KafkaClusterEmbedded {
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD)).thenReturn("gravitino");
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
File f = FileUtils.getFile(STORE_PATH);
diff --git
a/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/generic/TestGenericCatalogOperations.java
b/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/generic/TestGenericCatalogOperations.java
index 9af6e3c189..6bed860b0b 100644
---
a/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/generic/TestGenericCatalogOperations.java
+++
b/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/generic/TestGenericCatalogOperations.java
@@ -24,6 +24,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
@@ -115,6 +116,7 @@ public class TestGenericCatalogOperations {
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD)).thenReturn("gravitino");
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
File f = FileUtils.getFile(STORE_PATH);
diff --git
a/catalogs/catalog-model/src/test/java/org/apache/gravtitino/catalog/model/TestModelCatalogOperations.java
b/catalogs/catalog-model/src/test/java/org/apache/gravtitino/catalog/model/TestModelCatalogOperations.java
index 34fcc5c900..4ef7a9f3be 100644
---
a/catalogs/catalog-model/src/test/java/org/apache/gravtitino/catalog/model/TestModelCatalogOperations.java
+++
b/catalogs/catalog-model/src/test/java/org/apache/gravtitino/catalog/model/TestModelCatalogOperations.java
@@ -24,6 +24,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
@@ -121,6 +122,7 @@ public class TestModelCatalogOperations {
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD)).thenReturn("gravitino");
when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
when(config.get(VERSION_RETENTION_COUNT)).thenReturn(1L);
diff --git a/conf/gravitino.conf.template b/conf/gravitino.conf.template
index f5732377fc..af8a6c0651 100644
--- a/conf/gravitino.conf.template
+++ b/conf/gravitino.conf.template
@@ -70,6 +70,9 @@ gravitino.entity.store.relational.jdbcDriver = org.h2.Driver
gravitino.entity.store.relational.jdbcUser = gravitino
# The JDBC password
gravitino.entity.store.relational.jdbcPassword = gravitino
+# Maximum idle entity-store connections retained by this server (capped by
maxConnections).
+# Budget for it on every server, plus any JDBC catalog pools that use the same
database instance.
+# gravitino.entity.store.relational.maxIdleConnections = 10
# THE CONFIGURATION FOR Gravitino CATALOG
# The interval in milliseconds to evict the catalog cache
diff --git
a/core/src/jmh/java/org/apache/gravitino/cache/it/AbstractEntityStorageBenchmark.java
b/core/src/jmh/java/org/apache/gravitino/cache/it/AbstractEntityStorageBenchmark.java
index ba733f5f16..16e8eed9c5 100644
---
a/core/src/jmh/java/org/apache/gravitino/cache/it/AbstractEntityStorageBenchmark.java
+++
b/core/src/jmh/java/org/apache/gravitino/cache/it/AbstractEntityStorageBenchmark.java
@@ -20,11 +20,13 @@
package org.apache.gravitino.cache.it;
import static org.apache.gravitino.Configs.DEFAULT_ENTITY_RELATIONAL_STORE;
+import static
org.apache.gravitino.Configs.DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_CLEANUP_INTERVAL_SECS;
import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
@@ -291,6 +293,8 @@ public class AbstractEntityStorageBenchmark<E extends
Entity & HasIdentifier> {
Mockito.when(config.get(ENTITY_RELATIONAL_STORE)).thenReturn(DEFAULT_ENTITY_RELATIONAL_STORE);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PATH)).thenReturn(DB_DIR);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS))
+ .thenReturn(DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
Mockito.when(config.get(ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS)).thenReturn(3L);
diff --git a/core/src/main/java/org/apache/gravitino/Configs.java
b/core/src/main/java/org/apache/gravitino/Configs.java
index 5c016854cf..108f316219 100644
--- a/core/src/main/java/org/apache/gravitino/Configs.java
+++ b/core/src/main/java/org/apache/gravitino/Configs.java
@@ -55,6 +55,10 @@ public class Configs {
public static final String
ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTION_KEYS =
"gravitino.entity.store.relational.maxConnections";
+ /** Configuration key for the entity-store pool's maximum idle connections.
*/
+ public static final String
ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS_KEY =
+ "gravitino.entity.store.relational.maxIdleConnections";
+
public static final String
ENTITY_RELATIONAL_JDBC_BACKEND_MAX_WAIT_MILLIS_CONNECTION_KEY =
"gravitino.entity.store.relational.maxWaitMillis";
@@ -96,6 +100,9 @@ public class Configs {
public static final int DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS =
100;
+ /** Default maximum idle connections retained by the entity-store pool on
each server. */
+ public static final int DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS
= 10;
+
public static final int DEFAULT_GRAVITINO_AUTHORIZATION_THREAD_POOL_SIZE =
100;
public static final int DEFAULT_BULK_MAX_ITEMS = 100;
@@ -162,6 +169,15 @@ public class Configs {
.intConf()
.createWithDefault(DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS);
+ /** Maximum idle connections retained by the entity-store pool, capped by
maxConnections. */
+ public static final ConfigEntry<Integer>
ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS =
+ new
ConfigBuilder(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS_KEY)
+ .doc("The maximum number of idle connections retained by the JDBC
Backend pool")
+ .version(ConfigConstants.VERSION_2_0_0)
+ .intConf()
+ .checkValue(value -> value > 0,
ConfigConstants.POSITIVE_NUMBER_ERROR_MSG)
+
.createWithDefault(DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS);
+
public static final ConfigEntry<Long>
ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS =
new
ConfigBuilder(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_WAIT_MILLIS_CONNECTION_KEY)
.doc(
diff --git
a/core/src/main/java/org/apache/gravitino/storage/relational/session/SqlSessionFactoryHelper.java
b/core/src/main/java/org/apache/gravitino/storage/relational/session/SqlSessionFactoryHelper.java
index 05c92c0769..2b52ec1964 100644
---
a/core/src/main/java/org/apache/gravitino/storage/relational/session/SqlSessionFactoryHelper.java
+++
b/core/src/main/java/org/apache/gravitino/storage/relational/session/SqlSessionFactoryHelper.java
@@ -48,7 +48,6 @@ import
org.apache.ibatis.transaction.jdbc.JdbcTransactionFactory;
* should be initialized only once.
*/
public class SqlSessionFactoryHelper {
- private static final int JDBC_BACKEND_MAX_IDLE_CONNECTIONS = 10;
private static final int JDBC_BACKEND_MIN_IDLE_CONNECTIONS = 5;
private static final Duration JDBC_BACKEND_MIN_EVICTABLE_IDLE_TIME =
Duration.ofSeconds(30);
@@ -93,9 +92,14 @@ public class SqlSessionFactoryHelper {
dataSource.setDefaultAutoCommit(false);
dataSource.setMaxWaitMillis(
config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS));
-
dataSource.setMaxTotal(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS));
- dataSource.setMaxIdle(JDBC_BACKEND_MAX_IDLE_CONNECTIONS);
- dataSource.setMinIdle(JDBC_BACKEND_MIN_IDLE_CONNECTIONS);
+ int maxConnections =
config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS);
+ int configuredMaxIdle =
+
config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS);
+ int maxIdle =
+ maxConnections > 0 ? Math.min(maxConnections, configuredMaxIdle) :
configuredMaxIdle;
+ dataSource.setMaxTotal(maxConnections);
+ dataSource.setMaxIdle(maxIdle);
+ dataSource.setMinIdle(Math.min(JDBC_BACKEND_MIN_IDLE_CONNECTIONS,
maxIdle));
dataSource.setLogAbandoned(true);
dataSource.setRemoveAbandonedOnBorrow(true);
dataSource.setRemoveAbandonedTimeout(60);
diff --git a/core/src/test/java/org/apache/gravitino/TestConfig.java
b/core/src/test/java/org/apache/gravitino/TestConfig.java
index 560e7f05d3..f7c03df4b7 100644
--- a/core/src/test/java/org/apache/gravitino/TestConfig.java
+++ b/core/src/test/java/org/apache/gravitino/TestConfig.java
@@ -173,4 +173,18 @@ public class TestConfig {
ConfigEntry<Boolean> boolEntry = new
ConfigBuilder("boolean.key").booleanConf();
Assertions.assertTrue(boolEntry.readFrom(props));
}
+
+ @Test
+ public void testEntityStoreMaxIdleConnections() {
+ Config config = new Config(false) {};
+ Assertions.assertEquals(
+ 10,
config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS));
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> {
+
config.set(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS, 0);
+
config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS);
+ });
+ }
}
diff --git
a/core/src/test/java/org/apache/gravitino/authorization/TestAccessControlManager.java
b/core/src/test/java/org/apache/gravitino/authorization/TestAccessControlManager.java
index ac0089fb80..01d92dd44d 100644
---
a/core/src/test/java/org/apache/gravitino/authorization/TestAccessControlManager.java
+++
b/core/src/test/java/org/apache/gravitino/authorization/TestAccessControlManager.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_STORE;
@@ -150,6 +151,7 @@ public class TestAccessControlManager {
.thenReturn(String.format("jdbc:h2:file:%s;DB_CLOSE_DELAY=-1;MODE=MYSQL",
DB_DIR));
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_TRANSACTION_MAX_SKEW_TIME)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/authorization/TestOwnerManager.java
b/core/src/test/java/org/apache/gravitino/authorization/TestOwnerManager.java
index 832b00c993..e17d9b6f10 100644
---
a/core/src/test/java/org/apache/gravitino/authorization/TestOwnerManager.java
+++
b/core/src/test/java/org/apache/gravitino/authorization/TestOwnerManager.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_STORE;
@@ -110,6 +111,7 @@ public class TestOwnerManager {
.thenReturn(String.format("jdbc:h2:file:%s;DB_CLOSE_DELAY=-1;MODE=MYSQL",
DB_DIR));
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_TRANSACTION_MAX_SKEW_TIME)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/hook/TestFilesetHookDispatcher.java
b/core/src/test/java/org/apache/gravitino/hook/TestFilesetHookDispatcher.java
index 1a47a83025..4037f5d2c7 100644
---
a/core/src/test/java/org/apache/gravitino/hook/TestFilesetHookDispatcher.java
+++
b/core/src/test/java/org/apache/gravitino/hook/TestFilesetHookDispatcher.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_STORE;
@@ -240,6 +241,8 @@ public class TestFilesetHookDispatcher extends
TestOperationDispatcher {
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER))
.thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS))
+ .thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS))
.thenReturn(1000L);
Mockito.when(config.get(STORE_TRANSACTION_MAX_SKEW_TIME)).thenReturn(1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/policy/TestPolicyManager.java
b/core/src/test/java/org/apache/gravitino/policy/TestPolicyManager.java
index e588b24374..3ec76afae6 100644
--- a/core/src/test/java/org/apache/gravitino/policy/TestPolicyManager.java
+++ b/core/src/test/java/org/apache/gravitino/policy/TestPolicyManager.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_STORE;
@@ -242,6 +243,7 @@ public class TestPolicyManager {
.thenReturn(String.format("jdbc:h2:file:%s;DB_CLOSE_DELAY=-1;MODE=MYSQL",
DB_DIR));
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_TRANSACTION_MAX_SKEW_TIME)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/stats/TestStatisticManager.java
b/core/src/test/java/org/apache/gravitino/stats/TestStatisticManager.java
index aa0cb0fea5..46e82715d8 100644
--- a/core/src/test/java/org/apache/gravitino/stats/TestStatisticManager.java
+++ b/core/src/test/java/org/apache/gravitino/stats/TestStatisticManager.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_STORE;
@@ -109,6 +110,7 @@ public class TestStatisticManager {
.thenReturn(String.format("jdbc:h2:file:%s;DB_CLOSE_DELAY=-1;MODE=MYSQL",
DB_DIR));
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_TRANSACTION_MAX_SKEW_TIME)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/AbstractEntityStorageTest.java
b/core/src/test/java/org/apache/gravitino/storage/AbstractEntityStorageTest.java
index cac53eed61..7c8142d4b7 100644
---
a/core/src/test/java/org/apache/gravitino/storage/AbstractEntityStorageTest.java
+++
b/core/src/test/java/org/apache/gravitino/storage/AbstractEntityStorageTest.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
@@ -144,6 +145,7 @@ abstract class AbstractEntityStorageTest {
Mockito.when(config.get(ENTITY_RELATIONAL_STORE)).thenReturn(DEFAULT_ENTITY_RELATIONAL_STORE);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PATH)).thenReturn(DB_DIR);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
Mockito.when(config.get(ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS)).thenReturn(3L);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/relational/BackendTestExtension.java
b/core/src/test/java/org/apache/gravitino/storage/relational/BackendTestExtension.java
index a9377c8b59..9961e30bb1 100644
---
a/core/src/test/java/org/apache/gravitino/storage/relational/BackendTestExtension.java
+++
b/core/src/test/java/org/apache/gravitino/storage/relational/BackendTestExtension.java
@@ -25,6 +25,7 @@ import static
org.apache.gravitino.Configs.DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_C
import static
org.apache.gravitino.Configs.DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_WAIT_MILLISECONDS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_USER;
@@ -173,6 +174,7 @@ public class BackendTestExtension
.thenReturn(DEFAULT_ENTITY_RELATIONAL_STORE);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS))
.thenReturn(DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS))
.thenReturn(DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_WAIT_MILLISECONDS);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/relational/TestRelationalEntityStoreHierarchicalCache.java
b/core/src/test/java/org/apache/gravitino/storage/relational/TestRelationalEntityStoreHierarchicalCache.java
index 0ac25beede..9fa204a8d2 100644
---
a/core/src/test/java/org/apache/gravitino/storage/relational/TestRelationalEntityStoreHierarchicalCache.java
+++
b/core/src/test/java/org/apache/gravitino/storage/relational/TestRelationalEntityStoreHierarchicalCache.java
@@ -198,6 +198,8 @@ public class TestRelationalEntityStoreHierarchicalCache {
.thenReturn("org.h2.Driver");
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS))
.thenReturn(Configs.DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS);
+
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS))
+ .thenReturn(10);
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS))
.thenReturn(Configs.DEFAULT_RELATIONAL_JDBC_BACKEND_MAX_WAIT_MILLISECONDS);
Mockito.when(config.get(Configs.STORE_DELETE_AFTER_TIME)).thenReturn(20 *
60 * 1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestAuthMappers.java
b/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestAuthMappers.java
index 79995003a8..ac53847ef8 100644
---
a/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestAuthMappers.java
+++
b/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestAuthMappers.java
@@ -92,6 +92,8 @@ public class TestAuthMappers {
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER))
.thenReturn("org.h2.Driver");
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(20);
+
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS))
+ .thenReturn(10);
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS))
.thenReturn(1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestEntityChangeLogMapper.java
b/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestEntityChangeLogMapper.java
index 89fa8d6543..5e34cd9696 100644
---
a/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestEntityChangeLogMapper.java
+++
b/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestEntityChangeLogMapper.java
@@ -71,6 +71,8 @@ public class TestEntityChangeLogMapper {
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER))
.thenReturn("org.h2.Driver");
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(20);
+
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS))
+ .thenReturn(10);
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS))
.thenReturn(1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestUserRoleRelMapper.java
b/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestUserRoleRelMapper.java
index 38e9b044cc..6cebaa5a31 100644
---
a/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestUserRoleRelMapper.java
+++
b/core/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestUserRoleRelMapper.java
@@ -82,6 +82,8 @@ public class TestUserRoleRelMapper {
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER))
.thenReturn("org.h2.Driver");
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(20);
+
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS))
+ .thenReturn(10);
Mockito.when(config.get(Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS))
.thenReturn(1000L);
diff --git
a/core/src/test/java/org/apache/gravitino/storage/relational/session/TestSqlSession.java
b/core/src/test/java/org/apache/gravitino/storage/relational/session/TestSqlSession.java
index fd93824d10..15e824de5a 100644
---
a/core/src/test/java/org/apache/gravitino/storage/relational/session/TestSqlSession.java
+++
b/core/src/test/java/org/apache/gravitino/storage/relational/session/TestSqlSession.java
@@ -22,6 +22,7 @@ package org.apache.gravitino.storage.relational.session;
import static org.apache.gravitino.Configs.DEFAULT_ENTITY_RELATIONAL_STORE;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_USER;
@@ -78,6 +79,7 @@ public class TestSqlSession {
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_PASSWORD)).thenReturn("123");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
}
@@ -129,6 +131,68 @@ public class TestSqlSession {
assertEquals(Duration.ofSeconds(30).toMillis(),
dataSource.getMinEvictableIdleTimeMillis());
}
+ @Test
+ public void testConfiguredMaxIdleAndPoolCap() {
+ SqlSessionFactoryHelper.getInstance().close();
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(64);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(40);
+ try {
+ SqlSessionFactoryHelper.getInstance().init(config);
+ BasicDataSource dataSource =
+ (BasicDataSource)
+ SqlSessionFactoryHelper.getInstance()
+ .getSqlSessionFactory()
+ .getConfiguration()
+ .getEnvironment()
+ .getDataSource();
+ assertEquals(40, dataSource.getMaxTotal());
+ assertEquals(40, dataSource.getMaxIdle());
+ } finally {
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+ }
+ }
+
+ @Test
+ public void testMinIdleCappedByMaxIdle() {
+ SqlSessionFactoryHelper.getInstance().close();
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(3);
+ try {
+ SqlSessionFactoryHelper.getInstance().init(config);
+ BasicDataSource dataSource =
+ (BasicDataSource)
+ SqlSessionFactoryHelper.getInstance()
+ .getSqlSessionFactory()
+ .getConfiguration()
+ .getEnvironment()
+ .getDataSource();
+ assertEquals(3, dataSource.getMaxIdle());
+ assertEquals(3, dataSource.getMinIdle());
+ } finally {
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
+ }
+ }
+
+ @Test
+ public void testUnlimitedPoolStillLimitsIdleConnections() {
+ SqlSessionFactoryHelper.getInstance().close();
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(-1);
+ try {
+ SqlSessionFactoryHelper.getInstance().init(config);
+ BasicDataSource dataSource =
+ (BasicDataSource)
+ SqlSessionFactoryHelper.getInstance()
+ .getSqlSessionFactory()
+ .getConfiguration()
+ .getEnvironment()
+ .getDataSource();
+ assertEquals(-1, dataSource.getMaxTotal());
+ assertEquals(10, dataSource.getMaxIdle());
+ } finally {
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+ }
+ }
+
@Test
public void testGetSqlSessionFactoryWithoutInit() {
SqlSessionFactoryHelper.getInstance().close();
diff --git a/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
b/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
index 74156b1260..de3f9c8473 100644
--- a/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
+++ b/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
@@ -24,6 +24,7 @@ import static
org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_POLL_INTERVAL_SECS;
import static org.apache.gravitino.Configs.ENTITY_CHANGE_LOG_RETENTION_SECS;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS;
+import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_URL;
import static
org.apache.gravitino.Configs.ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS;
import static org.apache.gravitino.Configs.ENTITY_RELATIONAL_STORE;
@@ -167,6 +168,7 @@ public class TestTagManager {
.thenReturn(String.format("jdbc:h2:file:%s;DB_CLOSE_DELAY=-1;MODE=MYSQL",
DB_DIR));
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_DRIVER)).thenReturn("org.h2.Driver");
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_CONNECTIONS)).thenReturn(100);
+
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_MAX_IDLE_CONNECTIONS)).thenReturn(10);
Mockito.when(config.get(ENTITY_RELATIONAL_JDBC_BACKEND_WAIT_MILLISECONDS)).thenReturn(1000L);
Mockito.when(config.get(STORE_TRANSACTION_MAX_SKEW_TIME)).thenReturn(1000L);
Mockito.when(config.get(STORE_DELETE_AFTER_TIME)).thenReturn(20 * 60 *
1000L);
diff --git a/dev/charts/gravitino/resources/config/gravitino.conf
b/dev/charts/gravitino/resources/config/gravitino.conf
index b715195a70..d4c9f16a4c 100644
--- a/dev/charts/gravitino/resources/config/gravitino.conf
+++ b/dev/charts/gravitino/resources/config/gravitino.conf
@@ -51,6 +51,9 @@ gravitino.entity.store.relational.storagePath = {{
.Values.entity.storagePath }}
{{- if .Values.entity.maxConnections }}
gravitino.entity.store.relational.maxConnections = {{
.Values.entity.maxConnections }}
{{- end }}
+{{- if .Values.entity.maxIdleConnections }}
+gravitino.entity.store.relational.maxIdleConnections = {{
.Values.entity.maxIdleConnections }}
+{{- end }}
{{- if or .Values.lock.maxNodes .Values.lock.minNodes }}
diff --git a/dev/charts/gravitino/tests/configmap_test.yaml
b/dev/charts/gravitino/tests/configmap_test.yaml
index 761c007847..ef05708b05 100644
--- a/dev/charts/gravitino/tests/configmap_test.yaml
+++ b/dev/charts/gravitino/tests/configmap_test.yaml
@@ -275,6 +275,25 @@ tests:
path: data["gravitino.conf"]
pattern: "gravitino\\.entity\\.store\\.relational\\.maxConnections"
+ - it: renders entity.maxIdleConnections only when set
+ release:
+ name: gravitino
+ set:
+ entity:
+ maxIdleConnections: 64
+ asserts:
+ - matchRegex:
+ path: data["gravitino.conf"]
+ pattern:
"gravitino\\.entity\\.store\\.relational\\.maxIdleConnections = 64"
+
+ - it: does not render entity.maxIdleConnections when empty (default)
+ release:
+ name: gravitino
+ asserts:
+ - notMatchRegex:
+ path: data["gravitino.conf"]
+ pattern:
"gravitino\\.entity\\.store\\.relational\\.maxIdleConnections"
+
- it: renders lock.maxNodes and lock.minNodes only when set
release:
name: gravitino
diff --git a/dev/charts/gravitino/values.yaml b/dev/charts/gravitino/values.yaml
index b18b043a87..022b61e97a 100644
--- a/dev/charts/gravitino/values.yaml
+++ b/dev/charts/gravitino/values.yaml
@@ -144,6 +144,11 @@ entity:
## Leave unset to use the server default. Production recommendation: 200.
##
maxConnections: ""
+ ## Maximum idle JDBC connections the relational backend pool keeps on each
server, capped by
+ ## maxConnections. Leave unset to use the server default (10). Budget for
this value on every
+ ## replica, because idle connections are released slowly.
+ ##
+ maxIdleConnections: ""
## THE CONFIGURATION FOR Gravitino TREE LOCK
##
diff --git a/dev/performance/jdbc_idle_retention.py
b/dev/performance/jdbc_idle_retention.py
new file mode 100644
index 0000000000..9859b2a35c
--- /dev/null
+++ b/dev/performance/jdbc_idle_retention.py
@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Measure JDBC connection churn and HTTP tail latency under concurrent reads.
+
+Run the same workload against fresh server processes with only the idle limit
changed, for
+example::
+
+ python3 dev/performance/jdbc_idle_retention.py \\
+ --url http://localhost:8090/api/metalakes \\
+ --mysql-defaults-extra-file=/path/to/mysql-client.cnf \\
+ --clients 64 --warmup-seconds 3 --seconds 15 --rounds 3
+
+The MySQL client option file supplies the database credentials. Set
+GRAVITINO_BENCH_AUTHORIZATION to send an Authorization header. Use a catalog
read URL, such as
+the list-tables route, to test a JDBC catalog pool instead of the entity-store
pool.
+
+Each round prints successful throughput, p50/p99 latency, failures, and the
change in MySQL's
+server-wide Connections and Threads_created counters. Those counters include
this script's own
+status queries and any other database traffic, so compare runs on an otherwise
quiet database with
+the same client count, data, JVM settings and node count, and only after
confirming that no
+requests failed.
+
+Self-test::
+
+ cd dev/performance && PYTHONDONTWRITEBYTECODE=1 python3 -m unittest
test_jdbc_idle_retention
+"""
+
+import argparse
+import concurrent.futures
+import http.client
+import json
+import math
+import os
+import subprocess
+import sys
+import time
+from urllib.parse import urlsplit
+
+
+def percentile(samples, fraction):
+ """Return a nearest-rank percentile in milliseconds."""
+ if not samples:
+ return None
+ ordered = sorted(samples)
+ return ordered[math.ceil(fraction * len(ordered)) - 1]
+
+
+def parse_mysql_status(output):
+ """Parse MySQL's two-column, tab-separated status output."""
+ return {
+ name: int(value)
+ for name, value in (line.split("\t") for line in output.splitlines())
+ }
+
+
+def mysql_status(defaults_file):
+ """Read server-wide connection counters using credentials from a MySQL
option file."""
+ result = subprocess.run(
+ [
+ "mysql",
+ f"--defaults-extra-file={defaults_file}",
+ "--batch",
+ "--skip-column-names",
+ "-e",
+ "SHOW GLOBAL STATUS WHERE Variable_name IN ('Connections',
'Threads_created')",
+ ],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ return parse_mysql_status(result.stdout)
+
+
+def run_phase(url, clients, duration, authorization):
+ """Send keep-alive GET requests from each client for one timed phase."""
+ parsed = urlsplit(url)
+ connection_type = (
+ http.client.HTTPSConnection
+ if parsed.scheme == "https"
+ else http.client.HTTPConnection
+ )
+ path = parsed.path or "/"
+ if parsed.query:
+ path += "?" + parsed.query
+ headers = {"Authorization": authorization} if authorization else {}
+ deadline = time.monotonic() + duration
+
+ def worker():
+ latencies = []
+ errors = 0
+ connection = None
+ try:
+ while time.monotonic() < deadline:
+ if connection is None:
+ connection = connection_type(
+ parsed.hostname, parsed.port, timeout=10
+ )
+ start = time.perf_counter()
+ try:
+ connection.request("GET", path, headers=headers)
+ response = connection.getresponse()
+ response.read()
+ elapsed_ms = (time.perf_counter() - start) * 1000
+ if 200 <= response.status < 300:
+ latencies.append(elapsed_ms)
+ else:
+ errors += 1
+ except (OSError, http.client.HTTPException):
+ errors += 1
+ connection.close()
+ connection = None
+ finally:
+ if connection is not None:
+ connection.close()
+ return latencies, errors
+
+ start = time.monotonic()
+ with concurrent.futures.ThreadPoolExecutor(max_workers=clients) as
executor:
+ results = list(executor.map(lambda _: worker(), range(clients)))
+ elapsed = time.monotonic() - start
+ latencies = [
+ latency for client_latencies, _ in results for latency in
client_latencies
+ ]
+ errors = sum(client_errors for _, client_errors in results)
+ return {
+ "requests": len(latencies),
+ "errors": errors,
+ "throughput_per_second": round(len(latencies) / elapsed, 1),
+ "p50_ms": percentile(latencies, 0.50),
+ "p99_ms": percentile(latencies, 0.99),
+ }
+
+
+def main():
+ """Run warmup and measured rounds against an already-started Gravitino
server."""
+ parser = argparse.ArgumentParser(
+ description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter
+ )
+ parser.add_argument(
+ "--url",
+ required=True,
+ help="full URL of a GET route, such as
http://localhost:8090/api/metalakes";,
+ )
+ parser.add_argument("--mysql-defaults-extra-file", required=True)
+ parser.add_argument("--clients", type=int, default=64)
+ parser.add_argument("--warmup-seconds", type=float, default=3)
+ parser.add_argument("--seconds", type=float, default=15)
+ parser.add_argument("--rounds", type=int, default=3)
+ args = parser.parse_args()
+ if (
+ args.clients < 1
+ or args.warmup_seconds <= 0
+ or args.seconds <= 0
+ or args.rounds < 1
+ ):
+ parser.error("clients, warmup-seconds, seconds, and rounds must be
positive")
+ if urlsplit(args.url).scheme not in ("http", "https"):
+ parser.error("url must use http or https")
+
+ authorization = os.environ.get("GRAVITINO_BENCH_AUTHORIZATION")
+ had_errors = False
+ for round_number in range(1, args.rounds + 1):
+ run_phase(args.url, args.clients, args.warmup_seconds, authorization)
+ before = mysql_status(args.mysql_defaults_extra_file)
+ result = run_phase(args.url, args.clients, args.seconds, authorization)
+ after = mysql_status(args.mysql_defaults_extra_file)
+ result["round"] = round_number
+ had_errors |= result["errors"] > 0
+ result["mysql_connections_delta"] = after["Connections"] -
before["Connections"]
+ result["mysql_threads_created_delta"] = (
+ after["Threads_created"] - before["Threads_created"]
+ )
+ print(json.dumps(result, sort_keys=True), flush=True)
+ if had_errors:
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dev/performance/test_jdbc_idle_retention.py
b/dev/performance/test_jdbc_idle_retention.py
new file mode 100644
index 0000000000..abd8c72ba9
--- /dev/null
+++ b/dev/performance/test_jdbc_idle_retention.py
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from threading import Thread
+from unittest.mock import patch
+
+from jdbc_idle_retention import mysql_status, parse_mysql_status, percentile,
run_phase
+
+
+class TestJdbcIdleRetention(unittest.TestCase):
+ def test_percentile(self):
+ self.assertIsNone(percentile([], 0.99))
+ self.assertEqual(1, percentile([4, 1, 3, 2], 0.25))
+ self.assertEqual(4, percentile([4, 1, 3, 2], 0.99))
+
+ def test_parse_mysql_status(self):
+ self.assertEqual(
+ {"Connections": 37, "Threads_created": 4},
+ parse_mysql_status("Connections\t37\nThreads_created\t4\n"),
+ )
+
+ @patch("jdbc_idle_retention.subprocess.run")
+ def test_mysql_status(self, run):
+ run.return_value.stdout = "Connections\t37\nThreads_created\t4\n"
+ self.assertEqual(
+ {"Connections": 37, "Threads_created": 4},
mysql_status("/tmp/mysql.cnf")
+ )
+ self.assertEqual(
+ "--defaults-extra-file=/tmp/mysql.cnf", run.call_args.args[0][1]
+ )
+
+ def test_run_phase(self):
+ class Handler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def do_GET(self):
+ self.send_response(200)
+ self.send_header("Content-Length", "0")
+ self.end_headers()
+
+ def log_message(self, *_args):
+ pass
+
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+ thread = Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ try:
+ result = run_phase(
+ f"http://127.0.0.1:{server.server_port}/api/metalakes";, 4,
0.5, None
+ )
+ self.assertGreater(result["requests"], 0)
+ self.assertEqual(0, result["errors"])
+ self.assertGreaterEqual(result["p99_ms"], 0)
+ finally:
+ server.shutdown()
+ server.server_close()
+ thread.join()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/docs/gravitino-server-config.md b/docs/gravitino-server-config.md
index 965e1a71c8..672dd01340 100644
--- a/docs/gravitino-server-config.md
+++ b/docs/gravitino-server-config.md
@@ -279,20 +279,21 @@ PostgreSQL, and the setup procedure for both is in
The driver, user, and password properties are required whenever the URL is not
`jdbc:h2`.
-| Configuration Item | Description
| Default Value |
-|----------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
-| `gravitino.entity.store` | Entity storage
implementation. `relational` is the only supported value.
| `relational` |
-| `gravitino.entity.store.relational` | Relational storage
implementation. `JDBCBackend` is the only supported value, and it covers H2,
MySQL, and PostgreSQL.
| `JDBCBackend` |
-| `gravitino.entity.store.relational.jdbcUrl` | Database URL the
backend connects to.
| `jdbc:h2` |
-| `gravitino.entity.store.relational.jdbcDriver` | Driver class name.
Place the driver jar in `${GRAVITINO_HOME}/libs/`.
| `org.h2.Driver` |
-| `gravitino.entity.store.relational.jdbcUser` | Database username.
| `gravitino` |
-| `gravitino.entity.store.relational.jdbcPassword` | Database password.
| `gravitino` |
-| `gravitino.entity.store.relational.storagePath` | Where embedded H2 keeps
its files. A relative value resolves against `${GRAVITINO_HOME}`. The default
sits inside the deployment directory, so an upgrade that replaces that
directory discards the data. Change it. | `${GRAVITINO_HOME}/data/jdbc` |
-| `gravitino.entity.store.relational.maxConnections` | Maximum size of the
JDBC connection pool.
| `100` |
-| `gravitino.entity.store.relational.maxWaitMillis` | Maximum wait in
milliseconds for a connection from the pool.
| `1000` |
-| `gravitino.entity.store.maxTransactionSkewTimeMs` | Maximum transaction
skew in milliseconds.
| `2000` |
-| `gravitino.entity.store.deleteAfterTimeMs` | How long in
milliseconds deleted and superseded rows are kept. Accepts 10 minutes to 30
days.
| `604800000` (7 days) |
-| `gravitino.entity.store.versionRetentionCount` | Number of entity
versions kept, including the current one. Accepts 1 to 10.
| `1` |
+| Configuration Item | Description
| Default Value |
+|--------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+| `gravitino.entity.store` | Entity storage
implementation. `relational` is the only supported value.
| `relational` |
+| `gravitino.entity.store.relational` | Relational storage
implementation. `JDBCBackend` is the only supported value, and it covers H2,
MySQL, and PostgreSQL.
| `JDBCBackend` |
+| `gravitino.entity.store.relational.jdbcUrl` | Database URL the
backend connects to.
| `jdbc:h2` |
+| `gravitino.entity.store.relational.jdbcDriver` | Driver class name.
Place the driver jar in `${GRAVITINO_HOME}/libs/`.
| `org.h2.Driver` |
+| `gravitino.entity.store.relational.jdbcUser` | Database username.
| `gravitino` |
+| `gravitino.entity.store.relational.jdbcPassword` | Database password.
| `gravitino` |
+| `gravitino.entity.store.relational.storagePath` | Where embedded H2
keeps its files. A relative value resolves against `${GRAVITINO_HOME}`. The
default sits inside the deployment directory, so an upgrade that replaces that
directory discards the data. Change it. | `${GRAVITINO_HOME}/data/jdbc` |
+| `gravitino.entity.store.relational.maxConnections` | Maximum size of the
JDBC connection pool.
| `100` |
+| `gravitino.entity.store.relational.maxIdleConnections` | Maximum idle
connections retained per server; capped by `maxConnections`. Budget for it on
every server, plus any JDBC catalog pools that use the same database instance.
| `10` |
+| `gravitino.entity.store.relational.maxWaitMillis` | Maximum wait in
milliseconds for a connection from the pool.
| `1000` |
+| `gravitino.entity.store.maxTransactionSkewTimeMs` | Maximum transaction
skew in milliseconds.
| `2000` |
+| `gravitino.entity.store.deleteAfterTimeMs` | How long in
milliseconds deleted and superseded rows are kept. Accepts 10 minutes to 30
days.
| `604800000` (7 days) |
+| `gravitino.entity.store.versionRetentionCount` | Number of entity
versions kept, including the current one. Accepts 1 to 10.
| `1` |
#### Caching
diff --git a/docs/how-to-use-relational-backend-storage.md
b/docs/how-to-use-relational-backend-storage.md
index 80e055d45b..005649f1a1 100644
--- a/docs/how-to-use-relational-backend-storage.md
+++ b/docs/how-to-use-relational-backend-storage.md
@@ -31,6 +31,27 @@ gravitino.entity.store.relational.jdbcPassword = {password}
`gravitino.entity.store` and `gravitino.entity.store.relational` already
default to `relational`
and `JDBCBackend`. Leave them alone.
+For concurrent metadata reads, tune
`gravitino.entity.store.relational.maxIdleConnections`
+alongside `maxConnections`. The default retains up to 10 idle connections per
server; the
+effective limit never exceeds `maxConnections`. Count this limit once per
server when calculating
+the database connection budget, and also count the pools of any JDBC catalogs
that point at the
+same database instance.
+
+This limit, not idle-time eviction, is what decides whether connections are
reused. When a request
+returns a connection while the pool already holds `maxIdleConnections` idle
connections, the pool
+closes it at once, and a later request has to open a new one. When the number
of connections in
+use keeps swinging by more than this limit, physical connections keep being
closed and reopened,
+which adds latency. Setting it at or above the number of connections in use at
steady peak
+concurrency avoids that. For example, with 64 concurrent clients listing
metalakes against MySQL,
+a limit of `10` reopened about 1,400 to 2,700 connections every 15 seconds,
`32` about 270 to 440,
+and `64` almost none. Raise it only as far as the database connection budget
of all servers allows.
+
+Idle connections are released slowly. The pool's evictor runs every ten
minutes and checks at most
+three idle connections per run, closing those idle for more than 30 seconds.
After a burst, a server
+can therefore keep close to `maxIdleConnections` open for a long time; with a
limit of `64`, it
+takes more than three hours to shrink back to the minimum of five. Budget for
the full value on
+every server rather than treating it as a short-lived peak.
+
The values to use, and the driver each one needs:
| Database | JDBC URL
| Driver Class | Driver Jar |
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index fc86b677aa..c1e17bb59d 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -50,15 +50,16 @@ Pass any JDBC pool property that Gravitino does not define
by adding the `gravit
When using the JDBC catalog you must provide `jdbc-url`, `jdbc-driver`,
`jdbc-user`, and `jdbc-password`. Common catalog properties are listed
[here](./gravitino-server-config.md#catalog-properties-configuration);
ClickHouse adds no extra catalog-scoped keys.
-| Configuration item | Description
| Default value | Required |
-|-------------------------|-----------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | JDBC URL, for example
`jdbc:clickhouse://localhost:8123` | (none) | Yes |
-| `jdbc-driver` | JDBC driver class, for example
`com.clickhouse.jdbc.ClickHouseDriver` | (none) | Yes |
-| `jdbc-user` | JDBC user name
| (none) | Yes |
-| `jdbc-password` | JDBC password
| (none) | Yes |
-| `jdbc.pool.min-size` | Minimum pool size
| `2` | No |
-| `jdbc.pool.max-size` | Maximum pool size
| `10` | No |
-| `jdbc.pool.max-wait-ms` | Max wait time for a connection
| `30000` | No |
+| Configuration item | Description
| Default value |
Required |
+|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `jdbc-url` | JDBC URL, for example
`jdbc:clickhouse://localhost:8123`
| (none) | Yes |
+| `jdbc-driver` | JDBC driver class, for example
`com.clickhouse.jdbc.ClickHouseDriver`
|
(none) | Yes |
+| `jdbc-user` | JDBC user name
| (none) | Yes
|
+| `jdbc-password` | JDBC password
| (none) | Yes
|
+| `jdbc.pool.min-size` | Minimum pool size
| `2` | No
|
+| `jdbc.pool.max-size` | Maximum pool size
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
+| `jdbc.pool.max-wait-ms` | Max wait time for a connection
| `30000` | No
|
### Create a ClickHouse Catalog
diff --git a/docs/jdbc-doris-catalog.md b/docs/jdbc-doris-catalog.md
index 2417f79642..74ea49106b 100644
--- a/docs/jdbc-doris-catalog.md
+++ b/docs/jdbc-doris-catalog.md
@@ -41,15 +41,16 @@ more details.
Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
Doris catalog has the following properties:
-| Configuration item | Description
| Default value | Required |
-|-------------------------|---------------------------------------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:9030` | (none) | Yes
|
-| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver`. | (none) | Yes |
-| `jdbc-user` | The JDBC user name.
| (none) | Yes |
-| `jdbc-password` | The JDBC password.
| (none) | Yes |
-| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default. | `2` | No |
-| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default. | `10` | No |
-| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default. | `30000` | No |
+| Configuration item | Description
| Default value |
Required |
+|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:9030`
| (none) |
Yes |
+| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver`.
| (none) | Yes
|
+| `jdbc-user` | The JDBC user name.
| (none) | Yes
|
+| `jdbc-password` | The JDBC password.
| (none) | Yes
|
+| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No
|
+| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
+| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No
|
Before using the Doris Catalog, you must download the corresponding JDBC
driver to the `catalogs/jdbc-doris/libs` directory.
Gravitino doesn't package the JDBC driver for Doris due to licensing issues.
diff --git a/docs/jdbc-hologres-catalog.md b/docs/jdbc-hologres-catalog.md
index 7b6ab04652..5bea59a76d 100644
--- a/docs/jdbc-hologres-catalog.md
+++ b/docs/jdbc-hologres-catalog.md
@@ -43,15 +43,16 @@ Check the relevant data source configuration in [data
source properties](https:/
If you use a JDBC catalog, you must provide `jdbc-url`, `jdbc-driver`,
`jdbc-database`, `jdbc-user` and `jdbc-password` to catalog properties.
Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
Hologres catalog has the following properties:
-| Configuration item | Description
|
Default value | Required |
-|----------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | JDBC URL for connecting to the database. For example,
`jdbc:postgresql://hgprecn-cn-xxx.hologres.aliyuncs.com:80/my_database` |
(none) | Yes |
-| `jdbc-driver` | The driver of the JDBC connection. Must be
`org.postgresql.Driver`.
| (none) | Yes |
-| `jdbc-database` | The database name. This is mandatory for Hologres.
|
(none) | Yes |
-| `jdbc-user` | The JDBC user name (AccessKey ID or database
username).
| (none) | Yes |
-| `jdbc-password` | The JDBC password (AccessKey Secret or database
password). |
(none) | Yes |
-| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2` by
default. | `2`
| No |
-| `jdbc.pool.max-size` | The maximum number of connections in the pool. `10`
by default. |
`10` | No |
+| Configuration item | Description
| Default value | Required |
+|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `jdbc-url` | JDBC URL for connecting to the database. For example,
`jdbc:postgresql://hgprecn-cn-xxx.hologres.aliyuncs.com:80/my_database`
| (none) | Yes |
+| `jdbc-driver` | The driver of the JDBC connection. Must be
`org.postgresql.Driver`.
| (none) |
Yes |
+| `jdbc-database` | The database name. This is mandatory for Hologres.
| (none) | Yes |
+| `jdbc-user` | The JDBC user name (AccessKey ID or database
username).
| (none) |
Yes |
+| `jdbc-password` | The JDBC password (AccessKey Secret or database
password).
| (none) | Yes
|
+| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2` by
default.
| `2` | No |
+| `jdbc.pool.max-size` | The maximum number of connections in the pool. `10`
by default.
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
:::caution
Hologres uses the PostgreSQL JDBC Driver (version 42.3.2 or later
recommended). You need to download the PostgreSQL JDBC Driver and place it in
the `catalogs/jdbc-hologres/libs` directory under the Gravitino distribution
(e.g., `distribution/package/catalogs/jdbc-hologres/libs` or
`distribution/package-all/catalogs/jdbc-hologres/libs`).
diff --git a/docs/jdbc-mysql-catalog.md b/docs/jdbc-mysql-catalog.md
index 12156b4854..9c5126e4ef 100644
--- a/docs/jdbc-mysql-catalog.md
+++ b/docs/jdbc-mysql-catalog.md
@@ -41,15 +41,16 @@ When using Gravitino with Trino, pass the Trino MySQL
connector configuration us
If you use a JDBC catalog, you must provide `jdbc-url`, `jdbc-driver`,
`jdbc-user` and `jdbc-password` to catalog properties.
Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
MySQL catalog has the following properties:
-| Configuration item | Description
| Default value | Required |
-|-------------------------|--------------------------------------------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:3306` | (none) | Yes
|
-| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver` or `com.mysql.cj.jdbc.Driver`. | (none) | Yes
|
-| `jdbc-user` | The JDBC user name.
| (none) | Yes |
-| `jdbc-password` | The JDBC password.
| (none) | Yes |
-| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default. | `2` | No |
-| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default. | `10` | No
|
-| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default. | `30000` | No |
+| Configuration item | Description
| Default value |
Required |
+|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:3306`
| (none) |
Yes |
+| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver` or `com.mysql.cj.jdbc.Driver`.
| (none) | Yes
|
+| `jdbc-user` | The JDBC user name.
| (none) | Yes
|
+| `jdbc-password` | The JDBC password.
| (none) | Yes
|
+| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No
|
+| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
+| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No
|
:::caution
Gravitino does not package the MySQL JDBC driver, because MySQL Connector/J is
diff --git a/docs/jdbc-oceanbase-catalog.md b/docs/jdbc-oceanbase-catalog.md
index 0896fa6983..1d26c0b083 100644
--- a/docs/jdbc-oceanbase-catalog.md
+++ b/docs/jdbc-oceanbase-catalog.md
@@ -39,15 +39,16 @@ Check the relevant data source configuration in [data
source properties](https:/
If you use a JDBC catalog, you must provide `jdbc-url`, `jdbc-driver`,
`jdbc-user` and `jdbc-password` to catalog properties.
Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
OceanBase catalog has the following properties:
-| Configuration item | Description
| Default value | Required |
-|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:2881` or `jdbc:oceanbase://localhost:2881`
| (none) | Yes |
-| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver` or `com.mysql.cj.jdbc.Driver` or
`com.oceanbase.jdbc.Driver`. | (none) | Yes |
-| `jdbc-user` | The JDBC user name.
| (none) | Yes |
-| `jdbc-password` | The JDBC password.
| (none) | Yes |
-| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No |
-| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No |
-| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No |
+| Configuration item | Description
| Default value |
Required |
+|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:2881` or `jdbc:oceanbase://localhost:2881`
| (none) |
Yes |
+| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver` or `com.mysql.cj.jdbc.Driver` or
`com.oceanbase.jdbc.Driver`.
| (none) | Yes |
+| `jdbc-user` | The JDBC user name.
| (none) | Yes
|
+| `jdbc-password` | The JDBC password.
| (none) | Yes
|
+| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No
|
+| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
+| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No
|
:::caution
Gravitino does not package the OceanBase JDBC driver due to licensing, so you
diff --git a/docs/jdbc-postgresql-catalog.md b/docs/jdbc-postgresql-catalog.md
index 216f6ae8bf..60ff95dd1b 100644
--- a/docs/jdbc-postgresql-catalog.md
+++ b/docs/jdbc-postgresql-catalog.md
@@ -39,16 +39,17 @@ When using Gravitino with Trino, pass the Trino PostgreSQL
connector configurati
If you use JDBC catalog, you must provide `jdbc-url`, `jdbc-driver`,
`jdbc-user` and `jdbc-password` to catalog properties.
Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
PostgreSQL catalog has the following properties:
-| Configuration item | Description
| Default value | Required |
-|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | A valid PostgreSQL JDBC URL, for example
`jdbc:postgresql://localhost:5432/pg_database?sslmode=require`. If the URL
omits the database, set `jdbc-database`. | (none) | Yes |
-| `jdbc-driver` | The driver of the JDBC connection. For example
`org.postgresql.Driver`.
| (none) | Yes |
-| `jdbc-database` | The database of the JDBC connection. Derived from
`jdbc-url` when omitted. An explicit value must be nonblank and match the URL
database when both are provided. | (none) | Only if absent from
`jdbc-url` |
-| `jdbc-user` | The JDBC user name.
| (none) | Yes |
-| `jdbc-password` | The JDBC password.
| (none) | Yes |
-| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No |
-| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No |
-| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No |
+| Configuration item | Description
| Default value |
Required |
+|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|--------------------------------|
+| `jdbc-url` | A valid PostgreSQL JDBC URL, for example
`jdbc:postgresql://localhost:5432/pg_database?sslmode=require`. If the URL
omits the database, set `jdbc-database`. | (none)
| Yes |
+| `jdbc-driver` | The driver of the JDBC connection. For example
`org.postgresql.Driver`.
| (none) | Yes
|
+| `jdbc-database` | The database of the JDBC connection. Derived from
`jdbc-url` when omitted. An explicit value must be nonblank and match the URL
database when both are provided. | (none) | Only
if absent from `jdbc-url` |
+| `jdbc-user` | The JDBC user name.
| (none) | Yes
|
+| `jdbc-password` | The JDBC password.
| (none) | Yes
|
+| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No
|
+| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
+| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No
|
:::caution
Download the corresponding JDBC driver to the `catalogs/jdbc-postgresql/libs`
directory.
diff --git a/docs/jdbc-starrocks-catalog.md b/docs/jdbc-starrocks-catalog.md
index d3c976c450..f23221a632 100644
--- a/docs/jdbc-starrocks-catalog.md
+++ b/docs/jdbc-starrocks-catalog.md
@@ -40,15 +40,16 @@ more details.
Besides the [common catalog
properties](./gravitino-server-config.md#catalog-properties-configuration), the
StarRocks catalog has the following properties:
-| Configuration item | Description
| Default value | Required |
-|-------------------------|---------------------------------------------------------------------------------------------------|---------------|----------|
-| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:9030` | (none) | Yes
|
-| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver`. | (none) | Yes |
-| `jdbc-user` | The JDBC user name.
| (none) | Yes |
-| `jdbc-password` | The JDBC password.
| (none) | Yes |
-| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default. | `2` | No |
-| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default. | `10` | No |
-| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default. | `30000` | No |
+| Configuration item | Description
| Default value |
Required |
+|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `jdbc-url` | JDBC URL for connecting to the database. For
example, `jdbc:mysql://localhost:9030`
| (none) |
Yes |
+| `jdbc-driver` | The driver of the JDBC connection. For example,
`com.mysql.jdbc.Driver`.
| (none) | Yes
|
+| `jdbc-user` | The JDBC user name.
| (none) | Yes
|
+| `jdbc-password` | The JDBC password.
| (none) | Yes
|
+| `jdbc.pool.min-size` | The minimum number of connections in the pool. `2`
by default.
| `2` | No
|
+| `jdbc.pool.max-size` | The maximum number of connections in the pool.
`10` by default.
| `10` | No
|
+| `jdbc.pool.max-idle` | Maximum idle connections retained per catalog per
server; capped by `jdbc.pool.max-size`; takes precedence over
`gravitino.bypass.maxIdle`. Idle connections are not evicted by default. | `8`
| No |
+| `jdbc.pool.max-wait-ms` | The maximum Duration that the pool will wait for a
connection to be returned. `30000` by default.
| `30000` | No
|
Before using the StarRocks Catalog, you must download the corresponding JDBC
driver to the `catalogs/jdbc-starrocks/libs` directory.
Gravitino doesn't package the JDBC driver for StarRocks due to licensing
issues.
