This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e0050c41211 [test](external) Fix removed DLF and thread warm-up
assumptions (#69156)
e0050c41211 is described below
commit e0050c41211372e813107f13366d03fe45c005d6
Author: Gabriel <[email protected]>
AuthorDate: Fri Oct 16 16:41:18 2026 +0800
[test](external) Fix removed DLF and thread warm-up assumptions (#69156)
### What problem does this PR solve?
Two external regression suites encode assumptions that no longer hold:
- `hive_on_hms_and_dlf` still expects DLF 1.0 to work as an HMS
metastore. Catalog creation now intentionally reports
`hive.metastore.type = dlf is no longer supported`. Replace these
obsolete success scenarios with an explicit rejection check using
minimal properties, without cloud credentials. Keep the complete
supported HMS storage/authentication matrix.
- `test_paimon_write_thread_lifecycle` compares the minimum of its first
two phases with the minimum of its final two. A process-thread series
such as `[100, 121, 121, 121]` therefore fails despite reaching a
plateau. Require a three-phase stable warm-up window, with at most six
warm-up phases, then freeze its low-water baseline and run six further
equal-work phases. Compare the final two low-water marks against that
fixed baseline, retaining JVM +2/process +4 tolerances. This also
rejects sustained growth of one thread per phase starting only after
warm-up while tolerating one isolated background spike.
The row-count assertion follows the actual number of disjoint INSERT
batches. The thread suite now performs 108–144 INSERT statements instead
of 60; this additional measurement budget is intentional. Process-wide
background activity can still affect its counters.
### Validation
- Executed the actual Groovy thread suite through a local replay harness
with mocked SQL/HTTP/metric boundaries: 12/12 scenarios passed,
including the observed plateau, steady and delayed slow leaks, a single
leaking backend, late stabilization, and sampling spikes.
- Compiled the complete Hive script and replayed its new negative-test
DSL; independently checked that removed-metastore validation precedes
missing-URI validation.
- Verified the supported HMS invocation matrix is unchanged; `git diff
--check` passed.
- Live external HMS/Paimon end-to-end runs remain for CI. Replay
validates test logic, not live writer behavior.
### Release note
None. Regression tests only.
### Check List (For Author)
- [x] Regression test changes
- [x] Manual test: Groovy script replay described above
- [x] No product behavior change
- [x] No documentation needed
---
.../test_paimon_write_thread_lifecycle.groovy | 84 +++++++++++++---------
.../hive_on_hms_and_dlf.groovy | 52 ++++----------
2 files changed, 64 insertions(+), 72 deletions(-)
diff --git
a/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_thread_lifecycle.groovy
b/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_thread_lifecycle.groovy
index 420f53cd18b..daa60ccefcb 100644
---
a/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_thread_lifecycle.groovy
+++
b/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_thread_lifecycle.groovy
@@ -95,57 +95,73 @@ suite("test_paimon_write_thread_lifecycle",
"p0,external,paimon,nonConcurrent")
sql """use ${dbName}"""
try {
- // Warm common writer and metrics paths before measuring
phase-to-phase growth. The
- // blocking scheduler may still assign later writes to native workers
that have not entered
- // the JVM yet, so a bounded one-time increase after warm-up is not
evidence of a leak.
- for (int round = 0; round < 12; round++) {
- sql """
- INSERT INTO t_thread_lifecycle
- SELECT number + ${round * 1000}, repeat('w', 32)
- FROM numbers("number" = "1000")
- """
- }
- sleep(3000)
-
- def writePhase = { int firstRound ->
- for (int round = firstRound; round < firstRound + 12; round++) {
+ def jvmPhases = []
+ def processPhases = []
+ int nextRound = 0
+ def writeAndSamplePhase = {
+ for (int round = nextRound; round < nextRound + 12; round++) {
sql """
INSERT INTO t_thread_lifecycle
SELECT number + ${round * 1000}, repeat('x', 32)
FROM numbers("number" = "1000")
"""
}
- }
-
- def jvmPhases = []
- def processPhases = []
- for (int phase = 0; phase < 4; phase++) {
- writePhase(12 + phase * 12)
+ nextRound += 12
sleep(5000)
jvmPhases.add(minimumThreadCounts(jvmThreadCounts))
processPhases.add(minimumThreadCounts(processThreadCounts))
- logger.info("Paimon thread phase ${phase + 1}:
jvm=${jvmPhases[-1]}, "
+ logger.info("Paimon thread phase ${jvmPhases.size()}:
jvm=${jvmPhases[-1]}, "
+ "process=${processPhases[-1]}")
}
- assertEquals(60000L,
+ // Native workers can initialize after the first writes. Require a
stable window instead
+ // of treating a single increase followed by a plateau as a
continuously leaking writer.
+ def stableWindow = { phases, long tolerance ->
+ backendEndpoints.keySet().every { backendId ->
+ def counts = phases.takeRight(3).collect { sample ->
sample[backendId] as long }
+ counts.max() - counts.min() <= tolerance
+ }
+ }
+ boolean warmedUp = false
+ for (int phase = 0; phase < 6; phase++) {
+ writeAndSamplePhase()
+ if (jvmPhases.size() >= 3 && stableWindow(jvmPhases, 2L)
+ && stableWindow(processPhases, 4L)) {
+ warmedUp = true
+ break
+ }
+ }
+ assertTrue(warmedUp, "Thread counts did not stabilize during bounded
warm-up: "
+ + "jvm=${jvmPhases}, process=${processPhases}")
+
+ int warmupPhases = jvmPhases.size()
+ def baseline = { phases ->
+ backendEndpoints.keySet().collectEntries { backendId ->
+ [(backendId): phases.takeRight(3).collect { sample ->
sample[backendId] as long }.min()]
+ }
+ }
+ def jvmBaseline = baseline(jvmPhases)
+ def processBaseline = baseline(processPhases)
+ // Freeze the low-water marks before additional writes. Moving the
baseline on each phase
+ // would hide slow, persistent leaks that fit within the per-window
sampling tolerance.
+ for (int phase = 0; phase < 6; phase++) {
+ writeAndSamplePhase()
+ }
+
+ assertEquals(nextRound * 1000L,
(sql """SELECT COUNT(*) FROM t_thread_lifecycle""")[0][0] as
long)
+ // Use the final two low-water marks to tolerate one isolated
background-thread spike.
+ // Six validation phases also expose one new process thread per phase
starting after warm-up.
backendEndpoints.keySet().each { backendId ->
- // Judge persistent growth between equal measured phases.
Comparing the early and late
- // low-water marks allows a bounded worker-pool warm-up or an
isolated background-thread
- // spike, while a leaked thread cannot disappear in a later phase.
def jvmCounts = jvmPhases.collect { sample -> sample[backendId] as
long }
def processCounts = processPhases.collect { sample ->
sample[backendId] as long }
- def earlyJvmFloor = jvmCounts.take(2).min()
- def lateJvmFloor = jvmCounts.drop(2).min()
- def earlyProcessFloor = processCounts.take(2).min()
- def lateProcessFloor = processCounts.drop(2).min()
-
- assertTrue(lateJvmFloor <= earlyJvmFloor + 2,
- "JVM threads kept growing on backend ${backendId}:
phases=${jvmCounts}")
- assertTrue(lateProcessFloor <= earlyProcessFloor + 4,
- "Process threads kept growing on backend ${backendId}:
phases=${processCounts}")
+ assertTrue(jvmCounts.drop(warmupPhases).takeRight(2).min() <=
jvmBaseline[backendId] + 2,
+ "JVM threads grew after warm-up on backend ${backendId}: "
+ + "baseline=${jvmBaseline[backendId]},
phases=${jvmCounts}")
+ assertTrue(processCounts.drop(warmupPhases).takeRight(2).min() <=
processBaseline[backendId] + 4,
+ "Process threads grew after warm-up on backend
${backendId}: "
+ + "baseline=${processBaseline[backendId]},
phases=${processCounts}")
}
} finally {
sql """drop catalog if exists ${catalogName}"""
diff --git
a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
index 9fddd9ba979..94cfa991ffe 100644
---
a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
+++
b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy
@@ -19,6 +19,20 @@ import java.util.concurrent.ThreadLocalRandom
suite("hive_on_hms_and_dlf", "p2,external") {
+ // DLF 1.0 is no longer an HMS metastore. Check its explicit rejection
without cloud
+ // credentials, while retaining the HMS storage and authentication matrix
below.
+ sql """DROP CATALOG IF EXISTS hive_dlf_unsupported_catalog"""
+ test {
+ sql """
+ CREATE CATALOG hive_dlf_unsupported_catalog PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.type' = 'dlf'
+ )
+ """
+ exception "hive.metastore.type = dlf is no longer supported: " +
+ "Alibaba Cloud DLF 1.0 as an HMS thrift metastore has been
removed. Supported types: hms."
+ }
+
// Keep generated identifiers under the FE 64-character limit for long
prefixes.
def buildDbName = { String prefix ->
def suffix = String.format("_db%d%03d", System.currentTimeMillis(),
ThreadLocalRandom.current().nextInt(1000))
@@ -501,17 +515,6 @@ suite("hive_on_hms_and_dlf", "p2,external") {
"hdfs.authentication.kerberos.keytab" =
"${keytab_root_dir}/hive-presto-master.keytab"
"""
- String dlf_access_key = context.config.otherConfigs.get("dlf_access_key")
- String dlf_secret_key = context.config.otherConfigs.get("dlf_secret_key")
- /**************** DLF *******************/
-
- String dlf_oss_properties = """
- "oss.endpoint" = "${oss_endpoint}",
- "oss.access_key" = "${dlf_access_key}",
- "oss.secret_key" = "${dlf_secret_key}",
- """
-
-
String hms_kerberos_old_prop = """
"hive.metastore.uris" = "thrift://${externalEnvIp}:9583",
"hadoop.security.authentication" = "kerberos",
@@ -626,31 +629,4 @@ suite("hive_on_hms_and_dlf", "p2,external") {
testQueryAndInsert(hms_type_properties + hms_kerberos_new_prop +
hdfs_kerberos_properties, "hive_hms_hdfs_kerberos_test", db_location)
testQueryAndInsert(hms_type_properties + hms_kerberos_new_prop +
hdfs_new_kerberos_properties, "hive_hms_hdfs_new_kerberos_test", db_location)
-
- /**************** DLF *******************/
- String dlf_warehouse =
"oss://selectdb-qa-datalake-test/hive-dlf-oss-warehouse"
- String hive_dlf_catalog_base_properties = """
- 'type'='hms',
- 'hive.metastore.type'='dlf',
- "dlf.uid" = "203225413946383283",
- "dlf.catalog.id" = "p2_regression_case",
- "dlf.access_key" = "${dlf_access_key}",
- "dlf.secret_key" = "${dlf_secret_key}"
- """
- String dlf_region = "cn-beijing"
- String dlf_region_param = """
- "dlf.region" = "${dlf_region}",
- """
- String dlf_endpoint = context.config.otherConfigs.get("dlf_endpoint")
- String dlf_endpoint_properties = """
- "dlf.endpoint" = "${dlf_endpoint}",
- """
- db_location = "${dlf_warehouse}/hive/dlf-oss/" + System.currentTimeMillis()
- testQueryAndInsert(dlf_region_param + hive_dlf_catalog_base_properties,
"hive_dlf_oss_test", db_location)
- db_location = "${dlf_warehouse}/hive/dlf-oss/" + System.currentTimeMillis()
- testQueryAndInsert(dlf_region_param + dlf_oss_properties +
hive_dlf_catalog_base_properties, "hive_dlf_oss_test_with_oss", db_location)
-
- // not has region param
- db_location = "${dlf_warehouse}/hive/dlf-oss/" + System.currentTimeMillis()
- testQueryAndInsert(dlf_endpoint_properties +
hive_dlf_catalog_base_properties, "hive_dlf_oss_test", db_location)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
