This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 6f1538c6ee0 branch-4.0: [fix](test) deflake several branch-4.0 P2
regression cases (#64693)
6f1538c6ee0 is described below
commit 6f1538c6ee0522a8dd871b6047693f3f2b2997ae
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Jun 24 16:07:23 2026 +0800
branch-4.0: [fix](test) deflake several branch-4.0 P2 regression cases
(#64693)
## What problem does this PR solve?
Deflake several pre-existing flaky/failing cases in the branch-4.0 P2
regression suite (surfaced by an internal P2 run). None of these are
caused by product behavior changes.
1. **opensky_p2** `count` / `avgDistance` / `totalDistance` /
`mostBusyOrigin` — backport of #62447: the S3 source `csv.gz` was
actually a `tar.gz`, so after it was corrected the loaded row count
changed (+30 rows) while the committed `.out` still held the old values.
Update the four `.out` files and re-enable the `NumberTotalRows ==
NumberLoadedRows` load assertion.
2. **segcompaction_p2/test_segcompaction_agg_keys** — backport of
#61026: the AGGREGATE `REPLACE` winner for a duplicate key is not
deterministic. S3 load parallelizes into segments whose boundaries are
non-deterministic (adaptive memtable flush / memory pressure) and the
read-merge keeps the first row by segment order, so which of the
duplicate `col_0=47` rows wins varies between runs. Assert exactly one
row plus the value being one of the legitimate outcomes, and delete the
brittle `.out`.
3. **inverted_index_p2/test_show_data**
(`test_show_data_with_compaction`) — it compared index sizes with
`inverted_index_compaction_enable` on vs off for exact equality. For
identical data the on-disk size can differ slightly due to file/merge
layout, so compare within a 10% tolerance instead, which still catches
gross index bloat or corruption.
4. **compaction/test_base_compaction_with_dup_key_max_file_size_limit**
— the test builds a specific rowset layout via manual compactions and
expects a manual base compaction to be rejected with `E-808` (input
rowset exceeds `base_compaction_dup_key_max_file_size_mbytes`). When the
BE global `disable_auto_compaction` is false, background compaction
races the manual steps and reshapes the rowsets, and an in-test
short-circuit masked the result. Disable auto compaction cluster-wide
for the duration of the test (restored in `finally`) and assert on the
real response.
5. **compaction/test_single_replica_compaction** — the
`waitForCompaction` and `getTabletStatus` closures assigned `code`/`out`
without `def`, which the regression framework's script-source guard
rejects (`defined global variables in script are not allowed: code`).
#55933 declared only `process`; declare the remaining locals too.
## Release note
None
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01QuX5zypeHzV1ZgT9o8BzoF
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
---
.../data/opensky_p2/sql/avgDistance.out | 2 +-
regression-test/data/opensky_p2/sql/count.out | 2 +-
.../data/opensky_p2/sql/mostBusyOrigin.out | 18 ++---
.../data/opensky_p2/sql/totalDistance.out | 2 +-
.../test_segcompaction_agg_keys.out | 3 -
...paction_with_dup_key_max_file_size_limit.groovy | 76 +++++++++++++---------
.../test_single_replica_compaction.groovy | 10 +--
.../suites/inverted_index_p2/test_show_data.groovy | 13 +++-
regression-test/suites/opensky_p2/load.groovy | 2 +-
.../test_segcompaction_agg_keys.groovy | 12 +++-
10 files changed, 87 insertions(+), 53 deletions(-)
diff --git a/regression-test/data/opensky_p2/sql/avgDistance.out
b/regression-test/data/opensky_p2/sql/avgDistance.out
index 35c0077a6f1..778e74cc8df 100644
--- a/regression-test/data/opensky_p2/sql/avgDistance.out
+++ b/regression-test/data/opensky_p2/sql/avgDistance.out
@@ -1,4 +1,4 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !avgDistance --
-1040768
+1040771
diff --git a/regression-test/data/opensky_p2/sql/count.out
b/regression-test/data/opensky_p2/sql/count.out
index 7db64bddff4..d2958cd8306 100644
--- a/regression-test/data/opensky_p2/sql/count.out
+++ b/regression-test/data/opensky_p2/sql/count.out
@@ -1,4 +1,4 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !count --
-66010789
+66010819
diff --git a/regression-test/data/opensky_p2/sql/mostBusyOrigin.out
b/regression-test/data/opensky_p2/sql/mostBusyOrigin.out
index a1cb7bacd58..e50cfe3860a 100644
--- a/regression-test/data/opensky_p2/sql/mostBusyOrigin.out
+++ b/regression-test/data/opensky_p2/sql/mostBusyOrigin.out
@@ -1,10 +1,10 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !mostBusyOrigin --
-KORD 745006 1545579
+KORD 745007 1545581
KDFW 696702 1358356
KATL 667286 1169451
KDEN 582709 1287105
-KLAX 581949 2628301
+KLAX 581952 2628335
KLAS 447789 1336521
KPHX 428558 1344938
KSEA 412592 1757171
@@ -17,13 +17,13 @@ KMSP 346010 1286682
LFPG 344748 2205349
EGLL 341370 3215745
EHAM 340272 2115434
-KEWR 337695 1826368
+KEWR 337696 1826380
KPHL 320762 1291422
OMDB 308855 2855438
UUEE 307098 1554257
KBOS 304416 1621088
LEMD 291787 1694186
-YSSY 272977 1875510
+YSSY 272979 1875620
KMIA 265121 1925366
ZGSZ 263497 745210
EDDM 256691 1360465
@@ -39,7 +39,7 @@ KFLL 223447 1465818
KDAL 212055 1081956
KDCA 207883 1012912
LIRF 207047 1427062
-PANC 206005 2524856
+PANC 206007 2524906
LTFJ 205415 859915
KDTW 204020 1106339
VABB 201679 1301971
@@ -47,12 +47,12 @@ OTHH 200797 3759551
KMDW 200796 1232101
KSAN 198003 1495154
KPDX 197760 1269035
-SBGR 197623 2042769
+SBGR 197624 2042850
VOBL 189011 1042172
LEBL 188956 1282274
-YBBN 188010 1254268
+YBBN 188011 1254324
LSZH 187934 1571073
-YMML 187642 1869850
+YMML 187643 1869929
RCTP 184466 2774386
KSNA 180045 778417
EGKK 176420 1693763
@@ -84,7 +84,7 @@ KAPA 140776 419958
KHOU 138985 1068669
KTPA 138033 1338877
KFFZ 137333 55312
-NZAA 136091 1581353
+NZAA 136092 1581418
YPPH 133916 1272160
RJBB 133522 1804668
EDDL 133018 1264868
diff --git a/regression-test/data/opensky_p2/sql/totalDistance.out
b/regression-test/data/opensky_p2/sql/totalDistance.out
index 53c175c8baf..9c38aa77ee0 100644
--- a/regression-test/data/opensky_p2/sql/totalDistance.out
+++ b/regression-test/data/opensky_p2/sql/totalDistance.out
@@ -1,4 +1,4 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !totalDistance --
-68700204389
+68700432352
diff --git
a/regression-test/data/segcompaction_p2/test_segcompaction_agg_keys.out
b/regression-test/data/segcompaction_p2/test_segcompaction_agg_keys.out
deleted file mode 100644
index 4d375f3ccb5..00000000000
--- a/regression-test/data/segcompaction_p2/test_segcompaction_agg_keys.out
+++ /dev/null
@@ -1,3 +0,0 @@
--- This file is automatically generated. You should know what you did if you
want to edit this
--- !select_default --
-47 Lychee Lychee Plum Banana Lychee Lychee Cherry Pineapple
Banana Watermelon Mango Apple Apple Peach Raspberry Grapes
Raspberry Raspberry Kiwi Orange Apple Plum Blueberry
Strawberry Orange Raspberry Strawberry Lemon Orange
Blueberry Apple Peach Banana Kiwi Orange Banana Strawberry
Lemon Mango Orange Peach Avocado Pineapple Kiwi Lemon Grapes
Strawberry Grapes Lychee
diff --git
a/regression-test/suites/compaction/test_base_compaction_with_dup_key_max_file_size_limit.groovy
b/regression-test/suites/compaction/test_base_compaction_with_dup_key_max_file_size_limit.groovy
index a006ea406d6..c40fb2dc592 100644
---
a/regression-test/suites/compaction/test_base_compaction_with_dup_key_max_file_size_limit.groovy
+++
b/regression-test/suites/compaction/test_base_compaction_with_dup_key_max_file_size_limit.groovy
@@ -65,27 +65,44 @@
suite("test_base_compaction_with_dup_key_max_file_size_limit", "p2") {
}
}
}
- try {
- String backend_id;
- def backendId_to_backendIP = [:]
- def backendId_to_backendHttpPort = [:]
- getBackendIpHttpPort(backendId_to_backendIP,
backendId_to_backendHttpPort);
-
- backend_id = backendId_to_backendIP.keySet()[0]
- def (code, out, err) =
show_be_config(backendId_to_backendIP.get(backend_id),
backendId_to_backendHttpPort.get(backend_id))
-
- logger.info("Show config: code=" + code + ", out=" + out + ", err=" +
err)
- assertEquals(code, 0)
- def configList = parseJson(out.trim())
- assert configList instanceof List
-
- boolean disableAutoCompaction = true
- for (Object ele in (List) configList) {
- assert ele instanceof List<String>
- if (((List<String>) ele)[0] == "disable_auto_compaction") {
- disableAutoCompaction = Boolean.parseBoolean(((List<String>)
ele)[2])
- }
+ String backend_id;
+ def backendId_to_backendIP = [:]
+ def backendId_to_backendHttpPort = [:]
+ getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
+
+ // Set a BE config on every backend (used to disable auto compaction
cluster-wide).
+ def set_be_param = { paramName, paramValue ->
+ for (String id in backendId_to_backendIP.keySet()) {
+ def beIp = backendId_to_backendIP.get(id)
+ def bePort = backendId_to_backendHttpPort.get(id)
+ def (rcode, rout, rerr) = curl("POST",
String.format("http://%s:%s/api/update_config?%s=%s";, beIp, bePort, paramName,
paramValue))
+ assertTrue(rout.contains("OK"))
}
+ }
+
+ backend_id = backendId_to_backendIP.keySet()[0]
+ def (code, out, err) =
show_be_config(backendId_to_backendIP.get(backend_id),
backendId_to_backendHttpPort.get(backend_id))
+ logger.info("Show config: code=" + code + ", out=" + out + ", err=" + err)
+ assertEquals(code, 0)
+ def configList = parseJson(out.trim())
+ assert configList instanceof List
+
+ // Capture the original cluster-wide disable_auto_compaction so it can be
restored.
+ boolean originalDisableAutoCompaction = false
+ for (Object ele in (List) configList) {
+ assert ele instanceof List<String>
+ if (((List<String>) ele)[0] == "disable_auto_compaction") {
+ originalDisableAutoCompaction =
Boolean.parseBoolean(((List<String>) ele)[2])
+ }
+ }
+
+ try {
+ // This test deterministically builds a [0-3] (>1G) single base rowset
via manual
+ // compactions, then expects a manual base compaction to be REJECTED
with E-808
+ // (input rowset exceeds
base_compaction_dup_key_max_file_size_mbytes). Background
+ // auto compaction would race those manual steps and reshape the
rowsets, making the
+ // result flaky, so disable it cluster-wide for the duration of the
test.
+ set_be_param("disable_auto_compaction", "true")
def triggerCompaction = { be_host, be_http_port, compact_type,
tablet_id ->
// trigger compactions for all tablets in ${tableName}
@@ -98,15 +115,14 @@
suite("test_base_compaction_with_dup_key_max_file_size_limit", "p2") {
String command = sb.toString()
logger.info(command)
def process = command.execute()
- code = process.waitFor()
- err = IOGroovyMethods.getText(new BufferedReader(new
InputStreamReader(process.getErrorStream())));
- out = process.getText()
- logger.info("Run compaction: code=" + code + ", out=" + out + ",
disableAutoCompaction " + disableAutoCompaction + ", err=" + err)
- if (!disableAutoCompaction) {
- return "Success, " + out
- }
- assertEquals(code, 0)
- return out
+ def ccode = process.waitFor()
+ def cerr = IOGroovyMethods.getText(new BufferedReader(new
InputStreamReader(process.getErrorStream())));
+ def cout = process.getText()
+ logger.info("Run compaction: code=" + ccode + ", out=" + cout + ",
err=" + cerr)
+ // curl exit code 0 means the HTTP request completed; the E-808
rejection is
+ // carried in the response body (cout), which the caller asserts
on.
+ assertEquals(ccode, 0)
+ return cout
}
sql """ DROP TABLE IF EXISTS ${tableName}; """
@@ -190,5 +206,7 @@
suite("test_base_compaction_with_dup_key_max_file_size_limit", "p2") {
def rowCount = sql "select count(*) from ${tableName}"
assertTrue(rowCount[0][0] != rows)
} finally {
+ // Restore the original cluster-wide auto-compaction setting.
+ set_be_param("disable_auto_compaction",
originalDisableAutoCompaction.toString())
}
}
diff --git
a/regression-test/suites/compaction/test_single_replica_compaction.groovy
b/regression-test/suites/compaction/test_single_replica_compaction.groovy
index 53f241dac31..10705ee2568 100644
--- a/regression-test/suites/compaction/test_single_replica_compaction.groovy
+++ b/regression-test/suites/compaction/test_single_replica_compaction.groovy
@@ -100,9 +100,9 @@ suite("test_single_compaction_p2", "p2") {
String command = sb.toString()
logger.info(command)
- process = command.execute()
- code = process.waitFor()
- out = process.getText()
+ def process = command.execute()
+ def code = process.waitFor()
+ def out = process.getText()
logger.info("Get compaction status: code=" + code + ", out=" + out)
assertEquals(code, 0)
def compactionStatus = parseJson(out.trim())
@@ -129,8 +129,8 @@ suite("test_single_compaction_p2", "p2") {
String command = sb.toString()
logger.info(command)
def process = command.execute()
- code = process.waitFor()
- out = process.getText()
+ def code = process.waitFor()
+ def out = process.getText()
logger.info("Get compaction status: code=" + code + ", out=" + out)
assertEquals(code, 0)
def tabletStatus = parseJson(out.trim())
diff --git a/regression-test/suites/inverted_index_p2/test_show_data.groovy
b/regression-test/suites/inverted_index_p2/test_show_data.groovy
index 5d86c1e7c05..db76dcb2915 100644
--- a/regression-test/suites/inverted_index_p2/test_show_data.groovy
+++ b/regression-test/suites/inverted_index_p2/test_show_data.groovy
@@ -813,7 +813,13 @@ suite("test_show_data_with_compaction", "p2") {
assertTrue(another_with_index_size != "wait_timeout")
logger.info("with_index_size is {}, another_with_index_size is {}",
with_index_size, another_with_index_size)
- assertEquals(another_with_index_size, with_index_size)
+ // Index compaction merges per-segment index files; for identical data
the total
+ // on-disk size may differ slightly from the non-compacted layout
(merge/file overhead).
+ // Compare within 10% tolerance instead of exact equality to avoid
flakiness, while
+ // still catching gross index bloat or corruption.
+ assertTrue(Math.abs(with_index_size - another_with_index_size)
+ <= 0.1 * Math.max(with_index_size,
another_with_index_size),
+ "index size mismatch beyond 10% tolerance:
with_index=${with_index_size}, without_index=${another_with_index_size}")
set_be_config.call("inverted_index_compaction_enable", "true")
@@ -824,7 +830,10 @@ suite("test_show_data_with_compaction", "p2") {
def data_size_2 = create_table_run_compaction_and_wait(tableName)
logger.info("data_size_1 is {}, data_size_2 is {}", data_size_1,
data_size_2)
- assertEquals(data_size_1, data_size_2)
+ // Same rationale as above: compare index sizes within 10% tolerance,
not exact equality.
+ assertTrue(Math.abs(data_size_1 - data_size_2)
+ <= 0.1 * Math.max(data_size_1, data_size_2),
+ "index size mismatch beyond 10% tolerance:
data_size_1=${data_size_1}, data_size_2=${data_size_2}")
} finally {
// sql "DROP TABLE IF EXISTS ${tableWithIndexCompaction}"
diff --git a/regression-test/suites/opensky_p2/load.groovy
b/regression-test/suites/opensky_p2/load.groovy
index d00be91d3fd..26dcee6ae27 100644
--- a/regression-test/suites/opensky_p2/load.groovy
+++ b/regression-test/suites/opensky_p2/load.groovy
@@ -58,7 +58,7 @@ suite("load"){
log.info("Stream load result: ${result}".toString())
def json = parseJson(result)
assertEquals("success", json.Status.toLowerCase())
- // assertEquals(json.NumberTotalRows,
json.NumberLoadedRows)
+ assertEquals(json.NumberTotalRows, json.NumberLoadedRows)
assertTrue(json.NumberLoadedRows > 0 && json.LoadBytes > 0)
}
}
diff --git
a/regression-test/suites/segcompaction_p2/test_segcompaction_agg_keys.groovy
b/regression-test/suites/segcompaction_p2/test_segcompaction_agg_keys.groovy
index 3c09af9c542..d16627f956f 100644
--- a/regression-test/suites/segcompaction_p2/test_segcompaction_agg_keys.groovy
+++ b/regression-test/suites/segcompaction_p2/test_segcompaction_agg_keys.groovy
@@ -86,7 +86,17 @@ suite("test_segcompaction_agg_keys") {
}
}
- qt_select_default """ SELECT * FROM ${tableName} WHERE col_0=47; """
+ // Cannot use qt_select_default here: S3 Load parallelizes across
multiple BE workers,
+ // each creating separate segments with non-deterministic sequence
numbers.
+ // REPLACE aggregation picks the value from the segment with the
highest sequence,
+ // so the winner among the 12 source rows with col_0=47 is not
guaranteed across runs.
+ def selectResult = sql """ SELECT * FROM ${tableName} WHERE col_0=47;
"""
+ assertEquals(1, selectResult.size(), "Expected exactly 1 row for
col_0=47 after REPLACE aggregation")
+ def row = selectResult[0].collect { it?.toString() }
+ def possibleResult1 = ["47", "Lychee", "Lychee", "Plum", "Banana",
"Lychee", "Lychee", "Cherry", "Pineapple", "Banana", "Watermelon", "Mango",
"Apple", "Apple", "Peach", "Raspberry", "Grapes", "Raspberry", "Raspberry",
"Kiwi", "Orange", "Apple", "Plum", "Blueberry", "Strawberry", "Orange",
"Raspberry", "Strawberry", "Lemon", "Orange", "Blueberry", "Apple", "Peach",
"Banana", "Kiwi", "Orange", "Banana", "Strawberry", "Lemon", "Mango", "Orange",
"Peach", "Avocado", "Pineapple", "Ki [...]
+ def possibleResult2 = ["47", "Banana", "Watermelon", "Lychee",
"Blueberry", "Raspberry", "Strawberry", "Grapes", "Watermelon", "Lemon",
"Lemon", "Pineapple", "Watermelon", "Peach", "Kiwi", "Lychee", "Peach",
"Pineapple", "Raspberry", "Grapes", "Lychee", "Raspberry", "Peach", "Kiwi",
"Pineapple", "Apple", "Lemon", "Lychee", "Pineapple", "Blueberry", "Blueberry",
"Avocado", "Cherry", "Kiwi", "Cherry", "Watermelon", "Plum", "Banana", "Peach",
"Pineapple", "Apple", "Strawberry", "Avo [...]
+ assertTrue(row == possibleResult1 || row == possibleResult2,
+ "Result for col_0=47 does not match either expected outcome.\nGot:
${row}")
String[][] tablets = sql """ show tablets from ${tableName}; """
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
