This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dcf82859fd0 [test](regression) Stabilize test_auto_new_recycle and
test_sql_digest_generation (#70308)
dcf82859fd0 is described below
commit dcf82859fd0f9d660f62d04168df2945c7bfd9cf
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Nov 6 11:43:27 2026 +0800
[test](regression) Stabilize test_auto_new_recycle and
test_sql_digest_generation (#70308)
### Background
An auto-partition table with `partition.retention_count = N` keeps only
its N latest historical partitions: the FE's `DynamicPartitionScheduler`
drops older ones on every run, once per
`dynamic_partition_check_interval_seconds` (600 by default).
`partition_p0/auto_partition/test_auto_new_recycle` lowers that interval
to 1 s and checks the partition count after each load or ALTER.
`audit/test_sql_digest_generation` checks that the audit log gets an
`sql_digest` only for queries slower than the FE config
`sql_digest_generation_threshold_ms`, once with the threshold at 0 and
once at 100000. Both suites are in group `nonConcurrent`.
### What problem does this PR solve?
Issue Number: None
Related PR: #65375 and #65646 (the branch-4.1 recycle-suite fixes by
@shuke987, ported here), #65343 (the branch-4.1 digest-suite fix by
@shuke987, ported here), #65435 (master's earlier move of the recycle
suite's day-boundary guard)
PR Type: Test
Problem Summary:
Both suites rely on timing the product does not promise. The recycle
suite checks the partition count after a fixed `sleep(8000)` (5 s after
an ALTER), while the drop happens asynchronously at an unbounded time
after the config change; branch-4.1's NonConcurrent pipeline failed at
the first such check repeatedly (#65375). The digest suite's threshold-0
check needs the query's elapsed time to be above 0 ms. This PR gives
both suites their branch-4.1/4.2 form: the recycle suite polls the
partition count every second, for at most 1200 s, and widens its
day-boundary guard to cover the poll; the digest query adds `sleep(1)`.
Both files become identical to branch-4.2.
### Details
- **Failure:** On master's public NonConcurrent pipeline neither suite
has failed on its own since mid-July: of about 8,000 runs each, the
recycle suite failed 3 times and the digest suite 87 times, all in
builds whose cluster had died (most suites failing). The failures that
motivated the fixes are on branch-4.1's pipeline (#65375); #65343 made
the digest query deterministic there.
- **Cause:** `ADMIN SET FRONTEND CONFIG` sets the scheduler's new
interval at once, but the scheduler sleeps in 5-second slices and
notices the change only at the next slice. It then runs
`executeDynamicPartition` over every registered table before the drop.
So the drop lands 0 to 5 s plus one full pass after the config change,
and a pass longer than about 3 s fails the 8-second check. In the digest
suite, `AuditLogHelper` writes a digest only when `elapseMs >
sql_digest_generation_threshold_ms`, and nothing bounds the elapsed time
of `select *` on a two-row table from below (9 to 15 ms locally). With
`sleep(1)` on each row it takes about 2 s, still far below the second
check's 100000 ms.
- **Stability evidence:** The recycle checks now wait for the drop and
fail only when the count misses the target for 1200 s. The poll runs on
the suite thread, because the framework sets
`Awaitility.pollInSameThread()` (#68338), so it does not leak
connections. Shortening one of master's waits to 1 s reproduces the
branch-4.1 failure shape, and the ported suite passes; mutants show that
both suites still catch wrong results (see Testing).
### Testing
All runs used `./run-regression-test.sh --run -d
partition_p0/auto_partition -s test_auto_new_recycle` or `--run -d audit
-s test_sql_digest_generation` against a local single-FE, single-BE
cluster built from master.
| Suite | Version | Result |
|---|---|---|
| `test_auto_new_recycle` | master | passes |
| `test_auto_new_recycle` | this PR | passes; the polls reached the
target after 1, 2, 1, 3 and 3 s |
| `test_auto_new_recycle` | master, with the wait before the
10-partition check cut to 1 s | fails as #65375 describes: `expected:
<101> but was: <10>` (master's `assertEquals` has its arguments swapped,
so 101 partitions were still there) |
| `test_auto_new_recycle` | this PR, expecting 11 partitions, poll
capped at 20 s | fails as it should: `ConditionTimeoutException` |
| `test_sql_digest_generation` | master | passes |
| `test_sql_digest_generation` | this PR | passes; the query took 2,024
and 2,025 ms |
| `test_sql_digest_generation` | this PR, second threshold 1000 ms |
fails as it should: `Count of sql_digest changed` |
The PR NonConcurrent pipeline runs both suites; cloud_p0 skips
`nonConcurrent` suites.
### Notes for Reviewers
- The day-boundary guard before the final `now()`-based section goes
from 100 s to 1300 s, as in #65646, so that it covers the 1200 s poll
plus the insert and checks. It sleeps only when that section would
otherwise start within 1300 s of midnight.
- Four more suites fixed in the same branch-4.1 batch need nothing on
master: #65340 (master #68449 enables the publish debug point on all
FEs, superseding the open port #65556), #65929 (master #68329 fixed the
actual cause, a connection leak), #65885 (master #69723 removed the V1
scanner whose message it also accepted), and #67363 (ported as #67402).
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: the port (both suites taken from branch-4.2), the local
verification runs, and this description
- Human review: not stated
- Prompts: The author asked to continue moving branch-4.2's changes back
to master, where six branch-4.1 test fixes were recorded as one task.
After the check showed that master already covers four of them and has
not failed on the other two, the author chose to port those two exactly
as branch-4.2 has them and to verify them on a local cluster.
Co-authored-by: shuke <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../suites/audit/test_sql_digest_generation.groovy | 4 +--
.../auto_partition/test_auto_new_recycle.groovy | 35 ++++++++++++----------
2 files changed, 21 insertions(+), 18 deletions(-)
diff --git a/regression-test/suites/audit/test_sql_digest_generation.groovy
b/regression-test/suites/audit/test_sql_digest_generation.groovy
index f02f89da20e..382ab07d145 100644
--- a/regression-test/suites/audit/test_sql_digest_generation.groovy
+++ b/regression-test/suites/audit/test_sql_digest_generation.groovy
@@ -43,7 +43,7 @@ suite("test_sql_digest_generation", "nonConcurrent") {
Long prevSqlDigestCount = Long.valueOf(prevSqlDigestCountResult[0][0])
logger.info("prev sql_digest count: " + prevSqlDigestCount)
- sql "select * from audit_log_behavior"
+ sql "select *, sleep(1) from audit_log_behavior"
Thread.sleep(10000)
sql """call flush_audit_log()"""
@@ -66,7 +66,7 @@ suite("test_sql_digest_generation", "nonConcurrent") {
Long prevSqlDigestCount = Long.valueOf(prevSqlDigestCountResult[0][0])
logger.info("prev sql_digest count: " + prevSqlDigestCount)
- sql "select * from audit_log_behavior"
+ sql "select *, sleep(1) from audit_log_behavior"
Thread.sleep(10000)
sql """call flush_audit_log()"""
diff --git
a/regression-test/suites/partition_p0/auto_partition/test_auto_new_recycle.groovy
b/regression-test/suites/partition_p0/auto_partition/test_auto_new_recycle.groovy
index f9027423a07..c48a1674013 100644
---
a/regression-test/suites/partition_p0/auto_partition/test_auto_new_recycle.groovy
+++
b/regression-test/suites/partition_p0/auto_partition/test_auto_new_recycle.groovy
@@ -15,6 +15,9 @@
// specific language governing permissions and limitations
// under the License.
+import java.util.concurrent.TimeUnit
+import org.awaitility.Awaitility
+
suite("test_auto_new_recycle", "nonConcurrent") {
sql "drop table if exists auto_recycle"
test {
@@ -133,6 +136,16 @@ suite("test_auto_new_recycle", "nonConcurrent") {
);
"""
+ def waitPartitionCount = { int expectedCount ->
+ Awaitility.await().atMost(1200, TimeUnit.SECONDS).pollInterval(1,
TimeUnit.SECONDS).until {
+ res = sql "show partitions from auto_recycle"
+ logger.info("partition count: ${res.size()}, expected:
${expectedCount}")
+ res.size() == expectedCount
+ }
+ res = sql "show partitions from auto_recycle"
+ assertEquals(expectedCount, res.size())
+ }
+
sql """
insert into auto_recycle select date_add('2020-01-01 00:00:00', interval
number day) from numbers("number" = "100");
"""
@@ -144,9 +157,7 @@ suite("test_auto_new_recycle", "nonConcurrent") {
assertTrue(res[0][1].contains('"partition.retention_count" = "3"'))
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '1') """
- sleep(8000)
- res = sql "show partitions from auto_recycle"
- assertEquals(res.size(), 3)
+ waitPartitionCount.call(3)
// before everytime insert data, we should avoid recycle because if they
conflict, insert will fail
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '600') """
@@ -155,15 +166,11 @@ suite("test_auto_new_recycle", "nonConcurrent") {
insert into auto_recycle select date_add('2020-01-01 00:00:00', interval
number day) from numbers("number" = "100");
"""
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '1') """
- sleep(8000)
- res = sql "show partitions from auto_recycle"
- assertEquals(res.size(), 3)
+ waitPartitionCount.call(3)
qt_sql0 "select * from auto_recycle order by k0"
sql "alter table auto_recycle set ('partition.retention_count' = '1')"
- sleep(5000)
- res = sql "show partitions from auto_recycle"
- assertEquals(res.size(), 1)
+ waitPartitionCount.call(1)
sql "alter table auto_recycle set ('partition.retention_count' = '10')"
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '600') """
@@ -172,9 +179,7 @@ suite("test_auto_new_recycle", "nonConcurrent") {
insert into auto_recycle select date_add('2022-01-01 00:00:00', interval
number day) from numbers("number" = "100");
"""
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '1') """
- sleep(8000)
- res = sql "show partitions from auto_recycle"
- assertEquals(res.size(), 10)
+ waitPartitionCount.call(10)
qt_sql1 "select * from auto_recycle order by k0"
// verify not consider future partition when do recycle
@@ -191,14 +196,12 @@ suite("test_auto_new_recycle", "nonConcurrent") {
"""
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '600') """
sleep(8000)
- waitUntilSafeExecutionTime("NOT_CROSS_DAY_BOUNDARY", 100)
+ waitUntilSafeExecutionTime("NOT_CROSS_DAY_BOUNDARY", 1300)
sql """
insert into auto_recycle select date_add(now(), interval number-5 day)
from numbers("number" = "8");
"""
sql """ admin set frontend config
('dynamic_partition_check_interval_seconds' = '1') """
- sleep(8000)
- res = sql "show partitions from auto_recycle"
- assertEquals(res.size(), 6) // [-5, -1] -> [-3, -1], [0, 2] -> [0, 2]
+ waitPartitionCount.call(6) // [-5, -1] -> [-3, -1], [0, 2] -> [0, 2]
res = sql "select * from auto_recycle order by k0"
assertEquals(res.size(), 6)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
