This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d6c5a8da688 [test](regression) Stabilize update_after_create_rollup
and add test_writer_profile (#70309)
d6c5a8da688 is described below
commit d6c5a8da68853d2265cd6afd0656f27ccf6382ca
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Nov 7 05:42:23 2026 +0800
[test](regression) Stabilize update_after_create_rollup and add
test_writer_profile (#70309)
### Background
`query_p0/update/update_after_create_rollup` checks partial `UPDATE` on
a merge-on-write table that has a rollup. `ALTER TABLE ... ADD ROLLUP`
starts an FE job that sends one alter task per tablet to the BEs, and
`SHOW ALTER TABLE ROLLUP` reports the job's state. Separately, a load
through the move-memtable sink (`enable_memtable_on_sink_node`, on by
default) can add a `DeltaWriterV2 <tablet>` and a `MemTableWriter
<tablet>` child to its profile for each tablet; since #64620 the BE
attaches them only when the session's `profile_level` is at least 2, the
default.
### What problem does this PR solve?
Issue Number: None
Related PR: #65310 (the branch-4.1 fix of the rollup suite by @shuke987,
ported here), #59863 (added test_writer_profile on branch-4.0, by
@0AyanamiRei), #65232 (pins its first half to profile level 1 on
branch-4.1, by @shuke987), #64620 (the master gate that
test_writer_profile covers), #64378 (made profile level 2 the default),
#61842 (moved the rollup suite from nereids_p0 to query_p0)
PR Type: Test
Problem Summary:
`update_after_create_rollup` sleeps 10 s after `ADD ROLLUP` and then
requires EXPLAIN of an unrelated GROUP BY to pick `rollup1`, so it fails
whenever the rollup job takes longer; as in #65310, it now waits until
the job reports FINISHED. `test_writer_profile`, which branch-4.x has
and master lacks, checks that a load's profile hides the writer details
at profile level 1 and shows them at level 2; master has had that gate
since #64620, with no test. The rollup suite becomes identical to
branch-4.2; `test_writer_profile` also waits for `COMPLETE` profiles and
honors TLS.
### Details
- **Failure:** `update_after_create_rollup` has not failed on its own in
master's public pipelines: since June 30, P0 ran it 7,257 times and
cloud_p0 739 times, and all 55 failures were in builds where most suites
failed. A forced slow rollup job reproduces it (see Testing).
- **Cause:** The rollup job converts all 54 tablets on the BEs, so how
long it takes depends on the BE's alter task queue, while the suite
allows a fixed 10 s. The EXPLAIN check also depends on the optimizer
choosing the rollup, which suites such as `rollup_p0/test_rollup_agg`
cover.
- **Stability evidence:** `getAlterRollupFinalState` polls `SHOW ALTER
TABLE ROLLUP` every second for up to 300 s and returns the final state,
so a CANCELLED job still fails. With each tablet's alter task delayed by
5 s, master's suite fails at the EXPLAIN check and the ported one passes
after 91 s.
- **Scenarios:** In `test_writer_profile`, an `INSERT INTO ... SELECT`
at `profile_level=1`, whose profile must contain neither `DeltaWriterV2`
nor `MemTableWriter`, then the same INSERT at `profile_level=2`, whose
profile must contain both; both checks wait for a `COMPLETE` profile,
which the BE reports after the INSERT returns; cloud mode is skipped.
- **What existing tests miss:** The closest suite,
`query_profile/test_load_channel_profile`, turns
`enable_memtable_on_sink_node` off and checks only that level 3 shows
writer details; no suite checks the level-1 gate of the move-memtable
sink.
- **Fails when broken:** Without `set profile_level=1`, the first half
fails with `should not contain DeltaWriterV2`, as a missing gate would
make it; with the second INSERT at level 1, the second half fails with
`should contain DeltaWriterV2`.
- **External dependencies:** Both INSERTs read `load/data by line.json`
from the regression bucket through the `S3` TVF.
### Testing
All runs used `./run-regression-test.sh --run -d query_p0/update -s
update_after_create_rollup` or `--run -d query_profile -s
test_writer_profile` against a local single-FE, single-BE cluster built
from master.
| Suite | Version | Result |
|---|---|---|
| `update_after_create_rollup` | master | passes |
| `update_after_create_rollup` | this PR | passes; the job was FINISHED
about 1 s after `ADD ROLLUP` |
| `update_after_create_rollup` | master, with BE debug point
`SchemaChangeJob::process_alter_tablet.leave.sleep` (5 s per tablet) |
fails: `Explain and check failed, expect contains 'rollup1'`, with the
job RUNNING at 9/54 tasks |
| `update_after_create_rollup` | this PR, same debug point | passes
after waiting 91 s |
| `update_after_create_rollup` | this PR, with `CANCEL ALTER TABLE
ROLLUP` right after `ADD ROLLUP` | fails as it should: `expected:
<FINISHED> but was: <CANCELLED>` |
| `test_writer_profile` | this PR | passes; each profile was `COMPLETE`
at the first read, the level-2 one has a `DeltaWriterV2` and a
`MemTableWriter` child, the level-1 one neither |
| `test_writer_profile` | this PR without #65232's `set profile_level=1`
| fails: `should not contain DeltaWriterV2 ==> expected: <false> but
was: <true>` |
| `test_writer_profile` | this PR, second INSERT at level 1 | fails as
it should: `should contain DeltaWriterV2 ==> expected: <true> but was:
<false>` |
My local regression config's OSS key is rejected (`InvalidAccessKeyId`),
so the `test_writer_profile` runs used a copy whose INSERT reads
`numbers("number" = "2")` instead of the `S3` TVF; nothing else differs.
The PR's NonConcurrent pipeline runs that suite as committed, and P0 and
cloud_p0 run the rollup suite.
### Notes for Reviewers
- After the CI review, `test_writer_profile` finds each INSERT's profile
by a marker through `ProfileAction`, where branch-4.2 reads the newest
LOAD profile over plain HTTP 500 ms after the INSERT.
- Two more suites from the same branch-4.1 batch need nothing on master:
master's renamed `test_jdbc_refresh_catalog_schema_refresh_non_blocking`
(#65126) already has #65335's fixes, and master #65696 replaced #65234's
suite with an FE unit test.
Co-authored-by: shuke <[email protected]>
Co-authored-by: Refrain <[email protected]>
---
.../update/update_after_create_rollup.groovy | 11 +--
.../query_profile/test_writer_profile.groovy | 92 ++++++++++++++++++++++
2 files changed, 93 insertions(+), 10 deletions(-)
diff --git
a/regression-test/suites/query_p0/update/update_after_create_rollup.groovy
b/regression-test/suites/query_p0/update/update_after_create_rollup.groovy
index 1442e235742..d1dbaaaffe4 100644
--- a/regression-test/suites/query_p0/update/update_after_create_rollup.groovy
+++ b/regression-test/suites/query_p0/update/update_after_create_rollup.groovy
@@ -123,16 +123,7 @@ suite('update_after_create_rollup') {
sql """
ALTER TABLE mow_table ADD ROLLUP rollup1(event_date, event_time,
user_id, country, update_time)
"""
- sleep(10000)
- explain {
- sql('''
- SELECT event_date, country, count(*)
- FROM mow_table
- WHERE event_date = '2025-09-22'
- GROUP BY event_date, country
- ''')
- contains "rollup1"
- }
+ assertEquals("FINISHED", getAlterRollupFinalState("mow_table"))
// Test 1: UPDATE column not in rollup1 (city)
sql 'UPDATE mow_table SET city = "beijing" WHERE user_id = 2000'
diff --git a/regression-test/suites/query_profile/test_writer_profile.groovy
b/regression-test/suites/query_profile/test_writer_profile.groovy
new file mode 100644
index 00000000000..161bb601dd6
--- /dev/null
+++ b/regression-test/suites/query_profile/test_writer_profile.groovy
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.action.ProfileAction
+
+suite('test_writer_profile', "nonConcurrent") {
+ //cloud-mode
+ if (isCloudMode()) {
+ return
+ }
+
+ sql "set enable_profile=true;"
+ sql "set profile_level=1;"
+
+ def s3Endpoint = getS3Endpoint()
+ def s3Region = getS3Region()
+ def ak = getS3AK()
+ def sk = getS3SK()
+ def s3Uri = "s3://${getS3BucketName()}/load/data by line.json"
+
+ sql "drop table if exists t;"
+ sql """
+ CREATE TABLE t(
+ a INT,
+ b INT
+ )
+ DUPLICATE KEY(a)
+ PROPERTIES("replication_num" = "1");
+ """
+
+ // The INSERTs below run on the master FE, which keeps their profiles.
ProfileAction reads them
+ // with HttpCliAction, which uses HTTPS when the regression config sets
enableTLS.
+ def master = sql_return_maparray("SHOW FRONTENDS").find {
it.IsMaster.toString() == "true" }
+ assertNotNull(master, "Cannot locate the master FE")
+ def profileAction = new ProfileAction(context,
+ new InetSocketAddress(master.Host.toString(),
master.HttpPort.toString().toInteger()))
+
+ // BE reports the writer profiles after the INSERT returns, so wait until
the profile of this
+ // INSERT, found by its marker, is COMPLETE before checking which writer
details it has.
+ def insertAndGetProfile = {
+ def marker =
"test_writer_profile_${UUID.randomUUID().toString().replace('-', '')}"
+ sql """
+ /* ${marker} */ INSERT INTO t
+ SELECT * FROM S3(
+ "uri" = "$s3Uri",
+ "s3.access_key" = "$ak",
+ "s3.secret_key" = "$sk",
+ "s3.endpoint" = "${s3Endpoint}",
+ "s3.region" = "${s3Region}",
+ "format" = "json",
+ "read_json_by_line" = "true"
+ );
+ """
+ def profileString = profileAction.getProfileBySql(marker, [], 60000L,
500L)
+ logger.info(profileString)
+ return profileString
+ }
+
+ try {
+ def profileString = insertAndGetProfile()
+ assertFalse(profileString.contains("DeltaWriterV2"), "should not
contain DeltaWriterV2")
+ assertFalse(profileString.contains("MemTableWriter"), "should not
contain MemTableWriter")
+ } finally {
+ sql "set enable_profile=false;"
+ }
+
+ sql "set enable_profile=true;"
+ sql "set profile_level=2;"
+
+ try {
+ def profileString = insertAndGetProfile()
+ assertTrue(profileString.contains("DeltaWriterV2"), "should contain
DeltaWriterV2")
+ assertTrue(profileString.contains("MemTableWriter"), "should contain
MemTableWriter")
+ } finally {
+ sql "set enable_profile=false;"
+ sql "set profile_level=1;"
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
