github-actions[bot] commented on code in PR #69797: URL: https://github.com/apache/doris/pull/69797#discussion_r4597985001
########## regression-test/suites/fault_injection_p0/test_overlapping_segments_merge.groovy: ########## @@ -0,0 +1,161 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// One load that flushes several memtables writes a rowset whose segments overlap on keys, and +// reading it merges the segments before the rowsets. Each table below gets one such rowset in +// which every key sits in both segments, and the later segment holds the newer rows: +// - AGG_KEYS: every key folds into one row, REPLACE takes the newer segment's value; +// - merge-on-read UNIQUE_KEYS: the newer segment's row wins; +// - UNIQUE_KEYS with a sequence map: each column group takes the row with its own largest +// sequence value, which is in a different segment for each group. +// The load may write either of its Blocks first, so the expected AGG_KEYS and UNIQUE_KEYS rows come +// from the rows in storage order. The queries read with Blocks of 3 rows, so key groups are cut by +// Block boundaries, and the results must stay the same after a full compaction rewrites the rowset. +suite("test_overlapping_segments_merge", "nonConcurrent") { + sql "DROP TABLE IF EXISTS test_overlapping_segments_merge_agg" + sql "DROP TABLE IF EXISTS test_overlapping_segments_merge_uniq" + sql "DROP TABLE IF EXISTS test_overlapping_segments_merge_seq_map" + sql """ + CREATE TABLE test_overlapping_segments_merge_agg ( + k INT, + s BIGINT SUM, + r INT REPLACE + ) AGGREGATE KEY(k) + DISTRIBUTED BY HASH(k) BUCKETS 1 + PROPERTIES ("replication_num" = "1", "disable_auto_compaction" = "true") + """ + sql """ + CREATE TABLE test_overlapping_segments_merge_uniq ( + k INT, + v INT + ) UNIQUE KEY(k) + DISTRIBUTED BY HASH(k) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1", + "enable_unique_key_merge_on_write" = "false", + "disable_auto_compaction" = "true" + ) + """ + sql """ + CREATE TABLE test_overlapping_segments_merge_seq_map ( + k INT, + v1 INT, + s1 BIGINT, + v2 INT, + s2 BIGINT + ) UNIQUE KEY(k) + DISTRIBUTED BY HASH(k) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1", + "enable_unique_key_merge_on_write" = "false", + "light_schema_change" = "true", + "sequence_mapping.s1" = "v1", + "sequence_mapping.s2" = "v2", + "disable_auto_compaction" = "true" + ) + """ + + try { + // MemTable.need_flush flushes every Block into its own segment. With Blocks of 20 rows, + // numbers 0..19 and 20..39 become two segments that both hold keys 0..19 + // (k = number % 20), in the order the load writes the Blocks. + GetDebugPoint().enableDebugPointForAllBEs("MemTable.need_flush") + sql "SET batch_size = 20" Review Comment: [P2] Force writer-side flushes in the AGG/MoR fixture With a single insert task using the legacy sink (for example, cloud mode with the fuzzy `parallel_pipeline_task_num=1` setting), these 40 rows do not produce two writer batches: `VNodeChannel::init` floors its RPC batch size at 8192, so it buffers both 20-row source Blocks into one receiver write. `MemTableWriter::write` checks `MemTable.need_flush` only after inserting that entire write, producing one segment; the `hasMergeLayout` assertion then fails before any AGG/MoR/sequence-map or compaction checks. The single-task setting at line 124 occurs after these loads. This is a distinct fixture from the TopN test already changed to 8192/16384 batching. Construct separate writer-side writes on both sink implementations and set the required load parallelism before INSERT. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
