github-actions[bot] commented on code in PR #70408:
URL: https://github.com/apache/doris/pull/70408#discussion_r4596846325


##########
be/src/exec/common/variant_util.cpp:
##########
@@ -857,8 +900,14 @@ Status 
VariantCompactionUtil::aggregate_variant_extended_info(
 
 // get the subpaths and sparse paths for the variant column
 void VariantCompactionUtil::get_subpaths(int32_t max_subcolumns_count,
+                                         VariantSerializeTypePB serialize_type,
                                          const PathToNoneNullValues& stats,
                                          VariantCompactionPaths& 
paths_set_info) {
+    // Count 0 means no limit under V1, and no dynamic subcolumn under V2 
(every path stays sparse).
+    if (max_subcolumns_count == 0 &&
+        serialize_type == VariantSerializeTypePB::VARIANT_SERIALIZE_V2) {
+        return;

Review Comment:
   [P1] Fold historical extracted paths into the V2 sparse output. A new FE can 
create a V2 count-zero column on an old BE, which ignores the new optional 
field and stores it as V1 with every dynamic path extracted. Insert {"a":1}, 
upgrade that BE, then perform a light ADD COLUMN and load a different path such 
as {"b":2}: the higher-version V2 rowset becomes the compaction target. Old 
segments still open with their own V1 schema, so 
check_variant_serialize_type_layout does not reject them. Returning here leaves 
sparse_path_set empty: _create_sparse_merge_reader skips a, while the V2 output 
schema creates no dynamic subcolumns and ROOT_ONLY does not reconstruct object 
children. Since the statistics check only visits output paths, it also misses 
the disappearance of a. Compaction can therefore silently discard that 
committed value. Populate sparse_path_set so historical extracted paths are 
folded into the sparse output, and cover V1 input rowsets compacted into a V2 
count-zero target.



##########
be/src/exec/common/variant_util.cpp:
##########
@@ -1179,6 +1228,13 @@ Status 
VariantCompactionUtil::get_extended_compaction_schema(
         }
     }
 
+    // The rows this compaction writes, compared with 
variant_subcolumn_extraction_min_rows below.
+    // A load decides per segment; here the whole output stands for its 
segments.
+    int64_t input_rows = 0;
+    for (const auto& rs : rowsets) {
+        input_rows += rs->num_rows();

Review Comment:
   [P2] Base deferred extraction on output segment rows. This sum counts rows 
before compaction merges or filters them, and it also combines rows that may go 
into separate output files. For example, with 
variant_subcolumn_extraction_min_rows=5, two merge-on-read UNIQUE KEY rowsets 
containing the same three keys stay sparse when loaded, but this code selects 
subcolumns from six input rows even though compaction writes only three rows. 
The ordinary writer then takes its root-only path and writes the selected 
subcolumns without invoking select_variant_paths, so the actual file size never 
corrects the decision. Use output segment row counts (and fold paths back into 
sparse below the threshold), with coverage for duplicate elimination and output 
splitting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to