eldenmoon commented on PR #70408:
URL: https://github.com/apache/doris/pull/70408#issuecomment-6998961193

   [P1] Non-atomic RESTORE can cause compaction to drop VARIANT fields
   Restoring a V1 count=0 backup into an existing V2 count=0 table can leave 
both layouts at the same schema version. If compaction picks the V1 schema, it 
can mark a sparse-only path for extraction without creating a subcolumn for it. 
The sparse merge then drops the value. For example, after restoring {"a":1} and 
inserting {"b":2}, compaction can lose b=2 while preserving the row count.
   
   The current guard only covers V2 targets, so this can happen on a fully 
upgraded cluster. Please handle the V1-target case before merge and add a 
RESTORE → INSERT → compaction regression test that checks the field value. 
[Compaction 
code](https://github.com/apache/doris/blob/7583d8b979103817dca8deb070964adbd520992b/be/src/exec/common/variant_util.cpp#L1291-L1335)
   
   
   [P1] Persisted SQL can fail to reparse after upgrade
   Existing Routine Load jobs, views, and materialized views can contain 
VARIANT expressions with variant_enable_typed_paths_to_sparse=true, 
variant_doc_materialization_min_rows, or variant_doc_hash_shard_count. The new 
parser rejects these forms. Replaying a Routine Load job can set it to 
CANCELLED, and view queries or MV refreshes can fail when Doris reparses their 
definitions.
   
   Please preserve compatibility for persisted SQL before merge and add a 
replay test using an old definition. [Routine Load 
replay](https://github.com/apache/doris/blob/7583d8b979103817dca8deb070964adbd520992b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L2109-L2163)
   
   
   [P2] Compaction applies the extraction threshold to input rows
   For ordinary VARIANT columns, compaction compares 
variant_subcolumn_extraction_min_rows with the sum of input rowset rows. With a 
threshold of 100, it can still extract subcolumns when 120 input rows 
deduplicate to 60 output rows. Splitting the output across segments can produce 
the same mismatch. This changes when extraction happens and can increase write 
and storage costs. The existing review comment on this behavior still applies. 
[Threshold 
calculation](https://github.com/apache/doris/blob/7583d8b979103817dca8deb070964adbd520992b/be/src/exec/common/variant_util.cpp#L1229-L1238)
   
   
   [Other] Mixed-version writes with a lowered be_exec_version
   The default be_exec_version=19 prevents 4.0 and 4.1 BEs from accepting 
normal load blocks. However, the FE still creates V2 columns when that setting 
is lowered. If a load reaches an older BE with a compatible execution version, 
the BE can ignore variant_serialize_type and write count=0 using V1 semantics.
   
   Is this configuration supported? If so, please either prevent older BEs from 
writing V2 count=0 columns or add a mixed-version load test that verifies 
compaction preserves sparse-only fields when it selects the V1 schema.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to