Doris-Breakwater commented on issue #70687:
URL: https://github.com/apache/doris/issues/70687#issuecomment-6945325040

   Breakwater-GitHub-Analysis-Slot: slot_ba74eaf8ca14
   
   **Initial assessment:** This warrants BE storage-iterator/CSE triage because 
the supplied trace shows a fatal column-type assertion in Storage Top-N. Source 
inspection at the exact reported commit, 
`4a3fce2a8fc1000b90f14793241aeeb574402e7e`, identifies a concrete 
placeholder-preservation path consistent with the crash. This is a static 
analysis; I have not reproduced the query or validated a fix. The issue 
currently has no labels or assignee.
   
   **What the code establishes**
   
   1. A newly allocated scanner block initializes virtual slots as 
`ColumnNothing(0)`: 
[`ScannerContext::get_free_block`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/exec/scan/scanner_context.cpp#L299-L310)
 constructs the block from slot descriptors, and 
[`SlotDescriptor::get_empty_mutable_column`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/runtime/descriptors.cpp#L192-L197)
 supplies the placeholder. Clearing the block preserves that runtime column 
class.
   2. Storage key Top-N requires ordered rowset output. 
[`BetaRowsetReader::is_merge_iterator`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/storage/rowset/beta_rowset_reader.h#L61-L64)
 selects the merge path when ordered output is required, the reader covers more 
than one segment, and the rowset is marked as having overlapping segments. 
Overlap across separate rowsets alone does not establish this condition.
   3. SegmentIterator normally materializes virtual values before returning its 
output; see 
[`_materialization_of_virtual_column`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/storage/segment/segment_iterator.cpp#L3920-L3958).
 However, 
[`VMergeIteratorContext::copy_rows(Block*)`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/storage/iterator/vgeneric_iterators.cpp#L130-L156)
 inserts those source values into the caller's existing destination columns 
without replacing placeholder destinations. `ColumnNothing` inherits 
[`IColumnDummy::insert_range_from`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/core/column/column_dummy.h#L81-L85),
 which only increments the size and ignores the source values. Thus, even 
materialized segment output can become a nonempty `ColumnNothing` in merge 
output.
   4. 
[`VCollectIterator::_topn_next`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/storage/iterator/vcollect_iterator.cpp#L294-L305)
 already creates concrete virtual columns for its candidate accumulator. After 
reading the rowset, it calls `mutable_block.add_rows` at line 396. A 
placeholder in the source then reaches 
[`ColumnVector::insert_range_from`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/be/src/core/column/column_vector.cpp#L360-L373),
 whose checked cast matches the supplied crash stack.
   
   This suggests a more specific failure mechanism than a segment simply 
returning an unmaterialized expression: **the merge destination can preserve 
the placeholder and discard already materialized values, with the fatal cast 
occurring later in Top-N**. The posted stack identifies Top-N's copy as the 
abort site; it does not itself show a cast inside VMergeIterator or identify 
the source runtime type.
   
   **Information needed to confirm the reported execution**
   
   - Minimal `CREATE TABLE`, failing SQL, and data/load steps. Include the key 
model, sort/cluster keys, nullability, `ORDER BY`/`LIMIT`, and the repeated 
expression that becomes a virtual slot.
   - `EXPLAIN VERBOSE` and relevant session settings, especially 
`experimental_enable_virtual_slot_for_cse` and `enable_segment_limit_pushdown`; 
confirm both FE and BE build commits and any local patches.
   - The BE fatal log immediately preceding the stack, especially `Bad cast 
from type:... to ...`. The supplied stack identifies the destination class but 
omits the source class.
   - The actual affected tablet/rowset and segment metadata showing multiple 
overlapping segments in the same reader. The stack header's `tablet id: 0` is 
insufficient to identify that rowset. Explain how loads/flushes and compaction 
produce or preserve the layout, plus reproduction frequency.
   
   **Recommended next steps**
   
   - In an isolated reproduction, capture the virtual slot ordinal, declared 
datatype, runtime column class, and size at segment output, immediately 
before/after merge `copy_rows`, and before Top-N `add_rows`. A concrete source 
becoming a `Nothing` merge destination would directly confirm this path.
   - For a session-scoped isolation/workaround trial, use `SET 
experimental_enable_virtual_slot_for_cse = false;` and re-plan the query, 
removing any query hint that overrides it. At this commit, [`the CSE pushdown 
rule is explicitly gated by that 
setting`](https://github.com/apache/doris/blob/4a3fce2a8fc1000b90f14793241aeeb574402e7e/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVirtualColumnsIntoOlapScan.java#L160-L186).
 Verify the virtual slots disappear and compare results. This bypasses the 
identified CSE entry condition; it is not a tested fix for this report.
   - A PR should establish concrete, correctly typed merge destinations before 
copying materialized virtual values, including initial allocation and block 
reuse. Normalizing only the Top-N accumulator is already present and does not 
address this path. Replacing a nonempty placeholder with default values 
afterward would lose expression results.
   - Add a merge-iterator regression with materialized child virtual columns 
and a placeholder caller destination, checking actual values and runtime types. 
Add SQL coverage that reliably produces overlapping segments in one rowset, 
exercises Storage Top-N with CSE enabled, and checks results against CSE 
disabled. Cover ascending/descending reads, nullable/constant expressions, 
block reuse, and final batches returned with EOF. A no-crash assertion alone 
would miss the value loss.
   
   The source path is a strong root-cause candidate. Confirming that it caused 
this particular abort still requires the missing runtime evidence and 
reproduction; no affected release range or existing fixed version is 
established here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to