Doris-Breakwater commented on issue #70489: URL: https://github.com/apache/doris/issues/70489#issuecomment-6807874946
Code inspection supports this as a BE/storage iterator correctness defect affecting virtual columns, rather than an ANN-specific failure. I checked the cited master commit `9e598d38789571a7fc30652aa054f035bd4f1184` and the public `4.1.4` tag (`ad35a140c7fd0b842f18c23300bac581f7d04326`). Both initialize concrete virtual-column storage only in the Top-N accumulator. I have not independently executed the reported reproducer or validated the proposed patch. The relevant mechanism is confirmed in source: 1. The read block deliberately starts with `ColumnNothing` for virtual slots, while retaining the declared data type. `_topn_next` clones that block and replaces only the clone's virtual columns, leaving the original as the rowset read buffer. See [read-block construction](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/storage/schema.cpp#L285-L298) and [Top-N initialization and reads](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/storage/iterator/vcollect_iterator.cpp#L294-L332). 2. The rowset merge path requires an ordered result, more than one segment in the reader, and overlapping-segment metadata. Its `copy_rows` appends into the caller's existing columns; it does not replace the destination columns. Appending to `ColumnNothing` advances its size without retaining values. See [merge selection](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/storage/rowset/beta_rowset_reader.h#L61-L64), [merge copying](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/storage/iterator/vgeneric_iterators.cpp#L130-L156), and [dummy-column insertion](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/core/column/column_dummy.h#L81-L85). 3. Top-N subsequently calls the contiguous-range overload of `MutableBlock::add_rows`, which appends the placeholder source into a concrete destination. Matching declared data types do not make those physical column implementations compatible. This explains the reported bad-cast failure. See [the append call](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/storage/iterator/vcollect_iterator.cpp#L393-L396) and [the append implementation](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/core/block/block.cpp#L1170-L1181). The proposed fix direction is sound: initialize the read buffer's virtual columns with their concrete expression types before cloning, so both the merge destination and accumulator have suitable storage. Keep this local to Top-N initialization. Direct segment reads reset virtual slots at each batch and materialize them before returning, which supports compatibility with the non-overlapping path. See [segment reset and materialization](https://github.com/apache/doris/blob/9e598d38789571a7fc30652aa054f035bd4f1184/be/src/storage/segment/segment_iterator.cpp#L3910-L3958). For a backport, preserve each branch's column-position mapping: master uses read-schema ordinals, whereas the inspected `4.1.4` implementation uses `vir_col_idx_to_type`. Please open the offered PR with the regression test. Useful acceptance checks are: - Exercise the real merge iterator through `_topn_next`, starting with placeholder virtual slots and interleaved keys from two segments. Demonstrate failure before the fix and success afterward, asserting both selected row IDs and their associated virtual values. - Cover multiple batches/rowsets, ascending and descending order, and empty or fully filtered input. Include controls without virtual columns and with non-overlapping segments. - Cover an ordinary pushed-down expression as well as the ANN case; verify nullable results and constant-expression materialization where supported. These checks establish value preservation, beyond simply removing the exception. For end-to-end confirmation, please attach the exact `4.1.4` build commit, executable DDL/load/query steps with concrete vectors, the full BE bad-cast message and stack trace, and `EXPLAIN VERBOSE` showing storage Top-N and the virtual scan slot. Include the relevant rowset's segment count/overlap metadata: a large load alone does not guarantee this layout. The reported unit-test source and before/after output would directly validate the iterator-level diagnosis; the SQL example alone does not establish that a particular plan selected this path. No runtime profile was supplied or analyzed, and no source code was changed during this triage. Breakwater-GitHub-Analysis-Slot: slot_b14ac24806f0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
