morningman commented on code in PR #69547:
URL: https://github.com/apache/doris/pull/69547#discussion_r4365298253
##########
be/src/format/table/iceberg_reader_mixin.h:
##########
@@ -1226,6 +1245,22 @@ Status
IcebergReaderMixin<BaseReader>::_register_missing_equality_delete_column(
return Status::OK();
}
+template <typename BaseReader>
+void IcebergReaderMixin<BaseReader>::_register_derived_equality_delete_column(
+ const std::string& name, const DataTypePtr& delete_key_type) {
+ DORIS_CHECK(delete_key_type != nullptr);
+ // The key is extracted from its source column after the read. Until then
a placeholder keeps
+ // the carrier at the batch row count, so the reader filters it with the
other columns.
+ auto placeholder = delete_key_type->create_column();
+ placeholder->insert_default();
+ this->register_synthesized_column_handler(
Review Comment:
Fixed in 5acba3e2c97. The keys that the reader does not read — a key
extracted from the projected root struct, and the second key under a physical
root that another key's carrier reads — are no longer synthesized columns. The
reader block has no column for them during the read, so the lazy-read
classifier never sees a name without a tuple slot, and `on_after_read_block`
appends them once the batch has its final row count
(`_materialize_nested_equality_delete_columns` appends the extracted leaf).
The missing-key carrier from #65502 had the same problem, and not only on
Parquet: its synthesized handler appended the column after the reader had taken
its column count, and the lazy reads of both Parquet and ORC erase columns
beyond that count after filtering, so the key's block position went stale. It
is now appended after the read as well, on both formats.
Tests: lazy-read variants with a filter on `id` for a key taken from the
projected root (both column orders), for keys sharing an unprojected root and
for keys taken from the projected root, plus a missing key with and without the
filter on Parquet and ORC. Without the fix the four nested cases abort at
`vparquet_group_reader.cpp:829` (`column_size == cz`, 2 vs. 3) and both lazy
missing-key cases fail with `invalid input position, position=2`; the
production slot lookup itself is compiled out under BE_TEST. On reachability:
master FE sets `parquet_timestamp_semantics_version = 1` and Iceberg scan
semantics v2, which routes Iceberg scans to FileScannerV2, so the V1 readers
are reached when a BE that was upgraded first serves a plan of an older FE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
