github-actions[bot] commented on code in PR #65446:
URL: https://github.com/apache/doris/pull/65446#discussion_r3710265344
##########
be/src/format_v2/parquet/reader/native_column_reader.cpp:
##########
@@ -183,27 +202,34 @@ Status NativeColumnReader::create(
return Status::InvalidArgument("Invalid native parquet top-level
column id {} for {}",
column_schema.local_id,
column_schema.name);
}
- auto* field =
const_cast<NativeFieldSchema*>(native_schema.get_column(column_schema.local_id));
- DORIS_CHECK(field != nullptr);
- if (field->name != column_schema.name &&
- !(field->field_id >= 0 && field->field_id ==
column_schema.parquet_field_id)) {
+ const auto* metadata_field =
native_schema.get_column(column_schema.local_id);
+ DORIS_CHECK(metadata_field != nullptr);
+ if (metadata_field->name != column_schema.name &&
+ !(metadata_field->field_id >= 0 &&
+ metadata_field->field_id == column_schema.parquet_field_id)) {
return Status::Corruption(
"Native/metadata parquet schema mismatch at column {}:
native={}, arrow={}",
- column_schema.local_id, field->name, column_schema.name);
+ column_schema.local_id, metadata_field->name,
column_schema.name);
}
auto type = projected_type(column_schema, projection);
+ auto native_reader = std::unique_ptr<NativeColumnReader>(
+ new NativeColumnReader(column_schema, type, profile));
+ // Footer metadata is cached and shared across scans. Keep per-request
timestamp semantics on a
+ // reader-owned copy so mixed Paimon TIMESTAMP/TIMESTAMP_LTZ columns
cannot contaminate it.
+ native_reader->_native_field_schema = *metadata_field;
Review Comment:
[P2] Avoid rebuilding the full native schema for every row group
`metadata_field` is the complete top-level subtree, so this assignment
recursively clones all children and `sync_native_field_types()` walks that tree
again. `NativeColumnReader::create()` runs for each predicate/output root in
every surviving row group (and dictionary probing can create another reader),
even when the projection selects one leaf. A wide nested column therefore adds
O(row groups x subtree size) allocations and traversal to row-group open.
Please construct an immutable request-owned overlaid schema once per top-level
root, or keep the request type overlay separate from cached metadata, and share
it with row-group readers.
##########
be/src/format_v2/column_data.h:
##########
@@ -398,6 +402,13 @@ inline bool is_child_projected(const LocalColumnIndex*
projection, int32_t local
inline Status merge_local_column_index(LocalColumnIndex* target, const
LocalColumnIndex& source) {
DORIS_CHECK(target != nullptr);
DORIS_CHECK(target->index == source.index);
+ if (!target->timestamp_is_adjusted_to_utc.has_value()) {
Review Comment:
[P1] Keep semantic-only filter descendants when a full projection wins
When every visible child is missing/default-only, mapper fallback preserves
the full file subtree; `needs_nested_file_projection()` then returns false, so
`add_scan_column()` leaves this root as `project_all_children=true`. If the
same root has a filter-only Paimon `TIMESTAMP_LTZ` descendant, this merge
copies only the current node's marker and returns immediately, so the
descendant marker is dropped. `attach_timestamp_semantics()` cannot recreate it
because that filter-only field is absent from the visible mapping. The physical
INT96 leaf can then be decoded with plain `TIMESTAMP` semantics, making value
predicates evaluate in the wrong time domain. This is distinct from the
resolved `column_mapper.cpp:1845` case: its present visible sibling keeps the
target partial, so the merge descends. Please retain semantic-only descendants
even when a full projection dominates physical selection, and cover a
missing/default-only selected child plus a filter-only LTZ sibling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
