github-actions[bot] commented on code in PR #69032:
URL: https://github.com/apache/doris/pull/69032#discussion_r4553008540
##########
be/src/format_v2/parquet/parquet_column_schema.cpp:
##########
@@ -329,6 +329,29 @@ Status build_parquet_column_schema(const
NativeFieldDescriptor& schema,
return Status::OK();
}
+bool apply_spatial_binary_types(format::ColumnDefinition* column,
ParquetColumnSchema* schema) {
+ DORIS_CHECK(column->local_id == schema->local_id);
+ // A table-mapped BINARY leaf may have a spatial native annotation. Update
the split-local
+ // reader schema used by both dictionary and plain-page readers, leaving
the footer cache alone.
+ const auto physical_type =
remove_nullable(schema->type)->get_primitive_type();
+ if ((physical_type == TYPE_GEOMETRY || physical_type == TYPE_GEOGRAPHY) &&
+ remove_nullable(column->type)->get_primitive_type() == TYPE_VARBINARY)
{
+ schema->type = schema->type->is_nullable() ? column->type :
remove_nullable(column->type);
+ schema->type_descriptor.doris_type = schema->type;
+ return true;
+ }
+ bool changed = false;
+ for (size_t i = 0; i < column->children.size(); ++i) {
+ changed |= apply_spatial_binary_types(&column->children[i],
schema->children[i].get());
+ }
+ // A STRUCT containing an overridden leaf needs the same field type in
both schemas.
+ if (changed) {
+ rebuild_logical_complex_type(schema, true);
+ column->type = make_nullable(schema->type);
Review Comment:
[P2] Preserve timestamp siblings during BINARY schema overrides
With `enable_mapping_timestamp_tz=false`, an equality-delete file for
`payload STRUCT<k:BINARY,ts:TIMESTAMPTZ>` whose `k` has a spatial Parquet
annotation reaches `_load_equality_delete_file()`'s timestamp pass first. That
retags the `ColumnDefinition` timestamp and parent to TIMESTAMPTZ, while the
native reader tree still has DATETIMEV2. Rebuilding from that native tree here
overwrites the parent with DATETIMEV2 but leaves the child's timestamp marker
intact. `file_block_template` is then created from the stale parent;
`ParquetReader::open()` reapplies the marker to the native tree, and the first
`get_block()` fails `check_block_column_type` because its timestamp child is
TIMESTAMPTZ. Thus an ordinary scan fails when these two nested fields are
equality keys, in either key order, even with an ordinary BINARY data file.
Apply BINARY overrides before timestamp retagging or preserve all current
sibling semantics when synchronizing the trees, and cover the combined
nested-key case. T
his differs from the earlier missing-BINARY-override thread: the override now
runs but undoes a sibling's rewrite. Source tracing only; no Doris test ran.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
