github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4254819836
##########
be/src/format_v2/parquet/native_schema_desc.cpp:
##########
@@ -882,6 +882,11 @@ std::pair<DataTypePtr, bool>
NativeFieldDescriptor::convert_to_doris_type(
// sub-second values are silently truncated by the target SerDe.
ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMEV2,
nullable, 0, scale);
} else if (logicalType.__isset.TIMESTAMP) {
+ if (logicalType.TIMESTAMP.unit.__isset.NANOS &&
!logicalType.TIMESTAMP.isAdjustedToUTC) {
+ // Materialize nanoseconds before any table cast; DATETIMEV2 would
discard the remainder.
+ ans.first =
DataTypeFactory::instance().create_data_type(TYPE_TIMESTAMP_NS, nullable);
Review Comment:
[P2] Preserve fused nullable selection for nanosecond columns. This switches
optional unadjusted NANOS fields from ColumnDateTimeV2 to the distinct
ColumnTimeStampNs, but visit_nullable_expandable_column in
format_v2/parquet/reader/native/column_chunk_reader.cpp does not include that
class. An integer-filtered scan projecting ts with alternating NULL/value rows
therefore loses supports_fused_nullable_selection and falls into
decode_selected_values<true>, invoking the decoder and SerDe once per non-NULL
run instead of once per page fragment. This regresses the existing sparse
nullable scalar path for large files. Register ColumnTimeStampNs with the
existing in-place expansion dispatcher and cover a filtered nullable NANOS page
with alternating NULLs, checking that HybridSelectionNullFallbackBatches stays
zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
