Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4256217906
##########
be/src/format/transformer/vorc_transformer.cpp:
##########
@@ -349,6 +355,11 @@ std::unique_ptr<orc::Type>
VOrcTransformer::_build_orc_type(
type = orc::createPrimitiveType(orc::DATE);
break;
}
+ case TYPE_TIMESTAMP_NS: {
+ type = orc::createPrimitiveType(orc::TIMESTAMP);
+ type->setAttribute("iceberg.timestamp-unit", "NANOS");
Review Comment:
The TVF/V2 portion is fixed in 19ef150201f4. Schema RPCs now use V2 ORC
schema discovery, preserving the NANOS annotation in scalar and nested
TIMESTAMP_NS targets before column mapping. Filesystem information honors the
RPC range. Local BE tests and an actual S3 TVF round trip pass, including
array/struct/map fields and range-boundary values; the TVF schema golden was
generated by running the regression suite. Legacy LOAD and format-v1 decoding
remain explicitly out of scope, so this thread stays open.
##########
be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp:
##########
@@ -84,10 +109,14 @@ Status VIcebergPartitionWriter::open(RuntimeState* state,
RuntimeProfile* profil
open_status = parquet_compression_type.error();
break;
}
- ParquetFileOptions parquet_options = {.compression_type =
parquet_compression_type.value(),
- .parquet_version =
TParquetVersion::PARQUET_1_0,
- .parquet_disable_dictionary =
false,
- .enable_int96_timestamps =
false};
+ // Parquet 1.0 coerces Arrow nanoseconds to microseconds, including
nested fields.
+ ParquetFileOptions parquet_options = {
+ .compression_type = parquet_compression_type.value(),
+ .parquet_version =
contains_nanosecond_timestamp(_schema.root_struct())
Review Comment:
The TVF/V2 portion is fixed in 19ef150201f4. Schema RPCs now use V2 Parquet
schema discovery, preserving unadjusted TIMESTAMP(NANOS) in scalar and nested
TIMESTAMP_NS targets instead of first choosing DATETIMEV2(6). Filesystem
information honors the RPC range. Local plain/dictionary BE tests and an actual
S3 TVF round trip pass, including array/struct/map fields and range-boundary
values; the TVF schema golden was generated by running the regression suite.
Legacy LOAD and format-v1 decoding remain explicitly out of scope, so this
thread stays open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
