Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4255723965
##########
be/src/format/transformer/vorc_transformer.cpp:
##########
@@ -349,6 +355,11 @@ std::unique_ptr<orc::Type>
VOrcTransformer::_build_orc_type(
type = orc::createPrimitiveType(orc::DATE);
break;
}
+ case TYPE_TIMESTAMP_NS: {
+ type = orc::createPrimitiveType(orc::TIMESTAMP);
+ type->setAttribute("iceberg.timestamp-unit", "NANOS");
Review Comment:
I will fix ORC TVF schema discovery and the format-v2 read path so a NANOS
annotation reaches the TIMESTAMP_NS target without an intermediate microsecond
column. The legacy LOAD / format-v1 decoding path is explicitly out of scope
for this PR; I will keep this thread open to make that remaining part visible.
##########
be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp:
##########
@@ -84,10 +109,14 @@ Status VIcebergPartitionWriter::open(RuntimeState* state,
RuntimeProfile* profil
open_status = parquet_compression_type.error();
break;
}
- ParquetFileOptions parquet_options = {.compression_type =
parquet_compression_type.value(),
- .parquet_version =
TParquetVersion::PARQUET_1_0,
- .parquet_disable_dictionary =
false,
- .enable_int96_timestamps =
false};
+ // Parquet 1.0 coerces Arrow nanoseconds to microseconds, including
nested fields.
+ ParquetFileOptions parquet_options = {
+ .compression_type = parquet_compression_type.value(),
+ .parquet_version =
contains_nanosecond_timestamp(_schema.root_struct())
Review Comment:
I will fix Parquet TVF schema discovery and the format-v2 read path so
unadjusted NANOS maps to TIMESTAMP_NS without intermediate microsecond
rounding. Legacy LOAD / format-v1 decoding is explicitly out of scope; I will
keep this thread open for that remaining part.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java:
##########
@@ -156,6 +156,10 @@ private static ConnectorType toDecimalType(DecimalType
type) {
private static ConnectorType toTimestampType(DataType dataType) {
int scale = DEFAULT_TIMESTAMP_SCALE;
if (dataType instanceof TimestampType) {
+ // Keep all source fractional digits instead of truncating
high-precision wall-clock values.
+ if (((TimestampType) dataType).getPrecision() >
MAX_TIMESTAMP_SCALE) {
+ return ConnectorType.of("TIMESTAMP_NS");
Review Comment:
This range-compatibility issue is intentionally out of scope, as agreed with
the author. High-precision external timestamps will retain the TIMESTAMP_NS
mapping; this change will not introduce an opt-in mapping switch or a fallback
to DATETIMEV2 for out-of-range values. I will leave the thread open rather than
mark it fixed.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType
fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ",
ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2",
ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
Review Comment:
Mixed-version FE/BE compatibility is intentionally out of scope for this PR,
as agreed with the author. I will not add an older-backend fallback or
compatibility gate, and will leave this thread open rather than mark it fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
