github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4247525820
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -671,6 +679,19 @@ private long toMicros(LocalDateTime dateTime, Type
icebergType, boolean utcCarri
return instant.getEpochSecond() * 1_000_000L + instant.getNano() /
1000L;
}
+ private static Literal<Long> toNanos(LocalDateTime dateTime, Type
icebergType) {
+ if (((Types.TimestampNanoType) icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ try {
+ // A raw Long is a microsecond literal in Iceberg; preserve the
nanosecond unit explicitly.
+ // Unzoned values use a UTC wall-clock carrier, never the session
zone.
+ return
Expressions.nanos(IcebergPartitionUtils.toEpochNanos(dateTime));
Review Comment:
[P1] Preserve historical DATE bounds before nanos pushdown. After a valid v3
DATE-to-`timestamp_ns` promotion, a retained DATE file containing `2024-01-02`
still has four-byte manifest bounds equal to epoch-day 19724. This branch
pushes nanos equality at 1704153600000000000, but the [pinned Iceberg 1.11
conversion](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/api/src/main/java/org/apache/iceberg/types/Conversions.java#L156-L164)
widens those bounds to the unscaled long 19724. `InclusiveMetricsEvaluator`
therefore prunes the matching file: the equality query returns no rows while an
unfiltered native scan correctly returns midnight. Both SDK and cached planning
use this evaluator. Interpret retained DATE bounds in the promoted domain or
conservatively suppress this pushdown, and test old files with real bounds;
strict CAST checks cannot repair file pruning.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType
fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ",
ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2",
ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
Review Comment:
[P2] Fence nanosecond scans from older backend readers. During a cloud
smooth upgrade, a live source BE from the PR base remains query eligible and
already supports the internal `TIMESTAMP_NS` carrier. However, its V2 Parquet
reader infers unadjusted NANOS as `DATETIMEV2(6)`, truncates the fraction, then
casts that result to the new FE's `TIMESTAMP_NS` target: `.123456789` becomes
`.123456000`. A plain optional unpartitioned nanos projection does not trigger
`requiresCurrentScanSemantics`, so `PluginDrivenScanNode` skips its
smooth-source guard and scheduling admits that BE. Base and head also share
execution version 16, so the existing internal-type gate cannot distinguish
their file semantics. Enforce the required reader capability/version for
scalar, nested and filter-only nanos fields, and cover an old source backend in
planning tests.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType
fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ",
ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2",
ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
+ // TIMESTAMP_NS is a wall-clock type; zoned nanoseconds need a
distinct carrier.
+ return ConnectorType.of(((Types.TimestampNanoType)
primitive).shouldAdjustToUTC()
+ ? "UNSUPPORTED" : "TIMESTAMP_NS");
Review Comment:
[P2] Fail on out-of-range historical DATE promotion. For an unpartitioned v3
table that promotes an optional DATE field to `timestamp_ns` while retaining a
Parquet DATE file containing `1000-01-01`, this mapping now exposes the column
as `TIMESTAMP_NS`. The field-ID mapper builds ordinary `CAST(DATEV2 AS
TIMESTAMP_NS)`; with default `enable_strict_cast=false`, its overflow becomes
NULL and `TableReader` successfully returns that NULL. The [Iceberg
schema-evolution contract](https://iceberg.apache.org/spec/#schema-evolution)
requires a runtime failure outside the promoted range. This metadata is
accepted by the pinned parser and can be published through
`TableMetadata.Builder.setCurrentSchema`/`TableOperations.commit`, despite the
narrower `UpdateSchema` convenience API. Use checked promotion or explicitly
reject the unsupported conversion, and cover retained DATE files with an
out-of-range value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
