Gabriel39 commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4254161930
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java:
##########
@@ -671,6 +679,19 @@ private long toMicros(LocalDateTime dateTime, Type
icebergType, boolean utcCarri
return instant.getEpochSecond() * 1_000_000L + instant.getNano() /
1000L;
}
+ private static Literal<Long> toNanos(LocalDateTime dateTime, Type
icebergType) {
+ if (((Types.TimestampNanoType) icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ try {
+ // A raw Long is a microsecond literal in Iceberg; preserve the
nanosecond unit explicitly.
+ // Unzoned values use a UTC wall-clock carrier, never the session
zone.
+ return
Expressions.nanos(IcebergPartitionUtils.toEpochNanos(dateTime));
Review Comment:
Fixed the metadata-cleanup lifecycle gap in fd85964c3b4. The converter now
treats gaps in Iceberg schema IDs as incomplete history and conservatively
suppresses nanos predicates; retained history is captured once for the check.
Added actual snapshot expiration with cleanExpiredMetadata(true), verifying all
converter modes and SDK/manifest-cache planning preserve the retained DATE
file. Complete history still permits fields without DATE promotion.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java:
##########
@@ -150,12 +150,16 @@ private static ConnectorType
fromPrimitive(Type.PrimitiveType primitive,
return ConnectorType.of("TIMESTAMPTZ",
ICEBERG_DATETIME_SCALE_MS, 0);
}
return ConnectorType.of("DATETIMEV2",
ICEBERG_DATETIME_SCALE_MS, 0);
+ case TIMESTAMP_NANO:
+ // TIMESTAMP_NS is a wall-clock type; zoned nanoseconds need a
distinct carrier.
+ return ConnectorType.of(((Types.TimestampNanoType)
primitive).shouldAdjustToUTC()
+ ? "UNSUPPORTED" : "TIMESTAMP_NS");
Review Comment:
Fixed the remaining file-filter path in fd85964c3b4. Localized file-to-table
casts now carry the schema-evolution flag, so non-strict SQL settings cannot
turn DATE overflow into NULL before IS NOT NULL filtering. The real Parquet
reader test failed before the fix and now passes with filtered/unfiltered scans
and strict/non-strict settings. The related 371 BE tests pass.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -479,6 +491,13 @@ static Object parsePartitionValueFromString(String
valueStr, Type icebergType, Z
case DATE:
// Iceberg date is days since epoch (1970-01-01).
return (int) LocalDate.parse(valueStr,
DateTimeFormatter.ISO_LOCAL_DATE).toEpochDay();
+ case TIMESTAMP_NANO:
+ LocalDateTime timestampNs =
LocalDateTime.parse(valueStr.replace('T', ' '),
Review Comment:
Fixed the explicit-offset static-partition case in fd85964c3b4. Nanosecond
partition parsing now converts explicit offsets to session-local civil time,
matching the sink cast. Tests cover UTC, Asia/Shanghai and America/New_York
with nine-digit fractions, plus a real Iceberg overwrite retaining the adjacent
partition. The Parquet and ORC regression suite passed locally with the final
FE/BE builds; expected output was generated by actual execution. All 325
related FE tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
