github-actions[bot] commented on code in PR #68934:
URL: https://github.com/apache/doris/pull/68934#discussion_r4256804111
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java:
##########
@@ -163,9 +164,12 @@ public Timestamp getTimestamp(int index, int precision) {
case MICROSECOND:
return Timestamp.fromMicros(value);
case NANOSECOND:
- return Timestamp.fromEpochMillis(
- Math.floorDiv(value, 1_000_000L),
- (int) Math.floorMod(value,
1_000_000L));
+ // Paimon serializers trust target precision.
Truncate the positive
+ // fractional remainder so negative epochs and
leading zero digits
+ // obey the same narrowing rule, including
nested values and keys.
+ int nanos = (int) Math.floorMod(value,
1_000_000L);
+ return
Timestamp.fromEpochMillis(Math.floorDiv(value, 1_000_000L),
+ nanos / nanosPerUnit * nanosPerUnit);
Review Comment:
[P2] Reject narrowing that leaves the readable nanos range. For an existing
Paimon `TIMESTAMP(7/8)`, Doris exposes the sink column as `TIMESTAMP_NS` and
accepts `1677-09-21 00:12:43.145224192` (`Long.MIN_VALUE`). This branch
truncates it to `.145224100`/`.145224190`, respectively 92ns/2ns below the
signed-nanos minimum. [Paimon's
serializer](https://github.com/apache/paimon/blob/release-1.4.2/paimon-common/src/main/java/org/apache/paimon/data/serializer/TimestampSerializer.java#L49-L57)
can persist both millisecond-plus-tail values, and `tableRow`/`writer.write`
adds no signed-nanos check, so the insert can commit; subsequent Doris JNI
scans overflow in `TypeNativeBytes.convertToTimestampNs`. Check that the
normalized result remains in the readable range and reject before
`writer.write`, including nested fields, with precision-7/8 lower-bound
write/read coverage. This differs from the existing historical-row range
thread: a supported in-range Doris input creates the unreadable val
ue, and rejecting that write requires no mapping fallback.
##########
be/src/format_v2/parquet/parquet_scan.cpp:
##########
@@ -1579,6 +1579,7 @@ bool get_typed_dictionary_raw_values(PrimitiveType
primitive_type, const IColumn
GET_TYPED_DICTIONARY_VALUES(TYPE_DATEV2);
GET_TYPED_DICTIONARY_VALUES(TYPE_DATETIMEV2);
GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMPTZ);
+ GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMP_NS);
Review Comment:
[P2] Preserve fused dictionary projection for nanos columns. For a flat,
fully dictionary-encoded NANOS column, `SELECT ts FROM t WHERE ts >=
CAST('2024-01-02 03:04:05.123456789' AS TIMESTAMP_NS)` now reaches
dictionary-ID filtering with `ColumnTimeStampNs`.
`try_filter_and_project_fixed_width_dictionary` still omits that class, so each
survivor goes into a temporary ID column, is copied and revalidated by
`append_dictionary_values`, and is gathered in a separate pass.
`try_simd_insert_parquet_dictionary_indices` also omits `TYPE_TIMESTAMP_NS`,
forcing the scalar gather. The previous `ColumnDateTimeV2` carrier used the
fused filter/value gather, so normal dictionary scans incur extra passes and
buffer traffic. Register the nanos type in both dispatchers and add a projected
nanos dictionary-predicate test checking exact values and
`DictionaryPredicateFusedProjectedRows`. This is separate from the repaired
nullable-selection thread: it occurs without NULLs in dictionary predicate pr
ojection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
