OIiveirra commented on code in PR #68024:
URL: https://github.com/apache/doris/pull/68024#discussion_r4045030264
##########
regression-test/data/external_table_p0/paimon/paimon_timestamp_types.out:
##########
@@ -5,6 +5,31 @@
-- !c2 --
1 2024-01-02T10:04:05.100 2024-01-02T10:04:05.120 2024-01-02T10:04:05.123
2024-01-02T10:04:05.123400 2024-01-02T10:04:05.123450
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
2024-01-02T10:04:05.100 2024-01-02T10:04:05.120 2024-01-02T10:04:05.123
2024-01-02T10:04:05.123400 2024-01-02T10:04:05.123450
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
+-- !ts9_eq_orc --
+1
+1
Review Comment:
Resolved. The native ORC predicate path no longer duplicates the source row,
and the generated snapshots now contain exactly one matching `id = 1` without
using `DISTINCT`. The updated `paimon_timestamp_types` regression passes with
both scanner modes.
##########
regression-test/data/external_table_p0/paimon/paimon_timestamp_types.out:
##########
@@ -53,6 +78,27 @@
-- !c2 --
1 2024-01-02T10:04:05.100 2024-01-02T10:04:05.120 2024-01-02T10:04:05.123
2024-01-02T10:04:05.123400 2024-01-02T10:04:05.123450
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
2024-01-02T10:04:05.100 2024-01-02T10:04:05.120 2024-01-02T10:04:05.123
2024-01-02T10:04:05.123400 2024-01-02T10:04:05.123450
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
2024-01-02T10:04:05.123456 2024-01-02T10:04:05.123456
+-- !ts9_eq_orc --
Review Comment:
Resolved. Native ORC timestamp decoding now uses the same
nanosecond-to-microsecond truncation contract as the JNI/Paimon path, and the
generated snapshots produce the same values and row sets for both
`force_jni_scanner` settings. The updated `paimon_timestamp_types` regression
passes locally.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -502,22 +502,12 @@ std::optional<DateV2Value<DateTimeV2ValueType>>
datetime_v2_from_orc_millis(
}
const auto extra_nanos = std::max<int32_t>(nanos_tail, 0);
constexpr int64_t NANOS_PER_MICROSECOND = 1000;
- constexpr int64_t MICROS_PER_SECOND = 1000000;
// Stripe statistics split the timestamp into milliseconds and the
remaining nanoseconds. Use
- // the same half-up rule as row decoding so zone-map pruning observes
identical values.
- const auto rounded_extra_microseconds =
- (extra_nanos + NANOS_PER_MICROSECOND / 2) / NANOS_PER_MICROSECOND;
- const auto microseconds_with_carry = millis_remainder * 1000 +
rounded_extra_microseconds;
- // Calendar bounds depend on the target timezone, so only reject
arithmetic overflow here and
- // let the converted value below decide whether the statistic is
representable by Doris.
- int64_t rounded_seconds;
- if (__builtin_add_overflow(seconds, microseconds_with_carry /
MICROS_PER_SECOND,
- &rounded_seconds)) {
- return std::nullopt;
- }
- const auto microseconds = cast_set<uint64_t>(microseconds_with_carry %
MICROS_PER_SECOND);
+ // the same truncation as row decoding so zone-map pruning observes
identical values.
+ const auto microseconds =
Review Comment:
Resolved in `8b1f7a577e2`. Timestamp MIN/MAX pushdown now falls back to a
row scan when ORC exposes the conservative missing-tail sentinels
(`minimumNanos == 0` or `maximumNanos == 999999`). Added coverage for both
`TIMESTAMP` and `TIMESTAMP_INSTANT`; all 29 targeted timestamp BE tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
