Gabriel39 commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4060050882
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonColumnValue.java:
##########
@@ -167,7 +168,8 @@ public LocalDate getDate() {
@Override
public LocalDateTime getDateTime() {
- Timestamp ts = record.getTimestamp(idx, dorisType.getPrecision());
+ Timestamp ts = truncateTimestampPrecision(
Review Comment:
[P1] Repair pre-epoch narrowing before the SDK schema cast loses the
original value.
The fixed-width helper handles a raw GenericRow correctly, but actual
evolved-file reads pass through RawFileSplitRead -> DataFileRecordReader ->
CastedRow -> TimestampToTimestampCastRule before this getter. Paimon 1.3.1
converts 1969-12-31 23:59:59.600000 from p6 to p0 using epoch-millisecond
division toward zero, producing 1970-01-01 00:00:00. This helper then receives
that already-corrupted value and cannot recover the expected 1969-12-31
23:59:59, even with no predicate.
I reproduced the SDK cast followed by this PR's getter; both return the
epoch. The new preEpochValue test passes a raw GenericRow directly and bypasses
the failing schema-cast stage. Please address the SDK conversion boundary and
add a forced-JNI regression against a historical-schema file containing a
negative-epoch fractional timestamp. Disabling executeFilter alone cannot fix
this wrong-value case.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -332,7 +332,9 @@ private void resetDatetimeV2Precision() {
if (index != -1) {
DataType dataType = table.rowType().getTypeAt(index);
if (dataType instanceof TimestampType) {
- types[i].setPrecision(((TimestampType)
dataType).getPrecision());
+ int paimonPrecision = ((TimestampType)
dataType).getPrecision();
+ requiresDatetimeV2PrecisionRepair |=
types[i].getPrecision() > paimonPrecision;
Review Comment:
[P1] Detect file-to-table precision narrowing rather than a mismatch between
two current-schema types.
For a normal TIMESTAMP(6) -> TIMESTAMP(4) evolution, FE's PaimonTypeMapping
already produces DATETIMEV2(4), and the JNI bridge preserves that scale. This
comparison is therefore 4 > 4, so requiresDatetimeV2PrecisionRepair remains
false and initReader still supplies the predicates to executeFilter(). With
Paimon 1.3.1, an old value 2025-01-01 00:00:01.000001 survives the SDK's faulty
p4 truncation and fails equality against 2025-01-01 00:00:01 before
PaimonColumnValue can repair it.
I verified this with the PR's actual resetDatetimeV2Precision implementation
and getter plus Paimon 1.3.1 CastExecutors/PredicateBuilder: repair=false,
predicate=false, while the getter produces the expected whole-second value.
This is a component-level reproduction, not a full cluster run. Please derive
the guard from the historical file schema/current schema relationship, or
conservatively prevent affected predicates from executing before repair, and
add a forced-JNI p6 -> p4/p5 leading-zero predicate regression. The existing
positive p6 -> p0 cases do not expose this gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
