github-actions[bot] commented on code in PR #65446:
URL: https://github.com/apache/doris/pull/65446#discussion_r3709007866
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java:
##########
@@ -120,6 +120,11 @@ public HudiScanPlanProvider(Map<String, String>
properties, ConnectorContext con
this.context = context;
}
+ @Override
+ public boolean usesHiveParquetInt96TimeZone() {
+ return true;
Review Comment:
[P1] Keep Hudi JNI scans on the same INT96 timezone policy
Returning `true` here attaches `hive.parquet.time-zone` to every Hudi scan,
but only the native Parquet reader consumes field 35. COW switches to
`HudiJniReader` under `force_jni_scanner`, and MOR/log slices also use that JNI
arm; `HudiJniReader` does not forward field 35, while `JniTableReader` injects
the query session `time_zone` and `HadoopHudiColumnValue` uses it for
`TimestampWritableV2`/epoch values. A COW HDFS range forced to JNI cannot even
receive the catalog key through the indirect Hadoop-property route, and MOR can
mix native and JNI timestamp semantics within one scan. Please either propagate
the catalog INT96 timezone through Hudi JNI or avoid advertising/applying it to
ranges that use JNI, and add native-vs-JNI coverage with different
catalog/session zones.
##########
be/src/core/data_type_serde/data_type_serde.cpp:
##########
@@ -496,8 +522,20 @@ Status decode_timestamp_orc_values(IColumn& nested_column,
const OrcDecodedColum
}
auto& value =
reinterpret_cast<DateV2Value<DateTimeV2ValueType>&>(data[old_data_size + row]);
- value.from_unixtime(orc_batch->data[source_row], timezone);
-
value.set_microsecond(cast_set<uint64_t>(orc_batch->nanoseconds[source_row] /
1000));
+ RoundedOrcTimestamp timestamp;
+ auto status = round_orc_timestamp_to_microseconds(
Review Comment:
[P2] Round plain ORC TIMESTAMP in wall-clock space
This decoder serves plain `TIMESTAMP`, but the new helper adds the
fractional carry to the epoch second before `from_unixtime(..., timezone)`.
Across a DST transition that changes the civil result: in
`America/Los_Angeles`, `1636275599 + 999999500ns` is local
`01:59:59.999999500`, yet carrying the epoch crosses the rollback and decodes
as `01:00:00` instead of the civil half-up result `02:00:00` (the second
occurrence of the same local value rounds differently). Plain TIMESTAMP
statistics are intentionally reconstructed in UTC wall-clock coordinates in
`orc_reader.cpp`, so metadata can say `02:00` while row decoding says `01:00`,
which can also make pruning unsound. Please keep instant-domain carry for
`TIMESTAMP_INSTANT`, but apply the carry to the converted civil value for plain
`TIMESTAMP`, with spring-forward/fall-back row and metadata tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
