Gabriel39 commented on code in PR #65446:
URL: https://github.com/apache/doris/pull/65446#discussion_r3567608825
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -503,7 +503,15 @@ DateV2Value<DateTimeV2ValueType>
datetime_v2_from_orc_millis(int64_t millis, int
millis_remainder += 1000;
}
const auto extra_nanos = std::max<int32_t>(nanos_tail, 0);
- const auto microseconds = cast_set<uint64_t>(millis_remainder * 1000 +
extra_nanos / 1000);
+ constexpr int64_t NANOS_PER_MICROSECOND = 1000;
+ constexpr int64_t MICROS_PER_SECOND = 1000000;
+ // Stripe statistics split the timestamp into milliseconds and the
remaining nanoseconds. Use
+ // the same half-up rule as row decoding so zone-map pruning observes
identical values.
+ const auto rounded_extra_microseconds =
+ (extra_nanos + NANOS_PER_MICROSECOND / 2) / NANOS_PER_MICROSECOND;
Review Comment:
Fixed in f2d8de4eb14. ORC DATETIMEV2 timestamp MIN/MAX pushdown now requires
a fixed-offset session timezone, so named zones with DST folds fall back to row
aggregation. TIMESTAMPTZ remains eligible. Added a Los Angeles fold regression
plus fixed-offset and TIMESTAMPTZ positive coverage; the full NewOrcReaderTest
suite passes 138/138 remotely.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java:
##########
@@ -677,6 +682,17 @@ protected boolean getEnableMappingTimestampTz() {
return false;
}
+ protected String getHiveParquetTimeZone() throws UserException {
+ TableIf table = getTargetTable();
+ if (table instanceof ExternalTable) {
+ CatalogIf<?> catalog = ((ExternalTable) table).getCatalog();
+ if (catalog instanceof HMSExternalCatalog) {
+ return ((HMSExternalCatalog) catalog).getHiveParquetTimeZone();
Review Comment:
Fixed in f2d8de4eb14 by restricting hive.parquet.time-zone propagation to
HMSExternalTable with DLAType.HIVE. HMS-backed Iceberg and standalone Iceberg
HMS catalogs now both use the default raw INT96 semantics. Added FE coverage
for the canonical Hive path and the HMS Iceberg exclusion;
FileQueryScanNodeTest passes 6/6 remotely.
##########
be/src/format_v2/table_reader.cpp:
##########
@@ -674,10 +674,14 @@ Status
TableReader::create_file_reader(std::unique_ptr<FileReader>* reader) {
const bool enable_mapping_timestamp_tz = _scan_params != nullptr &&
_scan_params->__isset.enable_mapping_timestamp_tz &&
_scan_params->enable_mapping_timestamp_tz;
+ const std::string hive_parquet_time_zone =
+ _scan_params != nullptr &&
_scan_params->__isset.hive_parquet_time_zone
+ ? _scan_params->hive_parquet_time_zone
+ : "";
if (_format == FileFormat::PARQUET) {
*reader = std::make_unique<format::parquet::ParquetReader>(
_system_properties, _current_task->data_file, _io_ctx,
_scanner_profile,
- _global_rowid_context, enable_mapping_timestamp_tz);
+ _global_rowid_context, enable_mapping_timestamp_tz,
hive_parquet_time_zone);
Review Comment:
Addressed by f2d8de4eb14 through the Hive-only scope chosen for
hive.parquet.time-zone. HMS-backed Iceberg scans no longer receive this
parameter, so both data files and equality-delete files use the same default
raw INT96 decoding and cannot diverge on delete keys. The FE test verifies that
HMS Iceberg scan params leave the field unset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
