github-actions[bot] commented on code in PR #65446:
URL: https://github.com/apache/doris/pull/65446#discussion_r3567544898
##########
be/src/format_v2/table_reader.cpp:
##########
@@ -674,10 +674,14 @@ Status
TableReader::create_file_reader(std::unique_ptr<FileReader>* reader) {
const bool enable_mapping_timestamp_tz = _scan_params != nullptr &&
_scan_params->__isset.enable_mapping_timestamp_tz &&
_scan_params->enable_mapping_timestamp_tz;
+ const std::string hive_parquet_time_zone =
+ _scan_params != nullptr &&
_scan_params->__isset.hive_parquet_time_zone
+ ? _scan_params->hive_parquet_time_zone
+ : "";
if (_format == FileFormat::PARQUET) {
*reader = std::make_unique<format::parquet::ParquetReader>(
_system_properties, _current_task->data_file, _io_ctx,
_scanner_profile,
- _global_rowid_context, enable_mapping_timestamp_tz);
Review Comment:
The new scan parameter is still not applied to Iceberg equality-delete
Parquet files. Normal data-file reads go through this
`TableReader::create_file_reader()` path and pass `hive_parquet_time_zone` into
`ParquetReader`, but `IcebergTableReader::_read_equality_delete_file()` copies
the same `_scan_params` and then `_create_delete_file_reader()` constructs its
Parquet reader with only `system_properties, file_description, io_ctx,
_scanner_profile`, leaving the new timezone argument at the default empty
string. Since `FileQueryScanNode` now serializes this property for HMS-backed
Iceberg tables, an HMS Iceberg table with
`hive.parquet.time-zone=Asia/Shanghai` can decode INT96 data-file values with
the catalog timezone while an INT96 timestamp equality-delete key is decoded as
raw/UTC. The delete predicate is then built from a different DATETIMEV2 value
and can fail to remove matching rows. Please pass the same
`hive_parquet_time_zone` into Parquet delete-file readers, and add cover
age for an Iceberg equality delete on an INT96 timestamp key.
##########
be/src/format_v2/table_reader.cpp:
##########
@@ -674,10 +674,14 @@ Status
TableReader::create_file_reader(std::unique_ptr<FileReader>* reader) {
const bool enable_mapping_timestamp_tz = _scan_params != nullptr &&
_scan_params->__isset.enable_mapping_timestamp_tz &&
_scan_params->enable_mapping_timestamp_tz;
+ const std::string hive_parquet_time_zone =
+ _scan_params != nullptr &&
_scan_params->__isset.hive_parquet_time_zone
+ ? _scan_params->hive_parquet_time_zone
+ : "";
if (_format == FileFormat::PARQUET) {
*reader = std::make_unique<format::parquet::ParquetReader>(
_system_properties, _current_task->data_file, _io_ctx,
_scanner_profile,
- _global_rowid_context, enable_mapping_timestamp_tz);
+ _global_rowid_context, enable_mapping_timestamp_tz,
hive_parquet_time_zone);
Review Comment:
The new scan parameter is still not applied to Iceberg equality-delete
Parquet files. Normal data-file reads go through this
`TableReader::create_file_reader()` path and pass `hive_parquet_time_zone` into
`ParquetReader`, but `IcebergTableReader::_read_equality_delete_file()` copies
the same `_scan_params` and then `_create_delete_file_reader()` constructs its
Parquet reader with only `system_properties, file_description, io_ctx,
_scanner_profile`, leaving the new timezone argument at the default empty
string. Since `FileQueryScanNode` now serializes this property for HMS-backed
Iceberg tables, an HMS Iceberg table with
`hive.parquet.time-zone=Asia/Shanghai` can decode INT96 data-file values with
the catalog timezone while an INT96 timestamp equality-delete key is decoded as
raw/UTC. The delete predicate is then built from a different DATETIMEV2 value
and can fail to remove matching rows. Please pass the same
`hive_parquet_time_zone` into Parquet delete-file readers, and add cover
age for an Iceberg equality delete on an INT96 timestamp key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
