Gabriel39 commented on code in PR #68649:
URL: https://github.com/apache/doris/pull/68649#discussion_r4139954228
##########
be/src/core/data_type_serde/data_type_datev2_serde.cpp:
##########
@@ -65,22 +62,44 @@ Status decode_date_orc_values(const DataTypeSerDe& serde,
IColumn& column,
orc_serde_utils::orc_decode_row_count(orc_view.rows,
orc_view.selected_rows);
std::vector<int32_t> date_values;
date_values.resize(output_rows);
- auto& date_dict = date_day_offset_dict::get();
for (size_t row = 0; row < output_rows; ++row) {
const auto source_row = orc_serde_utils::orc_source_row_at(row,
orc_view.selected_rows);
- const auto date =
date_dict[cast_set<int>(orc_batch->data[source_row])];
- date_values[row] = cast_set<int32_t>(date.daynr() -
DORIS_DATE_EPOCH_DAYNR);
+ // The payload of a null slot is undefined, so it is zeroed rather
than range-checked;
+ // `read_column_from_decoded_values` never decodes it.
+ if (view.null_map != nullptr && view.null_map[row] != 0) {
+ date_values[row] = 0;
+ continue;
+ }
+ // ORC DATE is days since 1970-01-01 in the proleptic Gregorian
calendar, the same encoding
+ // `decode_epoch_days()` expects, but ORC hands it over in an int64
batch. Reject a value
+ // that does not fit the int32 view here, while the real file value is
still available for
+ // the message, instead of narrowing it blindly.
+ const int64_t file_days = orc_batch->data[source_row];
Review Comment:
Fixed in 15768f335c1. ORC COUNT now collects all DATE leaves of the argument
and validates their statistics in every selected stripe. Missing or unsafe
ranges fall back to materialization, including ranges that cross the
unrepresentable interior day. Added failing-before/passing-after coverage for
scalar, ARRAY, STRUCT and MAP arguments, lower/upper overflow, all-null leaves,
valid ranges and COUNT(*). The updated related ASAN selection passes all 243
tests.
##########
be/src/format_v2/parquet/parquet_reader.cpp:
##########
@@ -1175,6 +1175,28 @@ Status ParquetReader::get_aggregate_result(const
format::FileAggregateRequest& r
}
const auto& count_projection = request.columns[0].projection;
const auto& root_schema = projected_root_schema(_state->file_schema,
count_projection);
+ if (remove_nullable(root_schema.type)->get_primitive_type() ==
TYPE_DATEV2) {
Review Comment:
Fixed in 15768f335c1. Strict Parquet COUNT now recursively validates each
projected DATE leaf before the levels-only shortcut. Non-strict complex COUNT
retains its existing behavior; scalar DATE still requires validation in either
mode. Added failing-before/passing-after ARRAY, STRUCT and MAP coverage, plus
valid/all-null leaves and a partial STRUCT projection whose invalid DATE
sibling is not read. All 243 related ASAN tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
