This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2a4191c1dc9 [improvement](be) Use zero NaN counts for Parquet
floating-point pruning (#69655)
2a4191c1dc9 is described below
commit 2a4191c1dc94e789e1725d6cb345f46c31a17272
Author: Gabriel <[email protected]>
AuthorDate: Thu Oct 29 16:27:31 2026 +0800
[improvement](be) Use zero NaN counts for Parquet floating-point pruning
(#69655)
### What problem does this PR solve?
Problem Summary:
Parquet V2 treats every FLOAT/DOUBLE column as having an unknown NaN
count. Finite min/max bounds omit NaNs, so the existing comparison and
NOT IN guards correctly retain groups that might contain a matching NaN.
This also prevents pruning when a writer explicitly supplies
`nan_count=0`.
For example, a column chunk with `min=max=1` and a trustworthy zero NaN
count cannot match `x > 1`, `x >= 2`, `x <> 1`, or `x NOT IN (1)`.
Previously the reader decoded those rows anyway. This change makes that
zero proof available to the existing ZoneMap evaluator.
### Optimization approach
- Add standard optional Statistics field 9, `nan_count`, matching the
[Apache Parquet format
definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift).
Compared the touched definition on master and branch-3.1/4.0/4.1/4.2:
existing field IDs retain their meanings and field 9 is unused on those
Doris branches. Older readers skip the optional field, and existing
files retain the conservative fallback.
- Retain a zero-count proof in converted FLOAT/DOUBLE statistics only
when the count is explicitly zero, the value/null counts are consistent,
and the decoded bounds are usable (or the chunk is all NULL). Missing,
positive, negative, or excessive counts do not clear the NaN-unknown
flag. Invalid/NaN/unordered bounds remain unusable.
- Pass the proof alongside the row-group ZoneMap. For PageIndex
evaluation, reuse only the proof from the exact owning row group and
physical leaf column. It covers every page of that chunk. Both
single-slot and compound OR/AND page evaluation use this scoped proof.
- Keep the existing comparator and NOT IN semantics. Integer/string
columns and legacy floating files avoid the additional footer decode in
the page proof path. Writer support and additional column orders are
outside this reader change.
### Optimization effect
| Per reader query | Absent count (previous fallback) | Explicit zero
count |
| --- | ---: | ---: |
| RawRowsRead | 524,288 | 4,096 (-99.22%) |
| RowGroupsFilteredByMinMax | 0 | 127 |
| Actual local read bytes | 6,358,496 | 98,444 (-98.45%) |
| Median wall time | 6.908 ms | 0.690 ms (~10.0x) |
| Median process CPU time | 6.868 ms | 0.687 ms (~10.0x) |
Measured using Clang 20 RELEASE, warm local filesystem cache, one pinned
CPU. The absent-count samples range from 6.787 to 7.252 ms and
zero-count samples from 0.673 to 0.730 ms. All returned IDs and
per-query work counters match their expectations across all samples.
The opt-in `ParquetScanTest.ZeroNanCountReaderMicrobench` uses the
actual V2 reader and corresponds to:
```sql
SELECT id FROM parquet_source WHERE d > 1;
```
Fixture: 128 row groups x 4,096 rows; the first 127 groups contain
`d=1`, the last contains `d=2`. FLOAT, DOUBLE, and an integer row
identifier are stored using uncompressed PLAIN DataPageV2 with
dictionary and PageIndex disabled. Two files have identical data pages
and differ only in the optional zero-count fields. The absent-count
variant exercises the same conservative behavior as the pre-change
reader. Each query checks every returned ID and the pruning counters.
Seven samples of eight reader queries per variant follow warmup,
alternating variant order. The benchmark records wall time, process CPU
time, actual local read bytes, RawRowsRead, and
RowGroupsFilteredByMinMax. Results describe a local reader
microbenchmark; full SQL timings depend on storage, parallelism, and the
rest of the query.
### Validation and reproduction
- Related BE suites: 372 tests passed; the opt-in benchmark was skipped
in that run and then passed separately.
- The three initial reader regressions first failed on unmodified master
because zero counts did not reduce RawRowsRead or increase pruning.
Their result rows were already correct; the missing behavior was
pruning.
- Generated C++ Thrift code compiled successfully with the optional
field. Old files without the field remain covered by the complete reader
suite.
- clang-format 16 and BE build-hygiene checks passed. Changed-line
clang-tidy passed on all four affected C++ files.
Coverage includes FLOAT/DOUBLE; zero/positive/absent/negative/excessive
counts; GT/GE/NE/NOT IN; invalid null counts and bounds; all NULL/all
NaN/mixed NULL-finite-NaN rows; dictionary on/off; DataPageV1/V2;
PageIndex on/off; column/row-group proof scope; and compound OR page
pruning when one branch lacks a zero proof. Mixed NaN cases are also
checked against a statistics-disabled file.
Correctness suites:
```bash
BUILD_TYPE_UT=RELEASE ./run-be-ut.sh -j8 --run \
--filter='*Parquet*:*ColumnMapper*:*TableReader*'
```
Opt-in benchmark:
```bash
DORIS_PARQUET_NAN_COUNT_BENCHMARK=1 BUILD_TYPE_UT=RELEASE \
./run-be-ut.sh -j8 --run
--filter=ParquetScanTest.ZeroNanCountReaderMicrobench
```
### Release note
Improve Parquet V2 floating-point Min-Max pruning when trustworthy zero
NaN counts are available.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (reader microbenchmark)
- Behavior changed:
- [x] Yes. Proven NaN-free floating chunks/pages can be skipped by
existing comparison and NOT IN predicates.
- Does this need documentation?
- [x] No. Internal reader optimization using existing file metadata.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/format_v2/parquet/parquet_statistics.cpp | 99 ++++--
be/src/format_v2/parquet/parquet_statistics.h | 13 +-
be/test/format_v2/parquet/parquet_scan_test.cpp | 363 ++++++++++++++++++++-
.../format_v2/parquet/parquet_statistics_test.cpp | 51 +++
gensrc/thrift/parquet.thrift | 4 +-
5 files changed, 494 insertions(+), 36 deletions(-)
diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp
b/be/src/format_v2/parquet/parquet_statistics.cpp
index 62dc379cd06..3e15a5bc3f4 100644
--- a/be/src/format_v2/parquet/parquet_statistics.cpp
+++ b/be/src/format_v2/parquet/parquet_statistics.cpp
@@ -942,10 +942,12 @@ std::shared_ptr<segment_v2::ZoneMap>
make_zonemap_from_statistics(
}
void add_slot_zonemap(ZoneMapEvalContext* ctx, int slot_index, const
DataTypePtr& data_type,
- std::shared_ptr<segment_v2::ZoneMap> zone_map) {
+ std::shared_ptr<segment_v2::ZoneMap> zone_map, bool
has_zero_nan_count) {
DORIS_CHECK(ctx != nullptr);
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
slot_zone_map.set_data_type_from_parquet(data_type);
+ // A finite min=max=1 alone cannot exclude NaN > 1; a validated zero count
can.
+ slot_zone_map.floating_nan_count_unknown &= !has_zero_nan_count;
slot_zone_map.zone_map = std::move(zone_map);
ctx->slots.emplace(slot_index, std::move(slot_zone_map));
}
@@ -1015,6 +1017,15 @@ ParquetColumnStatistics
ParquetStatisticsUtils::TransformColumnStatistics(
result.has_null_count = false;
result.has_null = true;
}
+ // Accept only an explicit zero in consistent floating statistics.
Missing/positive/invalid
+ // counts, impossible null counts, and NaN or unordered bounds cannot
establish this proof.
+ const auto physical_type = column_schema.type_descriptor.physical_type;
+ result.has_zero_nan_count =
+ (physical_type == tparquet::Type::FLOAT || physical_type ==
tparquet::Type::DOUBLE) &&
+ statistics->__isset.nan_count && statistics->nan_count == 0 &&
+ (!statistics->__isset.null_count ||
+ (statistics->null_count >= 0 && statistics->null_count <=
column_value_count)) &&
+ (result.has_min_max || !result.has_not_null);
return result;
}
@@ -1035,6 +1046,49 @@ bool ParquetStatisticsUtils::NativeBloomFilterExcludes(
namespace {
+ParquetColumnStatistics native_footer_column_statistics(const
tparquet::FileMetaData& metadata,
+ const
tparquet::RowGroup& row_group,
+ const
ParquetColumnSchema& column_schema,
+ const cctz::time_zone*
timezone,
+ bool
null_count_trusted) {
+ if (column_schema.leaf_column_id < 0 ||
+ std::cmp_greater_equal(column_schema.leaf_column_id,
row_group.columns.size())) {
+ return {};
+ }
+ const auto& chunk = row_group.columns[column_schema.leaf_column_id];
+ if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics) {
+ return {};
+ }
+ const auto& column = chunk.meta_data;
+ const auto statistics = detail::sanitize_native_footer_statistics(
+ column_schema.type_descriptor, column.statistics,
+ detail::has_supported_type_defined_order(metadata,
column_schema.leaf_column_id));
+ return ParquetStatisticsUtils::TransformColumnStatistics(
+ column_schema, &statistics, column.num_values, timezone,
null_count_trusted);
+}
+
+bool native_footer_has_zero_nan_count(const tparquet::FileMetaData& metadata,
+ const tparquet::RowGroup& row_group,
+ const ParquetColumnSchema& column_schema,
+ const cctz::time_zone* timezone, bool
null_count_trusted) {
+ const auto physical = column_schema.type_descriptor.physical_type;
+ if ((physical != tparquet::Type::FLOAT && physical !=
tparquet::Type::DOUBLE) ||
+ column_schema.leaf_column_id < 0 ||
+ std::cmp_greater_equal(column_schema.leaf_column_id,
row_group.columns.size())) {
+ return false;
+ }
+ const auto& chunk = row_group.columns[column_schema.leaf_column_id];
+ // Avoid decoding footer bounds again for integer/string columns and
legacy floating files.
+ if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics ||
+ !chunk.meta_data.statistics.__isset.nan_count ||
+ chunk.meta_data.statistics.nan_count != 0) {
+ return false;
+ }
+ return native_footer_column_statistics(metadata, row_group, column_schema,
timezone,
+ null_count_trusted)
+ .has_zero_nan_count;
+}
+
void collect_filtered_leaf_ids(const ParquetColumnSchema& column_schema,
const format::LocalColumnIndex* projection,
std::set<int>* leaf_column_ids) {
@@ -1092,24 +1146,11 @@ bool check_native_statistics(const
tparquet::FileMetaData& metadata,
column_schema->leaf_column_id >=
static_cast<int>(row_group.columns.size())) {
return;
}
- const auto& chunk = row_group.columns[column_schema->leaf_column_id];
- std::shared_ptr<segment_v2::ZoneMap> zone_map;
- if (chunk.__isset.meta_data) {
- const auto& column_metadata = chunk.meta_data;
- std::optional<tparquet::Statistics> safe_statistics;
- if (column_metadata.__isset.statistics) {
- safe_statistics = detail::sanitize_native_footer_statistics(
- column_schema->type_descriptor,
column_metadata.statistics,
- detail::has_supported_type_defined_order(metadata,
-
column_schema->leaf_column_id));
- }
- zone_map = ParquetStatisticsUtils::MakeZoneMap(
- ParquetStatisticsUtils::TransformColumnStatistics(
- *column_schema,
- safe_statistics.has_value() ? &*safe_statistics :
nullptr,
- column_metadata.num_values, timezone,
null_count_trusted));
- }
- add_slot_zonemap(ctx, slot_index, column_schema->type,
std::move(zone_map));
+ const auto statistics = native_footer_column_statistics(metadata,
row_group, *column_schema,
+ timezone,
null_count_trusted);
+ add_slot_zonemap(ctx, slot_index, column_schema->type,
+ ParquetStatisticsUtils::MakeZoneMap(statistics),
+ statistics.has_zero_nan_count);
};
VExprContextSPtrs direct_conjuncts;
@@ -1881,13 +1922,14 @@ RowRange native_page_row_range(const
tparquet::OffsetIndex& offset_index, size_t
class NativePageIndexPredicateEvaluator {
public:
NativePageIndexPredicateEvaluator(
- const tparquet::FileMetaData& metadata,
+ const tparquet::FileMetaData& metadata, const tparquet::RowGroup&
row_group,
const std::unordered_map<int, NativeParquetPageIndex>&
page_indexes,
const std::vector<std::unique_ptr<ParquetColumnSchema>>&
file_schema,
const format::FileScanRequest& request, int64_t row_group_rows,
ParquetPruningStats* pruning_stats, const cctz::time_zone*
timezone,
bool null_count_trusted)
: _metadata(metadata),
+ _row_group(row_group),
_page_indexes(page_indexes),
_file_schema(file_schema),
_request(request),
@@ -1912,6 +1954,7 @@ public:
private:
struct SlotPageZoneMaps {
DataTypePtr data_type;
+ bool has_zero_nan_count = false;
std::vector<RowRange> ranges;
std::vector<std::shared_ptr<segment_v2::ZoneMap>> zone_maps;
};
@@ -1964,7 +2007,8 @@ private:
std::vector<RowRange> ranges;
for (size_t page_idx = 0; page_idx < pages->ranges.size(); ++page_idx)
{
ZoneMapEvalContext ctx;
- add_slot_zonemap(&ctx, slot_index, pages->data_type,
pages->zone_maps[page_idx]);
+ add_slot_zonemap(&ctx, slot_index, pages->data_type,
pages->zone_maps[page_idx],
+ pages->has_zero_nan_count);
if (expr->evaluate_zonemap_filter(ctx) !=
ZoneMapFilterResult::kNoMatch) {
append_row_range(pages->ranges[page_idx], &ranges);
}
@@ -1999,6 +2043,10 @@ private:
const auto& indexes = index_it->second;
SlotPageZoneMaps pages;
pages.data_type = column_schema->type;
+ // The owning column chunk's zero covers all its pages. A count from
another row group or
+ // column would be unsafe, e.g. if this column has finite bounds but
also contains a NaN.
+ pages.has_zero_nan_count = native_footer_has_zero_nan_count(
+ _metadata, _row_group, *column_schema, _timezone,
_null_count_trusted);
pages.ranges.reserve(indexes.offset_index.page_locations.size());
pages.zone_maps.reserve(indexes.offset_index.page_locations.size());
for (size_t page_idx = 0; page_idx <
indexes.offset_index.page_locations.size();
@@ -2020,6 +2068,7 @@ private:
}
const tparquet::FileMetaData& _metadata;
+ const tparquet::RowGroup& _row_group;
const std::unordered_map<int, NativeParquetPageIndex>& _page_indexes;
const std::vector<std::unique_ptr<ParquetColumnSchema>>& _file_schema;
const format::FileScanRequest& _request;
@@ -2088,6 +2137,8 @@ Status select_row_group_ranges_by_native_page_index(
continue;
}
const auto& indexes = index_it->second;
+ const bool has_zero_nan_count = native_footer_has_zero_nan_count(
+ metadata, row_group, *column_schema, timezone,
null_count_trusted);
std::vector<RowRange> filter_ranges;
bool usable = true;
for (size_t page_idx = 0; page_idx <
indexes.offset_index.page_locations.size();
@@ -2103,7 +2154,7 @@ Status select_row_group_ranges_by_native_page_index(
}
ZoneMapEvalContext ctx;
add_slot_zonemap(&ctx, slot_index, column_schema->type,
- ParquetStatisticsUtils::MakeZoneMap(statistics));
+ ParquetStatisticsUtils::MakeZoneMap(statistics),
has_zero_nan_count);
if (VExprContext::evaluate_zonemap_filter(conjuncts, ctx) !=
ZoneMapFilterResult::kNoMatch) {
append_row_range(page_range, &filter_ranges);
@@ -2123,8 +2174,8 @@ Status select_row_group_ranges_by_native_page_index(
}
}
- NativePageIndexPredicateEvaluator evaluator(metadata, page_indexes,
file_schema, request,
- row_group_rows, pruning_stats,
timezone,
+ NativePageIndexPredicateEvaluator evaluator(metadata, row_group,
page_indexes, file_schema,
+ request, row_group_rows,
pruning_stats, timezone,
null_count_trusted);
for (const auto& conjunct : multi_slot_conjuncts) {
auto conjunct_ranges = evaluator.evaluate(conjunct->root());
diff --git a/be/src/format_v2/parquet/parquet_statistics.h
b/be/src/format_v2/parquet/parquet_statistics.h
index 9afb336f7ac..2f36d2950d4 100644
--- a/be/src/format_v2/parquet/parquet_statistics.h
+++ b/be/src/format_v2/parquet/parquet_statistics.h
@@ -95,12 +95,13 @@ struct ParquetPruningStats {
};
struct ParquetColumnStatistics {
- Field min_value; // column minimum value converted to Doris
type
- Field max_value; // column maximum value
- bool has_null = false; // whether NULL exists
- bool has_not_null = false; // whether non-NULL values exist
- bool has_null_count = false; // whether null_count is valid
- bool has_min_max = false; // whether min/max is valid after conversion
+ Field min_value; // column minimum value converted to
Doris type
+ Field max_value; // column maximum value
+ bool has_null = false; // whether NULL exists
+ bool has_not_null = false; // whether non-NULL values exist
+ bool has_null_count = false; // whether null_count is valid
+ bool has_min_max = false; // whether min/max is valid after
conversion
+ bool has_zero_nan_count = false; // validated proof covering this column's
statistics
bool has_any_statistics() const { return has_null_count || has_min_max; }
};
diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp
b/be/test/format_v2/parquet/parquet_scan_test.cpp
index af4a6fe0633..0a5339cbccd 100644
--- a/be/test/format_v2/parquet/parquet_scan_test.cpp
+++ b/be/test/format_v2/parquet/parquet_scan_test.cpp
@@ -31,8 +31,10 @@
#include <chrono>
#include <cstdlib>
#include <cstring>
+#include <ctime>
#include <filesystem>
#include <fstream>
+#include <iostream>
#include <limits>
#include <memory>
#include <numeric>
@@ -1347,17 +1349,19 @@ std::shared_ptr<arrow::Array> build_list_array() {
return finish_array(&builder);
}
-void write_table(const std::string& file_path, const
std::shared_ptr<arrow::Table>& table,
- int64_t row_group_size, bool enable_dictionary = false,
- bool enable_page_index = false, bool enable_statistics = true,
- std::optional<::parquet::Encoding::type> encoding =
std::nullopt) {
+void write_table(
+ const std::string& file_path, const std::shared_ptr<arrow::Table>&
table,
+ int64_t row_group_size, bool enable_dictionary = false, bool
enable_page_index = false,
+ bool enable_statistics = true,
+ std::optional<::parquet::Encoding::type> encoding = std::nullopt,
+ ::parquet::ParquetDataPageVersion page_version =
::parquet::ParquetDataPageVersion::V2) {
auto file_result = arrow::io::FileOutputStream::Open(file_path);
ASSERT_TRUE(file_result.ok()) << file_result.status();
std::shared_ptr<arrow::io::FileOutputStream> out = *file_result;
::parquet::WriterProperties::Builder builder;
builder.version(::parquet::ParquetVersion::PARQUET_2_6);
- builder.data_page_version(::parquet::ParquetDataPageVersion::V2);
+ builder.data_page_version(page_version);
builder.compression(::parquet::Compression::UNCOMPRESSED);
if (enable_dictionary) {
builder.enable_dictionary();
@@ -1379,6 +1383,46 @@ void write_table(const std::string& file_path, const
std::shared_ptr<arrow::Tabl
row_group_size,
builder.build()));
}
+void write_footer_nan_counts(const std::string& file_path,
+ const std::vector<std::optional<int64_t>>&
counts) {
+ std::ifstream input(file_path, std::ios::binary | std::ios::ate);
+ DORIS_CHECK(input.good());
+ const auto size = static_cast<std::streamoff>(input.tellg());
+ std::vector<uint8_t> bytes(cast_set<size_t>(size));
+ input.seekg(0);
+ input.read(reinterpret_cast<char*>(bytes.data()), size);
+ DORIS_CHECK(input.good());
+ uint32_t footer_size = decode_fixed32_le(bytes.data() + bytes.size() - 8);
+ const size_t footer_offset = bytes.size() - 8 - footer_size;
+ tparquet::FileMetaData metadata;
+ DORIS_CHECK(deserialize_thrift_msg(bytes.data() + footer_offset,
&footer_size, true, &metadata)
+ .ok());
+ size_t index = 0;
+ for (auto& row_group : metadata.row_groups) {
+ for (auto& chunk : row_group.columns) {
+ DORIS_CHECK(index < counts.size());
+ if (counts[index].has_value()) {
+ chunk.meta_data.statistics.__set_nan_count(*counts[index]);
+ }
+ ++index;
+ }
+ }
+ DORIS_CHECK(index == counts.size());
+ std::vector<uint8_t> footer;
+ ThriftSerializer serializer(/*compact=*/true, 1024);
+ DORIS_CHECK(serializer.serialize(&metadata, &footer).ok());
+ bytes.resize(footer_offset);
+ bytes.insert(bytes.end(), footer.begin(), footer.end());
+ std::array<uint8_t, 4> encoded_size {};
+ encode_fixed32_le(encoded_size.data(), cast_set<uint32_t>(footer.size()));
+ bytes.insert(bytes.end(), encoded_size.begin(), encoded_size.end());
+ bytes.insert(bytes.end(), {'P', 'A', 'R', '1'});
+ std::ofstream output(file_path, std::ios::binary | std::ios::trunc);
+ output.write(reinterpret_cast<const char*>(bytes.data()), bytes.size());
+ output.close();
+ DORIS_CHECK(output.good());
+}
+
template <typename T>
std::vector<uint8_t> build_parquet_bloom_filter(const T* values, size_t count)
{
format::parquet::native::BlockSplitBloomFilter bloom_filter;
@@ -1966,10 +2010,319 @@ protected:
return request;
}
+ struct FloatingScanResult {
+ ColumnInt32::Container ids;
+ int64_t raw_rows = 0;
+ int64_t filtered_groups = 0;
+ int64_t filtered_page_rows = 0;
+ int64_t read_bytes = 0;
+ };
+
+ void scan_floating(int column_id, const VExprContextSPtr& conjunct,
FloatingScanResult* result,
+ int second_column_id = -1) {
+ const auto bytes_before =
DorisMetrics::instance()->local_bytes_read_total->value();
+ RuntimeProfile profile("profile");
+ auto reader = create_reader(0, -1, &profile);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ use_schema_order_positions(request.get(), schema);
+ format::FileScanRequestBuilder builder(request.get());
+
ASSERT_TRUE(builder.add_predicate_column(format::LocalColumnId(column_id)).ok());
+
ASSERT_TRUE(builder.add_non_predicate_column(format::LocalColumnId(2)).ok());
+
request->predicate_only_columns.push_back(format::LocalColumnId(column_id));
+ if (second_column_id >= 0) {
+
ASSERT_TRUE(builder.add_predicate_column(format::LocalColumnId(second_column_id)).ok());
+
request->predicate_only_columns.push_back(format::LocalColumnId(second_column_id));
+ }
+ ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok());
+ ASSERT_TRUE(conjunct->open(&state).ok());
+ Defer close([&] { conjunct->close(); });
+ request->conjuncts.push_back(conjunct);
+ ASSERT_TRUE(reader->open(request).ok());
+ bool eof = false;
+ while (!eof) {
+ Block block = build_file_block(schema);
+ size_t rows = 0;
+ ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
+ const auto& ids =
int32_data_column(*block.get_by_position(2).column).get_data();
+ result->ids.insert(result->ids.end(), ids.begin(), ids.end());
+ }
+ result->raw_rows = counter_value(profile, "RawRowsRead");
+ result->filtered_groups = counter_value(profile,
"RowGroupsFilteredByMinMax");
+ result->filtered_page_rows = counter_value(profile,
"FilteredRowsByPage");
+ result->read_bytes =
+ DorisMetrics::instance()->local_bytes_read_total->value() -
bytes_before;
+ }
+
std::filesystem::path _test_dir;
std::string _file_path;
};
+TEST_F(ParquetScanTest, ZeroNanCountRestoresFloatingFooterPruning) {
+ std::vector<float> floats(64, 1.0F);
+ std::vector<double> doubles(64, 1.0);
+ std::fill(floats.begin() + 32, floats.end(), 2.0F);
+ std::fill(doubles.begin() + 32, doubles.end(), 2.0);
+ std::vector<int32_t> ids(64);
+ std::iota(ids.begin(), ids.end(), 0);
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("f", arrow::float32(), false),
+ arrow::field("d", arrow::float64(), false),
+ arrow::field("id", arrow::int32(), false)}),
+ {build_float_array(floats), build_double_array(doubles),
build_int32_array(ids)});
+ ColumnInt32::Container expected;
+ for (int id = 32; id < 64; ++id) {
+ expected.push_back(id);
+ }
+ for (const auto version :
+ {::parquet::ParquetDataPageVersion::V1,
::parquet::ParquetDataPageVersion::V2}) {
+ for (const auto count : std::vector<std::optional<int64_t>>
{std::nullopt, 0, 1, -1, 33}) {
+ SCOPED_TRACE(count.has_value() ? std::to_string(*count) :
"missing");
+ write_table(_file_path, table, 32, false, false, true,
std::nullopt, version);
+ write_footer_nan_counts(_file_path,
+ {count, count, std::nullopt, count, count,
std::nullopt});
+ const auto check = [&]<PrimitiveType Type>(int column_id) {
+ const auto one = Field::create_field<Type>(1);
+ const auto two = Field::create_field<Type>(2);
+ std::vector<VExprContextSPtr> predicates {
+ create_floating_function_conjunct(column_id, Type,
"gt", TExprOpcode::GT,
+ one),
+ create_floating_function_conjunct(column_id, Type,
"ge", TExprOpcode::GE,
+ two),
+ create_floating_function_conjunct(column_id, Type,
"ne", TExprOpcode::NE,
+ one),
+ create_floating_in_conjunct(column_id, Type, {one},
true)};
+ for (const auto& predicate : predicates) {
+ FloatingScanResult result;
+ scan_floating(column_id, predicate, &result);
+ EXPECT_EQ(result.ids, expected);
+ EXPECT_EQ(result.raw_rows, count == 0 ? 32 : 64);
+ EXPECT_EQ(result.filtered_groups, count == 0 ? 1 : 0);
+ }
+ };
+ check.template operator()<TYPE_FLOAT>(0);
+ check.template operator()<TYPE_DOUBLE>(1);
+ }
+ }
+}
+
+TEST_F(ParquetScanTest, ZeroNanCountRestoresFloatingPageIndexPruning) {
+ std::vector<float> floats(128, 1.0F);
+ std::vector<double> doubles(128, 1.0);
+ std::fill(floats.begin() + 64, floats.end(), 2.0F);
+ std::fill(doubles.begin() + 64, doubles.end(), 2.0);
+ std::vector<int32_t> ids(128);
+ std::iota(ids.begin(), ids.end(), 0);
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("f", arrow::float32(), false),
+ arrow::field("d", arrow::float64(), false),
+ arrow::field("id", arrow::int32(), false)}),
+ {build_float_array(floats), build_double_array(doubles),
build_int32_array(ids)});
+ ColumnInt32::Container expected;
+ for (int id = 64; id < 128; ++id) {
+ expected.push_back(id);
+ }
+ for (const auto version :
+ {::parquet::ParquetDataPageVersion::V1,
::parquet::ParquetDataPageVersion::V2}) {
+ for (const auto count : std::vector<std::optional<int64_t>>
{std::nullopt, 0, 1, -1, 129}) {
+ write_table(_file_path, table, 128, false, true, true,
std::nullopt, version);
+ write_footer_nan_counts(_file_path, {count, count, std::nullopt});
+ const auto check = [&]<PrimitiveType Type>(int column_id) {
+ FloatingScanResult result;
+ scan_floating(
+ column_id,
+ create_floating_function_conjunct(column_id, Type,
"gt", TExprOpcode::GT,
+
Field::create_field<Type>(1)),
+ &result);
+ EXPECT_EQ(result.ids, expected);
+ EXPECT_EQ(result.filtered_groups, 0);
+ EXPECT_EQ(result.raw_rows, count == 0 ? 64 : 128);
+ EXPECT_EQ(result.filtered_page_rows, count == 0 ? 64 : 0);
+ };
+ check.template operator()<TYPE_FLOAT>(0);
+ check.template operator()<TYPE_DOUBLE>(1);
+ }
+ }
+}
+
+TEST_F(ParquetScanTest, ZeroNanCountRespectsOrBranchesInPageIndex) {
+ std::vector<float> floats(128, 1.0F);
+ std::vector<double> doubles(128, 1.0);
+ std::fill(floats.begin() + 64, floats.end(), 2.0F);
+ std::fill(doubles.begin() + 64, doubles.end(), 2.0);
+ std::vector<int32_t> ids(128);
+ std::iota(ids.begin(), ids.end(), 0);
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("f", arrow::float32(), false),
+ arrow::field("d", arrow::float64(), false),
+ arrow::field("id", arrow::int32(), false)}),
+ {build_float_array(floats), build_double_array(doubles),
build_int32_array(ids)});
+ for (const auto double_count : std::vector<std::optional<int64_t>> {0,
std::nullopt}) {
+ write_table(_file_path, table, 128, false, true);
+ write_footer_nan_counts(_file_path, {0, double_count, std::nullopt});
+ auto f = create_floating_function_conjunct(0, TYPE_FLOAT, "gt",
TExprOpcode::GT,
+
Field::create_field<TYPE_FLOAT>(1));
+ auto d = create_floating_function_conjunct(1, TYPE_DOUBLE, "gt",
TExprOpcode::GT,
+
Field::create_field<TYPE_DOUBLE>(1));
+ auto disjunction = create_compound_conjunct(TExprOpcode::COMPOUND_OR,
f->root(), d->root());
+ FloatingScanResult result;
+ scan_floating(0, disjunction, &result, 1);
+ ColumnInt32::Container expected;
+ for (int row = 64; row < 128; ++row) {
+ expected.push_back(row);
+ }
+ EXPECT_EQ(result.ids, expected);
+ EXPECT_EQ(result.filtered_groups, 0);
+ EXPECT_EQ(result.raw_rows, double_count == 0 ? 64 : 128);
+ EXPECT_EQ(result.filtered_page_rows, double_count == 0 ? 64 : 0);
+ }
+}
+
+TEST_F(ParquetScanTest, NanCountProofIsScopedToItsColumnAndRowGroup) {
+ // A zero for the finite FLOAT column or the first row group must not hide
a DOUBLE NaN.
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("f", arrow::float32(), false),
+ arrow::field("d", arrow::float64(), false),
+ arrow::field("id", arrow::int32(), false)}),
+ {build_float_array({1, 1, 1, 1}),
+ build_double_array({1, 1, 1,
std::numeric_limits<double>::quiet_NaN()}),
+ build_int32_array({0, 1, 2, 3})});
+ for (const auto count : std::vector<std::optional<int64_t>> {std::nullopt,
1, -1, 3}) {
+ write_table(_file_path, table, 2);
+ write_footer_nan_counts(_file_path, {0, 0, std::nullopt, 0, count,
std::nullopt});
+ FloatingScanResult result;
+ scan_floating(1,
+ create_floating_function_conjunct(1, TYPE_DOUBLE, "gt",
TExprOpcode::GT,
+
Field::create_field<TYPE_DOUBLE>(1)),
+ &result);
+ EXPECT_EQ(result.ids, ColumnInt32::Container({3}));
+ EXPECT_EQ(result.raw_rows, 2);
+ EXPECT_EQ(result.filtered_groups, 1);
+ }
+}
+
+TEST_F(ParquetScanTest, NanCountPreservesNullAndNanRowsAcrossEncodings) {
+ arrow::FloatBuilder floats;
+ arrow::DoubleBuilder doubles;
+ for (int row = 0; row < 12; ++row) {
+ if (row < 4 || row == 8) {
+ ASSERT_TRUE(floats.AppendNull().ok());
+ ASSERT_TRUE(doubles.AppendNull().ok());
+ } else {
+ const double value = row < 8 || row == 10 ?
std::numeric_limits<double>::quiet_NaN()
+ : (row == 9 ? 1.0 : 2.0);
+ ASSERT_TRUE(floats.Append(static_cast<float>(value)).ok());
+ ASSERT_TRUE(doubles.Append(value).ok());
+ }
+ }
+ std::vector<int32_t> ids(12);
+ std::iota(ids.begin(), ids.end(), 0);
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("f", arrow::float32()),
arrow::field("d", arrow::float64()),
+ arrow::field("id", arrow::int32(), false)}),
+ {finish_array(&floats), finish_array(&doubles),
build_int32_array(ids)});
+ const ColumnInt32::Container expected {4, 5, 6, 7, 10, 11};
+ for (const auto version :
+ {::parquet::ParquetDataPageVersion::V1,
::parquet::ParquetDataPageVersion::V2}) {
+ for (const bool dictionary : {false, true}) {
+ for (const bool page_index : {false, true}) {
+ for (int mode = 0; mode < 3; ++mode) {
+ // Mode 0 is the statistics-disabled oracle; mode 1
supplies positive counts,
+ // and mode 2 leaves them absent. Neither may lose NaNs
hidden by finite bounds.
+ write_table(_file_path, table, 4, dictionary, page_index,
mode != 0,
+ std::nullopt, version);
+ if (mode == 1) {
+ write_footer_nan_counts(_file_path, {0, 0,
std::nullopt, 4, 4, std::nullopt,
+ 1, 1,
std::nullopt});
+ }
+ const auto check = [&]<PrimitiveType Type>(int column_id) {
+ FloatingScanResult result;
+ scan_floating(column_id,
+ create_floating_function_conjunct(
+ column_id, Type, "gt",
TExprOpcode::GT,
+ Field::create_field<Type>(1)),
+ &result);
+ EXPECT_EQ(result.ids, expected);
+ };
+ check.template operator()<TYPE_FLOAT>(0);
+ check.template operator()<TYPE_DOUBLE>(1);
+ }
+ }
+ }
+ }
+}
+
+TEST_F(ParquetScanTest, ZeroNanCountReaderMicrobench) {
+ if (std::getenv("DORIS_PARQUET_NAN_COUNT_BENCHMARK") == nullptr) {
+ GTEST_SKIP() << "Set DORIS_PARQUET_NAN_COUNT_BENCHMARK=1 to run the
reader microbenchmark";
+ }
+ constexpr int GROUPS = 128;
+ constexpr int GROUP_ROWS = 4096;
+ constexpr int ITERATIONS = 8;
+ constexpr int SAMPLES = 7;
+ std::vector<float> floats(GROUPS * GROUP_ROWS, 1.0F);
+ std::vector<double> doubles(GROUPS * GROUP_ROWS, 1.0);
+ std::fill(floats.end() - GROUP_ROWS, floats.end(), 2.0F);
+ std::fill(doubles.end() - GROUP_ROWS, doubles.end(), 2.0);
+ std::vector<int32_t> ids(GROUPS * GROUP_ROWS);
+ std::iota(ids.begin(), ids.end(), 0);
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("f", arrow::float32(), false),
+ arrow::field("d", arrow::float64(), false),
+ arrow::field("id", arrow::int32(), false)}),
+ {build_float_array(floats), build_double_array(doubles),
build_int32_array(ids)});
+ const auto missing_path = (_test_dir / "missing.parquet").string();
+ const auto zero_path = (_test_dir / "zero.parquet").string();
+ _file_path = missing_path;
+ write_table(_file_path, table, GROUP_ROWS);
+ std::filesystem::copy_file(missing_path, zero_path);
+ std::vector<std::optional<int64_t>> counts;
+ for (int group = 0; group < GROUPS; ++group) {
+ counts.insert(counts.end(), {0, 0, std::nullopt});
+ }
+ write_footer_nan_counts(zero_path, counts);
+ // Compare identical data pages with/without the optional proof. Alternate
order after warmup.
+ for (int sample = -1; sample < SAMPLES; ++sample) {
+ for (int turn = 0; turn < 2; ++turn) {
+ const bool zero = (turn + sample + 1) % 2 == 0;
+ _file_path = zero ? zero_path : missing_path;
+ int64_t nanos = 0;
+ std::clock_t cpu_ticks = 0;
+ int64_t read_bytes = 0;
+ for (int iteration = 0; iteration < ITERATIONS; ++iteration) {
+ FloatingScanResult result;
+ auto predicate = create_floating_function_conjunct(
+ 1, TYPE_DOUBLE, "gt", TExprOpcode::GT,
Field::create_field<TYPE_DOUBLE>(1));
+ const auto cpu_start = std::clock();
+ const auto start = std::chrono::steady_clock::now();
+ scan_floating(1, predicate, &result);
+ nanos += std::chrono::duration_cast<std::chrono::nanoseconds>(
+ std::chrono::steady_clock::now() - start)
+ .count();
+ cpu_ticks += std::clock() - cpu_start;
+ ASSERT_EQ(result.ids.size(), GROUP_ROWS);
+ for (int row = 0; row < GROUP_ROWS; ++row) {
+ ASSERT_EQ(result.ids[row], (GROUPS - 1) * GROUP_ROWS +
row);
+ }
+ ASSERT_EQ(result.raw_rows, zero ? GROUP_ROWS : GROUPS *
GROUP_ROWS);
+ ASSERT_EQ(result.filtered_groups, zero ? GROUPS - 1 : 0);
+ read_bytes += result.read_bytes;
+ }
+ if (sample >= 0) {
+ std::cout << "PARQUET_NAN_COUNT_WORK sample=" << sample << "
zero=" << zero
+ << " wall_ns=" << nanos / ITERATIONS << " cpu_ns="
+ << static_cast<double>(cpu_ticks) / CLOCKS_PER_SEC *
1e9 / ITERATIONS
+ << " read_bytes=" << read_bytes / ITERATIONS
+ << " raw_rows=" << (zero ? GROUP_ROWS : GROUPS *
GROUP_ROWS)
+ << " filtered_groups=" << (zero ? GROUPS - 1 : 0) <<
'\n';
+ }
+ }
+ }
+}
+
TEST_F(ParquetScanTest,
FloatingPredicatesPreserveDorisSemanticsWithRealBloomFilters) {
write_floating_parquet_file_with_bloom_filters(_file_path);
diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp
b/be/test/format_v2/parquet/parquet_statistics_test.cpp
index 5065219d01e..dd65f1c0144 100644
--- a/be/test/format_v2/parquet/parquet_statistics_test.cpp
+++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp
@@ -774,6 +774,57 @@ TEST(NativeParquetStatisticsTest,
FloatingNanEqualityKeepsFiniteOnlyFooterAndPag
uint64_t
{0x7ff8000000000002ULL});
}
+TEST(NativeParquetStatisticsTest,
ZeroNanCountRequiresConsistentFloatingStatistics) {
+ const auto check = []<PrimitiveType Type, typename
DataType>(tparquet::Type::type physical) {
+ using T = typename PrimitiveTypeTraits<Type>::CppType;
+ format::parquet::ParquetColumnSchema schema;
+ schema.type = std::make_shared<DataType>();
+ schema.type_descriptor.doris_type = schema.type;
+ schema.type_descriptor.physical_type = physical;
+ const T one = 1;
+ tparquet::Statistics stats;
+ stats.__set_null_count(0);
+ stats.__set_min_value(std::string(reinterpret_cast<const char*>(&one),
sizeof(T)));
+ stats.__set_max_value(stats.min_value);
+ const auto transform = [&](const tparquet::Statistics& input, int64_t
rows = 2,
+ bool trust_nulls = true) {
+ return
format::parquet::ParquetStatisticsUtils::TransformColumnStatistics(
+ schema, &input, rows, nullptr, trust_nulls);
+ };
+ EXPECT_FALSE(transform(stats).has_zero_nan_count);
+ for (const int64_t count : {-1, 0, 1, 3}) {
+ stats.__set_nan_count(count);
+ EXPECT_EQ(transform(stats).has_zero_nan_count, count == 0);
+ }
+ stats.__set_nan_count(0);
+ EXPECT_FALSE(transform(stats, -1).has_zero_nan_count);
+ auto missing_null_count = stats;
+ missing_null_count.__isset.null_count = false;
+ EXPECT_TRUE(transform(missing_null_count).has_zero_nan_count);
+ for (const int64_t nulls : {-1, 3}) {
+ auto invalid = stats;
+ invalid.__set_null_count(nulls);
+ EXPECT_FALSE(transform(invalid).has_zero_nan_count);
+ EXPECT_FALSE(transform(invalid, 2, false).has_zero_nan_count);
+ }
+ auto all_null = stats;
+ all_null.__set_null_count(2);
+ EXPECT_TRUE(transform(all_null).has_zero_nan_count);
+ EXPECT_FALSE(transform(all_null).has_not_null);
+ const T nan = std::numeric_limits<T>::quiet_NaN();
+ auto invalid_bounds = stats;
+ invalid_bounds.__set_min_value(std::string(reinterpret_cast<const
char*>(&nan), sizeof(T)));
+ EXPECT_FALSE(transform(invalid_bounds).has_zero_nan_count);
+ const T zero = 0;
+ invalid_bounds = stats;
+ invalid_bounds.__set_max_value(
+ std::string(reinterpret_cast<const char*>(&zero), sizeof(T)));
+ EXPECT_FALSE(transform(invalid_bounds).has_zero_nan_count);
+ };
+ check.template operator()<TYPE_FLOAT,
DataTypeFloat32>(tparquet::Type::FLOAT);
+ check.template operator()<TYPE_DOUBLE,
DataTypeFloat64>(tparquet::Type::DOUBLE);
+}
+
TEST(NativeParquetStatisticsTest, InvalidNullableDecimalBoundsDisableMinMax) {
format::parquet::ParquetColumnSchema column_schema;
column_schema.type = make_nullable(std::make_shared<DataTypeDecimal32>(2,
0));
diff --git a/gensrc/thrift/parquet.thrift b/gensrc/thrift/parquet.thrift
index 1c18eeb796a..70af288fd12 100644
--- a/gensrc/thrift/parquet.thrift
+++ b/gensrc/thrift/parquet.thrift
@@ -304,6 +304,8 @@ struct Statistics {
7: optional bool is_max_value_exact;
/** If true, min_value is the actual minimum value for a column */
8: optional bool is_min_value_exact;
+ /** Optional NaN count for floating types; an absent count does not prove
the absence of NaNs. */
+ 9: optional i64 nan_count;
}
/** Empty structs to use as logical type annotations */
@@ -1320,4 +1322,4 @@ struct FileCryptoMetaData {
/** Retrieval metadata of key used for encryption of footer,
* and (possibly) columns **/
2: optional binary key_metadata
-}
\ No newline at end of file
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
