jayzhan211 commented on code in PR #27500:
URL: https://github.com/apache/datafusion/pull/27500#discussion_r4655921017
##########
datafusion/common/src/config.rs:
##########
@@ -1451,9 +1451,31 @@ config_namespace! {
/// This applies only when DataFusion infers the table schema. Tables
with
/// a user-supplied schema are not promoted because the Parquet footer
is
/// not read at DDL time, so dictionary pages cannot be detected per
column.
+ ///
+ /// Setting `dictionary_read_max_distinct_values` also enables
dictionary
+ /// reads. When both options are set, the cap controls which plain
columns
+ /// are promoted.
/// See <https://github.com/apache/datafusion/issues/24112>
pub enable_rle_to_dictionary: bool, default = false
+ /// (reading) If set, promote top-level plain string and binary
columns to
+ /// dictionaries when every data page is dictionary encoded and the
+ /// per-file sum of row-group cardinality upper bounds is within this
cap.
Review Comment:
Optional cleanup: files without page encoding statistics are never promoted
under the cap, while `enable_rle_to_dictionary` alone does promote them.
parquet-rs only started writing these stats in 55.0.0, so this covers files
written by DataFusion 46 and earlier, and also DuckDB 1.1.3 output. A DuckDB
file with a 4-value string column (NDV written) reads as `Utf8View` at `cap =
100`, but as `Dictionary(Int32, Utf8)` with the legacy flag. A writer that
falls back to PLAIN lists the same `encodings` as one that doesn't, so I don't
see a safe fallback for these files. Saying so in the docs would save users
from wondering why the cap does nothing:
```suggestion
/// dictionaries when every data page is dictionary encoded and the
/// per-file sum of row-group cardinality upper bounds is within
this cap.
/// Files without page encoding statistics, such as those written by
/// parquet-rs before 55.0.0 (DataFusion 46 and earlier) or DuckDB
1.1, are
/// not promoted.
```
`configs.md` and `information_schema.slt` would need regenerating.
##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -1429,7 +1471,11 @@ mod tests {
if let Some(s) = stats {
builder = builder.set_statistics(s);
}
- builder.set_num_values(num_rows).build().unwrap()
+ builder
Review Comment:
@Rich-T-kid thanks for applying these! The page-encoding check I suggested
last round makes `dictionary_promotion_uses_non_null_count_for_missing_ndv`
fail on this head (`left: {}`, `right: {"col_0"}`), because the synthetic
chunks in this helper have a dictionary page offset but no page encoding stats.
Sorry, I only ran the slt with that suggestion. CI skipped the Rust test jobs
for this push, so it hasn't shown up there yet. Giving the helper's dictionary
chunks an all-`RLE_DICTIONARY` mask fixes it; `cargo test -p
datafusion-datasource-parquet` then passes 313/313 and clippy is clean:
```suggestion
if dictionary_page_offset.is_some() {
builder = builder.set_page_encoding_stats_mask(
parquet::basic::EncodingMask::new_from_encodings(
[parquet::basic::Encoding::RLE_DICTIONARY].iter(),
),
);
}
builder
```
##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -39,13 +40,74 @@ use parquet::file::metadata::{
use parquet::file::statistics::Statistics as ParquetStatistics;
use parquet::schema::types::{ColumnDescriptor, SchemaDescriptor};
use std::any::Any;
+use std::collections::HashSet;
use std::sync::Arc;
/// Minimum fraction of row groups that must report NDV statistics for the
/// merged result to be `Inexact` rather than `Absent`, as the estimate
/// would be too unreliable otherwise.
const PARTIAL_NDV_THRESHOLD: f64 = 0.75;
+fn dictionary_promotion_candidates(
+ metadata: &ParquetMetaData,
+ schema_descr: &SchemaDescriptor,
+ max_distinct_values: Option<usize>,
+) -> HashSet<String> {
+ (0..schema_descr.num_columns())
+ .filter_map(|col_idx| {
+ let column = schema_descr.column(col_idx);
+ let parts = column.path().parts();
+ if parts.len() != 1 {
+ return None;
+ }
+
+ let row_groups = metadata.row_groups();
+ let eligible = match max_distinct_values {
+ None => row_groups
+ .iter()
+ .any(|rg|
rg.column(col_idx).dictionary_page_offset().is_some()),
+ Some(max) => row_groups
+ .iter()
+ .try_fold(0_u128, |upper_bound, row_group| {
+ let row_count =
u64::try_from(row_group.num_rows()).ok()?;
+ let column = row_group.column(col_idx);
+ // A chunk whose dictionary outgrew the writer's page
limit
+ // continues in PLAIN pages, and reads slower as a
dictionary
+ // than as plain values.
+ let all_dictionary_pages = column
+ .dictionary_page_offset()
+ .is_some()
+ &&
column.page_encoding_stats_mask().is_some_and(|mask| {
+
mask.is_only(parquet::basic::Encoding::RLE_DICTIONARY)
+ || mask.is_only(
+
parquet::basic::Encoding::PLAIN_DICTIONARY,
Review Comment:
Optional cleanup: no test covers the `PLAIN_DICTIONARY` arm of the check,
because DataFusion's writer always emits `RLE_DICTIONARY`. If that arm is
removed, every test still passes. parquet-mr v1 files use it, and
`parquet-testing` already has one. This case passes on this head and fails with
the arm removed (`BinaryView`). It goes after `DROP TABLE threshold_declared;`:
<details>
<summary>slt case (run with the lines that follow it in the file)</summary>
```
# parquet-mr format v1 writes PLAIN_DICTIONARY data pages; they count as
# dictionary encoded. The file has no NDV, so its 1000 rows are the estimate.
statement ok
set datafusion.execution.parquet.dictionary_read_max_distinct_values = 1000;
statement ok
CREATE EXTERNAL TABLE threshold_plain_dictionary STORED AS PARQUET LOCATION
'../../parquet-testing/data/plain-dict-uncompressed-checksum.parquet';
query T
SELECT DISTINCT arrow_typeof(binary_field) FROM threshold_plain_dictionary;
----
Dictionary(Int32, Binary)
statement ok
DROP TABLE threshold_plain_dictionary;
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]