jayzhan211 commented on code in PR #27500:
URL: https://github.com/apache/datafusion/pull/27500#discussion_r4650110263


##########
datafusion/common/src/config.rs:
##########
@@ -1451,9 +1451,30 @@ config_namespace! {
         /// This applies only when DataFusion infers the table schema. Tables 
with
         /// a user-supplied schema are not promoted because the Parquet footer 
is
         /// not read at DDL time, so dictionary pages cannot be detected per 
column.
+        ///
+        /// Setting `dictionary_read_max_distinct_values` also enables 
dictionary
+        /// reads. When both options are set, the cap controls which plain 
columns
+        /// are promoted.
         /// See <https://github.com/apache/datafusion/issues/24112>
         pub enable_rle_to_dictionary: bool, default = false
 
+        /// (reading) If set, promote top-level plain string and binary 
columns to
+        /// dictionaries when every nonempty row group has a dictionary page 
and the
+        /// per-file sum of row-group cardinality upper bounds is within this 
cap.
+        /// Missing NDVs use the non-null row count when available and the row 
count
+        /// otherwise. Values repeated across row groups may be counted more 
than once.
+        /// The cap does not demote a dictionary declared by the same file; 
mixed
+        /// dictionary/plain schemas across files still merge as plain. 
Computed during
+        /// schema inference.
+        ///
+        /// Large row-group dictionaries can be slower because each output 
batch can
+        /// retain the whole dictionary. The crossover is workload-dependent: 
in one
+        /// 20-million-row group-by benchmark with 40-byte strings, the case 
with about
+        /// 100,000 distinct values per row group was faster than plain reads, 
while the
+        /// case with about 295,000 was slower. Use a conservative cap and 
benchmark
+        /// representative queries.

Review Comment:
   Optional cleanup: the cap counts the per-file sum, but these numbers are per 
row group, so a reader who sets `cap = 100000` from them won't promote the 
faster file. That benchmark wrote five row groups per file, so its per-file 
sums were 500,000 and about 1,460,000; with `cap = 100000` the 100k file stays 
`Utf8View`, and it is promoted only at `cap = 500000`. Stating the numbers in 
the cap's unit:
   
   ```suggestion
           /// Large row-group dictionaries can be slower because each output 
batch can
           /// retain the whole dictionary. The crossover is 
workload-dependent: in one
           /// 20-million-row group-by benchmark with 40-byte strings and five 
row groups
           /// per file, files with a per-file sum of about 500,000 (100,000 
per row group)
           /// were faster than plain reads, while files with about 1,460,000 
(about
           /// 295,000 per row group) were slower. Use a conservative cap and 
benchmark
           /// representative queries.
   ```
   
   `configs.md` and `information_schema.slt` would need regenerating.



##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -39,13 +40,62 @@ use parquet::file::metadata::{
 use parquet::file::statistics::Statistics as ParquetStatistics;
 use parquet::schema::types::{ColumnDescriptor, SchemaDescriptor};
 use std::any::Any;
+use std::collections::HashSet;
 use std::sync::Arc;
 
 /// Minimum fraction of row groups that must report NDV statistics for the
 /// merged result to be `Inexact` rather than `Absent`, as the estimate
 /// would be too unreliable otherwise.
 const PARTIAL_NDV_THRESHOLD: f64 = 0.75;
 
+fn dictionary_promotion_candidates(
+    metadata: &ParquetMetaData,
+    schema_descr: &SchemaDescriptor,
+    max_distinct_values: Option<usize>,
+) -> HashSet<String> {
+    (0..schema_descr.num_columns())
+        .filter_map(|col_idx| {
+            let column = schema_descr.column(col_idx);
+            let parts = column.path().parts();
+            if parts.len() != 1 {
+                return None;
+            }
+
+            let row_groups = metadata.row_groups();
+            let eligible = match max_distinct_values {
+                None => row_groups
+                    .iter()
+                    .any(|rg| 
rg.column(col_idx).dictionary_page_offset().is_some()),
+                Some(max) => row_groups
+                    .iter()
+                    .try_fold(0_u128, |upper_bound, row_group| {
+                        let row_count = 
u64::try_from(row_group.num_rows()).ok()?;
+                        let column = row_group.column(col_idx);
+                        if row_count > 0 && 
column.dictionary_page_offset().is_none() {
+                            return None;
+                        }

Review Comment:
   @Rich-T-kid thanks for the updates! This check still promotes a column chunk 
whose dictionary outgrew the writer's page limit, so its later pages are PLAIN. 
With `cap = 40000`, a file of 100-byte strings with 30k distinct values per row 
group is promoted, and its queries run 1.2–2.5x slower than plain reads. 
Requiring every data page to be dictionary-encoded, as your reopen comment on 
#24113 describes, keeps that file plain. `page_encoding_stats_mask` is decoded 
by default, so the check needs no extra I/O:
   
   ```suggestion
                           // A chunk whose dictionary outgrew the writer's 
page limit
                           // continues in PLAIN pages, and reads slower as a 
dictionary
                           // than as plain values.
                           let all_dictionary_pages = column
                               .dictionary_page_offset()
                               .is_some()
                               && 
column.page_encoding_stats_mask().is_some_and(|mask| {
                                   
mask.is_only(parquet::basic::Encoding::RLE_DICTIONARY)
                                       || mask.is_only(
                                           
parquet::basic::Encoding::PLAIN_DICTIONARY,
                                       )
                               });
                           if row_count > 0 && !all_dictionary_pages {
                               return None;
                           }
   ```
   
   Files without page encoding stats then stay plain under the cap. The docs 
that say "every nonempty row group has a dictionary page" would change to 
"every data page is dictionary encoded".
   
   <details>
   <summary>Measurements and a regression test (fails on this head with 
`Dictionary(Int32, Utf8)`, passes with the change)</summary>
   
   Data: 4 files × 2M rows, one row group each, `cat = concat(repeat('x', 95), 
lpad(CAST(value % 30000 AS VARCHAR), 5, '0'))`, NDV written, default 
`dictionary_page_size_limit` (1 MiB < 30k × 100 B, so the writer falls back to 
PLAIN). Median of 5, three rounds, datafusion-cli release-nonlto, M-series 
laptop. Results are identical in both modes.
   
   | query | plain | `cap = 40000` (promoted) |
   |---|---:|---:|
   | `SELECT cat, count(*), sum(v) FROM t GROUP BY cat` | 0.096–0.133 s | 
0.142–0.248 s |
   | `SELECT count(DISTINCT cat) FROM t` | 0.074–0.113 s | 0.121–0.190 s |
   | `SELECT count(*) FROM t WHERE cat LIKE '%12345'` | 0.037–0.045 s | 
0.076–0.112 s |
   
   With the change, the files stay `Utf8View`. The same data written with a 64 
MiB page limit (no fallback) is still promoted and still wins on the group-by 
(0.090–0.126 s → 0.042–0.062 s).
   
   Test, placed after `DROP TABLE threshold_no_dict_page;`:
   
   ```
   # A column whose dictionary page outgrew its size limit continues in PLAIN
   # pages, so it stays plain even when its NDV is within the cap.
   query I
   COPY (SELECT column1 AS fallback_page FROM (VALUES ('a'), ('b')))
   TO 
'test_files/scratch/parquet_rle_to_dictionary/threshold_fallback_page/p.parquet'
   STORED AS PARQUET OPTIONS (
     'format.dictionary_page_size_limit' 1,
     'format.write_batch_size' 1,
     'format.data_page_row_count_limit' 1
   );
   ----
   2
   
   statement ok
   CREATE EXTERNAL TABLE threshold_fallback_page STORED AS PARQUET LOCATION 
'test_files/scratch/parquet_rle_to_dictionary/threshold_fallback_page/';
   
   query T
   SELECT DISTINCT arrow_typeof(fallback_page) FROM threshold_fallback_page;
   ----
   Utf8View
   
   statement ok
   DROP TABLE threshold_fallback_page;
   ```
   </details>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to