Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3916700865
##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -439,6 +448,68 @@ impl<'a> DFParquetMetadata<'a> {
.coerce()
})
.unwrap_or(schema);
+
+ let schema = if self.enable_rle_to_dictionary {
+ let schema_descr = file_metadata.schema_descr();
+ // Top-level columns that have a dictionary page in at least one
row group.
+ let dict_cols: HashSet<String> = metadata
+ .row_groups()
+ .iter()
+ .flat_map(|rg| {
+ rg.columns()
+ .iter()
+ .enumerate()
+ .filter_map(|(col_idx, col)| {
+ col.dictionary_page_offset()?;
Review Comment:
these exist in the PR thread. I didn't do a good job distinguishing when the
flag was on or off.
I can enable the flag again and run the benchmarks. from the previous times
that I did run this it caused regressions in query performance as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
