Rich-T-kid commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5370432706
TLDR: the current state of this PR is that it enables direct reads of RLE_DICTIONARY-encoded Parquet columns into dictionary arrays. The main issue right now is a performance regression on some queries, since the flag currently promotes every variable-length column to a dictionary type regardless of cardinality, causing high cardinality columns (5-15 million) to degrade query performance. I still think it's worth keeping the flag: it lets us benchmark dictionary-encoding-related changes, as @alamb pointed out [here](https://github.com/apache/datafusion/pull/24418#pullrequestreview-4965416771), and this PR lays the groundwork for the follow-up PRs in #24111 that will add fine-grained, per-column tuning. The flag will also be turned off by default @alamb @adriangb the PR isn't ready for review I'm more so checking that this is the correct direction to be moving in -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
