adriangb commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5282764688
> no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings. Why only RLE and not dictionaries as well? How does this compare to / relate to the `schema_force_view_types` option? It also looks like this goes through `infer_schema` right? A lot of code paths never touch that (`CREATE EXTERNAL TABLE (a VARCHAR)`, any custom table providers, etc.). I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into `Utf8View`. That would be applicable to all DataFusion users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
