Rich-T-kid commented on PR #23187: URL: https://github.com/apache/datafusion/pull/23187#issuecomment-5193036812
So I think some nice follow ups would be 1. expose a parquet dictionary encoding read flag to users 2. allow a way to read only specific string columns as dictionary encoded (this may already be possible I need to check) by hand we can experiment with transforming low-card string group keys into `Dict<k,v>` and then #22891 will be in a position to be implemented. it may be possible to use column statistics as a guide for "low cardinality" columns but due to the state of statistics in datafusion this may or may not be accurate. ideally this will be tunable as either a hard set value or a percentage. There are some other issues that come to mind like schema management but I think its best to leave that in #22891's thread -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
