Rich-T-kid commented on PR #24418: URL: https://github.com/apache/datafusion/pull/24418#issuecomment-5333679568
> Thank you @Rich-T-kid -- this looks quite clever and can probably generate an observable speedup for end to end benchmarks that read data as parquet dictionaries. Did you try that? > > Also, I wonder if it would be easier to test some of these dictionary related improvements if you added a config setting to treat string columns in parquet files as dictionary (rather than Stringview) 🤔 @alamb no not exactly. I have made some made tweaks to the existing tpch benchmarks by changing datatypes to dict and theyve shown improvement. I have this PR https://github.com/apache/datafusion/pull/24227 that implements your suggestion https://github.com/apache/datafusion/issues/24111#issuecomment-5206277128 , it would be nice if we could run the tpch benchmarks on it while having the flag enabled. I think we could see some very nice results. Ill tweak this adjust this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
