adriangb commented on PR #23851: URL: https://github.com/apache/datafusion/pull/23851#issuecomment-6935790425
@DarkWanderer @alamb I wonder if this couldn’t fit better inside of arrow-rs? Or as some sort of option to tell arrow-rs how to do the reads? Or would it make sense to do this only as part of the row filter machinery (maybe it is already, I haven’t checked the code)? This certainly seems like the right idea in principle, but we need to make sure it works in practice. > in some cases it produces small regression due to dictionary check overhead could you elaborate a bit? maybe we can help resolve this. fwiw we also have a benchmark mode to simulate s3 like latencies. we can run those if you think it might be an improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
