adriangb commented on PR #23851:
URL: https://github.com/apache/datafusion/pull/23851#issuecomment-6935790425

   @DarkWanderer @alamb I wonder if this couldn’t fit better inside of 
arrow-rs? Or as some sort of option to tell arrow-rs how to do the reads? Or 
would it make sense to do this only as part of the row filter machinery (maybe 
it is already, I haven’t checked the code)?
   
   This certainly seems like the right idea in principle, but we need to make 
sure it works in practice.
   
   > in some cases it produces small regression due to dictionary check overhead
   
   could you elaborate a bit? maybe we can help resolve this.
   
   fwiw we also have a benchmark mode to simulate s3 like latencies. we can run 
those if you think it might be an improvement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to