jayshrivastava commented on issue #23376: URL: https://github.com/apache/datafusion/issues/23376#issuecomment-5047433301
The binary search seems reasonable. I suspect the way to implement it is to still use a CASE expression over the ranges but make the CASE expression evaluation support a mode where it does a binary search when the cases are ordered. I assume each case will be something like a <= expr < b where a and b are range partition split points. Today, I believe CASE expressions do a linear search IIUC so I would expect this to perform a lot better than the CASE hash for hash partitioning (also we would not have to pay the cost of hashing). I should also mention there's people considering removing / changing the CASE hash(expr) logic for performance reasons: https://github.com/apache/datafusion/issues/21207#issuecomment-5029901073 However, I think that work should not block your work here. We can always refactor it later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
