alamb commented on issue #8842:
URL: https://github.com/apache/arrow-rs/issues/8842#issuecomment-5591951388
I think the key for that particular issue, the trick will be to figure out
what the API is to express predicates (the current predicate API assumes the
expressions get RecordBatches in, but then implement decoding directly on data
you'll probably need a parquet specific expression encoding -- I would
recommend looking at Vortex and Iceberg for ideas
I recommend starting simple (column equality to a constant, and ANDs) and
work out the details there. The API should be extensible enough to support more
things down the line but not needed
We also need to demonstrate that operating on encoded data is actually more
performant -- in this case that it is faster than the existing predicate
pushdown -- so benchmarks would also be good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
