bharath-techie commented on issue #10655:
URL: https://github.com/apache/arrow-rs/issues/10655#issuecomment-5603726418
So for external indices on large files, I'm planning to write up a separate
issue in datafusion where we can further discuss how datafusion formats can be
accelerated by external indices [ different from existing approaches such as a
join (or) materialize entire bitset/rowSelection ]. TL;DR So OpenSearch uses
two different approaches with datafusion for parquet :
1 ) Basically create a new plan at row group boundaries [ external index
returns a bitset for the RG range + next set bit to skip any irrelevant RGs ] ,
so any random access based skips will be at row group level.
2 ) We use the patched arrow forward reader where we traverse doc by doc [
i don't think this is a general use case for either arrow-rs / datafusion ]
But apart from above use cases - any case where you can skip ahead within RG
/ page - which we haven't pruned already during creation of initial access plan
- will benefit from this method , especially for random access friendly
encodings, as we can skip decode till a particular doc id. [ Maybe lance /
vortex might have more use cases ]
Apart from dynamic filter, sort merge join , lookup join etc where we will
have monotonically increasing rows to match can benefit from this I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
