dwsmith1983 commented on issue #5655: URL: https://github.com/apache/datafusion-comet/issues/5655#issuecomment-6458494677
Narrowed the scope of this issue, since most of what it asked for is already on main. When this was filed, every split of a DV-backed file built an access plan for the whole file and reserved memory for all of its selectors. That changed in #5365 before it merged. `build_access_plan` now takes the split's `FileRange`, and only the row groups the split owns get selectors: https://github.com/apache/datafusion-comet/blob/967d09b52694d64c423012ed7dcc8f7ec6e20413/native/core/src/execution/delta_dv.rs#L163-L254 It keeps file-global row positions across the groups it skips, validates the deletion vector against the whole file, follows DataFusion's half-open dictionary-page boundaries, and reserves memory only for the selectors it keeps. The `split_access_plan_*` tests cover each of these: https://github.com/apache/datafusion-comet/blob/967d09b52694d64c423012ed7dcc8f7ec6e20413/native/core/src/execution/delta_dv.rs#L1278-L1523 After that, #6945 (closes #5656) stopped native-format decoding from expanding the bitmap row by row, which makes each decode cheaper. Neither change removes the repeated deletion vector work. `attach_access_plan` still fetches the blob, checks its CRC, deserializes the full bitmap and validates its cardinality once per split: https://github.com/apache/datafusion-comet/blob/967d09b52694d64c423012ed7dcc8f7ec6e20413/native/core/src/execution/delta_dv.rs#L646-L678 So the issue now covers decoding the DV once and sharing it across a file's splits, plus a split-file benchmark to show whether that is worth doing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
