benibus commented on PR #33732: URL: https://github.com/apache/arrow/pull/33732#issuecomment-1421624442
I attempted to implement the newer `ScanNode` methods. However, I ran into some issues that have prevented feature parity with the legacy API. Primarily, the methods that use `ScanOptions` are able to prevent loading top-level fields if none of its children are expected to be materialized. However, I haven't figured out a good way to do this using `FragmentScanRequest` since it only provides `FieldPath`s (relative to the inspected fragment). I suppose there's a possibility that the tests in `FileFormatScanNodeMixin` are expecting too much of the format implementation, though. From what I can tell, CSV is the only other format that implements the new API and even then it doesn't support nested fields - so the relevant tests in that fixture haven't been used anywhere before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
