xiaoh1024 opened a new issue, #6070: URL: https://github.com/apache/datafusion-comet/issues/6070
### What is the problem the feature request solves? Vortex scans currently execute through the Vortex Spark reader. Native scan support would allow Comet to read Vortex files directly and feed the resulting Arrow batches into downstream Comet operators. ### Describe the potential solution Follow the contrib architecture proposed for Lance in #4632 and #4633, including the [maintainer recommendation to keep format-specific integration in contrib](https://github.com/apache/datafusion-comet/pull/4633#issuecomment-4940130267). Spark and the Vortex Spark connector would remain responsible for scan planning, including file discovery, projection, and filter pushdown. A Vortex contrib module would consume the resulting scan information through the existing Comet contrib interfaces and execute supported scans using `vortex-datafusion`. Build-time and runtime support would be opt-in. Scans outside the supported scope would retain the Vortex Spark reader. The companion API change, [vortex-data/vortex#9956](https://github.com/vortex-data/vortex/pull/9956), exposes the full table schema and accepted predicates. This lets the native reader preserve filtering semantics when columns used only by predicates have been pruned from the scan output. Proposed contribution stages: 1. Add the optional Vortex contrib module and build support, following the precedent in #5728. 2. Add the native reader and Spark-side integration, together with supported-scan checks, fallback behavior, and end-to-end tests. 3. Extend scan capabilities and add performance optimizations in follow-up PRs. ### Additional context We have a working prototype with local and distributed correctness checks, including full output comparisons against Spark, and plan to contribute it in stages. The packaged prototype also encountered an Arrow C Data class conflict similar to the one [reported during the Lance integration](https://github.com/apache/datafusion-comet/issues/4632#issuecomment-4720970848). We would like to coordinate the packaging approach with that effort. Does this integration approach and proposed contribution split fit the direction of Comet’s contrib integrations? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
