schenksj commented on PR #4952:
URL:
https://github.com/apache/datafusion-comet/pull/4952#issuecomment-5330312849
@sunchao yes — and that's deliberate. The SPI is agnostic about who does the
planning.
`CometScanContrib.tryTransformV1` receives Spark's already-planned scan:
```scala
def tryTransformV1(
plan: SparkPlan,
session: SparkSession,
scanExec: FileSourceScanExec,
relation: HadoopFsRelation): Option[SparkPlan] = None
```
For a Delta table that `FileSourceScanExec` is produced by **delta-spark's
JVM planning** — by the
time the hook is called, Delta has already resolved the snapshot and
selected files, and
`relation.location` *is* Delta's own `FileIndex` (`TahoeLogFileIndex`,
`PreparedDeltaFileIndex`
after `PreprocessTableWithDVs`, `TahoeBatchFileIndex` for DML). A contrib
that wants to honour that
JVM-side selection can simply read the files out of that index and hand them
to a native reader —
no `delta-kernel-rs` involvement at all. Core neither knows nor cares which
it did.
So the two models you describe both sit behind this one hook, and can
coexist:
- **JVM-planned + native read** — take delta-spark's snapshot/file
selection, read natively.
- **Fully native** — replay the log with `delta-kernel-rs` and enumerate
files there.
The later parts of this series happen to take the second route, but that's a
choice of the Delta
contrib, not a constraint this PR imposes. Nothing in core names either
strategy: the JVM side
routes on a `CometContribScanMarker` type test, and the native side
dispatches the opaque
`contrib_scan` envelope purely on `type_url` — so a second implementation
could ship its own
message under its own `type_url` and be selected at runtime alongside the
existing one.
Worth flagging one real constraint, since it bears on "leverage this as a
foundation": within a
single build, contribs are consulted in `ServiceLoader` order and the
**first claim wins**, and core
cannot detect two contribs claiming the same scan (a claim is opaque — the
only way to know a second
would also have claimed is to ask it, which is what claiming is meant to
prevent). So two Delta
strategies registered at once would need to agree on which owns a given scan
— most naturally one
implementation with a config selecting the strategy, rather than two
competing registrations. The
ownership contract is documented on the trait, and @parthchandra raised
making core *warn* about
this as an optional follow-up.
---
🤖 This reply was drafted with [Claude Code](https://claude.com/claude-code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
