wirybeaver opened a new pull request, #2694:
URL: https://github.com/apache/datafusion-ballista/pull/2694
# Which issue does this PR close?
Part of #1677 (B1); does not close the roadmap.
# Rationale for this change
Ballista needs the provider's planned layout before it can align
storage-partitioned joins. This adds the scan contract and reference adapter;
planner activation belongs to later steps.
# What changes are included in this PR?
Add an immutable Parquet manifest provider with explicit Iceberg integer
buckets, group identities and row estimates. Transport the provider through the
logical codec and preserve source-group metadata at the dependency integration
points.
```text
provider manifest -> grouped file scan
|
client codec -> scheduler reconstructs the same read view
```
Draft dependency: the proposed DataFusion grouping contract is not released.
The integration currently uses the published `wirybeaver/datafusion` branch
`spj/distribution-contract` (55.2 backport), configured by the local override
helper.
# Are there any user-facing changes?
Adds the reference provider and an opt-in setting, disabled by default.
Files must remain immutable and accessible to eligible executors. This is not
an Iceberg catalog implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
