andygrove opened a new issue, #56: URL: https://github.com/apache/datafusion-spark/issues/56
Tracks the remaining work to finish moving `datafusion-spark` here from apache/datafusion, following apache/datafusion#27103. ### Build and CI - [ ] Add a workspace `Cargo.toml` (#45) - [ ] Set the crate version to 57.0.0. `datafusion-spark` 56.x is still released from `branch-56` in apache/datafusion, so 57.0.0 is the first release from this repository (#45 sets 56.0.0) - [ ] Decide whether `main` builds against DataFusion `main` at a pinned `rev` or against released DataFusion 56. The code was extracted from DataFusion `main`, and #45 reverts apache/datafusion#26703 to build against `branch-56`. apache/datafusion#27103 proposed a long-lived branch that tracks DataFusion `main` so Comet and Sail can test DataFusion before a release - [ ] Merge a first workflow (no workflow has run on the open pull requests yet) - [ ] `cargo fmt` and `cargo clippy` (#53) - [ ] Run the sqllogictest files (#51) - [ ] License header check (#50). apache/datafusion checks the same SPDX headers with HawkEye - [ ] Run `cargo test -p datafusion-spark` (unit tests and doctests) - [ ] Check that the crate builds with default features, with no default features, and with the `core` feature, like the `cargo check datafusion-spark features` job in apache/datafusion - [ ] Build the docs with `RUSTDOCFLAGS="-D warnings"` - [ ] Add the ASF action allowlist check, as in apache/datafusion-iceberg - [ ] Add a Dependabot configuration - [ ] Require the CI checks in `.asf.yaml` - [ ] Support `--complete` in the sqllogictest runner to regenerate expected output ### Documentation and repository files - [ ] README, `LICENSE.txt` and `NOTICE.txt` (#49) - [ ] The `audit-datafusion-spark-expression` agent skill and `AGENTS.md` (#52), including the [AI policy](https://datafusion.apache.org/contributor-guide/ai-policy.html)'s rule that contributions from fully autonomous agents are not accepted - [ ] Pull request template, and issue templates with a field for Spark versions - [ ] `CONTRIBUTING.md` - [ ] Update the imported docs. `crates/spark/README.md` still describes the crate as a submodule of DataFusion, documents `datafusion-cli --spark` (removed by apache/datafusion#27337), and links to a sqllogictest README that has moved. `crates/sqllogictest/test_files/README.md` also mentions `datafusion-cli --spark`, and links to #5 by its old URL (apache/datafusion#15914) ### Moving work from apache/datafusion - [ ] Post instructions in apache/datafusion#27103 for porting open pull requests - [ ] Port the PySpark runner for the sqllogictest files (apache/datafusion#27165) - [ ] Port or close the [open `spark` pull requests](https://github.com/apache/datafusion/pulls?q=is%3Apr+is%3Aopen+label%3Aspark) in apache/datafusion - [ ] Decide whether to transfer apache/datafusion#7301, apache/datafusion#17964, apache/datafusion#22638, apache/datafusion#23929, apache/datafusion#25118 and apache/datafusion#26185 - [ ] Merge apache/datafusion#27337 and apache/datafusion#27344 once CI runs here, and bring over any Spark changes that land in apache/datafusion before then ### Releases - [ ] Add release scripts under `dev/release/` and changelog generation - [ ] Release 57.0.0 after DataFusion 57.0.0 Until 57.0.0, fixes for `datafusion-spark` 56.x need to be backported to `branch-56` in apache/datafusion. ### Community - [ ] Announce the move on [email protected] - [ ] Decide whether to create a Slack channel - [ ] Agree who leads Spark CI, reviews and Spark version support ### Testing against Spark - [ ] Run the sqllogictest files against Apache Spark in CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
