ErikBPF opened a new pull request, #7593: URL: https://github.com/apache/datafusion-comet/pull/7593
## Which issue does this PR close? Closes #7455. ## Rationale for this change The Spark 4.2 profile currently inherits Delta 4.3.1, which targets Spark 4.1. Delta 4.4.1 publishes the matching `delta-spark_4.2_2.13` artifact. Hadoop 3.5 also introduces `fs.s3a.encryption.context`. Treating it as an unconditional discovery-test exemption is unsafe: a local Delta shallow clone can reference S3 data files without initializing that bucket during metadata reads. A real MinIO reproducer showed stock Spark rejecting malformed bucket context while `CometDeltaNativeScan` returned all four rows. The previous 281-case contrib run passed and missed that shape. ## What changes are included in this PR? - Pair Spark 4.2 with `io.delta:delta-spark_4.2_2.13:4.4.1`, leaving older profile pins unchanged. - Add the Spark 4.2 Delta contrib lane to nightly and `run-delta-tests` routing, including required-check aggregation. Run required MinIO coverage on Spark 4.2 as well as 3.5. - Add a dedicated context admission gate for selected S3 data and deletion-vector buckets. It uses existing propagated alias and credential-provider resolution and declines nonempty or unverifiable effective context without printing values. SSE-C/CSE decline precedence is unchanged. - Conservatively fall back to Spark for valid configured contexts too, until Comet reproduces Hadoop's setup validation. Unconfigured reads retain the native Delta scan. Document this behavior. - Add the cold metadata failure regression, provider/alias/redaction checks, and a MinIO local-clone regression covering valid and malformed global, short-bucket and long-bucket context settings. The integration test asserts external S3 `AddFile` paths and an unconfigured native-scan control. The separate kernel-backed `-Pcontrib-delta` surface is unchanged; this pairing is for `contrib/delta-spark` under `-Pdelta`. ## How are these changes tested? Validated on Apollo at `fea770f91a0c03f8e3a2d88c7275405fa47ab716` with a freshly built matching JNI: - Full Spark 4.2 / Delta 4.4.1 contrib run: **287 passed, zero failures/errors/cancellations/skips**. Native scan: 119; DML: 3; MinIO: 4; admission/reference checks: 161. - Spark 4.2 root-reactor test compilation and Spark 3.5 / Delta 3.3.2 root-reactor compilation with `-Pstrict-warnings` pass. - The unchanged original cross-store reproducer fails before the guard and passes afterward. A mutation that changes only the captured Comet error message causes exactly the parity regression to fail. - CI routing assertion RED/GREEN, suite registry, Actionlint and Spotless checks pass. - Independent review and [Hackerman source/static review](https://github.com/ErikBPF/datafusion-comet/pull/6#issuecomment-6766721334) completed. The bot did not independently execute JVM tests. ```bash (cd native && cargo build --locked --profile ci -j 8) mkdir -p native/target/release cp native/target/ci/libcomet.so native/target/release/libcomet.so ./mvnw -B -Prelease -Pspark-4.2,delta test \ -Dtest=none -DwildcardSuites=org.apache.comet.contrib.delta ``` The observed run used a 2 GiB test JVM and `ActiveProcessorCount=2`, with the repository's module-opening arguments, and required MinIO through `COMET_DELTA_S3_REQUIRED=1`. Archive checkouts skipped Git metadata extraction only. Fork GitHub Actions dispatch returns HTTP 422, “Actions has been disabled for this repository.” These are local Apollo results; no GitHub runtime success is claimed. ## Draft review follow-up Maintainer feedback on the fork requests DV-only bucket coverage, encryption-gate precedence tests, Hadoop-key-based test guards, a filesystem-cache comment, and clearer conservative-fallback compatibility wording. These are pending reconciliation while this PR remains a draft. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
