felipepessoto opened a new pull request, #13499:
URL: https://github.com/apache/gluten/pull/13499
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
Fixes #13496.
Add mandatory runtime validation to Delta Spark UT so a missing or disabled
Gluten configuration cannot silently produce a green baseline run.
- Check the actual Spark context's plugin configuration, installed Gluten
SQL rules, and session/thread enablement at test boundaries. Cover shared
Spark, Hive, the newer session-test fixture, Java/JUnit, and the known direct
factories. Java rules are transient so serializable test suites remain usable
in Spark closures.
- Inspect compiled suite inheritance independently of the hooks, and require
matching per-test evidence. Run a real native aggregation in every fork, with
process-specific ScalaTest report IDs.
- Enforce activation checks before baseline/quarantine/seed handling.
Missing, malformed, or failed evidence is fatal; canaries alone cannot make an
empty upstream run pass.
- Support explicit, reasoned exemptions and restored scoped disabling.
Upload evidence and report uncovered suites without assuming they are
Spark-free.
This is CI-only: no production changes and no Java agent. Fixture/factory
checks do not claim coverage of every additional session created through
unwrapped paths inside a test; that limitation is documented.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
- All 26 Python installer/gate unit tests pass.
- Applied the installer to Delta 4.4.1 and reran it idempotently: 194
shared-fixture import aliases, six Java suites, six direct-factory suites, and
the Hive/session-test hooks.
- Compiled the Scala runtime helpers and sbt meta-build helper in a focused
real Spark 4.1.0 / Delta 4.4.1 harness under JDK 17, using the existing Gluten
bundle from CI run https://github.com/apache/gluten/actions/runs/40756222035.
- Positive two-fork native run passed, including the five Scala guard unit
tests, scoped session/thread disabling, stopped-session cleanup, Java/JUnit
lifecycle and suite serialization, and an explicitly exempt vanilla suite. The
independent gate confirmed two native-validated JVMs, five checked tests, and
one exemption.
- Negative controls removing `GlutenPlugin` or setting
`spark.gluten.enabled=false` both failed the JVM run and independently failed
activation validation with exit code 2.
- Python formatting, changed-file Scala formatting, license headers, YAML
parsing, Bash syntax, and whitespace checks pass.
**Draft / CI follow-up:** the complete Delta matrix has not been rerun
locally. The full repository formatter encounters existing Windows symlink
stubs; changed Scala files were checked successfully using the repository
formatter and Maven wrapper with a scoped file selector.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: GitHub Copilot (gpt-6-astra).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
