vinishjail97 opened a new issue, #969: URL: https://github.com/apache/incubator-xtable/issues/969
### Feature Request / Improvement `.github/workflows/spark-runtime-validation.yml` (#964) runs `ITXTableSparkRuntimeBundle` on Spark 3.4.3 and 3.5.9, but the engine classpath passed to `spark-submit` always comes from the module's `provided` dependencies, which use `hudi-spark3.4-bundle`. So the Spark 3.5 leg runs against the Spark 3.4 Hudi bundle. Raised in https://github.com/apache/incubator-xtable/pull/964#discussion_r4238765679. Today the sync path only uses `hudi-common` and `hudi-java-client` from the bundle, Iceberg comes in as `iceberg-core`, and Delta on 3.5 uses the Spark-free Kernel, so the coverage gap is small. Still, users on Spark 3.5 bring `hudi-spark3.5-bundle` and `delta-spark` 3.x, and CI should run the same set. Constraint: the IT's own JVM must stay on Spark 3.4, because it writes the Delta source tables through `delta-core` 2.4. Only the `spark-submit` engine classpath should follow `matrix.spark_version`. Possible approach: a property for the engine Spark prefix, used only by the `engine-classpath` execution (e.g. a separate profile or a `dependency:copy` list), set from the workflow matrix, with `delta-spark` and `delta-storage` 3.x swapped in for 3.5. Other formats should be checked the same way, though Iceberg (`iceberg-core`) and Delta Kernel are Spark-independent today. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
