Hello, Follow-up to the regtest thread (https://lists.apache.org/thread/4bx31cfbcqfxzgpsddvc9kcfbn9l093y) and current PR (https://github.com/apache/polaris/pull/4588).
Currently we support both Spark 3 (https://github.com/apache/polaris/tree/main/plugins/spark/v3.5) and 4 (https://github.com/apache/polaris/tree/main/plugins/spark/v4.0) for Polaris spark client, however, only spark 3 has regtests. There was a concern with potentially increasing CI time, however, this later got proved to be not the case as "moving regtests to integration tests would not necessarily save time. In fact, it could potentially increase overall CI duration, since the longest running workflows are currently not the regtests". Before we can finalize the testing strategy for Polaris spark client, we need to decide if we want to proceed with the conversion (from docker based to JUnit based). The lack of regtests for spark 4 can potentially cause regression issues later. Also, as we are using JUnit, we can't trigger a actual 'spark-shell xxxx' to simulate the actual `--packages` and `--jars`. However, we can kind get them working by using `URLClassLoader` for `--jars` and `SparkSubmitUtils.resolvedMavenCoordinates` for `--packages`. The catch here is to be able to use `--packages`, we will need to `publishToMavenLocal` (which is project-isolation violation, as it will try to modify `~/.m2`). The suggest is to drop this test and only handle bundle jar via `URLClassLoader`. I am wondering how team would like to proceed as we can't leave spark 4 out there without proper JUnit for a long period of time. Thanks, Yong Zheng
