Hi Yong Zheng, Thanks for bringing this up! In short, I don't think it's worth the effort to make this conversion at the moment for the following reasons:
1. *It doesn't meaningfully improve CI time.* I think you mentioned this in the thread as well. Looking at one CI run as an example ( https://github.com/apache/polaris/actions/runs/24255532169/job/70826005994), the Spark Regression Test section only takes about *2m 35s*. Even if we add another Spark 4.x regression test, I don't think it would significantly increase the overall CI time—probably just another 2–3 minutes. The Runtime Service tests are still the slowest part of the pipeline, and their execution time is likely to continue growing. 2. *The regression tests provide a high level of confidence in correctness.* They remain the tests that most closely resemble our customers' actual environments, making them our last line of defense against regressions. That gives them significant value. Rather than spending effort trying to build simulations that provide similar coverage, I think it's better to keep these regression tests in place since they validate the real end-to-end behavior. Those are my thoughts, but I'm happy to discuss further if you see additional benefits that I'm missing. Best Regards, Yun On Sun, Jun 28, 2026 at 8:47 PM Yong Zheng <[email protected]> wrote: > Hello, > > Follow-up to the regtest thread ( > https://lists.apache.org/thread/4bx31cfbcqfxzgpsddvc9kcfbn9l093y) and > current PR (https://github.com/apache/polaris/pull/4588). > > Currently we support both Spark 3 ( > https://github.com/apache/polaris/tree/main/plugins/spark/v3.5) and 4 ( > https://github.com/apache/polaris/tree/main/plugins/spark/v4.0) for > Polaris spark client, however, only spark 3 has regtests. There was a > concern with potentially increasing CI time, however, this later got proved > to be not the case as "moving > regtests to integration tests would not necessarily save time. In fact, it > could potentially increase overall CI duration, since the longest running > workflows are currently not the regtests". > > Before we can finalize the testing strategy for Polaris spark client, we > need to decide if we want to proceed with the conversion (from docker based > to JUnit based). The lack of regtests for spark 4 can potentially cause > regression issues later. > > Also, as we are using JUnit, we can't trigger a actual 'spark-shell xxxx' > to simulate the actual `--packages` and `--jars`. > > However, we can kind get them working by using `URLClassLoader` for > `--jars` and `SparkSubmitUtils.resolvedMavenCoordinates` for `--packages`. > The catch here is to be able to use `--packages`, we will need to > `publishToMavenLocal` (which is project-isolation violation, as it will try > to modify `~/.m2`). The suggest is to drop this test and only handle bundle > jar via `URLClassLoader`. > > I am wondering how team would like to proceed as we can't leave spark 4 > out there without proper JUnit for a long period of time. > > Thanks, > Yong Zheng >
