LuciferYang opened a new issue, #9793: URL: https://github.com/apache/paimon/issues/9793
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation `paimon-spark-4.0` takes the shared Spark test bases from `paimon-spark-ut`'s test-jar, and that jar is compiled against the `spark4` profile baseline, currently Spark 4.1.2. Test bytecode built against 4.1.2 can name Spark symbols that a 4.0.3 runtime does not have, and then the class fails to link when the 4.0 suite loads it. The workaround for that is already in the tree. `paimon-spark/paimon-spark-4.0/src/test/scala/org/apache/spark/sql/streaming/StreamTestCheckAnswerWithTimeoutStub.scala` declares eight empty classes whose compiled names match `StreamTest` inner classes that exist only from Spark 4.1 on, so that test discovery can verify the shared classes on a 4.0 runtime. In the current baseline build, 15 classfiles in `paimon-spark-ut`'s test-jar carry references to those eight names, none of them from anything a Paimon test calls: scalac emits them into the constant pool through the mix-in forwarders of `StreamTest`. Two things are unsatisfying about it. It is per-symbol, so it has to be extended by hand whenever the baseline moves and the shared sources reach something else that 4.0 lacks. And it only holds while no test actually invokes the missing member, which nothing checks. ### Solution Compile the shared test sources a second time, against 4.0.3, and let `paimon-spark-4.0` consume that build: - add a `paimon-spark-ut-4.0` module that owns no sources of its own, adds `../paimon-spark-ut/src/test/{scala,java}` through `build-helper`, pins every Spark artifact to 4.0.3, and produces a test-jar only; - point `paimon-spark-4.0` at that test-jar instead of `paimon-spark-ut`'s; - delete the stub file. A class compiled against 4.0.3 cannot reference a symbol that 4.0.3 does not have, so there is nothing left for the stub to stand in for. `paimon-spark-4.1` needs none of this while the baseline is 4.1.x, and keeps consuming `paimon-spark-ut` directly. ### Anything else? The same mechanism is what #9265 needs for both 4.0 and 4.1 once the baseline moves to 4.2, but the 4.0 half stands on its own and is worth landing separately: it removes an existing workaround and changes how one CI lane compiles its tests, which is easier to review apart from a baseline bump. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
