DanielLeens commented on issue #10881: URL: https://github.com/apache/seatunnel/issues/10881#issuecomment-4436398502
Thanks for the report, and especially for including the temporary workaround. That workaround is very helpful here. I checked the current Spark starter packaging path, and this looks much more like a real packaging / compatibility bug than a config-only issue. SeaTunnel currently builds the Spark 3 starter around Scala 2.12.x, and the starter jar itself still embeds `scala/*` runtime classes. At the same time, YARN cluster mode will also run with the Spark distribution's own Scala runtime on the cluster side. That makes the failure mode you reported very plausible: the job picks up conflicting Scala class/signature versions in cluster mode, while client mode or manually removing the embedded `scala/*` classes avoids the collision. That lines up well with your symptom: - cluster mode fails with `expected 5.0 found 5.2`; - client mode works; - deleting the embedded Scala classes from `seatunnel-spark-3-starter.jar` also works. So I think this issue is worth keeping open as a real bug in the Spark starter packaging / classpath boundary. I do not see an existing fix for it in the current `dev` branch yet. The likely fix direction is to make the Spark starter packaging and classpath handling stricter for cluster mode, instead of requiring users to manually strip Scala classes from the jar. Thanks again for the precise reproduction and workaround. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
