jiangzho commented on PR #10: URL: https://github.com/apache/spark-kubernetes-operator/pull/10#issuecomment-2084386256
> It seems that we need a shim layer or multiple module like Iceberg Yes, that's one mid-term goal that we will target for operator v1.0, in order to achieve fully version agnostic. This PR proposes single submission worker based on latest spark-kubernetes - consider it's history, we tested the compatibility with Spark 3.2, 3.3, 3.4, 3.5. We can do the same for 4.0 to ensure no breaking change is introduced. This is the pattern adopted by most operator solutions, like Flink operator / Google Spark operator. I'm not saying this is the absolutely right way to go for longer term, but it could enable the first batch of evaluations on operator 0.1 while we work on the multi-submission worker mode. The challenges of multi-version submission worker mode involves * the operator image can be heavy (packaging multiple Spark jars) * runtime resource consumption can be higher, because we need multiple containers (per Spark version) to avoid jar conflicts in class path. * deployment (helm chart) of operator can be a bit more complex when users are more familiar with operator. i.e., users could choose to deploy operator with single submission worker mode, or a selection of Spark versions, or all known versions based on the need. Given this can we start with this PR for v0.1 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
