mosche commented on issue #23932: URL: https://github.com/apache/beam/issues/23932#issuecomment-1332228838
Thanks for answering @cozos 👍 > We have our spark cluster is in a kubernates based environment, how should we deploy the sdk service? The general idea is to deploy the SDK harness as sidecar to each Spark worker node. Have a look at `sidecars` in the [8s sparkoperator](https://googlecloudplatform.github.io/spark-on-k8s-operator/docs/api-docs.html). > Is there a difference between how sdk service should be deployed while running on Flink vs Spark cluster? It's more or less the same. Have a look at [Spotify's guide](https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/beam_guide.md) how to run Beam with the Flink operator and [SDK harness sidecars](https://github.com/spotify/flink-on-k8s-operator/blob/master/examples/beam/without_job_server/beam_flink_cluster.yaml#L11-L14). That should also help to better understand how to use Spark with sidecars. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
