Hi Buvana, You can submit a Beam Pipeline to Spark on k8s like any other Spark Pipeline using the spark-submit script.
Create an Uber Jar of your Beam code and provide it as the primary resource to spark-submit. Provide the k8s master and the container image to use as arguments to spark-submit. Refer https://spark.apache.org/docs/latest/running-on-kubernetes.html to know more about how to run Spark on k8s. The Beam pipeline will be translated to a Spark Pipeline using Spark APIs in Runtime. Regards, Rahul On Sat, Apr 11, 2020 at 4:38 AM Ramanan, Buvana (Nokia - US/Murray Hill) < [email protected]> wrote: > Hello, > > > > I newly joined this group and I went through the archive to see if any > discussion exists on submitting Beam pipelines to a SparkRunner on k8s. > > > > I run my Spark jobs on a k8s cluster in the cluster mode. Would like to > deploy my beam pipeline on a SparkRunner with k8s underneath. > > > > The Beam documentation: > > https://beam.apache.org/documentation/runners/spark/ > > does not discuss about k8s (though there is mention of Mesos and YARN). > > > > Can someone please point me to relevant material in this regard? Or, > provide the steps for running my beam pipeline in this configuration? > > > > Thank you, > > Regards, > > Buvana >
