Hi Kamil, there is a JIRA for this: https://issues.apache.org/jira/browse/BEAM-8970 It's theoretically possible but remains untested as far as I know :)
As I indicated in a comment, you can set --output_executable_path to create a jar that you can then submit to yarn via spark-submit. If you can get this working, I'd additionally like to script the jar submission in python to save users the extra step. Thanks, Kyle On Tue, Jun 23, 2020 at 9:16 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Hi all, > > I'm trying to run a Beam pipeline using Spark on YARN. My pipeline is > written in Python, so I need to use a portable runner. Does anybody know > how I should configure job server parameters, especially > --spark-master-url? Is there anything else I need to be aware of while > using such setup? > > If it makes a difference, I use Google Dataproc. > > Best, > Kamil >
