[
https://issues.apache.org/jira/browse/BEAM-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-12094:
--------------------------------
Summary: Support Spark 3 in spark_runner.py (was: Support Spark 3 in
spark_runner.py.)
> Support Spark 3 in spark_runner.py
> ----------------------------------
>
> Key: BEAM-12094
> URL: https://issues.apache.org/jira/browse/BEAM-12094
> Project: Beam
> Issue Type: Sub-task
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: P3
> Labels: portability-spark
> Fix For: 2.32.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> spark_runner.py is the Python wrapper for the Beam Spark runner. It requires
> the Beam Spark job server jar to operate. The default jar is Spark 2.
> To build and use the Spark 3 jar from source:
> # Build it from source with "./gradlew :runners:spark:3:job-server:shadowJar"
> # Use it in Python by passing the pipeline option
> "--spark_job_server_jar=/.../runners/spark/3/job-server/build/libs/beam-runners-spark-3-job-server-2.x.0-SNAPSHOT.jar"
> But this is cumbersome. We should provide an easy way to use the released
> Spark 3 job server jar.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
