Hi Dhiren,
Running via the Spark CLI doesn't work. You need to execute your Python
pipeline directly. The Beam job server will then submit to the Spark
cluster.
The Jar can't be found because you are working with the development
version, for which the jars haven't been released on Maven Central. As
Tomo mentioned, the error states that you have to build it manually. The
Beam pipeline will then pick it up.
Cheers,
Max
On 18.12.19 21:10, Tomo Suzuki wrote:
I don't use spark-job server but the error says you need to build the
JAR file by
cd C:\apache_beam; ./gradlew runners:spark:job-server:shadowJar
Did you try that?
On Wed, Dec 18, 2019 at 3:08 PM Dhiren Pachchigar
<[email protected]> wrote:
Hi Team,
I am trying to submit beam job in local spark with below command :-
spark-submit --master spark://192.168.0.106:7077 sample.py --runner=SparkRunner
Getting error :--
RuntimeError:
C:\apache_beam\runners\spark\job-server\build\libs\beam-runners-spark-job-server-2.19.0-SNAPSHOT.jar
not found. Please build the server with
cd C:\apache_beam; ./gradlew runners:spark:job-server:shadowJar
Could you please help me on this.
Regards,
Dhiren
