Thanks, but still have problem making remote worker on k8s work (important to point out that I had to create shared volume between nodes in order all have access to the same /tmp, since beam runner creates artifact staging files on the machine it is running on, and expects workers to read from it).
 
However, I get this error from executor:
 

INFO AbstractArtifactRetrievalService: GetManifest for /tmp/beam-artifact-staging/job_cca3e889-76d9-4c8a-a942-a64ddbd2dd1f/MANIFEST
INFO AbstractArtifactRetrievalService: GetManifest for /tmp/beam-artifact-staging/job_cca3e889-76d9-4c8a-a942-a64ddbd2dd1f/MANIFEST -> 0 artifacts
INFO ProcessEnvironmentFactory: Still waiting for startup of environment '/opt/spark/beam/sdks/python/container/build/target/launcher/linux_amd64/boot' for worker id 3-1
ERROR Executor: Exception in task 0.1 in stage 0.0 (TID 2)
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Process died with exit code 1
        at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
 
(note that job manifest has no artifacts in it)
 
I can see ports for enpoints (logging, artifact, ...) are open on the worker. Some debugging to boot.go and running it manually shows it doesn't return from "artifact.Materialize" function.
 
Any idea what could be wrong in setup?
 
Sent: Wednesday, November 06, 2019 at 5:45 PM
From: "Kyle Weaver" <kcwea...@google.com>
To: dev <dev@beam.apache.org>
Subject: Re: Command for Beam worker on Spark cluster
Where can I extract these parameters from?
 
 
> Also, how spark executor can find the port that grpc server is running on?
Not sure which grpc server you mean here.
 
On Wed, Nov 6, 2019 at 3:32 PM Matthew K. <softm...@gmx.com> wrote:
Thanks, still I need to pass parameters to the boot executable, such as, worker id, control endpoint, logging endpoint, etc.
 
Where can I extract these parameters from? (In apache_beam Python code, those can be extracted from StartWorker request parameters)
 
Also, how spark executor can find the port that grpc server is running on?
 
Sent: Wednesday, November 06, 2019 at 5:07 PM
From: "Kyle Weaver" <kcwea...@google.com>
To: dev <dev@beam.apache.org>
Subject: Re: Command for Beam worker on Spark cluster
In Docker mode, most everything's taken care of for you, but in process mode you have to do a lot of setup yourself. The command you're looking for is `sdks/python/container/build/target/launcher/linux_amd64/boot`. You will be required to have both that executable (which you can build from source using `./gradlew :sdks:python:container:build`) and a Python installation including Beam and other dependencies on all of your worker machines.
 
On Wed, Nov 6, 2019 at 2:24 PM Matthew K. <softm...@gmx.com> wrote:
Hi all,
 
I am trying to run *Python* beam pipeline on a Spark cluster. Since workers are running on separate nodes, I am using "PROCESS" for "evironment_type" in pipeline options, but I couldn't find any documentation on what "command" I should pass to "environment_config" to run on the worker, so executor can be able to communicate with.
 
Can someone help me on that?

Reply via email to