[
https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17144810#comment-17144810
]
Kamil Wasilewski commented on BEAM-8970:
----------------------------------------
Yes. I have the logging level increased in my code
"logging.basicConfig(level=logging.INFO)" and I can see job server logs without
starting it separately.
Here's the full output:
{code}
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.23.0.dev.
If the image is not available at local, we will try to pull from hub.docker.com
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.23.0.dev.
If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f6200843050> ====================
INFO:apache_beam.utils.subprocess_server:Using pre-built snapshot at
/home/kamil/projects/beam/runners/spark/job-server/build/libs/beam-runners-spark-job-server-2.23.0-SNAPSHOT.jar
INFO:apache_beam.utils.subprocess_server:Starting service with ['java' '-jar'
'/home/kamil/projects/beam/runners/spark/job-server/build/libs/beam-runners-spark-job-server-2.23.0-SNAPSHOT.jar'
'--spark-master-url' 'local[4]' '--artifacts-dir'
'/tmp/beam-tempzz1pqj9z/artifactsnzn6lhtl' '--job-port' '56209'
'--artifact-port' '0' '--expansion-port' '0']
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:12 INFO
org.apache.beam.runners.jobsubmission.JobServerDriver: ArtifactStagingService
started on localhost:42793'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:13 INFO
org.apache.beam.runners.jobsubmission.JobServerDriver: Java ExpansionService
started on localhost:35223'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:13 INFO
org.apache.beam.runners.jobsubmission.JobServerDriver: JobService started on
localhost:56209'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:13 INFO
org.apache.beam.runners.jobsubmission.JobServerDriver: Job server now running,
terminate with Ctrl+C'
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
['--iterations=10', '--number_of_counters=0',
'--number_of_counter_operations=0']
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:14 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Staging
artifacts for
{"sessionId":"job_6a782d43-25ff-4b65-80ca-e8714daa37c1","basePath":"/tmp/beam-tempzz1pqj9z/artifactsnzn6lhtl"}.'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:14 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Resolving
artifacts for
{"sessionId":"job_6a782d43-25ff-4b65-80ca-e8714daa37c1","basePath":"/tmp/beam-tempzz1pqj9z/artifactsnzn6lhtl"}.ref_Environment_default_environment_1.'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:14 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Getting 0
artifacts for
{"sessionId":"job_6a782d43-25ff-4b65-80ca-e8714daa37c1","basePath":"/tmp/beam-tempzz1pqj9z/artifactsnzn6lhtl"}.null.'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:14 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Artifacts
fully staged for
{"sessionId":"job_6a782d43-25ff-4b65-80ca-e8714daa37c1","basePath":"/tmp/beam-tempzz1pqj9z/artifactsnzn6lhtl"}.'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:15 INFO
org.apache.beam.runners.spark.SparkJobInvoker: Invoking job
BeamApp-kamil-0625095415-64173733_9e09a9b5-00e0-4df8-bb67-bd09e2e5b47d'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:15 INFO
org.apache.beam.runners.jobsubmission.JobInvocation: Starting job invocation
BeamApp-kamil-0625095415-64173733_9e09a9b5-00e0-4df8-bb67-bd09e2e5b47d'
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:15 INFO
org.apache.beam.runners.jobsubmission.PortablePipelineJarCreator: Creating jar
/home/kamil/pipeline.jar for job BeamApp-kamil-0625095415-64173733'
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
STOPPED
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
STARTING
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
RUNNING
INFO:apache_beam.utils.subprocess_server:b'20/06/25 11:54:15 ERROR
org.apache.beam.runners.jobsubmission.JobInvocation: Error during job
invocation
BeamApp-kamil-0625095415-64173733_9e09a9b5-00e0-4df8-bb67-bd09e2e5b47d.'
INFO:apache_beam.utils.subprocess_server:b'java.lang.IllegalArgumentException:
Expected exactly one jar on sun.misc.Launcher$AppClassLoader@70dea4e'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.runners.jobsubmission.PortablePipelineJarCreator.writeClassPathResources(PortablePipelineJarCreator.java:141)'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.runners.jobsubmission.PortablePipelineJarCreator.run(PortablePipelineJarCreator.java:99)'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:83)'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)'
INFO:apache_beam.utils.subprocess_server:b'\tat
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)'
INFO:apache_beam.utils.subprocess_server:b'\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)'
INFO:apache_beam.utils.subprocess_server:b'\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)'
INFO:apache_beam.utils.subprocess_server:b'\tat
java.lang.Thread.run(Thread.java:748)'
ERROR:root:java.lang.IllegalArgumentException: Expected exactly one jar on
sun.misc.Launcher$AppClassLoader@70dea4e
INFO:apache_beam.runners.portability.portable_runner:Job state changed to FAILED
Traceback (most recent call last):
File "/home/kamil/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 193, in
_run_module_as_main
"__main__", mod_spec)
File "/home/kamil/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 85, in
_run_code
exec(code, run_globals)
File
"/home/kamil/projects/beam/sdks/python/apache_beam/testing/load_tests/pardo_test.py",
line 221, in <module>
ParDoTest().run()
File
"/home/kamil/projects/beam/sdks/python/apache_beam/testing/load_tests/load_test.py",
line 152, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File
"/home/kamil/projects/beam/sdks/python/apache_beam/runners/portability/portable_runner.py",
line 546, in wait_until_finish
raise self._runtime_exception
RuntimeError: Pipeline
BeamApp-kamil-0625095415-64173733_9e09a9b5-00e0-4df8-bb67-bd09e2e5b47d failed
in state FAILED: java.lang.IllegalArgumentException: Expected exactly one jar
on sun.misc.Launcher$AppClassLoader@70dea4e
{code}
> Spark portable runner supports Yarn
> -----------------------------------
>
> Key: BEAM-8970
> URL: https://issues.apache.org/jira/browse/BEAM-8970
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Kyle Weaver
> Priority: P2
> Labels: portability-spark
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
