[
https://issues.apache.org/jira/browse/BEAM-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samuel Brotherton updated BEAM-11000:
-------------------------------------
Description:
Apologies if this is user error, as I am still learning Beam. I am attempting
to run a python-based pipeline on the Flink runner, with Flink and an external
beam image running inside docker-compose (see "Environment"). Whenever I submit
a job, I get the following error. Am I doing something wrong? Thanks in advance.
* Edit: Jira mangled the traceback, you can see it here:
https://gist.github.com/sbrother/b9f62580fbab84e2cfc925e0ea8ffc07
was:
Apologies if this is user error, as I am still learning Beam. I am attempting
to run a python-based pipeline on the Flink runner, with Flink and an external
beam image running inside docker-compose (see "Environment"). Whenever I submit
a job, I get the following error. Am I doing something wrong? Thanks in advance.
{code:java}
jobmanager_1 | 2020-10-01 20:23:07,622 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
JobGraph submission 0b199c080493017fd532138056d27d57
(BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,622
INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] -
Received JobGraph submission 0b199c080493017fd532138056d27d57
(BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,622
INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] -
Submitting job 0b199c080493017fd532138056d27d57
(BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,625
INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] -
Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at
akka://flink/user/rpc/jobmanager_3 .jobmanager_1 | 2020-10-01 20:23:07,626
INFO org.apache.flink.runtime.jobmaster.JobMaster [] -
Initializing job BeamApp-root-1001202304-4e0d2f62
(0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,627
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using
restart back off time strategy NoRestartBackoffTimeStrategy for
BeamApp-root-1001202304-4e0d2f62
(0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,630
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running
initialization on master for job BeamApp-root-1001202304-4e0d2f62
(0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,654
INFO org.apache.flink.runtime.jobmaster.JobMaster [] -
Successfully ran initialization on master in 24 ms.jobmanager_1 | 2020-10-01
20:23:07,655 ERROR org.apache.flink.runtime.dispatcher.StandaloneDispatcher
[] - Failed to submit job 0b199c080493017fd532138056d27d57.jobmanager_1 |
org.apache.flink.runtime.client.JobExecutionException: Could not instantiate
JobManager.jobmanager_1 | at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:398)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
~[?:1.8.0_265]jobmanager_1 | at
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | Caused by:
java.lang.NullPointerExceptionjobmanager_1 | at
java.util.Collections$UnmodifiableCollection.<init>(Collections.java:1028)
~[?:1.8.0_265]jobmanager_1 | at
java.util.Collections$UnmodifiableList.<init>(Collections.java:1304)
~[?:1.8.0_265]jobmanager_1 | at
java.util.Collections.unmodifiableList(Collections.java:1289)
~[?:1.8.0_265]jobmanager_1 | at
org.apache.flink.runtime.jobgraph.JobVertex.getOperatorCoordinators(JobVertex.java:352)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:232)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:814)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:228)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:269)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:242)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:229)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:119)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:103)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:284)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:272)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:140)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:84)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:388)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | ... 7 morejobmanager_1
| 2020-10-01 20:23:07,656 ERROR
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler [] - Unhandled
exception.jobmanager_1 |
org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
{code}
> NullPointerException in Flink manager when submitting job with
> environment_type=EXTERNAL
> ----------------------------------------------------------------------------------------
>
> Key: BEAM-11000
> URL: https://issues.apache.org/jira/browse/BEAM-11000
> Project: Beam
> Issue Type: Bug
> Components: beam-community
> Affects Versions: 2.24.0
> Environment: jobmanager:
> image: flink:1.11.2-scala_2.11
> ports:
> - "8081:8081"
> command: jobmanager
> environment:
> - |
> FLINK_PROPERTIES=
> jobmanager.rpc.address: jobmanager
> taskmanager:
> image: flink:1.11.2-scala_2.11
> depends_on:
> - jobmanager
> command: taskmanager
> environment:
> - |
> FLINK_PROPERTIES=
> jobmanager.rpc.address: jobmanager
> taskmanager.numberOfTaskSlots: 2
> beamworker:
> image: apache/beam_python3.8_sdk
> ports:
> - "50000:50000"
> command: --worker_pool
> Reporter: Samuel Brotherton
> Assignee: Aizhamal Nurmamat kyzy
> Priority: P2
>
> Apologies if this is user error, as I am still learning Beam. I am attempting
> to run a python-based pipeline on the Flink runner, with Flink and an
> external beam image running inside docker-compose (see "Environment").
> Whenever I submit a job, I get the following error. Am I doing something
> wrong? Thanks in advance.
> * Edit: Jira mangled the traceback, you can see it here:
> https://gist.github.com/sbrother/b9f62580fbab84e2cfc925e0ea8ffc07
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
