Samuel Brotherton created BEAM-11000:
----------------------------------------
Summary: NullPointerException in Flink manager when submitting job
with environment_type=EXTERNAL
Key: BEAM-11000
URL: https://issues.apache.org/jira/browse/BEAM-11000
Project: Beam
Issue Type: Bug
Components: beam-community
Affects Versions: 2.24.0
Environment: jobmanager:
image: flink:1.11.2-scala_2.11
ports:
- "8081:8081"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.11.2-scala_2.11
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 2
beamworker:
image: apache/beam_python3.8_sdk
ports:
- "50000:50000"
command: --worker_pool
Reporter: Samuel Brotherton
Assignee: Aizhamal Nurmamat kyzy
Apologies if this is user error, as I am still learning Beam. I am attempting
to run a python-based pipeline on the Flink runner, with Flink and an external
beam image running inside docker-compose (see "Environment"). Whenever I submit
a job, I get the following error. Am I doing something wrong? Thanks in advance.
{code:java}
jobmanager_1 | 2020-10-01 20:23:07,622 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
JobGraph submission 0b199c080493017fd532138056d27d57
(BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,622
INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] -
Received JobGraph submission 0b199c080493017fd532138056d27d57
(BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,622
INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] -
Submitting job 0b199c080493017fd532138056d27d57
(BeamApp-root-1001202304-4e0d2f62).jobmanager_1 | 2020-10-01 20:23:07,625
INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] -
Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at
akka://flink/user/rpc/jobmanager_3 .jobmanager_1 | 2020-10-01 20:23:07,626
INFO org.apache.flink.runtime.jobmaster.JobMaster [] -
Initializing job BeamApp-root-1001202304-4e0d2f62
(0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,627
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using
restart back off time strategy NoRestartBackoffTimeStrategy for
BeamApp-root-1001202304-4e0d2f62
(0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,630
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running
initialization on master for job BeamApp-root-1001202304-4e0d2f62
(0b199c080493017fd532138056d27d57).jobmanager_1 | 2020-10-01 20:23:07,654
INFO org.apache.flink.runtime.jobmaster.JobMaster [] -
Successfully ran initialization on master in 24 ms.jobmanager_1 | 2020-10-01
20:23:07,655 ERROR org.apache.flink.runtime.dispatcher.StandaloneDispatcher
[] - Failed to submit job 0b199c080493017fd532138056d27d57.jobmanager_1 |
org.apache.flink.runtime.client.JobExecutionException: Could not instantiate
JobManager.jobmanager_1 | at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:398)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
~[?:1.8.0_265]jobmanager_1 | at
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | Caused by:
java.lang.NullPointerExceptionjobmanager_1 | at
java.util.Collections$UnmodifiableCollection.<init>(Collections.java:1028)
~[?:1.8.0_265]jobmanager_1 | at
java.util.Collections$UnmodifiableList.<init>(Collections.java:1304)
~[?:1.8.0_265]jobmanager_1 | at
java.util.Collections.unmodifiableList(Collections.java:1289)
~[?:1.8.0_265]jobmanager_1 | at
org.apache.flink.runtime.jobgraph.JobVertex.getOperatorCoordinators(JobVertex.java:352)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:232)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:814)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:228)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:269)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:242)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:229)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:119)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:103)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:284)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:272)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:140)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:84)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:388)
~[flink-dist_2.11-1.11.2.jar:1.11.2]jobmanager_1 | ... 7 morejobmanager_1
| 2020-10-01 20:23:07,656 ERROR
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler [] - Unhandled
exception.jobmanager_1 |
org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
