AFAIK there is no theoretical limit on the size of the plan, it just
depends on the available resources.
The job submissions times out since it takes too long to deploy all the
operators that the job defines. With 300 flows, each with 6 operators
you're looking at potentially (1800 * parallelism) tasks that have to be
deployed. For each task Flink copies the user-code of /all/ flows to the
executing TaskManager, which the network may just not be handle in time.
I suggest to split your job into smaller batches or even run each of
them independently.
On 31.10.2017 16:25, Chan, Regina wrote:
Asking an additional question, what is the largest plan that the
JobManager can handle? Is there a limit? My flows don’t need to run in
parallel and can run independently. I wanted them to run in one single
job because it’s part of one logical commit on my side.
Thanks,
Regina
*From:*Chan, Regina [Tech]
*Sent:* Monday, October 30, 2017 3:22 PM
*To:* 'user@flink.apache.org'
*Subject:* Job Manager Configuration
Flink Users,
I have about 300 parallel flows in one job each with 2 inputs, 3
operators, and 1 sink which makes for a large job. I keep getting the
below timeout exception but I’ve already set it to a 30 minute time
out with a 6GB heap on the JobManager? Is there a heuristic to better
configure the job manager?
Caused by:
org.apache.flink.runtime.client.JobClientActorSubmissionTimeoutException:
Job submission to the JobManager timed out. You may increase
'akka.client.timeout' in case the JobManager needs more time to
configure and confirm the job submission.
*Regina Chan*
*Goldman Sachs**–*Enterprise Platforms, Data Architecture
*30 Hudson Street, 37th floor | Jersey City, NY 07302*((212) 902-5697**
