Hi, Marzi,
> But only one Task manager spins up and even when I submit a second job which
> requires more resources, The job is stuck in SCHEDULED state and no
> additional TMs get automatically assigned.
Would you like to check if there are enough resources in your yarn
cluster? Also, the JM log will help to figure out the root cause.
> Also I have tried to specify the number of TM when starting the yarn session
> like below, but the session starts with only 1 TM.
The "-n" or "-yn" has been dropped in Flink 1.12. [1] FLINK-15959
might be treated as a replacement but the progress has not been
started yet. As a workaround for your testing purpose, you can submit
a warmup job(e.g. WordCount with required parallelism) and increase
the "slotmanager.taskmanager-timeout" to ensure the TM will not
timeout fast.
[1] http://issues.apache.org/jira/browse/FLINK-15959
Best,
Yangze Guo
On Tue, Jul 6, 2021 at 8:41 AM Marzi K wrote:
>
> Hi All,
>
> I am exploring running Flink 1.12.0 on yarn and so far I have been able to
> start a yarn session and submit a job. But only one Task manager spins up and
> even when I submit a second job which requires more resources, The job is
> stuck in SCHEDULED state and no additional TMs get automatically assigned.
> Any pointers as why this is the case?
> Also I have tried to specify the number of TM when starting the yarn session
> like below, but the session starts with only 1 TM.
> ./bin/yarn-session.sh —queue -n 4 -jm 1024 -tm 4096
>
> The older Flink version had the -yn option in the below command but it’s
> missing from newer versions and I don’t see any replacement for it:
> ./bin/flink run -m yarn-cluster -p10 -yn 5 -yjm 1024 -ytm 4069 -yqu
> examples/batch/wordCount.jar
>
> Would appreciate ant pointers as this is not allowing me to test out HA
> performance on yarn.
>
> Best,
> Marzi
