Hi Guys, I’m struggling while initiating the task manager with flink 1.11.0 in AWS EMR but with older versions it is not. Let me put the full context here.
*When using Flink 1.9.1 and EMR 5.29.0* To create a long running session, we used the below command. *sudo flink-yarn-session -n <Number of TM> -s <Number of slot> -jm <memory> -tm <memory> -d* and followed by below command to run the final job. *flink run -m yarn-cluster -yid <flink sessionId> -yn <Number of TM> -ys <Number of slot> -yjm <memory> -ytm <memory> -c <ClassName> <Jar Path>* and if “n” is 6 then it is used to create 6 task managers to start the job, so whatever “n” is configured the result was that number of TM the job is being started. But Now when we scaled up with the configuration (*i.e. Flink 1.11.0 and EMR 6.1.0*) we are unable to achieve the desired values for TM. Please find the session Ids of new configuration, *sudo flink-yarn-session -Djobmanager.memory.process.size=<Memory in GB> -Dtaskmanager.memory.process.size=<Memory in GB> -n <no of TM> -s <No of slot/core> -d* And the final Job command *flink run -m yarn-cluster -yid <Flink sessionId> -c <ClassName> <Jar Path>* I have tried a lot of combinations, but nothing worked out so far. I request your help in this regard as the plan to have this configuration in *PRODUCTION* soon. Thanks in advance. Regards, -Deep
