Thanks for the info! It's very helpful.
-chad
On Sun, Aug 11, 2019 at 4:21 AM Zhu Zhu wrote:
> Hi Chad,
>
> We have (Blink) jobs each running with over 10 thousands of TMs.
> In our experience, the main regression caused by large scale TMs is the in
> TM allocation stage in ResourceManager, t
Hi Chad,
We have (Blink) jobs each running with over 10 thousands of TMs.
In our experience, the main regression caused by large scale TMs is the in
TM allocation stage in ResourceManager, that some times it fails to
allocate enough TMs before the allocation timeout.
It does not deteriorate much o
Hi Chad,
In our cases, 1~2k TMs with up to ~10k TM slots are used in one job. In
general, the CPU/memory of Job Manager should be increased with more TMs.
Regards,
Qi
> On Aug 11, 2019, at 2:03 AM, Chad Dombrova wrote:
>
> Hi,
> I'm still on my task management investigation, and I'm curious t
Hi,
I'm still on my task management investigation, and I'm curious to know how
many task managers people are reliably using with Flink. We're currently
using AWS | Thinkbox Deadline, and we're able to easily utilize over 300
workers, and I've heard from other customers who use several thousand, so
