Hi, did you observe the problem only under YARN 2.4.0? IIRC this version of YARN has some problems that can also lead to issues with Flink’s HA mode, and I would encourage you to upgrade YARN to 2.5 or higher. On a different note, there have been several improvements that we will release in Flink 1.1.4, not entirely sure if this is a known problem covered by the upcoming bugfix release. I will add Till to the discussion who worked a lot in this direction.
Best, Stefan > Am 07.12.2016 um 09:19 schrieb Michael Pisula <[email protected]>: > > Hi Guys, > > We are having a slight issue using Flink 1.1.3 (we also observed the > problem with 1.0.2) in Yarn 2.4.0. Whenever a TaskManager restarts, YARN > seems to reserve memory during the TaskManager restart, and not free the > memory again. We are using a CapacityScheduler with 2 queues, where the > queue in which our Flink Yarn Session runs has a guaranteed capacity of > 75%. What we are seeing, is that the amount of reserved memory is > exactly the amount of memory available in the queue after the > TaskManager is crashed. > > On our test system, further TaskManager restarts have been able to get > rid of the TaskManager again. When trying to replicate this on our > production system I was not successful, one difference being, that I > killed a TaskManager with no used slots in prod, while on the test > system jobs were restarted. > > Nothing enlightening in the logs, unfortunately. > > Is this something that anyone has experienced so far? > > Cheers, > > Michael > > > -- > Michael Pisula * [email protected] * +49-174-3180084 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > >
