Can you provide more details about your setup and master logs when this started to happen? Typically, when a framework declines an offer it sets a timeout filter which gives mesos a chance to aggregate free resources on a slave into one big offer.
On Tue, Sep 29, 2015 at 6:31 AM, Eren Güven <[email protected]> wrote: > Hello, > > I had an incident where mesos kept offering marathon insufficient > resources which resulted in lots of forever WAITING tasks in marathon. > Below are some example logs I saw. It kept repeating the same insufficient > offers even though the cluster had a lot of free resources like ~80cpus and > 45G mem. A master restart solved the issue but I was wondering what can > result in this situation or if anyone had any pointers on where to dig > further. > > Thanks! > > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (160.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (160.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 62.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (160.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (160.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 6.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (160.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT > SATISFIED (128.0 > 42.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (160.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 120.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (160.0 > 34.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0), > mem NOT SATISFIED (128.0 > 34.0) > > Cheers > Eren >
