>>> Noticed with jobs with short map tasks (8-12 secs), I rarely got morethan
>>>two containers per node, I'm curious if I'm not consuming resources
fast enough
Yes, we have noticed that too. I think because heartbeat between Mesos and RM
is 5 sec (I think) we may not get enough resources with short tasks. It is
worth investigation though.
Thanks,Yuliya
From: Darin Johnson <dbjohnson1...@gmail.com>
To: Dev <dev@myriad.incubator.apache.org>
Sent: Wednesday, April 13, 2016 8:34 AM
Subject: Observations on Fine Grained Scaling
I've been running a number of tests on the Fine Grained scaling aspect on
Myriad. Here's a few notes:
1. After the patches it seems stable, I'm able to run multiple terasort/pi
jobs and a few scalding jobs without difficulty.
2. Noticed with jobs with short map tasks (8-12 secs), I rarely got more
than two containers per node, I'm curious if I'm not consuming resources
fast enough. The issue goes away on the reduce side (able to get far
better utilization of offers). The issue can be lessened by increasing
mapred.splits.min.size and mapred.splits.max.size. This may be an issue
for things like Hive.
Darin
