> After the patches it seems stable, I'm able to run multiple terasort/pi > jobs and a few scalding jobs without difficulty. Great work, Darin. Glad to see FGS is now stable.
>Noticed with jobs with short map tasks (8-12 secs), I rarely got more > than two containers per node, I'm curious if I'm not consuming resources > fast enough. Yes. Perhaps we need to tune the rate at which Mesos sends out resource offers to frameworks. The default that we observe in Myriad is 5 seconds. However, if your job has many map tasks and Mesos offer is big enough to accommodate several of them, then you should ideally see lot more than 2-3 containers per node. Isn't that happening? How many map tasks does your job have? Thanks, Santosh On Wed, Apr 13, 2016 at 8:34 AM, Darin Johnson <dbjohnson1...@gmail.com> wrote: > I've been running a number of tests on the Fine Grained scaling aspect on > Myriad. Here's a few notes: > > 1. After the patches it seems stable, I'm able to run multiple terasort/pi > jobs and a few scalding jobs without difficulty. > 2. Noticed with jobs with short map tasks (8-12 secs), I rarely got more > than two containers per node, I'm curious if I'm not consuming resources > fast enough. The issue goes away on the reduce side (able to get far > better utilization of offers). The issue can be lessened by increasing > mapred.splits.min.size and mapred.splits.max.size. This may be an issue > for things like Hive. > > Darin >
