hi folks,
I'm trying to compare the performance of a Scalding job on Tez vs Hadoop
and understand a bit about the resource usage overheads on Tez. This job
reads around 6TB of input data and sets up a Scalding flow with 13 Hadoop
jobs. (20 Tez vertices)
The runtime of the job on Tez is around 2-3x better than that on Hadoop.
Tez run takes 15 mins (container reuse on) and 25 mins (reuse off). The
Hadoop job on the other hand takes around 48 mins.
Looking at the resource usage Megabyte Millis(total container uptime *
allocatedMemory), it seems like Tez is at-par with Hadoop (0.59% worse).
This seems strange given that Tez is running for 2-3x lesser time. I'm
trying to understand why this is the case and I could use any ideas /
suggestions that folks might have.
Some things I have checked out / confirmed:
- For the purpose of these tests, my container sizes in Hadoop and Tez
are identical (4G).
- Total no of containers:
- Hadoop seems to be spinning up around 46K containers over the 13
jobs.
- Tez spins up 32K tasks (and around 35K containers when reuse=false,
and 5 - 10K containers when reuse=true)
- While looking at the run with reuse=false, on comparing the container
runtime durations with the corresponding tasks's runtime, I noticed that a
decent number of containers were run for around 1s or more longer than the
task as part of it. Is the Tez AM holding on to containers for a duration
of time before starting up tasks on them? This contributes around 5% of the
MB_MILLIS in case of the Tez run.
- Tez seems to be spinning up around 3K extra containers that are very
short lived. This doesn't contribute a lot of overhead but seemed
interesting.
- I tried out a couple of minor speculative execution setting tweaks
(tez.shuffle-vertex-manager.max-src-fraction=0.95, min-src-fraction=0.9)
this made MB_MILLIS a little worse (~10%).
I looked at some of the resource usage related jiras like TEZ-3274 and
TEZ-3535. We're not hitting TEZ-3274. Wasn't sure based on reading TEZ-3535
if that might be a problem here.
Does anyone have suggestions what we could look into / explore? Not sure if
others have run into such scenarios while comparing resource usage numbers
on Hadoop / Tez where the runtimes are much better on Tez but usage isn't
too much better.
Thanks,
--
- Piyush
