setting tez.task.scale.memory.enabled=false; failed the job sooner. But the fix from TEZ-2575 worked!
Upgraded from 0.7.0 to 0.7.1 and applied https://github.com/apache/tez/commit/0e155e7185d1350f64dead488103777295ac76d1 Goes through without any fatal issues. Will continue testing / benchmarking further. thanks! -Gautam. On Wed, Feb 10, 2016 at 8:12 PM, Gopal Vijayaraghavan <gop...@apache.org> wrote: > > > Good to know there's a fix .. Is there a jira that talks about this > >issue? Coz I couldn't find one. > > https://github.com/apache/tez/commit/714461f47e6408ec331acd0ddd640335e6a7a0 > 6c > > > Also, it looks like Reducer 16 is the one failing - not Reducer 17. > > You can draw out the explain using https://github.com/t3rmin4t0r/lipwig > > PTF doesn't actually tell the UDAF name in the explain, so I'm guessing it > a ROW_NUMBER() <= 50 - because that's the only one which didn't get > optimized. > > I see absolutely no broadcast edges in this, so it's possible to disable > the weighted memory scaler in Tez to sort of dumb it down to MRv2 mode. > > set tez.task.scale.memory.enabled=false; > > *or* do extensive tuning for it (see > tez.task.scale.memory.additionalreservation.fraction.max). > > Cheers, > Gopal > > -- "If you really want something in this life, you have to work for it. Now, quiet! They're about to announce the lottery numbers..."
