Ming, in its current form, Tez Shuffle Handler is not setup to handle backwards compatibility with MapReduce. This could be easily added, the complication is that MapReduce hard-codes its auxiliary shuffle service to be named "mapreduce_shuffle". In addition, there are no compelling benefits for using the Tez Shuffle Handler with mapreduce except not having to have two services running. For a typical setup, yarn should define both mapreduce_shuffle and tez_shuffle auxiliary services and run them concurrently.
jeagles On Tue, May 23, 2017 at 7:28 PM, Ming Ma <[email protected]> wrote: > +1. Is Tez shuffle handler backward compatible with MR jobs? For a test > cluster running Tez and MR jobs, if Tez shuffle handler is backward > compatible, we can configure NM to only load Tez shuffle handler. > Otherwise, will need to configure NM to load both Tez and MR shuffle > handlers. > > On Tue, May 23, 2017 at 9:16 AM, Eric Badger <[email protected] > > > wrote: > > > +1 (non-binding) given the test failure followup jira ([TEZ-3735] Test > > failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA) > > Eric > > > > | > > | | > > [TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap - ASF > > JIRA > > | | > > > > | > > > > > > > > > > On Monday, May 22, 2017 3:46 PM, Jason Lowe > > <[email protected]> wrote: > > > > > > +1, looking forward to this functionality being available in an official > > release! > > Jason > > > > > > On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <[email protected]> > > wrote: > > > > > > This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334) > > into the master branch. Having done extensive testing and thanks to all > the > > feedback given, the Tez Shuffle Handler delivers on its promise to reduce > > shuffle times for the auto-reduce case and pave the way for more shuffle > > types in the future. > > > > Please view a patch of the proposed merge attached to the TEZ-3334 jira > > https://issues.apache.org/jira/browse/TEZ-3334 > > > > As to be the most useful and time efficient, please test and vote early > if > > you have any misgivings or questions about the readiness of the branch so > > as to not to delay the merge process and increase the amount of work on > my > > part to continue to maintain this branch with master. > > > > This vote will be open until Wednesday around the same time. > > > > Here is my +1. > > > > Regards, > > jeagles > > > > > > > > > > > > >
