You can also decompose your problem so that you have code that is testable outside the map-reduce process. This way, you can ensure all your "helper" classes/methods are behaving as you expect, and you can debug this locally. Once that is done, you are left with only debugging things related to MR plumbing. MRUnit is good for this as Brock pointed out. But even if you can't use that for some reason, knowing the helper logic is correct is a big help with complex jobs.
-paul -----Original Message----- From: Brock Noland [mailto:br...@cloudera.com] Sent: Thursday, March 29, 2012 12:08 PM To: mapreduce-user@hadoop.apache.org Subject: Re: Debug MR tasks impossible. Can I suggest unit testing with MRUnit: https://cwiki.apache.org/confluence/display/MRUNIT/Index https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial Brock On Thu, Mar 29, 2012 at 10:33 AM, Pedro Costa <psdc1...@gmail.com> wrote: > Hi, > > I'm trying to debug map and reduce tasks for a quite long time, and it seems > that it's impossible. MR are launched in new process and there's no way to > debug them. Even with IsolationRunner class it's impossible. This isn't good > because I really need to debug the class, to understand some changes that I > made to the code. > > I wonder how MapReduce programmers could debug the code that they > implemented? > > -- > Best regards, > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
