This is my first time on a cluster I'll try what Josh suggests now. J
On Fri, Mar 28, 2014 at 3:41 AM, Josh Wills <[email protected]> wrote: > > On Fri, Mar 28, 2014 at 1:22 AM, Gabriel Reid <[email protected]>wrote: > >> Hi Jeremy, >> >> On Thu, Mar 27, 2014 at 3:26 PM, Jeremy Lewi <[email protected]> wrote: >> > Hi >> > >> > I'm hitting the exception pasted below when using AvroPathPerKeyTarget. >> > Interestingly, my code works just fine when I run on a small dataset >> using >> > the LocalJobTracker. However, when I run on a large dataset using a >> hadoop >> > cluster I hit the exception. >> > >> >> Have you ever been able to successfully use the AvroPathPerKeyTarget >> on a real cluster, or is this the first try with it? >> >> I'm wondering if this could be a problem that's always been around (as >> the integration test for AvroPathPerKeyTarget also runs in the local >> jobtracker), or if this could be something new. >> > > +1-- Jeremy, if you force the job to run w/a single reducer on the cluster > (i.e., via groupByKey(1)), does it work? > > >> >> - Gabriel >> > >
