Hi Lewis, I think I narrowed the problem to SelectorEntryComparator class nested in GeneratorJob. In debugger during crash I noticed there a single instance of SelectorEntryComparator shared across multiple reducer tasks. The class is inherited from org.apache.hadoop.io.WritableComparator that has a few members unprotected for concurrent usage. At some point multiple threads may access those members in WritableComparator.compare call. I modified SelectorEntryComparator and it seems solved the problem but I am not sure if the change is appropriate and/or sufficient (covers GENERATE only?)
Original code: ============================ public static class SelectorEntryComparator extends WritableComparator { public SelectorEntryComparator() { super(SelectorEntry.class, true); } } Modified code: ============================ public static class SelectorEntryComparator extends WritableComparator { public SelectorEntryComparator() { super(SelectorEntry.class, true); } @Override synchronized public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { return super.compare(b1, s1, l1, b2, s2, l2); } } Regards, Vyacheslav Pascarel -----Original Message----- From: lewis john mcgibbney [mailto:lewi...@apache.org] Sent: Wednesday, June 21, 2017 1:41 PM To: user@nutch.apache.org Subject: [EXTERNAL] - Re: ERROR: Cannot run job worker! Hi Vyacheslav, Which version of Nutch are you using? 2.x? lewis On Wed, Jun 21, 2017 at 10:32 AM, <user-digest-h...@nutch.apache.org> wrote: > > > From: Vyacheslav Pascarel <vpasc...@opentext.com> > To: "user@nutch.apache.org" <user@nutch.apache.org> > Cc: > Bcc: > Date: Wed, 21 Jun 2017 17:32:15 +0000 > Subject: ERROR: Cannot run job worker! > Hello, > > I am writing an application that performs web site crawling using > Nutch REST services. The application: > > >