Thanks Yan! That definitely looks like it could be the culprit. I'll give that a shot.
FYI, I also found an instance of Log in one of my UDFs that wasn't static. >From what I gather, changing that to static will keep serialization from occurring, so I'm giving that a shot as well. On Tue, Sep 7, 2010 at 10:33 AM, Yan Zhou <y...@yahoo-inc.com> wrote: > Most likely you, when build PIG, were using the older version 1.0.3 of > log4j logger. See https://jira.jboss.org/browse/JBAS-1781. The root cause > might be due to https://issues.apache.org/jira/browse/PIG-1582. So the > first thing you probably need to do is upgrade your local copy of PIG trunk. > > Yan > > -----Original Message----- > From: Bill Graham [mailto:billgra...@gmail.com] > Sent: Tuesday, September 07, 2010 10:00 AM > To: pig-user@hadoop.apache.org > Subject: PigServer.executeBatch throwing NotSerializableException > > Hi, > > I've just deployed some new Pig jobs live (Pig version 0.7.0) and I'm > getting the error shown below. Has anyone seen this before? What's strange > is that I have a tier of 4 load-balanced machines that serve as my job > runners, all with identical code and hardware, but 2 of the four will fail > consistently with this error. The other 2 run the same job without problem. > I'm racking my brain to understand why... > > java.io.NotSerializableException: > org.apache.commons.logging.impl.Log4JLogger > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1156) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474) > at > > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) > at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326) > at java.util.HashMap.writeObject(HashMap.java:1000) > at sun.reflect.GeneratedMethodAccessor624.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461) > at > > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474) > at > > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474) > at > > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) > at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326) > at > > org.apache.pig.impl.util.ObjectSerializer.serialize(ObjectSerializer.java:40) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:511) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:246) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:131) > at > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:308) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:844) > at org.apache.pig.PigServer.execute(PigServer.java:837) > at org.apache.pig.PigServer.access$100(PigServer.java:107) > at org.apache.pig.PigServer$Graph.execute(PigServer.java:1089) > at org.apache.pig.PigServer.executeBatch(PigServer.java:290) > > thanks, > Bill >
