Johannes Herr created PIG-5475:
----------------------------------
Summary: Jar-Compatibility Problem with Hadoop 3.4.2
Key: PIG-5475
URL: https://issues.apache.org/jira/browse/PIG-5475
Project: Pig
Issue Type: Bug
Affects Versions: 0.18.0
Reporter: Johannes Herr
In https://issues.apache.org/jira/browse/PIG-5374 a dependency on the
Apache-Commons-3 class
"org.apache.commons.collections.buffer.CircularFifoBuffer" was introduced for
Pig 0.18. At the time Apache-Commons-3 was part of the classpath of Hadoop.
With https://issues.apache.org/jira/browse/HADOOP-15760 Hadoop has switched to
version 4 of Apache-Commons, which no longer contains the above-mentioned class.
We therefore now get a NoClassDefFoundError for some (not all) of our Pig-jobs.
This is the corresponding stacktrace:
{noformat}
13.11.2025 11:13:47 ERROR o.a.h.m.YarnChild [main] - Error running child :
java.lang.NoClassDefFoundError:
org/apache/commons/collections/buffer/CircularFifoBuffer
at
org.apache.pig.impl.io.InterRecordReader.skipUntilMarkerOrSplitEndOrEOF(InterRecordReader.java:95)
at
org.apache.pig.impl.io.InterRecordReader.nextKeyValue(InterRecordReader.java:179)
at org.apache.pig.impl.io.InterStorage.getNext(InterStorage.java:79)
at
org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoader.java:74)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.LoadFuncDecorator.getNext(LoadFuncDecorator.java:74)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:209)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:569)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.collections.buffer.CircularFifoBuffer
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 17 more
{noformat}
This are the corresponding classpath entries for 'common-collections':
{noformat}
[...]hadoop/share/hadoop/common/lib/commons-collections4-4.4.jar
[...]hadoop/share/hadoop/hdfs/lib/commons-collections4-4.4.jar
{noformat}
(Which explains the problem, since the class is only in version 3 of
commons-collections and no longer in version 4.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
