sankalpsthakur opened a new pull request, #44: URL: https://github.com/apache/pig/pull/44
Fixes [PIG-5475](https://issues.apache.org/jira/browse/PIG-5475). ## Problem `InterRecordReader` (used by `InterStorage` / `RandomSampleLoader`) calls `org.apache.commons.collections.buffer.CircularFifoBuffer` from commons-collections 3. That class was available because Hadoop shipped collections 3. Hadoop 3.4.2 ([HADOOP-15760](https://issues.apache.org/jira/browse/HADOOP-15760)) ships only commons-collections4. Map tasks then fail with: ``` java.lang.NoClassDefFoundError: org/apache/commons/collections/buffer/CircularFifoBuffer at org.apache.pig.impl.io.InterRecordReader.skipUntilMarkerOrSplitEndOrEOF(InterRecordReader.java:95) ``` Switching the call to collections4 `CircularFifoQueue` would break Hadoop 2 clusters that only provide collections 3. Pig's `pig-withouthadoop` jar does not bundle either jar. ## Fix Replace the bounded FIFO with an equivalent in-line byte sliding window. Behavior matches the old queue (oldest-to-newest compare, including signed bytes > 127). No extra runtime dependency on either collections 3 or 4. ## Tests - New `TestInterRecordReader`: marker at start, after a prefix, overlapping prefix, signed bytes, missing marker, split-end stop. - Added to `test/commit-tests`. - Existing `TestBinInterSedes` (including `testPrefixSyncMarkers`) still covers the scanner. ## ICLA Apache requires an ICLA before a contribution can be merged. This PR does not claim that an ICLA has been signed. Contributor: Sankalp Thakur ([sankalpsthakur](https://github.com/sankalpsthakur)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
