[
https://issues.apache.org/jira/browse/PIG-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5380:
------------------------------
Attachment: pig-5380-v02.patch
{quote}
Looking back on changes, I think I understand that this is a bigger bug than
this jira given all these self-spilling bags are designed on the premise that
no other threads would touch them (and thus lockings are dropped).
{quote}
This turned out to be false. PIG-5390 has the details.
bq. I think same change is required in InternalSortedBag as well as code is
exactly same and it can spill too
Made the same changes to InternalSortedBag.
Also while comparing the two source codes, I think I found one section that
was missing locking while reading from memory. Added.
> SortedDataBag hitting ConcurrentModificationException or producing incorrect
> output in a corner-case
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-5380
> URL: https://issues.apache.org/jira/browse/PIG-5380
> Project: Pig
> Issue Type: Bug
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Major
> Attachments: pig-5380-v01.patch, pig-5380-v02.patch
>
>
> User had a UDF that created large SortedDataBag. This UDF was failing with
> {noformat}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.readFromPriorityQ(SortedDataBag.java:346)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.next(SortedDataBag.java:322)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.hasNext(SortedDataBag.java:235)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
