[
https://issues.apache.org/jira/browse/PIG-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546329
]
Utkarsh Srivastava commented on PIG-31:
---------------------------------------
+1 to Ben's patch.
The point of the finalize method is to collect the temp files left around by
the big bag. It should not be calling content.clear().
If the users decide to call databag.clear() while they have an iterator open on
the bag, its only right that they get a CoMod exception.
Internally we don't call clear except at one place, where also, it is not
strictly necessary.
> concurrent modification error
> -----------------------------
>
> Key: PIG-31
> URL: https://issues.apache.org/jira/browse/PIG-31
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Olga Natkovich
> Priority: Critical
> Attachments: concurrentmod.patch
>
>
> Many users encountered this problem. The stack looks like:
> Error:java.lang.RuntimeException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at com.yahoo.pig.builtin.SUM.sum(SUM.java:58)
> at com.yahoo.pig.builtin.SUM.exec(SUM.java:25)
> at com.yahoo.pig.builtin.SUM.exec(SUM.java:21)
> at com.yahoo.pig.impl.eval.FuncEvalSpec$1.add(FuncEvalSpec.java:101)
> at
> com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.(GenerateSpec.java:140)
> at com.yahoo.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:52)
> at
> com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.add(GenerateSpec.java:217)
> at
> com.yahoo.pig.impl.eval.collector.UnflattenCollector.add(UnflattenCollector.java:42)
> at
> com.yahoo.pig.impl.eval.collector.DataCollector.addToSuccessor(DataCollector.java:79)
> at com.yahoo.pig.impl.eval.SimpleEvalSpec$1.add(SimpleEvalSpec.java:21)
> at
> com.yahoo.pig.impl.eval.GenerateSpec$CrossProductItem.exec(GenerateSpec.java:248)
> at com.yahoo.pig.impl.eval.GenerateSpec$1.add(GenerateSpec.java:61)
> at
> com.yahoo.pig.impl.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:144)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:355)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1707)
> We traced the issue to the same bag being shared between multiple operations
> and clearing it in one place caused problem in another. Will attach Ben's fix
> shortly. All unit and end-to-end tests passed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
