[
https://issues.apache.org/jira/browse/PIG-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600547#comment-13600547
]
Cheolsoo Park commented on PIG-3241:
------------------------------------
I seem to be able to reproduce ConcurrentModificationException in my 3-node
cluster (both MR1 and MR2). The only difference is that it happens a lot more
frequently in MR2 than in MR1.
Here is the script that I am using with 100M integers:
{code}
a = LOAD '1.txt' AS (i:int);
b = GROUP a ALL;
c = FOREACH b GENERATE TOP(1000, 0, a);
STORE c INTO 'out';
{code}
This doesn't give me exactly the same stack trace, but almost every mapper
fails with ConcurrentModificationException constantly in MR2. I am setting
mapPartAgg properties as follows:
{code}
pig.exec.mapPartAgg=true
pig.exec.mapPartAgg.minReduction=3
{code}
> ConcurrentModificationException in POPartialAgg
> -----------------------------------------------
>
> Key: PIG-3241
> URL: https://issues.apache.org/jira/browse/PIG-3241
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11
> Reporter: Lohit Vijayarenu
> Priority: Blocker
> Fix For: 0.12, 0.11.1
>
>
> While running few PIG scripts against Hadoop 2.0, I see consistently see
> ConcurrentModificationException
> {noformat}
> at java.util.HashMap$HashIterator.remove(HashMap.java:811)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregate(POPartialAgg.java:365)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregateSecondLevel(POPartialAgg.java:379)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.getNext(POPartialAgg.java:203)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:263)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:729)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> {noformat}
> It looks like there is rawInputMap is being modified while elements are
> removed from it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
