Surbhi Mungre created CRUNCH-494:
------------------------------------

             Summary: Unable to union large number of PCollections 
                 Key: CRUNCH-494
                 URL: https://issues.apache.org/jira/browse/CRUNCH-494
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.8.3
            Reporter: Surbhi Mungre
            Assignee: Josh Wills
            Priority: Minor


If you try to union large number of PCollections(~5K), then Crunch throws 
StackOverflowError exception. 

{noformat}
java.lang.StackOverflowError
        at 
com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:68)
        at 
com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:54)
        at com.google.common.collect.Iterators$12.<init>(Iterators.java:1072)
        at com.google.common.collect.Iterators.forArray(Iterators.java:1072)
        at 
com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:68)
        at 
com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:31)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:291)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
        at 
org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
{noformat}

Here is a simple test which can reproduce the issue. 
https://gist.github.com/anonymous/22f08511604341d0ffda




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to