[
https://issues.apache.org/jira/browse/CRUNCH-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335177#comment-14335177
]
Josh Wills commented on CRUNCH-494:
-----------------------------------
Second patch pushed to master; I hope that fixes the issue up to at least 100k
PCollections. ;-)
> Unable to union large number of PCollections
> ---------------------------------------------
>
> Key: CRUNCH-494
> URL: https://issues.apache.org/jira/browse/CRUNCH-494
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.3
> Reporter: Surbhi Mungre
> Assignee: Josh Wills
> Priority: Minor
> Fix For: 0.12.0
>
> Attachments: CRUNCH-494.patch, CRUNCH-494b.patch
>
>
> If you try to union large number of PCollections(~5K), then Crunch throws
> StackOverflowError exception.
> {noformat}
> java.lang.StackOverflowError
> at
> com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:68)
> at
> com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:54)
> at com.google.common.collect.Iterators$12.<init>(Iterators.java:1072)
> at com.google.common.collect.Iterators.forArray(Iterators.java:1072)
> at
> com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:68)
> at
> com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:31)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:291)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
> {noformat}
> Here is a simple test which can reproduce the issue.
> https://gist.github.com/anonymous/22f08511604341d0ffda
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
