[
https://issues.apache.org/jira/browse/CRUNCH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432265#comment-15432265
]
Josh Wills commented on CRUNCH-601:
-----------------------------------
No, I don't think you are-- in that situation, q should not be marked as empty
(or getSize() == 0), since during execution, it would at least emit a record
during the cleanup step, which should run regardless of whether or not the
upstream PCollection contained any data. So I'm still back to wondering why
Math.max(1L, newSize) doesn't do the trick here; I can give it a patch that
does that a try tomorrow to see what fails if no one beats me to it.
> Short PCollections in SparkPipeline get length null.
> ----------------------------------------------------
>
> Key: CRUNCH-601
> URL: https://issues.apache.org/jira/browse/CRUNCH-601
> Project: Crunch
> Issue Type: Bug
> Components: Spark
> Affects Versions: 0.13.0
> Environment: Running in local mode on Mac as well as in a ubuntu
> 14.04 docker container
> Reporter: Mikael Goldmann
> Assignee: Micah Whitacre
> Priority: Minor
> Attachments: CRUNCH-601.patch, CRUNCH-601b.patch, CRUNCH-601c.patch,
> SmallCollectionLengthTest.java
>
>
> I'll attach a file with a test that I would expect to pass but which fails.
> It creates five PCollection<String> of lengths 0, 1, 2, 3, 4 gets the
> lengths, runs the pipeline and prints the lengths. Finally it asserts that
> all lengths are non-null.
> I would expect it to print lengths 0, 1, 2, 3, 4 and pass.
> What it does is print lengths null, null, null, 3, 4 and fail.
> I think the underlying reason is the use of getSize() on an unmaterialized
> object and assuming that when the estimate that getSize() returns is 0, then
> the PCollection is guaranteed to be empty, which is false in some cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
