[
https://issues.apache.org/jira/browse/BEAM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560432#comment-15560432
]
ASF GitHub Bot commented on BEAM-735:
-------------------------------------
GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/1073
[BEAM-735] PAssertStreaming should make sure the assertion happened.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.txt).
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amitsela/incubator-beam BEAM-735
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1073.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1073
----
commit ca3aa78d9651d8d8fc981234c9f4707414dc9738
Author: Sela <[email protected]>
Date: 2016-10-09T18:38:14Z
PAssertStreaming shuold check an assertion happened.
commit 4adc3827443a6d8490f6b1799fce5c820c9484a5
Author: Sela <[email protected]>
Date: 2016-10-09T18:39:00Z
Test assert for skipped assertion..
commit 948fc9d0e894c06e555cf47fc3db48e6aad55008
Author: Sela <[email protected]>
Date: 2016-10-09T18:39:53Z
This name is more true to the natureof this test.
commit 346e85df4436a07a8dd30d793d7fa0ca4bf23806
Author: Sela <[email protected]>
Date: 2016-10-09T18:40:46Z
Fix according to new PAssertStreaming.
----
> PAssertStreaming should make sure the assertion happened.
> ---------------------------------------------------------
>
> Key: BEAM-735
> URL: https://issues.apache.org/jira/browse/BEAM-735
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Amit Sela
> Assignee: Amit Sela
>
> The Spark runner currently runs PAsserts via `PAssertStreaming` which groups
> into a single key and asserts the values on the worker (part of the "Lambda"
> in the Spark lingo).
> This could be a problem since Spark won't run if there's nothing to process -
> so that if for some reason the input is missed, say reading from Kafka latest
> or simply an empty topic, the assertion will be skipped and so we'll never
> fail (we would like to fail if there was no input, while we expected one).
> This might change once Spark provide a better support for the Beam model in
> streaming, but until then, it's best that our tests will consider this case
> as well.
> I'll add an aggregator and increment for assertion, at the end I'll make sure
> the aggregator is not 0, so that at least one assertion took place (if for
> some reason Spark kept on for a couple of more intervals it might execute the
> same assertion more then once, if the input is repeated).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
