[
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Sela resolved BEAM-769.
----------------------------
Resolution: Fixed
Fix Version/s: 0.4.0-incubating
This is generally resolved.
Flakes in streaming tests could still occur, but providing a truly robust
solution is a part of implementing triggers.
> Spark streaming tests fail on "nothing processed" if runtime env. is slow
> because timeout is hit before processing is done.
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Affects Versions: Not applicable
> Reporter: Daniel Halperin
> Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1586/
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1587/
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1588/
> {code}
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> The above tests use a hard-timeout (ungraceful stop) so if the runtime env.
> is slow enough so that the batch is not done, it'll stop anyway and assert
> and rightfully fail.
> It's difficult to create locally because I never had trouble on my laptop.
> Since Jenkins will be slow from time to time, it is reasonable enough to have
> a more robust solution here :
> # don't use checkpoint (Spark) if not necessary - only really necessary for
> one test in {{KafkaStreamingTest}} and {{ResumeFromCheckpointStreamingTest}}
> I think.
> # allow for graceful stop - will take longer for each test, but should allow
> the test to finish even if runtime env. is slow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
