[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595461#comment-15595461 ]
Luke Cwik commented on BEAM-769: -------------------------------- I would prefer a bigger timeout over having flaky tests if we couldn't make it deterministic in some way. If a test never flakes, people won't have to look at it. > Spark streaming tests fail on "nothing processed" if runtime env. is slow > because timeout is hit before processing is done. > --------------------------------------------------------------------------------------------------------------------------- > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark > Affects Versions: Not applicable > Reporter: Daniel Halperin > Assignee: Amit Sela > > https://builds.apache.org/job/beam_PostCommit_MavenVerify/1586/ > https://builds.apache.org/job/beam_PostCommit_MavenVerify/1587/ > https://builds.apache.org/job/beam_PostCommit_MavenVerify/1588/ > {code} > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > The above tests use a hard-timeout (ungraceful stop) so if the runtime env. > is slow enough so that the batch is not done, it'll stop anyway and assert > and rightfully fail. > It's difficult to create locally because I never had trouble on my laptop. > Since Jenkins will be slow from time to time, it is reasonable enough to have > a more robust solution here : > # don't use checkpoint (Spark) if not necessary - only really necessary for > one test in {{KafkaStreamingTest}} and {{ResumeFromCheckpointStreamingTest}} > I think. > # allow for graceful stop - will take longer for each test, but should allow > the test to finish even if runtime env. is slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)