[
https://issues.apache.org/jira/browse/BEAM-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892900#comment-16892900
]
Ryan Skraba commented on BEAM-6883:
-----------------------------------
More bad news -- it looks like the StreamingSourceMetricsTests is not testing
an UnboundedSource :(
By adding a breakpoint, you can see that it's producing a very-long-lived
PCollection, but it is IsBounded.BOUNDED -- if we were let it run, it's a batch
pipeline in the end. Watermarks are never advanced or calculated, and there's
nothing to indicate that the test should stop until the TestPipelineOptions
timeout occurs.
A couple of options:
1) Use GenerateSequence in a truly unbounded mode and test that the PCollection
is UNBOUNDED by removing the `to()` and `withMaxReadTime()` configuration. Add
a time function to make sure that the watermark advances to the end after 1000
elements. Due to the nature of GenerateSequence, you might see more than 1000
elements (we removed the `to()`) but probably not exactly 1000 -- the assertion
could be fixed, I suppose.
2) Use CreateStream or implement TestStream in SparkRunner, and have them
generate Read metrics.
Both of those would finish in under 10 seconds and test the intended
functionality -- as it is, this test isn't doing anything useful.
> StreamingSourceMetricsTest takes too long to finish
> ---------------------------------------------------
>
> Key: BEAM-6883
> URL: https://issues.apache.org/jira/browse/BEAM-6883
> Project: Beam
> Issue Type: Test
> Components: runner-spark
> Affects Versions: 2.11.0
> Reporter: Ismaël Mejía
> Assignee: Alexey Romanenko
> Priority: Minor
>
> This test is part of Spark's ValidatesRunner suite and it takes more than 10
> minutes to end.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
