Aviem Zur commented on BEAM-2671:

[~jkff] we've been running Spark runner in streaming mode using 2.0.0 
We've encountered the issue with the watermarks not advancing which was a no-go 
for us, since results were not emitted and eventually apps failed on out of 
memory exceptions. This was the reason for the fix in 

After integrating this fix (on our local fork) we were able to run applications 
Regarding running the current state of master, we are not, so hard to comment 
on if the failing tests cause real applications to fail, this will have to be 
assessed. What we do know is the tests are failing.

Regarding ValidatesRunner tests, while these do pass, IIRC the creation of the 
streaming tests for Spark runner were in fact due to insufficient tests for 
Spark runner's streaming mode. So while they are not perfect and flaky, they do 
provide value. Perhaps [~amitsela] can elaborate on these tests.

> CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark 
> runner
> --------------------------------------------------------------------------------
>                 Key: BEAM-2671
>                 URL: https://issues.apache.org/jira/browse/BEAM-2671
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Etienne Chauchot
>            Assignee: Jean-Baptiste Onofré
>             Fix For: 2.2.0
> Error message:
> Flatten.Iterables/FlattenIterables/FlatMap/ParMultiDo(Anonymous).out0: 
> Expected: iterable over [] in any order
>      but: Not matched: "late"

This message was sent by Atlassian JIRA

Reply via email to