Eugene Kirpichov created BEAM-1712:
--------------------------------------
Summary: TestPipeline.run doesn't actually waitUntilFinish
Key: BEAM-1712
URL: https://issues.apache.org/jira/browse/BEAM-1712
Project: Beam
Issue Type: Bug
Components: runner-flink, sdk-java-core, testing
Reporter: Eugene Kirpichov
Assignee: Stas Levin
Priority: Blocker
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java#L124
it calls waitUntilFinish() only if
1) run wasn't called
2) enableAutoRunIfMissing is true.
However in practice both of these are false.
1) run() is, in most tests, called. So effectively if you call .run() at all,
then this thing doesn't call waitUntilFinish().
2) enableAutoRunIfMissing() is set to true only via
TestPipeline.enableAutoRunIfMissing(), which is called only from its own unit
test.
This means that, for all tests that use TestPipeline - if the test waits until
finish, it's only because of the grace of the particular runner. Which is like
really bad.
We're lucky because in practice TestDataflowRunner, TestApexRunner,
TestSparkRunner in run() call themselves waitUntilFinish().
However, TestFlinkRunner doesn't - i.e. there currently might be tests that
actually fail in Flink runner, undetected.
The proper fix to this is to fix TestPipeline to always waitUntilFinish().
Currently testing a quick-fix in https://github.com/apache/beam/pull/2240 to
make sure Flink is safe.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
