[
https://issues.apache.org/jira/browse/BEAM-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881896#comment-16881896
]
Juta Staes commented on BEAM-7674:
----------------------------------
On python 2 there are three tests that are explicitly run as streaming job
{code:java}
"apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
"apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
"apache_beam.io.gcp.bigquery_test:BigQueryStreamingInsertTransformIntegrationTests.test_multiple_destinations_transform_streaming",
{code}
The first two use set the pipeline option for streaming themselves, as
[~angoenka] pointed out.
The last one does not do this, however in the options of this test is says to
use the streaming inserts method from bigquery:
[https://github.com/apache/beam/blob/fbd1f4cf7118c7b2fb4e3a4cf46646e98f3e3b8d/sdks/python/apache_beam/io/gcp/bigquery_test.py#L689]
My thoughts are that the distinction between batch and streaming is not
necessary in the gradle setup (as the tests define it themselves)
I think that if we use the setup as defined in the python 3 gradle files for
the python 2 setup, no behavior will change except for the bigquery streaming
test.
[~pabloem] Do you think that the
test_multiple_destinations_transform_streaming needs to be a streaming test or
is it fine just to use the streaming inserts method?
What are your thoughts on this?
> Define streaming ITs tests for direct runner in consistent way in Python 2
> and Python 3 suites.
> ------------------------------------------------------------------------------------------------
>
> Key: BEAM-7674
> URL: https://issues.apache.org/jira/browse/BEAM-7674
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Reporter: Valentyn Tymofieiev
> Assignee: Juta Staes
> Priority: Major
>
> Currently in Python 2 direct runner test suite some tests run in streaming
> mode:
> https://github.com/apache/beam/blob/fbd1f4cf7118c7b2fb4e3a4cf46646e98f3e3b8d/sdks/python/build.gradle#L130
> However in Python 3, we run both Batch and Streaming direct runner tests in
> Batch mode:
> https://github.com/apache/beam/blob/fbd1f4cf7118c7b2fb4e3a4cf46646e98f3e3b8d/sdks/python/test-suites/direct/py35/build.gradle#L32
> We should check whether we need to explicitly separate the tests into batch
> and streaming and define all directrunner suites consistently.
> cc: [~Juta]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
