[
https://issues.apache.org/jira/browse/BEAM-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Wasilewski updated BEAM-10659:
------------------------------------
Status: Resolved (was: Open)
> ParDo Python streaming load tests timeouts on 200-iterations case
> -----------------------------------------------------------------
>
> Key: BEAM-10659
> URL: https://issues.apache.org/jira/browse/BEAM-10659
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Kasia Kucharczyk
> Priority: P2
>
> Running Python Dataflow load test in streaming option timeouts on Jenkins on
> case 2:
> {code:java}
> 2GB 100 byte records 200 times
> {code}
> It
> [iterates|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/load_tests/pardo_test.py#L147]
> same ParDo step sequentially.
> Jenkins jobs has 2h timeout. Second case usually is
> [cancelled|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_05_00_47-15183151853043328210;mainTab=JOB_METRICS?project=apache-beam-testing]
> after 1h 47 min. The most suspicious metric here is throughput which in
> comparison to other jobs doesn't look steady. Sometimes there are spike after
> 1 hour of non action, or there are several spikes (to 30 000 elements/sec).
> [Python batch
> case|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_06_32_29-2466435392086580014;step=s1;mainTab=JOB_METRICS?project=apache-beam-testing]
> scenario takes ~56 minutes, with steady throughput ~7000 elements/sec for
> almost whole job run.
> In comparison [Java same test
> case|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-03_05_13_48-16554947290254286391;mainTab=JOB_GRAPH?project=apache-beam-testing]
> takes ~6 minutes. Here throughput goes up to ~100 000 elements/sec then
> after processing all elements it decreases.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
