[
https://issues.apache.org/jira/browse/BEAM-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035914#comment-17035914
]
Valentyn Tymofieiev commented on BEAM-9085:
-------------------------------------------
This is a regression in numpy. Downgrading numpy to 1.16.5 produces the same
results for me on Py 2 / Py 3.
pip install numpy==1.18.1
python -m timeit 'import numpy as np; [np.random.RandomState(i) for i in
range(10000)]'
> [1 loop, best of 5: 1.22 sec per loop
1.16.5 is much faster:
pip install numpy==1.16.5
python -m timeit 'import numpy as np; [np.random.RandomState(i) for i in
range(10000)]'
> 1 loop, best of 5: 17.8 msec per loop
Beam uses np.random.RandomState() during generation of Synthetic input[1],
which causes the slowdown.
Note that 1.16.5 is the latest numpy version with Python 2 support, but on
Python 3 newer versions are available.
[1]
https://github.com/apache/beam/blob/b3e06126405105836f9ae15a15d5e1c7189b7f6e/sdks/python/apache_beam/testing/synthetic_pipeline.py#L419
> Investigate performance difference between Python 2/3 on Dataflow
> -----------------------------------------------------------------
>
> Key: BEAM-9085
> URL: https://issues.apache.org/jira/browse/BEAM-9085
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Kamil Wasilewski
> Assignee: Valentyn Tymofieiev
> Priority: Major
>
> Tests show that the performance of core Beam operations in Python 3.x on
> Dataflow can be a few time slower than in Python 2.7. We should investigate
> what's the cause of the problem.
> Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A
> dashboard with runtime results can be found here [2].
> [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py
> [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
