[ 
https://issues.apache.org/jira/browse/BEAM-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035914#comment-17035914
 ] 

Valentyn Tymofieiev commented on BEAM-9085:
-------------------------------------------

This is a regression in numpy. Downgrading numpy to 1.16.5 produces the same 
results for me on Py 2 / Py 3.

pip install numpy==1.18.1
python -m timeit 'import numpy as np; [np.random.RandomState(i) for i in 
range(10000)]'
> [1 loop, best of 5: 1.22 sec per loop

1.16.5 is much faster:

pip install numpy==1.16.5
python -m timeit 'import numpy as np; [np.random.RandomState(i) for i in 
range(10000)]'
> 1 loop, best of 5: 17.8 msec per loop

Beam uses np.random.RandomState() during generation of Synthetic input[1], 
which causes the slowdown.

Note that 1.16.5 is the latest numpy version with Python 2 support, but on 
Python 3 newer versions are available.

[1] 
https://github.com/apache/beam/blob/b3e06126405105836f9ae15a15d5e1c7189b7f6e/sdks/python/apache_beam/testing/synthetic_pipeline.py#L419


> Investigate performance difference between Python 2/3 on Dataflow
> -----------------------------------------------------------------
>
>                 Key: BEAM-9085
>                 URL: https://issues.apache.org/jira/browse/BEAM-9085
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Kamil Wasilewski
>            Assignee: Valentyn Tymofieiev
>            Priority: Major
>
> Tests show that the performance of core Beam operations in Python 3.x on 
> Dataflow can be a few time slower than in Python 2.7. We should investigate 
> what's the cause of the problem.
> Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A 
> dashboard with runtime results can be found here [2].
> [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py
> [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to