Valentyn Tymofieiev commented on BEAM-9085:

This is a regression in numpy. Downgrading numpy to 1.16.5 produces the same 
results for me on Py 2 / Py 3.

pip install numpy==1.18.1
python -m timeit 'import numpy as np; [np.random.RandomState(i) for i in 
> [1 loop, best of 5: 1.22 sec per loop

1.16.5 is much faster:

pip install numpy==1.16.5
python -m timeit 'import numpy as np; [np.random.RandomState(i) for i in 
> 1 loop, best of 5: 17.8 msec per loop

Beam uses np.random.RandomState() during generation of Synthetic input[1], 
which causes the slowdown.

Note that 1.16.5 is the latest numpy version with Python 2 support, but on 
Python 3 newer versions are available.


> Investigate performance difference between Python 2/3 on Dataflow
> -----------------------------------------------------------------
>                 Key: BEAM-9085
>                 URL: https://issues.apache.org/jira/browse/BEAM-9085
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Kamil Wasilewski
>            Assignee: Valentyn Tymofieiev
>            Priority: Major
> Tests show that the performance of core Beam operations in Python 3.x on 
> Dataflow can be a few time slower than in Python 2.7. We should investigate 
> what's the cause of the problem.
> Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A 
> dashboard with runtime results can be found here [2].
> [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py
> [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536

This message was sent by Atlassian Jira

Reply via email to