[
https://issues.apache.org/jira/browse/ARROW-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347716#comment-16347716
]
Antoine Pitrou commented on ARROW-2071:
---------------------------------------
Experimenting in https://github.com/apache/arrow/pull/1541 :
* Python 2.7:
{code}
========================== slowest 15 test durations ===========================
285.93s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_serialization.py::test_custom_serialization
44.93s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_serialization.py::test_primitive_serialization
42.07s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_subscribe_deletions
29.15s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_subscribe
12.06s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_serialization.py::test_serialize_to_buffer
12.06s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_serialization.py::test_complex_serialization
8.30s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_use_one_memory_mapped_file
6.74s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_store_full
5.35s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_many_hashes
4.60s setup
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_use_one_memory_mapped_file
4.52s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_create_with_metadata
3.33s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_get
2.80s teardown
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_use_one_memory_mapped_file
2.68s call
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_create_existing
2.55s teardown
pyarrow-test-2.7/lib/python2.7/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_store_full
======= 510 passed, 46 skipped, 3 xfailed, 6 warnings in 509.55 seconds ========
{code}
* Python 3.6:
{code}
========================== slowest 15 test durations ===========================
236.05s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_serialization.py::test_custom_serialization
41.78s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_subscribe_deletions
38.78s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_serialization.py::test_primitive_serialization
28.87s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_subscribe
9.01s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_serialization.py::test_complex_serialization
8.90s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_serialization.py::test_serialize_to_buffer
8.30s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_use_one_memory_mapped_file
6.90s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_store_full
5.32s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_many_hashes
4.88s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_create_with_metadata
4.59s setup
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_use_one_memory_mapped_file
3.34s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_get
2.92s call
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_create_existing
2.82s teardown
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_use_one_memory_mapped_file
2.54s teardown
pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_plasma.py::TestPlasmaClient::test_store_full
======= 510 passed, 46 skipped, 3 xfailed, 8 warnings in 447.68 seconds ========
{code}
I'm quite surprised by the runtimes.
{{pyarrow/tests/test_serialization.py::test_custom_serialization}} only takes
1.7s here.
In the end the total time taken by Python tests was 15 minutes, while the whole
build was 41 minutes long.
> [Python] Reduce runtime of builds in Travis CI
> ----------------------------------------------
>
> Key: ARROW-2071
> URL: https://issues.apache.org/jira/browse/ARROW-2071
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 0.9.0
>
>
> For some reason, recently each Python build has been taking about 15 minutes
> to run. I speculate this is due to VM thrashing caused by reduced resources
> on the Travis CI workers, related to the problem I fixed in ARROW-2062.
> We should experiment, but it seems like perhaps this can be fixed either by:
> * Reducing the size of the Plasma store on Travis CI
> * Disabling valgrind in Plasma tests
> The slowness could be caused by something else, though, so we should
> investigate (and have pytest report slow tests in the logs)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
