[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Thanks to Kumar for contributing Windows compiler flags side of this (the point 
of this issue).

--
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[STINNER Victor]

STINNER Victor  added the comment:

Duration of the "Tests" step of https://github.com/python/cpython/pull/30890 

* GHA win32: 14 min 11 sec (test_peg_generator: 8 min 16 sec)
* GHA win64: 21 min 2 sec (test_peg_generator: 16 min 38 sec)
* Azure win32: 9 min 34 sec (test_peg_generator: 5 min 30 sec)
* Azure win64: 13 min 56 sec (test_peg_generator: 8 min 3 sec)

Duration of the "Tests" step of https://github.com/python/cpython/pull/31096 
(which includes Gregory's change):

* GHA win32: 8 min 30 sec (test_peg_generator: 5 min 1 sec)
* GHA win64: 8 min 29 sec (test_peg_generator: 4 min 52 sec)
* Azure win32: 9 min 54 sec (test_peg_generator: 4 min 19 sec)
* Azure win64: 7 min 57 sec (test_peg_generator: 3 min 32 sec)

test_peg_generator is way faster, and the total tests duration is shorter 
especially the maximum time: 21 min (before) => 10 min (after).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:

test_peg_generator is significantly less of the long tail on optimized builds 
now.  CI extremely noisy performance wise (times vary by 2-3x without any 
differences anways) so I can't easily judge if this made a notable difference 
in windows CI latency.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[miss-islington]

miss-islington  added the comment:


New changeset e8258608c28c65680253d0ca6167430e34c2fd87 by Miss Islington (bot) 
in branch '3.9':
[3.9] [3.10] bpo-46576: bpo-46524: Disable compiler optimization within 
test_peg_generator. (GH-31015) (GH-31089) (GH-31093)
https://github.com/python/cpython/commit/e8258608c28c65680253d0ca6167430e34c2fd87


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 10.0 -> 11.0
pull_requests: +29279
pull_request: https://github.com/python/cpython/pull/31093

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset f5ebec4d3e1199ec38b88920cfde8e460e5120dd by Gregory P. Smith in 
branch '3.10':
[3.10] bpo-46576: bpo-46524: Disable compiler optimization within 
test_peg_generator. (GH-31015) (GH-31089)
https://github.com/python/cpython/commit/f5ebec4d3e1199ec38b88920cfde8e460e5120dd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +29274
pull_request: https://github.com/python/cpython/pull/31089

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e by Gregory P. Smith in 
branch 'main':
bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. 
(#31015)
https://github.com/python/cpython/commit/164a017e13ee96bd1ea1ae79f5ac9e25fe83994e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +29197
pull_request: https://github.com/python/cpython/pull/31015

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:

If a decent parallelism CI systems are not available from github (they seem 
stuck at 2-3 threads per 
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners),
 an alternative approach could be to shard across multiple parallel CI tasks. 
Shard such that each one gets only one of the slow tests.

Unfortunately if each of these were a line item in Github's poor CI UI sharding 
a single config's tests across 5-10 tasks would be a nightmare to navigate on a 
PR.  I expect everyone would hate that.

Providing our own runners with decent parallelism could help: 
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

But we're always going to be bound by our longest tail test if we don't fix our 
test parallelism to be more granular.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Kumar Aditya]

Change by Kumar Aditya :


--
keywords: +patch
pull_requests: +29196
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/31017

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Gregory P. Smith]

Gregory P. Smith  added the comment:

re: slow tests in the first half of the list.  the same total amount of time is 
going to be spent regardless.  In our test suite on a modern fast 16 thread 
system, all but 10 tests are completed in parallel within the first 30 seconds. 
 The remaining ~10 take 10x+ that wall time more minutes.

So the most latency you will shave off on a modern system is probably <30 
seconds.  On a slower system the magnitude of that will remain the same in 
proportion.  CI systems are not workstations.  On -j1 or -j2 system I doubt it 
will make a meaningful difference at all.

Picture test execution as a utilization graph:

```
|ttt
|   
|   ttt
|  tt
+
```

The total area under that curve is going to remain the same no matter what so 
long as we execute everything.  Reordering the tests can pull the final long 
tail in a bit by pushing out the top layer.  You move more towards an optimal 
rectangle, but you're still limited by the area.  **The less -jN parallelism 
you have as CPU cores the less difference any reordering change makes.**

What actual parallelism do our Github CI systems offer?

The fundamental problem is that we do a LOT in our test suite and have no 
concept of what depends on what and thus _needs_ to be run.  So we run it all.  
For specialized tests like test_peg_generator and test_tools it should be easy 
to determine from a list of modified files if those tests are relevant.

That gets a lot more complicated to accurately express for things like 
test_multiprocessing and test_concurrent_futures.

test_peg_generator and test_tools are also *packages of tests* that themselves 
should be parallelized individually instead of considered a single serialized 
unit.

At work we even shard test methods within TestCase classes so that big ones can 
be split across test executor tasks: See the _setup_sharding() function in 
absltest here: 
https://github.com/abseil/abseil-py/blob/main/absl/testing/absltest.py#L2368

In absence of implementing an approach like that within test.regrtest to shard 
at a more granular level thus enabling us to approach the golden rectangle of 
optimal parallel test latency, we're left with manually splitting long running 
test module/packages up into smaller units to achieve a similar effect.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

See also https://bugs.python.org/issue46576 and 
https://github.com/python/cpython/pull/31015

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Do all of the tests use all of the slowly built extensions, or could the test 
file be split into multiple files run separately, each faster?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Eric Snow]

Eric Snow  added the comment:

On Tue, Jan 25, 2022 at 4:14 PM STINNER Victor  wrote:
> Currently, most CI run "make buildbottest" which uses -r option of 
> libregrtest: randomize tests order.

How hard would it be to first randomize the list and then move the
slow tests up to a random position in the first half (for example) of
the list?

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Mark Shannon]

Mark Shannon  added the comment:

It's plenty slow on linux as well.

I like the idea of starting the slower tests first.
The long tail of slow tests is annoying when running `make -j12 test`.

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Kumar Aditya]

Kumar Aditya  added the comment:

Another solution would be to shard the tests on windows i.e. run tests on two 
different jobs concurrently, edgedb does this. 
See https://github.com/edgedb/edgedb/actions/runs/1746736086

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[STINNER Victor]

STINNER Victor  added the comment:

> Would it be possible to skip these tests when no changes to the parser  
> related code are made to speed up tests?

Maybe, some CIs could export an environment variable which contains the list of 
modified files, and then each file would be able to decide if it should be 
skipped or not. The risk is skipping a test because the change looks 
"unrelated", whereas it causes the test to fail.

Currently, we only skip tests if no code is changed: if only the documentation 
is modified.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Kumar Aditya]

Kumar Aditya  added the comment:

Would it be possible to skip these tests when no changes to the parser  related 
code are made to speed up tests?

--
nosy: +kumaraditya303

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[STINNER Victor]

STINNER Victor  added the comment:

> In which case it would be generally helpful to start the longest-running 
> tests first.

Currently, most CI run "make buildbottest" which uses -r option of libregrtest: 
randomize tests order.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Tim Peters]

Tim Peters  added the comment:

As a general thing, I expect people on Windows always run the tests with 
multiple processes. In which case it would be generally helpful to start the 
longest-running tests first. As is, because of its late-in-the-alphabet name, 
"test_peg_generator" gets started late in the process and so typically keeps 
running for minutes and minutes after all other tests have completed. So just 
renaming it to  "test_0peg_generator" would cut minutes off the Windows wall 
clock time ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

The test needs to build a lot of C extensions with different parsers, and that 
compilation is what takes most of the time.

I don't think we should skip these tests by default on Windows, as it gives us 
valuable information (that the parser features work compiled on windows).

If you have an idea on how to speed the tests, I'm all ears :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[Tim Peters]

Change by Tim Peters :


--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46524] test_peg_generator takes 8 minutes on Windows

[STINNER Victor]

New submission from STINNER Victor :

test_peg_generator takes between 5 and 16 minutes on the Windows CI run on 
Python pull requests.

Example of timings on my PR https://github.com/python/cpython/pull/30890

* GitHub Action win32: 8 min 16 sec
* GitHub Action win64: 16 min 38 sec
* Azure Pipelines win32: 5 min 30 sec
* Azure Pipelines win64: 8 min 3 sec

Would it be possible to make these tests faster?

Or at least, would it be possible to skip these slow tests on Windows where 
spawing a process or running a C compiler is slower?

On my Fedora 35 (Linux) laptop, test_peg_generator takes 49.5 seconds. 
test_peg_generator only takes 736 ms if I disable TestCParser.

--
components: Tests
messages: 411662
nosy: lys.nikolaou, pablogsal, vstinner
priority: normal
severity: normal
status: open
title: test_peg_generator takes 8 minutes on Windows
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com