[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[STINNER Victor]

STINNER Victor  added the comment:

It seems like macOS job pass again on Azure Pipelines. I close the issue.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Steve Dower]

Steve Dower  added the comment:

Filed as issue38084

I recommend not investigating this issue any further until that one is resolved.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[STINNER Victor]

STINNER Victor  added the comment:

Steve: Would you mind to open a separated issue for the multiprocessing bug? 
multiprocessing is supposed to handle this case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Steve Dower]

Steve Dower  added the comment:

I suspect this code is a repro - it certainly locks up the host process 
reliably enough.

Perhaps if we unblock multiprocessing in the context of a crashed worker then 
it'll show what the underlying errors are?


import os
from multiprocessing import Pool

def f(x):
os._exit(0)
return "success"

if __name__ == '__main__':
with Pool(1) as p:
print(p.map(f, [1]))

--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Kyle Stanley]

Kyle Stanley  added the comment:

It looks like the Azure macOS tests timed out again in the recently opened 
PR-15688. Specifically, for test_multiprocessing_spawn and test_functools (both 
of which also timed out in PR-15651, which Victor mentioned earlier):

0:26:41 load avg: 2.89 [418/419/1] test_multiprocessing_spawn crashed (Exit 
code 1) -- running: test_functools (14 min 38 sec)
Timeout (0:20:00)!

0:32:03 load avg: 3.17 [419/419/2] test_functools crashed (Exit code 1)
Timeout (0:20:00)!

Build logs: 
https://dev.azure.com/Python/cpython/_build/results?buildId=49868&view=logs&j=18d1a34d-6940-5fc1-f55b-405e2fba32b1

As far as I can tell, PR-15688 should have had no direct influence on 
test_multiprocessing_spawn or test_functools.

> Maybe macOS on Azure is running slower and we should just increase the 
> timeout?

> Yeah, I agree that increasing the timeout shouldn't be the answer here. 

Since this seems to be affecting multiple PRs, would it be appropriate to 
attempt to increase the timeout duration as a temporary fix and open an issue 
for further investigation on the cause of the intermittent slowdown on those 
tests?

--
nosy: +aeros167

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Ned Deily]

Ned Deily  added the comment:

Yeah, I agree that increasing the timeout shouldn't be the answer here. I still 
have never seen failure modes like this when running my own tests.  The idea 
about CPUs is one worth pursuing although I usually run with -j3.  Also I 
wonder how much memory the VM is configured with.  Any way we can find out 
number of cpus and memory easily?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Steve Dower]

Steve Dower  added the comment:

> Maybe macOS on Azure is running slower and we should just increase the 
> timeout?

(Aside, why don't the macOS buildbots have a tag saying that? Took me ages to 
find them...)

I doubt it's running 6-7x slower. More likely something is causing one of the 
workers to crash at a point where the lock remains held instead of being 
released (I saw this at work the other week in a slightly different context, 
but same symptoms).

Could os._exit() at the wrong time cause it?

It also looks like Azure is running tests with 4 processes, but the buildbot 
(at least the one I'm looking at) is only using 2. So perhaps there are more 
conflicts from that?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[STINNER Victor]

STINNER Victor  added the comment:

> I'm still seeing this, maybe 1 in 20 builds, so it's semi-random. A new 
> deadlock, maybe?

It seems like only the jobs on Azure are killed by timeout. The jobs on macOS 
buildbots look fine. Maybe macOS on Azure is running slower and we should just 
increase the timeout?

The bug still occurs:

https://github.com/python/cpython/pull/15651
https://dev.azure.com/Python/cpython/_build/results?buildId=49786&view=results

0:49:27 load avg: 1.41 [419/419/6] test_threading crashed (Exit code 1)

6 tests failed:
test_concurrent_futures test_functools test_importlib
test_multiprocessing_forkserver test_multiprocessing_spawn
test_threading

The whole job was killed after 57 minutes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Steve Dower]

Steve Dower  added the comment:

I'm still seeing this, maybe 1 in 20 builds, so it's semi-random. A new 
deadlock, maybe?

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37245] Azure Pipeline 3.8 CI: multiple tests hung and timed out on macOS 10.13

[Ned Deily]

Ned Deily  added the comment:

FWIW, I tried reproducing with 3.8 at 996e52623af3854552d41751e0c2522bc0a7e84f 
(the PR 14000 checkin) on both a current 10.14.5 Mojave system and on a 10.13.6 
High Sierra system (the version used in the failed Azure test) and did not see 
any unusual failures.  I don't recall seeing a timeout like in 
test_concurrent_futures, at least recently!, but, if it is due to some race 
condition, there might be a more significant difference, like number of CPUs 
available, that might precipitate the failure.  I'll leave it up to you, 
Victor, on whether or how long to leave this issue open but I don't see that 
there is anything practical to do until it can be reproduced.

--
title: Azure Pipeline: sick macOS job on Python 3.8? -> Azure Pipeline 3.8 CI: 
multiple tests hung and timed out on macOS 10.13

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com