[issue37232] Parallel compilation fails because of low ulimit.

[Jakub Kulik]

Jakub Kulik  added the comment:

I am closing this as it is true that the limit on files of 256 is pretty low, 
and no matter how robust you make it, arbitrarily low ulimits will crash it 
anyway.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37232] Parallel compilation fails because of low ulimit.

[Jakub Kulik]

Jakub Kulik  added the comment:

I am not sure what you are asking now. compileall with -j0 does compile on all 
cores simultaneously right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37232] Parallel compilation fails because of low ulimit.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I guess you'd get a similar problem if you were compiling on all cores 
simultaneously?  256 simultaneous open files is very low.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37232] Parallel compilation fails because of low ulimit.

[Jakub Kulik]

Jakub Kulik  added the comment:

We have a limit of 256 opened files, which is not much, but I can up it and 
then it doesn't happen.

Mainly, I wanted to report that this might be happening now. But I guess not 
many people will face this problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37232] Parallel compilation fails because of low ulimit.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Can't you just up the ulimit? What the current ulimit for your user?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37232] Parallel compilation fails because of low ulimit.

[Ned Deily]

Change by Ned Deily :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37232] Parallel compilation fails because of low ulimit.

[Jakub Kulik]

New submission from Jakub Kulik :

When building and installing Python 3.8 on our sparc machine, the build breaks 
during the compileall stage with [Error 24] Too many open files. The problem is 
due to the recently enabled parallel compilation (issue36786).

When -j0 is passed to the compileall, executor starts as many threads as there 
are cpus, which is problem on our sparc machine with 512 CPUs and rather low 
ulimit on number of opened files. Compilation fails and is not able to recover 
from that. For the same reason, test_compileall also breaks and never ends.

I am able to fix both of these by patching both Makefile.pre.in and 
test_compileall.py with -j32 (instead of -j0), but that is not the best 
solution.

I think that the best solution to this would be to make compileall.py check for 
this limit and consider that when creating ProcessPoolExecutor (or, maybe even 
better, to make it recover when this problem occurs).

--
components: Build
messages: 345232
nosy: kulikjak
priority: normal
severity: normal
status: open
title: Parallel compilation fails because of low ulimit.
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com