[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Larry Hastings]

Larry Hastings added the comment:

Pull request accepted.  Please forward-merge, thanks!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Paul Moore]

Paul Moore added the comment:

> Maybe we can special-case pip uninstalling it from the site-packages folder? 
> *Paul* - any thoughts?

Sorry, I've been following this thread but it's been moving pretty fast, so I'm 
probably replying too late to be helpful now :-(

One alternative thought I had was to bundle vcruntime140.dll in a separate 
wheel, which other wheels can depend on. Then we get pip's usual dependency 
resolution to handle ensuring that the runtime is present.

It's possible to special-case vcruntime, what I'd do is modify distutils to 
omit vcruntime140.dll from the RECORD file - then nothing (pip, distlib, other 
install tools) views the vcruntime file as being "owned" by a particular 
package. I'm not overly keen on that solution, though, as it's clearly a hack 
and would be a maintainability issue going forward. But it does keep the code 
changes isolated to the core, rather than needing pip/setuptools changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

FYI: we're making a new release (right now!) with the patch applied, that 
should go out tomorrow.

If anyone spots anything important in the patch, I still really want to hear 
about it, but hopefully having something installable means we'll get at least a 
few days of testing before locking it in.

All my apologies for waiting until the last minute before giving up on the 
crusade to avoid including the versioned files in Python. The timing is 
unfortunate, but I'm sure we're going to have the best compatibility story 
possible right now. So thanks for indulging me, and I hope I haven't put too 
many people through too much anguish.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Roundup Robot]

Roundup Robot added the comment:

New changeset 8374472c6a6e by Steve Dower in branch '3.5':
Issue #25027: Reverts partial-static build options and adds vcruntime140.dll to 
Windows installation.
https://hg.python.org/cpython/rev/8374472c6a6e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Christoph Gohlke]

Changes by Christoph Gohlke :


Added file: http://bugs.python.org/file40402/test_dll_load_failed.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Christoph Gohlke]

New submission from Christoph Gohlke:

This issue was first mentioned at .

The attached script (test_dll_load_failed.py) builds up to 256 C extension 
modules and imports them. On Python 3.4.3 the script passes while on Python 
3.5.0rc3 the script fails with:

```
Traceback (most recent call last):
File "test_dll_load_failed.py", line 42, in 
import_module(name)
File "X:\Python35\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 666, in _load_unlocked
File "", line 577, in module_from_spec
File "", line 903, in create_module
File "", line 222, in _call_with_frames_removed
ImportError: DLL load failed: A dynamic link library (DLL) initialization 
routine failed.
```

Tested on Windows 7 and 10, 64 bit, with Python 3.5.0rc3, 32 and 64 bit.

Due to this issue the "scientific stack" is practically unusable. For example, 
Pandas unit tests error or abort. In a Jupyter notebook, the following simple 
imports fail (using current builds from 
):

```
In [1]:

import matplotlib.pyplot
import pandas
import statsmodels.api
---
ImportError   Traceback (most recent call last)
 in ()
  1 import matplotlib.pyplot
  2 import pandas
> 3 import statsmodels.api



X:\Python35\lib\site-packages\scipy\signal\__init__.py in ()
276 # The spline module (a C extension) provides:
277 # cspline2d, qspline2d, sepfir2d, symiirord1, symiirord2
--> 278 from .spline import *
279 
280 from .bsplines import *

ImportError: DLL load failed: A dynamic link library (DLL) initialization 
routine failed.
```

The cause of this issue is that as of Python 3.5.0rc1 C extension modules are 
linked statically to the multi-threaded runtime library (/MT) instead of the 
multi-threaded DLL runtime library (/MD). A process can not load more than 127 
statically-linked CRT DLLs using LoadLibrary due to a limit of fiber-local 
storage (FLS) as mentioned in the following links:




To put the 127 limit in perspective: the pywin32 package contains 51 C 
extension modules, pygame 36, scipy 65, and scikit-image 41. 

In addition to C extension modules, the 127 limit also applies to 
statically-linked CRT DLLs that are dynamically loaded via Ctypes or 
LoadLibrary.

--
components: Extension Modules, Windows
messages: 250169
nosy: cgohlke, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Python 3.5.0rc3 on Windows can not load more than 127 C extension modules
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Stefan Krah]

Changes by Stefan Krah :


--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Stefan Krah]

Stefan Krah added the comment:

If the scientific stack is unusable, I think this should be a release
blocker.

--
nosy: +skrah
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Larry Hastings]

Larry Hastings added the comment:

This is your wheelhouse, Steve.

--
assignee:  -> steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

Let me experiment today with a few of the proposals I posted in the other 
thread and get back to you.

I suspect someone will need to ship vcruntime.dll, and I'd rather it was the 
extension.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

vcruntime140.dll *is* a system library when installed properly, and if someone 
installs VCRedist then all the bundled ones should be ignored. Over time, I 
expect to see extensions appear that depend on vcruntime150.dll rather than 
140.dll, so it won't always be shared by all extensions.

However, if someone has vcruntime140.dll installed and an extension requires 
vcruntime150.dll, they will get errors unless that extension includes the 
correct version. We can't ship currently-nonexistent versions with Python 3.5, 
and if extensions have to depend on what's installed then Python 3.5 extensions 
will forever be tied to MSVC 14.0.

GPL code should either statically link or recommend their users install 
VCRedist separately. I'm not going to compromise compiler version independence 
because of one license.

The uninstall issue is something I hadn't considered. Maybe we can special-case 
pip uninstalling it from the site-packages folder? *Paul* - any thoughts?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Stefan Krah]

Stefan Krah added the comment:

Is Python-core built with /MD? I cannot see the flags in the buildbot
logs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

It shouldn't be - locale state is in the shared part of the CRT. That is one of 
the reasons I was keen to move to this model (everyone seems to think that 
FILE* is the only problem with mixing CRT versions :) )

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Stefan Krah]

Stefan Krah added the comment:

It seems to be /MTd. Sorry for the noise (and yay! for horizontal scrolling :).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Stefan Krah]

Stefan Krah added the comment:

The reason I asked: We had issues where extension modules linked
against a different CRT had isolated locale settings from the
interpreter, i.e., changes made by the module would not be seen
by the interpreter.

I don't know if this is still an issue with the new runtimes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[eryksun]

eryksun added the comment:

I think 3.5 should be distributed with vcruntime140.dll. It seems a waste for 
python.exe, python35.dll, and all of the extension modules and dependent DLLs 
to each take an FLS slot:

>>> import ctypes # +1 for _ctypes
>>> kernel32 = ctypes.WinDLL('kernel32')
>>> kernel32.FlsGetValue.restype = ctypes.c_void_p
>>> [x for x in range(128) if kernel32.FlsGetValue(x)]
[1, 2, 4, 5, 6, 8]

>>> import pyexpat, select, unicodedata, winsound
>>> import _bz2, _decimal, _elementtree, _hashlib
>>> import _lzma, _msi, _multiprocessing, _overlapped 
>>> import _socket, _sqlite3, _ssl, _tkinter
>>> [x for x in range(128) if kernel32.FlsGetValue(x)]
[1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 
23, 24, 25, 26, 27]

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

Okay, here's a proposal:

We bundle vcruntime140.dll with Python's normal install, so it's always there 
and extensions that use it do not need to ship anything.

When distutils._msvccompiler is used to build an extension with a *different* 
version of MSVC, it will copy the dependency or statically link (as in my 
attached patch).

This does not prevent us from changing the compiler used for 3.5, as long as we 
continue to ship both vcruntime140.dll and the newer version, and extensions 
build with newer compilers will include the dependency or pick up the bundled 
one if they are on a version that includes it.

(Extensions that use C++ and depend on msvcp###.dll will need to ship that 
themselves, obviously.)

I'll post a new patch shortly, but it's only a very small change from this one 
for distutils, and the rest is in the installer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

New patch. Mostly build and installer changes, but the 
distutils/_msvccompiler.py is also part of it.

I've run a full build and done basic testing with a full test run going now, 
but I don't have a clean machine handy to try it without the full CRT install, 
so that'll have to wait until tomorrow.

This *basically* reverts the build back to /MD for everything. The one 
exception is that distutils now knows which DLLs are shipped with Python and if 
a vcruntime is needed that isn't included, it will be put into the dist (or 
statically linked). So wheels created with 3.5.6 and MSVC 15.0 will still run 
against 3.5.0, even if the user has not installed the latest VCRedist.

--
Added file: http://bugs.python.org/file40418/25027_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

Kind-of... We use the same flags I described in my blog[1] so that we don't 
have any version-specific dependencies.

You should (might) see /MT in the build logs, but then we replace most of the 
static CRT with the dynamic (but versionless) one. The versioned parts 
(including the FlsAlloc call - module initialization is compiler version 
specific) are statically linked.

I'm going to try and update distutils to build with /MD again (sorry 
Christoph!) and include vcruntime###.dll in the output. That way, people who 
bdist_wheel will include all of their own dependencies and don't have to worry 
about whether users are on Python 3.5.0 or 3.9.9.

[1]: http://stevedower.id.au/blog/building-for-python-3-5/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Steve Dower]

Steve Dower added the comment:

Attached a fix for distutils that will include the required vcruntime DLL with 
the extension.

This is purely Python source code and only needs to be patched on the build 
machine.

I have tested with a numpy build from source (setup.py bdist_wheel) and it 
works correctly on a clean Win7 machine with only a Python 3.5.0rc3 install.

When multiple vcruntime###.dll files are available, the first one that is 
loaded wins. This means that if you can guarantee a particular import occurs 
first, you can omit the DLL elsewhere in your package. There is no way to 
determine this automatically, 

Because of the way I wrote the patch, if you build with DISTUTILS_USE_SDK set, 
you will get the old static linking unless you also define PY_VCRUNTIME_REDIST 
to the path of the file to include. If the redist file cannot be found (you 
probably don't have the compiler either, but assuming you do), you will get the 
old static linking. I think this is the right balance of "works-by-default" and 
"I know what I'm doing let me control it" (though now I put them next to each 
other, I could rename the variable to DISTUTILS_VCRUNTIME_REDIST).

Will work up a test for it, but wanted to get feedback on the approach first.

--
keywords: +patch
Added file: http://bugs.python.org/file40410/25027_1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25027] Python 3.5.0rc3 on Windows can not load more than 127 C extension modules

[Larry Hastings]

Changes by Larry Hastings :


--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com