[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

Distutils is now deprecated (see PEP 632) and all tagged issues are being 
closed. From now until removal, only release blocking issues will be considered 
for distutils.

If this issue does not relate to distutils, please remove the component and 
reopen it. If you believe it still requires a fix, most likely the issue should 
be re-reported at https://github.com/pypa/setuptools

--
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[P12 Pfx]

Change by P12 Pfx :


--
nosy: +P12 Pfx

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[miss-islington]

miss-islington  added the comment:


New changeset 0d20364b132014eec609b900997c34779a4d548c by Miss Islington (bot) 
in branch '3.8':
bpo-38597: Never statically link extension initialization code on Windows 
(GH-18724)
https://github.com/python/cpython/commit/0d20364b132014eec609b900997c34779a4d548c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[miss-islington]

miss-islington  added the comment:


New changeset 8a5f7ad5e423b74ea612e25472e5bff3adf1ea87 by Steve Dower in branch 
'3.7':
[3.7] bpo-38597: Never statically link extension initialization code on Windows 
(GH-18724) (GH-18759)
https://github.com/python/cpython/commit/8a5f7ad5e423b74ea612e25472e5bff3adf1ea87


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +18114
pull_request: https://github.com/python/cpython/pull/18759

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:


New changeset ce3a4984089b8e0ce5422ca32d75ad057b008074 by Steve Dower in branch 
'master':
bpo-38597: Never statically link extension initialization code on Windows 
(GH-18724)
https://github.com/python/cpython/commit/ce3a4984089b8e0ce5422ca32d75ad057b008074


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 11.0 -> 12.0
pull_requests: +18113
pull_request: https://github.com/python/cpython/pull/18758

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

> don't know if you still need it but here is what you requested.

Thanks. That basically confirms that something is interfering with distutils. 
Skipping the check entirely should avoid it.

--
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Xinfa Zhu]

Xinfa Zhu  added the comment:

Steve, don't know if you still need it but here is what you requested. Sorry 
for the slow move (I was working on something else). Seems mine is finding the 
x64 before the OneCore, though I don't know the significance.

Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] on 
win32
In[2]: import glob
In[3]: glob.glob(r"C:\Program Files (x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll",
 recursive=True)
Out[3]: 
['C:\\Program Files (x86)\\Microsoft Visual 
Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll',
 'C:\\Program Files (x86)\\Microsoft Visual 
Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\onecore\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll']

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Change by Steve Dower :


--
keywords: +patch
pull_requests: +18080
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/18724

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

In thinking about this, I think the best way forward is to just remove the 
logic that might statically link the initialization code, and instead commit to 
CPython releases always including vcruntime140.dll even if we switch to a newer 
version one day.

Hopefully third party distributions will do the same, though it should only 
matter for ABI3 modules that are not recompiled for the newer versions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

Hmm, just ran it on my own and it's finding the OneCore file first, which is 
not a problem (yet) but could become one.

So we at least need to replace the "**" with a "*" in 
Lib/distutils/_msvccompiler.py#L109. Not sure if that will help in this other 
case or not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

Thanks, it does seem like it's finding the correct MSVC, but is not finding the 
redistributable DLL:

> cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT ...

The "/MT" means to statically the CRT, and then we use link settings later to 
dynamically link the part installed in the OS (see 
https://stevedower.id.au/blog/building-for-python-3-5-part-two for more 
details).

The problem seems to be that this glob in distutils can't find the path listed 
below it:

"2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll"

"2019\BuildTools\VC\Redist\MSVC\14.24.28127\x64\Microsoft.VC142.CRT\vcruntime140.dll"

We swallow a lot of errors while doing this glob, so I assume there's something 
about your install that's affecting it (maybe related to it being BuildTools 
rather than Community or higher?)

Could you try running this code and see what result/error you get:

import glob
glob.glob(r"C:\Program Files (x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll",
 recursive=True)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Xinfa Zhu]

Xinfa Zhu  added the comment:

FYI when I reduced my package to 106 extension modules, I could run without the 
DLL error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Xinfa Zhu]

Xinfa Zhu  added the comment:

I have had wheel installed. Following this SO answer 
(https://stackoverflow.com/a/50314071/7269441), I added "import setuptools" 
before distutils (my setup.py attached).

Following is the log, where we see the VS path it is using. BTW, I am trying to 
reduce my package to <120 modules and test it.

(base) 
C:\Users\xinfa\Documents\code\ezcad-dev\beta>C:\Users\xinfa\AppData\Local\Continuum\anaconda3\python.exe
 setup.py build_ext
Include directory: C:\Users\xinfa\Documents\code\ezcad-dev\beta
Number of ext modules = 225
running build_ext
building 'ezcad.__main__' extension
C:\Program Files (x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c 
/nologo /Ox /W3 /GL /DNDEBUG /MT -IC:\Users\xinfa\Documents\code\ezcad-dev\beta 
-I. -IC:\Users\xinfa\AppData\Loca
l\Continuum\anaconda3\include 
-IC:\Users\xinfa\AppData\Local\Continuum\anaconda3\include "-IC:\Program Files 
(x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files 
(x86)\Windows Kits
\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows 
Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows 
Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows 
Kits\10\include\10.0.1836
2.0\winrt" "-IC:\Program Files (x86)\Windows 
Kits\10\include\10.0.18362.0\cppwinrt" /Tcezcad\__main__.c 
/Fobuild\temp.win-amd64-3.7\Release\ezcad\__main__.obj
__main__.c
C:\Program Files (x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\link.exe 
/nologo /INCREMENTAL:NO /LTCG /nodefaultlib:libucrt.lib ucrt.lib /DLL 
/MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPAT
H:C:\Users\xinfa\AppData\Local\Continuum\anaconda3\libs 
/LIBPATH:C:\Users\xinfa\AppData\Local\Continuum\anaconda3\PCbuild\amd64 
"/LIBPATH:C:\Program Files (x86)\Microsoft Visual 
Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\l
ib\x64" "/LIBPATH:C:\Program Files (x86)\Windows 
Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows 
Kits\10\lib\10.0.18362.0\um\x64" 
build\temp.win-amd64-3.7\Release\ezcad\__main__.obj /OUT:build\lib.win
-amd64-3.7\ezcad\__main__.cp37-win_amd64.pyd
   Creating library build\lib.win-amd64-3.7\ezcad\__main__.cp37-win_amd64.lib 
and object build\lib.win-amd64-3.7\ezcad\__main__.cp37-win_amd64.exp
Generating code
Finished generating code
version = 0.1.9

--
versions: +Python 3.7 -Python 3.6
Added file: https://bugs.python.org/file48918/setup.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

You should be able to install "wheel" without setuptools to get the bdist_wheel 
command.

Can you confirm that the build process is actually using that Visual Studio 
install? If it's going through the regular distutils detection process then it 
ought to be finding the right files.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Xinfa Zhu]

Xinfa Zhu  added the comment:

Thanks Steve. Here is what you requested.

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ ./Installer/vswhere.exe
Visual Studio Locator version 2.7.1+180c706d56 [query version 2.3.2200.14893]
Copyright (C) Microsoft Corporation. All rights reserved.

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ find . -name "vcvarsall.bat"
./2019/BuildTools/VC/Auxiliary/Build/vcvarsall.bat

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ find . -name "vcruntime140.dll"
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/onecore/x64/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/onecore/x86/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/x64/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/x86/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x86/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx86/x64/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx86/x86/vcruntime140.dll
./Installer/resources/app/ServiceHub/Services/Microsoft.VisualStudio.Setup.Service/vcruntime140.dll
./Installer/vcruntime140.dll

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio


I want mention that I have 301 extension modules. I used setuptools in my 
setup.py
from setuptools import setup
from setuptools.extension import Extension
python setup.py build_ext
python setup.py bdist_wheel
I bring the whl file to another computer, pip install, then launch the app and 
I get this error
ImportError: DLL load failed: A dynamic link library (DLL) initialization 
routine failed.

I tried using distutils instead, but it says: error: invalid command 
'bdist_wheel'
from distutils.core import setup
from distutils.extension import Extension

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

Okay, looking at _find_vcvarsall in distutils, I'm guessing that something 
about how your machines are set up means that the vcredist search is failing.

First, could you specify which versions of Visual Studio you have installed, 
and if possible which one is being found and used by your builds.

Then, if you can search your install to find both vcvarsall.bat and 
vcruntime140.dll (there will be a few of these) and post the paths, that may 
indicate if the layout isn't consistent.

Distutils will try and dynamically link to the runtime if you have the redist, 
and statically link it if you don't (though I don't remember why it doesn't 
just rely on the copy included with Python... probably future-proofing or 
licencing).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Xinfa Zhu]

Xinfa Zhu  added the comment:

I have a similar issue. Do we have an estimate how long it may take to fix this 
bug? Thanks. I can help but would need some mentoring.

--
nosy: +xinfazhu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Stefan Behnel]

Stefan Behnel  added the comment:

Cython doesn't interfere with the C compiler setup in any way, that's left
to distutils/setuptools (and the user).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

I haven't looked into _why_ yet, but the first PYD I grabbed from the GitHub 
link above has had the CRT statically linked. This is not the default (or it 
should not be), because when we made it the default this exact issue occurred :)

If somehow the default linking mode in distutils has changed, we should fix 
that. If it is being overridden by Setuptools or Cython then we should get 
those projects fixed.

Stefan/Paul - do Cython or Setuptools override compiler/linker settings like 
this at all? Most likely it's the /MT vs /MD option

--
nosy: +p-ganssle, scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Kevin Schlossser]

Kevin Schlossser  added the comment:

Thank you msg356892 for spear heading this for me. I family things to attend to 
so I apologize for opening this bug report and then walking away..

As far as recreating this issue. It simple to do. you can either use cython or 
you can put together a quick script that will make say 150 extensions. put one  
line of code in it and compile it.. The results always end up being the same 
123 extensions can be loaded on the 124th a TB happens.. It does not matter 
where the files are inside the package. it is always the 124th one that gets 
loaded for any one package. importing a pyd from another package does not 
change the number. The extensions have to be a direct descendant of one single 
package.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Yurii Leonov]

Yurii Leonov  added the comment:

Done:
.pyd files are added in 
https://github.com/Yuriy-Leonov/cython_imports_limit_issue/commit/2f9e7c02798fb52185dabfe6ce3811c439ca2839

folder with name "dist_example_with_error"

--
versions: +Python 3.6 -Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

Could you share just one of your .pyd files?

Without being able to see whether they are compiled incorrectly, it's hard to 
be sure whether this is the same cause as before. It certainly looks like 
distutils is still going to link correctly.

--
versions: +Python 3.7, Python 3.8, Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Юрий Леонов]

Юрий Леонов  added the comment:

Looks like I have same problem for Windows 10 (version 1809, build - 17763.864).

I created repository with steps for reproducing - 
https://github.com/Yuriy-Leonov/cython_imports_limit_issue

--
components:  -Interpreter Core, Windows
nosy: +Юрий Леонов
versions: +Python 3.6 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Steve Dower]

Steve Dower  added the comment:

There is an implicit cap due to the C runtime (specifically vcruntime140.dll) 
allocating fibre local storage on load, which seems like the one you are 
hitting. However, we discovered this before the first 3.5 release and fixed it.

How are you compiling your packages? Or where are you getting them from?

It's possible that someone else is building and statically linking the C 
runtime, which will cause this, but distutils (and hence CPython) should not be 
doing it by default.

--
components: +Distutils
nosy: +dstufft, eric.araujo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +Interpreter Core, Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components:  -Extension Modules

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38597] C Extension import limit

[Kevin Schlossser]

New submission from Kevin Schlossser :

System
Windows 7 x64 SP2
Ram 16GB
6 Core AMD @ 3.2ghz

CPython 3.7.2


C Extension (pyd) import cap.

There seems to be a cap on the number of extensions that a package is able to 
contain. I am able to import 123 extension modules that my package has but when 
i go to import number 124 i get the following traceback

ImportError: DLL load failed: A dynamic link library (DLL) initialization 
routine failed.

these extension modules are part of my package, importing an extension module 
from another package does not change this behavior. it is only when I import 
the 124th extension that is in my package does it occur. 

When I change the order of the imports the error does not follow the import. I 
end up getting the same error when the 124th extension gets loaded doesn't 
matter what extension it is. 

I have tried to see if maybe it was a module limit and I spread the imports 
across multiple files and it still fails when the 124th gets loaded. I also 
tried imp.load_dynamic and importlib.import_module and the same error occurs.

If there is a way to work around this limitation it would be very helpful.

--
components: Extension Modules
messages: 355425
nosy: Kevin Schlossser
priority: normal
severity: normal
status: open
title: C Extension import limit
type: resource usage
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com