[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Dickinson]

Change by Mark Dickinson :


--
nosy:  -mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[STINNER Victor]

STINNER Victor  added the comment:

The distutils bdist_wininst command has been removed in Python 3.10: see 
bpo-42802.

--
nosy: +vstinner
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Lawrence]

Mark Lawrence added the comment:

This is referenced from #4431 which has been closed for over six years but 
keeps getting comments.

--
nosy: +BreamoreBoy, steve.dower, tim.golden, zach.ware

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Lawrence]

Changes by Mark Lawrence breamore...@yahoo.co.uk:


--
components:  -Distutils2
nosy: +dstufft
versions: +Python 3.4, Python 3.5 -3rd party, Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Almar Klein]

Almar Klein added the comment:

Just checking in to point out a possible problem with the code that strips the 
MSVCR dependency from the embedded manifest. The used regexpr is too greedy: 
the first bit can trigger on an earlier assemblyIdentity tag, so that after 
the removal the manifest is no longer valid XML. 

The key problem is that the assemblyIdentity and the name attribute are 
allowed to be on a separate lines. To fix this I removed the re.DOTALL flag and 
replaced the second occurrence of .*? to also allow newlines: 

  pattern = re.compile(
- rassemblyIdentity.*?name=(|')Microsoft\.\
- rVC\d{2}\.CRT(|').*?(/|/assemblyIdentity),
- re.DOTALL)
+ rassemblyIdentity.*?name=(|')Microsoft\.\
+ rVC\d{2}\.CRT(|')(.|\r|\r)*?(/|/assemblyIdentity))


It is well possible that the problem does not causes any problems for the 
intended usage of the code. I'm using the code to strip other DLL's and ran 
into this issue (with tk85.dll to be precise).

- Almar

--
nosy: +almar

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

Actually, I think this is OK - the reference to the x86 is in the tests and 
those tests don't actually perform a build, just check the manifest is detected 
and stripped (ie, the test should still work fine on 64bit boxes).  Ideally the 
test could also check a manifest with a 64bit architecture, but I don't think 
that's really necessary...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Mark:  Possibly a stupid question, but in your commit (see snippet below), why 
is the processorArchitecture hard coded to x86?  Is it appropriate to replace 
this with amd64 for 64-bit builds, or should it always be x86?

+dependentAssembly
+  assemblyIdentity type=win32 name=Microsoft.VC90.CRT
+ version=9.0.21022.8 processorArchitecture=x86
+ publicKeyToken=
+  /assemblyIdentity
+/dependentAssembly

--
nosy: +mark.dickinson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

ack - that is a really good point.  IIRC it can be *.  I'll try and look at 
this over the next day or 2.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Sébastien Sablé]

Changes by Sébastien Sablé sa...@users.sourceforge.net:


--
nosy: +sable

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

To port the patch to packaging, go into your CPython 3.3 checkout and edit 
Lib/packaging/compiler/msvc9compiler.py (and its test :).

To port the patch to distutils2, clone http://hg.python.org/distutils2/ and 
edit distutils/compiler/msvc9compiler.py (same :).  Test with Python 2.4, 2.5, 
2.6 and 2.7.  Then hg update python3, hg merge default, test with Python 3.1, 
3.2 and 3.3.  Then you can push :)

If you don’t have the necessary Pythons or roundtuits, I’ll port the patch when 
I get a Windows VM.  There’s plenty of time before Python 3.3 is out.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

s/distutils/distutils2/ !

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset fb886858024c by Mark Hammond in branch '2.7':
Issue #7833: Ext. modules built using distutils on Windows no longer get a 
manifest
http://hg.python.org/cpython/rev/fb886858024c

New changeset 9caeb7215344 by Mark Hammond in branch '3.2':
Issue #7833: Ext. modules built using distutils on Windows no longer get a 
manifest
http://hg.python.org/cpython/rev/9caeb7215344

New changeset 3073ef853647 by Mark Hammond in branch 'default':
Issue #7833: Ext. modules built using distutils on Windows no longer get a 
manifest
http://hg.python.org/cpython/rev/3073ef853647

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

I pushed the changes to 2.7, 3.2 and 3.3.  I'm happy to help with 
distutils2/packaging but I'll need to do that later once I work out where to 
start :)  Therefore I'm not yet closing this issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

Looks good.

Style nit: I don’t put backslashes at end-of-lines in parens (in one re.compile 
call you have that).  Also, I use -- where I can’t use —.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

New version of the patch with the small tweaks requested plus a NEWS entry.

--
Added file: http://bugs.python.org/file23400/bug-7833-tweaks-plus-news.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

 Note the existing test doesn't actually perform a build so the new
 test also doesn't, but it does check the core logic
Sounds good to me.

+def manifest_setup_ldargs
I’d make all new methods private ones (i.e. leading underscore).

 an embedded manifests
Typo: extra s

 return None if not temp_manifest else (temp_manifest, mfid)
Using a ternary expression runs afoul of PEP 291: distutils should remain 
compatible with 2.3.  (I’m not sure it is right now, we use modern unittest 
methods in tests and all, but it is no reason for making it worse in new code :)

Your patch will also need an entry in Misc/NEWS; at first glance, there is no 
documentation file to edit.

Will you port the patch to packaging in 3.3?  I can do it if you don’t have the 
time, but I’m not set up yet to test on Windows, so I can ask you to test a 
patch.  Also, for the distutils2 backport (which I can do too), we would need 
to run tests with all versions from 2.4 to 3.3.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

Thanks for the review.  One note:

| +def manifest_setup_ldargs
| I’d make all new methods private ones (i.e. leading underscore).

They aren't strictly private and are designed to be overridden by subclasses 
(although in practice, subclassing the compiler is much harder than it should 
be, so pywin32 monkey-patches the instance.)  This is actually the entire point 
of my updated patch - to give setup.py authors some level of control over the 
manifest behaviour.

I do intend forward-porting to 3.3 and also to check it is is too late for 3.2 
(a quick check before implied it might be OK, but I'm not sure)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

 They aren't strictly private and are designed to be overridden by
 subclasses
OK.

 I do intend forward-porting to 3.3 and also to check it is is too late
 for 3.2 (a quick check before implied it might be OK, but I'm not sure)
2.7 and 3.2 are open for bug fixes, as indicated by the versions field of this 
bug (it’s actually a matrix: component distutils + versions 2.7, 3.2, 3.3, 
component distutils2 + version 3.3 == packaging and distutils2 + third-party == 
the distutils2 backport :)

--
assignee: tarek - mhammond

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

There is a already a test for manifest things in 
Lib/distutils/tests/test_msvc9compiler.py, and higher-level tests for building 
extension modules in Lib/distutils/tests/test_build_ext.py and 
Lib/distutils/tests/test_install.py.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

Let me add that I don’t know the MS toolchain at all, but if you tell me what a 
test should do (e.g. compile a C extension, open some compiled file and look 
for some bytes) I can write a test.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

My apologies Eric - I had completely overlooked those tests.  Attaching a new 
patch with a test.  Note the existing test doesn't actually perform a build so 
the new test also doesn't, but it does check the core logic (ie, that a 
manifest with only the msvcrt reference gets it scrubbed).

--
Added file: 
http://bugs.python.org/file23341/bug-7833-overridable-manifest-settings-with-test.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

I'm reluctant to commit to adding test infrastructure for the distutils build 
commands - if I've missed existing infrastructure and adding such tests would 
actually be relatively simple, please educate me!  Or if someone else would 
like to help with the infrastructure so I can test just this patch, that would 
be awesome.  But I don't think this fix should block on tests given it can 
easily be tested and verified manually.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] bdist_wininst installers fail to load extensions built with Issue4120 patch

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

Can the patch include regression tests?

--
components: +Distutils2
nosy: +alexis, eric.araujo
title: Bdist_wininst installers fail to load extensions built with Issue4120 
patch - bdist_wininst installers fail to load extensions built with Issue4120 
patch
versions: +3rd party, Python 3.2 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond skippy.hamm...@gmail.com added the comment:

This is biting people (including me :) so I'm going to try hard to get this 
fixed.  One user on the python-win32 mailing list resorts to rebuilding every 
3rd party module he uses with this patch to get things working again (although 
apps which use only builtin modules or pywin32 modules - which already hacks 
this basic functionality in - don't see the problem.)

I'm attaching a different patch that should have the same default effect as 
Christoph's, but also allows the behaviour to be overridden.  Actually 
overriding it is quite difficult as distutils isn't setup to easily allow such 
compiler customizations - but at least it *is* possible.  To test this out I 
hacked both the pywin32 and py2erxe build processes to use those customizations 
and it works fine and allows them both to customize the behaviour to meet 
various modules' requirements.

Finally, this whole thing is still fundamentally broken for extensions which 
need a manifest (eg, to reference the common controls or the 
requestedExecutionLevel cruft).  These extension will still need to include the 
CRT reference in their manifest and thus will need a copy of the CRT next to 
each of them.  I *think* this also means they basically get a private copy of 
the CRT - they are not sharing the CRT with Python, which means they are at 
risk of hitting problems such as trying to share FILE * objects.  In practice, 
this means such modules are probably better of just embedding the CRT 
statically.  This is the route I've taken for one pywin32 module so the module 
can have a manifest and still work without a complete, private copy of the CRT 
needing to live next to it.  But even with that problem I think this patch 
should land.

It would be great if someone can review and test this version of the patch and 
I'll check it in.

--
versions: +Python 3.3, Python 3.4 -Python 2.6
Added file: 
http://bugs.python.org/file23305/bug-7833-overridable-manifest-settings.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Garen]

Changes by Garen garen.par...@gmail.com:


--
nosy: +Garen

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Anton Benard]

Changes by Anton Benard acben...@gmail.com:


--
nosy: +abenard

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Eric Smith]

Changes by Eric Smith e...@trueblade.com:


--
nosy: +eric.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[John Cary]

John Cary jrobc...@gmail.com added the comment:

Just to follow up.  My case is an application that is almost
all statically linked, but it loads in the python library, and
at runtime it needs to load the tables module, and as distributed
by Python, I get the load-time error on Windows.

Using Christoph Gohlke's exe's works great for me, but I cannot
redistribute due to the linking in of szip for tables and
MKL for numpy.  So I build my own version of tables without
szip and numpy without MKL, but that failed until I applied 
Christoph's patch on Python.  (I also have to patch tables'
setup.py not to include szip.dll and zlib1.dll in dll_files.)

So the result is that with Christoph's patch, all is working
for me.  I hope it (or something similar) makes it into 2.7.

John Cary

--
nosy: +John.Cary

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

Christoph Gohlke cgoh...@uci.edu added the comment:

Actually, PyQt4 was not a good example since it is not build using distutils. 
Pywin32 and wxPython extensions are the only ones on my system specifying a 
dependency on Common Controls or MFC. No dependencies other than CRT, MFC, and 
Common Controls are specified in any assemblies of over 1000 pyd and dll files 
in my Python27 directory.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond mhamm...@users.sourceforge.net added the comment:

Thinking more about this, I think the approach of this patch is more complex 
than necessary.  I think a better patch would be one which *unconditionally* 
removes the manifest from extension modules.  For maximum flexibility though, 
we should probably allow a hook so setup.py can specify the name of (or 
contents of) a manifest file to use when linking with the default being None.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

Christoph Gohlke cgoh...@uci.edu added the comment:

The proposed patch was meant to be backwards compatible. Unconditionally 
removing the whole assembly/manifest from extensions could break extensions 
that have additional dependencies, such as MFC or Common Controls. PyQt4 
extensions for example depend on Common Controls. 

Btw, there is a discussion at 
http://psycopg.lighthouseapp.com/projects/62710/tickets/20 and 
http://groups.google.com/group/modwsgi/browse_thread/thread/137f88ac6927df59 
about the need to put the msvcr90 manifest back into the psycopg.pyd file in 
order to work properly under Apache's mod_wsgi, which is linked against msvcrt.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond mhamm...@users.sourceforge.net added the comment:

I'm wondering if, in practice, extensions which need a manifest can have the 
manifest being generated completely by the linker - ie, I expect that in most 
cases where something other than the CRT or MFC is needed in the manifest, the 
author will want to specify an explicit manifest file rather than one generated 
by the linker.

And worse, those extensions are going to be screwed anyway - the fact the 
manifest remains at all will mean they probably fail to load for reasons 
already discussed in this bug. IOW, not having a manifest at all is about the 
only way to ensure the module will be loaded correctly.

Re adding a manifest back in - I've struck the exact same problem with pywin32 
- any DLLs which are entry points into Python need to have a manifest - eg, 
pythoncomxx and a few other pywin32 ones.  This left me with a dilemma for 
pythoncom - as it is both an entry-point (ie, COM loads that DLL) and a 
regular Python module, I simultaneously needed a manifest (to work when loaded 
by COM) and needed to *not* have one (to work when loaded by Python).  My 
solution has been to introduce another DLL with a manifest and have COM servers 
register using that.  This strategy seems to be working well in all my tests.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond mhamm...@users.sourceforge.net added the comment:

I'm failing to get a new pywin32 out of the door due to this issue.  I've spent 
a few hours playing with this and I think the analysis is generally correct 
here.  The key thing is that when using distutils, the extensions end up with a 
manifest (albeit one without a reference to the CRT) whereas the extensions 
shipped with Python have no manifest at all.

I agree with Martin that it seems strange the CRT fails to be used even though 
the CRT is obviously already loaded, but it seems to be a fact.  I can't find 
much on this, but suspect it relates to the different activation contexts in 
use and how the activation contexts are designed to allow side-by-side loading 
of DLLs; Windows doesn't know if the version of the DLL already loaded is 
suitable.  I also guess that the fact the DLL has *any* manifest means they use 
a more strict interpretation of things (ie, refuse to use already loaded ones) 
whereas a dll with no manifest gets given a little more slack.

I can confirm that with the attached patch, pywin32 builds and installs fine on 
a machine without the CRT installed globally - so I'm +1 on this patch with one 
caveat:  The check for '.pyd' should either be expanded to include '.dll', or 
else the check should just use the 'target_desc == CCompiler.EXECUTABLE' 
condition already used.

I'm happy to make the change once I get some feedback and/or guidance about 
where I should check this in - I believe it is too late for python 2.6 which is 
a shame...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[anatoly techtonik]

anatoly techtonik techto...@gmail.com added the comment:

I see that this issue mentions --user-access-control option. Can somebody also 
check if issue8870 and issue8871 are related to this one?

--
nosy: +techtonik

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Koen van de Sande]

Koen van de Sande k...@tibed.net added the comment:

issue8870 and issue8871 are not related to this one. There, the UAC elevation 
fails, here the issue is with the MS runtimes, elevation is working fine.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

Christoph Gohlke cgoh...@uci.edu added the comment:

The bdist_wininst and DLL build issues also exist in Python 2.7b2. A patch 
against svn trunk is attached. 

The pywin32 v214 package fails as reported earlier when built with Python 
2.7b2. It installs and functions when built with this patch.

--
versions: +Python 2.7
Added file: 
http://bugs.python.org/file17286/msvc9compiler_stripruntimes_trunk.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Koen van de Sande]

Koen van de Sande k...@tibed.net added the comment:

There should be no manifest embedded into wininst, because then the cases which 
Issue 4120 fixed (a CRT installed into a local folder, instead of system-wide, 
due to limited access rights), will 'break' again: the installer can then no 
longer work unless there is a system-wide installation of the CRT.

Option #1, #2 and #3 all sound reasonable (and #2 is the current situation).

I have some doubts about option #4: it is a very specific use case, and then 
the whole benefit of issue 4120 is lost, because stripping runtimes would have 
to be removed again. Why is putting a separate manifest file next to the DLL 
not an option? Combined with #3 (allow extension developer to disable embedding 
of manifests) a separate manifest can fix the problem.

--
nosy: +koen

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Koen van de Sande]

Koen van de Sande k...@tibed.net added the comment:

Concerning the patch: what happens when the developer already added 
/MANIFEST:NO to the flags, and the code deduces that MSVCR9 is the only 
runtime, e.g. the case where /MANIFEST:NO is in the flags twice? Does the 
linker handle this OK, or does there need to be an additional check as to not 
have it twice?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

Christoph Gohlke cgoh...@uci.edu added the comment:

 I have some doubts about option #4: it is a very specific use case, and then 
 the whole benefit of issue 4120 is lost

Pythoncom are pywintypes are indeed special cases: Out of the 170 DLL files in 
my Python site-packages directory, these seem to be the only ones built with 
distutils. All other DLLs are apparently built without  Python involvement 
using make, nmake, CMake, or Visual Studio and most of them contain embedded 
manifests, which is the default when using nmake, CMake, or Visual Studio. 
Practically, to make a standalone distribution of any Python 2.6/3.1 
application with external DLL dependencies likely requires to provide external 
manifest files. The issue4120 patch does not change this situation and I don't 
see any sane way to patch Python/distutils that could. The main benefit of the 
issue4120 patch, as I see it, is that PYD files produced by distutils work in a 
standalone distribution without any further attention. 
Msvc9compiler_stripruntimes_revised.patch does not change this.

My reasoning for this patch (besides fixing the bdist_wininst installer issue) 
was to allow the popular pywin32 package to build without changes, and offer a 
way for other extension packages to exclude manifests from DLL files if 
required (apparently not that common). Alternatively one could provide a 
mechanism to embed specific manifests into DLLs. Is that currently possible? 
Then pywin32 setup.py could be fixed.


 Why is putting a separate manifest file next to the DLL not an option?

Because the pythoncom dll is currently installed into the Windows system 
directory. Putting manifest files there will pollute the system directory even 
more and possibly interfere with other system components if not done right (not 
tested). But again, pywin32 setup.py could be fixed to not install the 
DLL/manifest files into the system directory.


Which Python packages other than pywin32 build DLL files via distutils? I don't 
know any.

Can anyone provide a minimal setup.py script and C file that produces a DLL 
file for testing?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Mark Hammond mhamm...@users.sourceforge.net added the comment:

the pywin32 DLLs have 2 heads.  They are Python extension modules as well as 
regular DLLs.  They are built by distutils and therefore have no manifests - I 
think many packages use distutils to build their extension modules - it is just 
that they usually don't have a .dll extension.

I fear that simply adding a manifest to those DLLs will put them in the same 
position we have before issue4120 was addressed, and these .dlls do need a way 
to be installed into System32 (or somewhere else on the global PATH) to 
function as COM servers etc.  I need to experiment with this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Mark Hammond]

Changes by Mark Hammond mhamm...@users.sourceforge.net:


--
nosy: +mhammond

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

Christoph Gohlke cgoh...@uci.edu added the comment:

I thought one conclusion of the discussion on issue4120 was that any 
executable, which embeds Python and imports MSVCR9 dependent extensions, must 
now provide the manifest for the MSVCR9 runtimes, either embedded or as a 
separate file. See http://bugs.python.org/issue4120#msg94644 and responses. 
Why shouldn't this apply to wininst executables?

Another observation: importing the winsound extension in the postinstall script 
works even though it also depends on MSVCR90.DLL. Winsound.pyd does not have an 
embedded manifest. It is linked with /MANIFEST:NO. Apparently there is a 
difference between an empty manifest and no manifest. 

How about a patch that does:

1) not embed any manifest into PYD files if the manifest contains only a single 
MSVCR90 dependentAssembly element. 

2) remove the MSVCR90 dependentAssembly element from manifests embedded into 
PYD files if the manifest defines additional dependentAssembly elements (e.g. 
for MFC or Common Controls). This is essentially what the 
msvc9compiler_stripruntimes patch does now.

3) not embed manifests into any file (EXE, PYD, or DLL) if '/MANIFEST:NO' is 
defined in extra_link_args. Extension developers can then provide external 
manifest files.

4) not touch the default manifests embedded into EXE and DLL files. The 
msvc9compiler_stripruntimes currently produces DLLs that can not be used 
standalone, such as pythoncom26.dll from the pywin32 package (I can file a 
separate bug if requested). Pythoncom26.dll is meant to be placed in the 
system32 folder and to be used outside of a Python environment, i.e. from the 
Windows Scripting Host. Several pywin32 tests fail with the pythoncom26.dll 
built with the msvc9compiler_stripruntimes patch. Placing a MSVCR9 manifest 
file into the system32 folder next to the pythoncom26.dll is not an option.


A tentative patch against the Python 2.6 branch is attached. I will test it 
further.

(1) and (4) will solve the original pywin32 wininstaller problem without 
changing wininst.exe.

As it is now the installer and some functionality of the pywin32 package will 
likely break if pywin32 is built on Python 2.6.5.

--
Added file: 
http://bugs.python.org/file16121/msvc9compiler_stripruntimes_revised.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

New submission from Christoph Gohlke cgoh...@uci.edu:

Wininst-9.0.exe and wininst-9.0-amd64.exe are missing MSVCRT90 dependencies in 
the embedded manifest.

While testing installers of pywin32 http://sourceforge.net/projects/pywin32/ 
version 214 built with Python 2.6.4 and the 
msvc9compiler_stripruntimes_regexp2.diff patch from Issue4120 
http://bugs.python.org/issue4120, I noticed that the post_install script 
fails with an ImportError while trying to load extensions without embedded 
MSVCRT90 manifest:

Traceback (most recent call last):
  File string, line 601, in module
  File string, line 311, in install
  File string, line 149, in LoadSystemModule
ImportError: DLL load failed: The specified module could not be found.

The code that fails is:

mod = imp.load_module(modname, None, filename, ('.dll', 'rb', 
imp.C_EXTENSION))

where modname='pywintypes' and filename points to an existing 
'pywintypes26.dll' file. 

The post_install script runs fine when executed from the main python.exe 
interpreter.

A possible fix is to embed a MSVCRT90 dependency into wininst-9.0.exe and 
wininst-9.0-amd64.exe using the following linker switches:

/MANIFESTDEPENDENCY:type='Win32' name='Microsoft.VC90.CRT' 
version='9.0.21022.8' processorArchitecture='X86' 
publicKeyToken='1fc8b3b9a1e18e3b'

/MANIFESTDEPENDENCY:type='Win32' name='Microsoft.VC90.CRT' 
version='9.0.21022.8' processorArchitecture='amd64' 
publicKeyToken='1fc8b3b9a1e18e3b' 

See also http://msdn.microsoft.com/en-us/library/ew0y5khy.aspx

A patch against the Python 2.6 PCbuild\bdist_wininst.vcproj is attached. I 
tested it with Python 2.6.4, 32 and 64 bit, and pywin32 214 installers built 
with the following command:

  python.exe setup.py bdist_wininst --user-access-control=auto 
--install-script=wxpython_win_post_install.py

--
assignee: tarek
components: Distutils, Windows
files: bdist_wininst.vcproj.patch
keywords: patch
messages: 98694
nosy: cgohlke, tarek
severity: normal
status: open
title: Bdist_wininst installers fail to load extensions built with Issue4120 
patch
versions: Python 2.6
Added file: http://bugs.python.org/file16086/bdist_wininst.vcproj.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Martin v . Löwis]

Martin v. Löwis mar...@v.loewis.de added the comment:

This doesn't look right. IIUC, wininst.exe loads python26.dll, which in turn 
loads pywintypes.dll. So it should be python26.dll which loads the CRT.

IIUC, wininst.exe is linked statically, precisely to avoid a dependency on the 
CRT assembly.

--
nosy: +loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7833] Bdist_wininst installers fail to load extensions built with Issue4120 patch

[Christoph Gohlke]

Christoph Gohlke cgoh...@uci.edu added the comment:

The last line of my previous post should actually read 
  python.exe setup.py bdist_wininst

Anyway, here are three files (also attached) that can reproduce the problem:

1) setup.py

from distutils.core import setup, Extension
setup(name='testpyd', scripts = [testpyd_postinstall.py],
ext_modules=[Extension('testpyd', ['testpyd.c'],)],
options = {bdist_wininst: {install_script: 
testpyd_postinstall.py, user_access_control: auto},})

2) testpyd.c

#include Python.h
PyMethodDef methods[] = {{NULL, NULL},};
void inittestpyd() {(void)Py_InitModule(testpyd, methods);}

3) testpyd_postinstall.py

#!python
import testpyd


Build the installer with python 2.6 and Issue4120 patch applied:
python setup.py bdist_wininst

Run the installer:
dist\testpyd-0.0.0.win32-py2.6.exe

The postinstall script fails with:

Traceback (most recent call last):
  File string, line 1, in module
ImportError: DLL load failed: The specified module could not be found.

According to Sysinternals process monitor python26.dll is loaded from the 
system32 directory, the testpyd.pyd extension is found at the right place, and 
then the Windows path is searched in vain for MSVCR90.DLL.

Tested on Windows 7 Pro 64 bit.

--
Added file: http://bugs.python.org/file16088/test-bdist_wininst.zip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7833
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com