[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 80fc40a9ae47 by Martin Panter in branch '3.5':
Issue #22980: Skip a sysconfig test if _ctypes is not available.
https://hg.python.org/cpython/rev/80fc40a9ae47

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Larry Hastings]

Larry Hastings added the comment:

It's fixed!  So it's finally closed.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 1744d65705d0 by Yury Selivanov in branch '3.5':
whatsnew/3.5: Describe changes in issue #22980
https://hg.python.org/cpython/rev/1744d65705d0

New changeset cfbcb3a6a848 by Yury Selivanov in branch 'default':
Merge 3.5 (issue #22980, whatsnew/3.5)
https://hg.python.org/cpython/rev/cfbcb3a6a848

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Yury Selivanov]

Yury Selivanov added the comment:

Larry, Matthias, Steve, Berker - I've mentioned this issue in the whatsnew 
(applied Larry's patch with some modifications to address comments from Steve 
and Matthias).  Please review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

There's no dot before the debug marker on Windows. Otherwise, looks good to me. 
Thanks for writing this up.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Larry Hastings]

Larry Hastings added the comment:

Here's an attempt at a What's New section for this change.  I expect it's 
wrong!   Maybe someone can fix it.  Maybe it's actually better than not having 
one at all.

Can we maybe get a round or two of edits on this and get something in for 3.5 
final?

--
Added file: 
http://bugs.python.org/file40420/larry.whatsnew35.ext.module.suffix.diff.1.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Berker Peksag]

Berker Peksag added the comment:

Adding Yury since he and Elvis are working on Doc/whatsnew/3.5.rst and they 
might want to take a look at the latest patch.

--
nosy: +berker.peksag, yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

Only thing I'd add is that the extra tag is optional (on Windows at least), and 
Python will happily import extensions without it. But extensions with a 
mismatched tag won't be loaded.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

thanks for the draft!

I'm not sure how to describe this properly. The extension names are derived 
from https://wiki.debian.org/Multiarch/Tuples

and this again is derived from the GNU triplets/quadruplets.

there is no "cpu" and "os" part, depending on the architecture some ABI parts 
are either encoded in the "cpu" part or the "os" part.

So what about just enumerating the most common cases (i386-linux-gnu, 
x86_64-linux-gnu, arm-linux-gnueabi (still used for the old Raspberry Pi), 
arm-linux-gnueabihf), and then point to the "spec"? The above examples have 
some irregular cases, most other cases just follow the triplets.

I wouldn't mention x86_64-linux-gnux32 explicitly. Until now there are only 
unreleased or experimental distros.

Not sure if it is worth mentioning that this would allow distributing "fat" 
wheels.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Larry Hastings]

Larry Hastings added the comment:

I'm leaving this open just because we're apparently waiting on some "What's 
New" docs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Larry Hastings]

Larry Hastings added the comment:

I sure hope not.

--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Ned Deily]

Ned Deily added the comment:

I think we should add something to the 3.5 What's New document about these 
changes and which platforms are affected.  Otherwise is there anything left to 
do before closing?

--
priority: normal - deferred blocker

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 558335559383 by doko in branch 'default':
- #22980: fix triplet configure test for more targets
https://hg.python.org/cpython/rev/558335559383

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

 I'm not trying to discredit any use cases, I just don't see them.

so why do you see this on x86 for 32/64bit, but not for ARM 
soft-float/hard-float.  The example given was pretty clear.

 All Linux distributions I know place the 32-bit and 64-bit versions
 of shared libs into different directories rather than putting them all
 into a single dir and adding ABI flags to the .so files.

Well, then at least you don't know Debian, Ubuntu, and any of their derivates.

And even the Python default install on Linux installs into the same place.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

I'm not sure I like the way the simple idea Antoine has now resulted in a 
complete mess across platforms.

None of this is documented anywhere except on this ticket.

Since the change is a major one for anyone creating installers, packagers or 
otherwise dealing with compile extensions, I think we will need a PEP 
documenting the changes - and perhaps even have this discussed in retrospect on 
python-dev, possibly updating some of the changes already checked in.

Reopening the issue until this is done and the PEP accepted.

--
resolution: fixed - 
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 18:12, Nick Coghlan wrote:
 
 I can tell you exactly where these files need to live side by side: index 
 servers. We currently paper over it on PyPI for Windows and Mac OS X by 
 leveraging the implicitly defined ABI compatibility of the CPython binary 
 releases published on python.org, but that starts to break down as soon as 
 people are using interpreter binaries built locally or by redistributors 
 rather than by the CPython core development team. There's more to the 
 computing world than Intel CPU architectures, and more kernel APIs than 
 Linux, Darwin and Windows.

But that's a completely different use case than what's being
addressed in this ticket:

This ticket is about C extension files, not packages or custom
Python interpreter binary names.

For packages, I agree, we would benefit from a more standardized ABI
naming scheme that what setuptools or distutils currently have to offer.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

Nick filed issue #23966 to document these changes. Yes, these tags should be 
documented, so that installers don't have to guess (currently they are only 
exposed in importlib.machinery.EXTENSION_SUFFIXES. 

What you describe as a simple idea is just another hack, only addressing the 
issue on x86 platforms, not addressing this for soft-float/hard-float calling 
conventions, not addressing this for endianness, not addressing this for other 
platform ABIs. And for all these cases there are machines where you can run the 
variants on the same machine.  If you like to call this a mess, fine. But 
this is reality.  I'm not creating this mess,  I'm describing this and exposing 
it to the interpreter.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 17:30, Matthias Klose wrote:
 
 Matthias Klose added the comment:
 
 Nick filed issue #23966 to document these changes. Yes, these tags should be 
 documented, so that installers don't have to guess (currently they are only 
 exposed in importlib.machinery.EXTENSION_SUFFIXES. 
 
 What you describe as a simple idea is just another hack, only addressing 
 the issue on x86 platforms, not addressing this for soft-float/hard-float 
 calling conventions, not addressing this for endianness, not addressing this 
 for other platform ABIs. And for all these cases there are machines where you 
 can run the variants on the same machine.  If you like to call this a mess, 
 fine. But this is reality.  I'm not creating this mess,  I'm describing this 
 and exposing it to the interpreter.

The simple idea Antoine had was to be able to install C extensions
compiled for different bit architectures, but the *same platform*
into the same directory, which is similar to what we're doing for
Python byte code files.

The typical use case is to have a 32-bit version and a 64-bit version
on the same system.

It seems that the scope of this simple idea has now completely
blown up in trying to stuff all kinds of other platform features
into the binary names as well.

And while doing so, we now have different naming styles on different
platforms, require hand written configure files to support additional
platforms, and have yet another system for associating platform
information with binary Python files, in addition to
PEP 3149, PEP 425 and PEP 427.

See http://bugs.python.org/issue22980#msg232729

I don't think this is a good development and I can hardly imagine
a use case where all those different ABIs will have to live on the
same machine and in the same directory.

At the same time you are creating incompatibilities which did
not exist before, by requiring configure script fixes for unknown
platforms.

I'm -1 on these changes. I was +0 on Antoine's original idea,
since that addresses real life use case you can run into every
now and then.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Nick Coghlan]

Nick Coghlan added the comment:

I can tell you exactly where these files need to live side by side: index 
servers. We currently paper over it on PyPI for Windows and Mac OS X by 
leveraging the implicitly defined ABI compatibility of the CPython binary 
releases published on python.org, but that starts to break down as soon as 
people are using interpreter binaries built locally or by redistributors rather 
than by the CPython core development team. There's more to the computing world 
than Intel CPU architectures, and more kernel APIs than Linux, Darwin and 
Windows.

Fortunately, this platform ABI identification issue is not a new problem, but 
rather one where there's a rich history of existing expertise that we can draw 
from, rather than inventing our own custom solution that doesn't actually solve 
the problem properly. I agree it needs to be documented appropriately, perhaps 
including a PEP to explain *why* the changes are being made (rather than just 
documenting the end result), which is why I opened issue 23966.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

On 04/16/2015 05:56 PM, Marc-Andre Lemburg wrote:
 
 Marc-Andre Lemburg added the comment:
 
 On 16.04.2015 17:30, Matthias Klose wrote:

 Matthias Klose added the comment:

 Nick filed issue #23966 to document these changes. Yes, these tags should be 
 documented, so that installers don't have to guess (currently they are only 
 exposed in importlib.machinery.EXTENSION_SUFFIXES. 

 What you describe as a simple idea is just another hack, only addressing 
 the issue on x86 platforms, not addressing this for soft-float/hard-float 
 calling conventions, not addressing this for endianness, not addressing this 
 for other platform ABIs. And for all these cases there are machines where 
 you can run the variants on the same machine.  If you like to call this a 
 mess, fine. But this is reality.  I'm not creating this mess,  I'm 
 describing this and exposing it to the interpreter.
 
 The simple idea Antoine had was to be able to install C extensions
 compiled for different bit architectures, but the *same platform*
 into the same directory, which is similar to what we're doing for
 Python byte code files.
 
 The typical use case is to have a 32-bit version and a 64-bit version
 on the same system.
 
 It seems that the scope of this simple idea has now completely
 blown up in trying to stuff all kinds of other platform features
 into the binary names as well.
 
 And while doing so, we now have different naming styles on different
 platforms, require hand written configure files to support additional
 platforms, and have yet another system for associating platform
 information with binary Python files, in addition to
 PEP 3149, PEP 425 and PEP 427.
 
 See http://bugs.python.org/issue22980#msg232729
 
 I don't think this is a good development and I can hardly imagine
 a use case where all those different ABIs will have to live on the
 same machine and in the same directory.
 
 At the same time you are creating incompatibilities which did
 not exist before, by requiring configure script fixes for unknown
 platforms.
 
 I'm -1 on these changes. I was +0 on Antoine's original idea,
 since that addresses real life use case you can run into every
 now and then.

I'm disappointed that you discredit any other use case besides what you think as
the typical use case as not real life use case.  Maybe you are focused on x86
only, but if you've been to PyCon 2014, you should have a nice Raspberry Pi.
What do you run on it, a soft float, or a hard float distribution? How do you
distribute extensions for that? Yes, you can run both at the same time. There
are now the first 64bit Raspberry Pi like boards (https://www.96boards.org/).
Most of the SoCs can run ARM32 hard- and soft-float binaries, but not all, and
that's why AArch64 gets an ILP32 ABI too. Maybe you don't like the variety in
the ARM world, but's that real life.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 18:53, Matthias Klose wrote:
 I'm disappointed that you discredit any other use case besides what you think 
 as
 the typical use case as not real life use case.  Maybe you are focused on x86
 only, but if you've been to PyCon 2014, you should have a nice Raspberry Pi.
 What do you run on it, a soft float, or a hard float distribution? How do you
 distribute extensions for that? Yes, you can run both at the same time. There
 are now the first 64bit Raspberry Pi like boards (https://www.96boards.org/).
 Most of the SoCs can run ARM32 hard- and soft-float binaries, but not all, and
 that's why AArch64 gets an ILP32 ABI too. Maybe you don't like the variety in
 the ARM world, but's that real life.

I'm not trying to discredit any use cases, I just don't see them.

For package distributions you do need to make your distribution
files unique and it makes sense adding such complex ABI tags
to them, including even having to invest into manually maintaining
them.

However, for plain .so files that you have on your system (which will
mostly like not support more than 2-4 different architecture configurations
running at the same time), I don't currently see a need to make things
more complicated than necessary.

Perhaps you can point me to some use cases where the triple
platform tag is really useful.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Donald Stufft]

Donald Stufft added the comment:

 Perhaps you can point me to some use cases where the triple
 platform tag is really useful.

If I understand correctly (and ABI isn't my strong suite), it would be useful 
in the sense that you could utilize it to create a sort of fat wheel that 
included the .so's for multiple architectures and then pip could simply drop 
them all into place and have the interpreter decide which one to load. This is 
useful because maybe you have one .so in a wheel and 30 .py files, it's 
somewhat wasteful (both disk space and in cache efficiency) to have 10 
different wheel files and those 30 .py files duplicated when it could be 
possible to have a single one serving 10 different architectures.

To be clear, this ability doesn't yet exist in Wheel and I don't know of anyone 
pushing for it, but if Python is smart enough to load the right .so that makes 
fat wheels significantly easier to implement (in fact, you wouldn't need to add 
anything else to pip or the wheel spec to handle it I think).

--
nosy: +dstufft

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Ned Deily]

Ned Deily added the comment:

 Antoine's ticket is the first in two decades to request being
 able to install .so extension files side-by-side, so even if
 times and platforms change, people don't seem to have a big
 issues without this feature.

That's exactly what PEP 3149 was supposed to implement, isn't it?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 19:44, Donald Stufft wrote:
 
 Donald Stufft added the comment:
 
 Perhaps you can point me to some use cases where the triple
 platform tag is really useful.
 
 If I understand correctly (and ABI isn't my strong suite), it would be useful 
 in the sense that you could utilize it to create a sort of fat wheel that 
 included the .so's for multiple architectures and then pip could simply drop 
 them all into place and have the interpreter decide which one to load. This 
 is useful because maybe you have one .so in a wheel and 30 .py files, it's 
 somewhat wasteful (both disk space and in cache efficiency) to have 10 
 different wheel files and those 30 .py files duplicated when it could be 
 possible to have a single one serving 10 different architectures.

Well, it's even more wasteful if you have to download 100MB wheels
with all the different platforms when the dedicated wheel would just
need 1.5MB.

This approach has been considered a few times in distutils history
and no one really liked it because it would increase the download
requirements for the users a lot. It does make things easier for
the packages, though, but then again, this can also be had
by having different subdirs in the wheel or other package
format to address the issue of having name collisions.

Today, you usually have a web installer take care of grabbing
only the bits you need.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 19:14, Marc-Andre Lemburg wrote:
 However, for plain .so files that you have on your system (which will
 mostly like not support more than 2-4 different architecture configurations
 running at the same time), I don't currently see a need to make things
 more complicated than necessary.
 
 Perhaps you can point me to some use cases where the triple
 platform tag is really useful.

Antoine's ticket is the first in two decades to request being
able to install .so extension files side-by-side, so even if
times and platforms change, people don't seem to have a big
issues without this feature.

If you have a need, it's not really hard to build your extensions
for different architecture ABIs in different directories. We've
been doing this for years, just like everyone else.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 19:47, Ned Deily wrote:
 
 Ned Deily added the comment:
 
 Antoine's ticket is the first in two decades to request being
 able to install .so extension files side-by-side, so even if
 times and platforms change, people don't seem to have a big
 issues without this feature.
 
 That's exactly what PEP 3149 was supposed to implement, isn't it?

No, PEP 3149 is about the Python ABI, following PEP 3147,
which implements this for PYC files.

The intent is to be able to have mutliple *Python* ABI/API versions
installed side-by-side, not multiple platform ABI versions :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 20:17, Donald Stufft wrote:
 
 Donald Stufft added the comment:
 
 Well, it's even more wasteful if you have to download 100MB wheels
 with all the different platforms when the dedicated wheel would just
 need 1.5MB.
 
 I think it's going to vary greatly based on how many platforms you're 
 attempting to support and how big your .so's are compared to the rest of the 
 Wheel. You can also mix and match, do a single bundle for the most popular 
 platforms (which will mean that you're almost always serving out of cache) 
 but then do individual wheels for the less popular platforms to keep the file 
 size of the main wheel from bloating up with a bunch of .so's for platforms 
 which are unlikely to be needed very often.

Whatever you do, you're still going to force all your main users to
download things they don't need, so I don't see the argument of
optimizing downloads or caches.

 Another possible (future) benefit - Right now we have executable zip files, 
 but they can only reasonably contain pure Python files. There are rumblings 
 of making it so it's possible to import .so's from inside of an executable 
 zip file. If you bake in the platform ABI into the .so file name, it would 
 mean in that possible future you could have a single executable zip file that 
 just works across multiple platforms as long as you already have Python 
 installed.

Since you need special support for such ZIP files (either using dlopen
hacks or temporarily extracting them), you might as well deal with
the platform dependencies in that handler. No need to force the
platform tags on all your .so file for no apparent reason.

There's a very real use case for having multiple Python versions
installed which was the motivation for the PEPs I quoted, but this
development is one of those YAGNI features only very few people
will ever need.

 I do agree that pretty much every place someone would want to do this, could 
 possibly be implemented by having it look inside a per platform directory 
 (you could implement fat wheels for instance by having platform sub dirs, 
 same with a single executable zip file), however doing that causes 
 duplication because every place you deal with .so's then need to deal with 
 the question of platform ABI and have to come up with their own solution to 
 it, instead of having a central solution which is managed by Python itself 
 and can be re-used by all of these situations.

I'm not saying that having a central solution is wrong. All I'm
saying is that the implementations on this ticket are not within
the scope of the ticket and instead need a proper PEP to see where the
real use cases are and whether this particular way of doing things
is a way we all want to go.

We now have four ways of describing ABI flags in Python (well, actually
even more, since Linux, Windows and OX S use different approaches for
the platform ABI .so flags). This can't possibly be a good approach.

I can already see all the different OS vendors creating
their own little platform triplet extensions. In the end, it's rather
likely that an extension compiled with eg. openSUSE won't run on Fedora or
Debian anymore and vice-versa; and one compiled with vanilla Python
probably won't run on Apples' Python anymore for similar reasons.
Not a good perspective. This is going to make distributing binaries
harder, not easier.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
nosy:  -brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.04.2015 20:21, Ned Deily wrote:
 
 Ned Deily added the comment:
 
 No, PEP 3149 is about the Python ABI, following PEP 3147,
 which implements this for PYC files.
 
 The intent is to be able to have mutliple *Python* ABI/API versions
 installed side-by-side, not multiple platform ABI versions :-)
 
 Well, for all practical purposes, the platform *is* part of the ABI :=)

Yes, but if all your files on your box share the same ABI, do you
really want to have all of them come with an extra name extension ?
I mean: If all your apples are green, would you write green on them to
remember ? ;-)

All Linux distributions I know place the 32-bit and 64-bit versions
of shared libs into different directories rather than putting them all
into a single dir and adding ABI flags to the .so files.
Windows does this too. FreeBSD as well.

Why should Python behave differently ? Just because we can is not
really a good answer, IMO.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 17.04.2015 00:51, Donald Stufft wrote:
 
 Since you need special support for such ZIP files (either using dlopen
 hacks or temporarily extracting them), you might as well deal with
 the platform dependencies in that handler. No need to force the
 platform tags on all your .so file for no apparent reason.
 
 There are other reasons as have already been mentioned, this is just yet
 another reason (and on it's own I'd agree it's not a sufficiently compelling
 use case), but when I see a pattern of things which all need the same thing
 then that speaks to me that it should live someplace centrally instead of
 having each one reimplement it. 

Sure, but whatever the central implementation is going to be,
it doesn't necessarily have to require sticking platform ABI flags
on all .so files, even those which will never need to be installed
side-by-side. The more paths you need to stat when searching
a shared mod, the slower Python will get.

There's a very simple trick which some packages used in the
past for sumo distributions - you simply modify the __path__
attribute of the package to point to the platform dependent
files in the __init__.py file and Python will then automagically
use the right C extensions.

To simplify this, the platform triplets and other platform ABI flags
could be made available via the sys or sysconfig module for importers
and other tools to pick up.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Donald Stufft]

Donald Stufft added the comment:

 Whatever you do, you're still going to force all your main users to
 download things they don't need, so I don't see the argument of
 optimizing downloads or caches.

pip caches downloads by default, many systems are starting to utilize that
cache in order to stop repeat downloads of the same file. This would make it
so that if you had a shared pip cache amongst many archiectures or platforms
(which people are starting to do, especially with sharing caches with virtual
boxes running on their own machines or services like Travis-CI) you'd only have
to download that file from PyPI once ever.

Looking at a few of the top projects on PyPI in terms of download count we
have:

Of those, only really lxml is large enough that adding a second or third or
fourth copy of the .so is a really meaningful increase in size and since we
wouldn't be making a fat wheel mandatory lxml could just decide not to build
one. As far as I can tell we don't actually optimize for maximizing the amount
downloading (otherwise we'd use something better than a .zip file).

 Since you need special support for such ZIP files (either using dlopen
 hacks or temporarily extracting them), you might as well deal with
 the platform dependencies in that handler. No need to force the
 platform tags on all your .so file for no apparent reason.

There are other reasons as have already been mentioned, this is just yet
another reason (and on it's own I'd agree it's not a sufficiently compelling
use case), but when I see a pattern of things which all need the same thing
then that speaks to me that it should live someplace centrally instead of
having each one reimplement it. 

 I'm not saying that having a central solution is wrong. All I'm
 saying is that the implementations on this ticket are not within
 the scope of the ticket and instead need a proper PEP to see where the
 real use cases are and whether this particular way of doing things
 is a way we all want to go.

I don't care if it gets added as part of this ticket, another ticket, or as
a PEP. I'm just listing where it'd be useful for the kinds of things I do.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Donald Stufft]

Donald Stufft added the comment:

 Well, it's even more wasteful if you have to download 100MB wheels
 with all the different platforms when the dedicated wheel would just
 need 1.5MB.

I think it's going to vary greatly based on how many platforms you're 
attempting to support and how big your .so's are compared to the rest of the 
Wheel. You can also mix and match, do a single bundle for the most popular 
platforms (which will mean that you're almost always serving out of cache) but 
then do individual wheels for the less popular platforms to keep the file size 
of the main wheel from bloating up with a bunch of .so's for platforms which 
are unlikely to be needed very often.

Another possible (future) benefit - Right now we have executable zip files, but 
they can only reasonably contain pure Python files. There are rumblings of 
making it so it's possible to import .so's from inside of an executable zip 
file. If you bake in the platform ABI into the .so file name, it would mean in 
that possible future you could have a single executable zip file that just 
works across multiple platforms as long as you already have Python installed.

I do agree that pretty much every place someone would want to do this, could 
possibly be implemented by having it look inside a per platform directory (you 
could implement fat wheels for instance by having platform sub dirs, same with 
a single executable zip file), however doing that causes duplication because 
every place you deal with .so's then need to deal with the question of platform 
ABI and have to come up with their own solution to it, instead of having a 
central solution which is managed by Python itself and can be re-used by all of 
these situations.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Ned Deily]

Ned Deily added the comment:

 No, PEP 3149 is about the Python ABI, following PEP 3147,
 which implements this for PYC files.

 The intent is to be able to have mutliple *Python* ABI/API versions
 installed side-by-side, not multiple platform ABI versions :-)

Well, for all practical purposes, the platform *is* part of the ABI :=)

So, if we have been supporting multiple P@P 3147 extension modules since 3.2, I 
don't see this as a risky change.  I don't think anyone is advocating 
installing distributions with dozens of extension module variants as a general 
practice.  But it seems like there are times when it would be useful to have 
the capability to have more than one and this seems like a safe and logical 
extension to what PEP 3147 already provides.  I don't have a strong opinion 
about other platforms.

For OS X, because of the complexity and usefulness of mixing and matching 
various fat CPU archs and OS X ABIs (deployment target), pip already supports 
selecting the proper wheel to download and wheel creators are tagging with the 
right metadata to make the right decisions, so I don't think the changes here 
bring much added value for OS X users, except for two things: (1) we now 
support PEP 3147 ext file names (which for some reason was never fully 
implemented on OS X and should have been) which is useful for the original PEP 
3147 use cases (for example, if someone wants to distribute non-debug and debug 
versions of ext modules); (2) the addition of '-darwin' to the PEP 3147 ext 
file name allows for this additional use case of allowing multiple platform 
extensions to be stored in the same directory without fear of name clash (even 
if only one is eventually installed).  I think both are reasonable and safe 
changes for OS X.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Nick Coghlan]

Nick Coghlan added the comment:

Matthias's latest patch looks good to me - I'm filing a separate issue about 
the fact that the native and cross-platform build info isn't really exposed 
properly at the Python level, and is confusing in various ways.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 11c4f3f936e7 by doko in branch 'default':
- Issue #22980: Under Linux, GNU/KFreeBSD and the Hurd, C extensions now include
https://hg.python.org/cpython/rev/11c4f3f936e7

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset df6c73f0e375 by doko in branch 'default':
- #22980: fix typo in Lib/test/test_sysconfig.py triplet test
https://hg.python.org/cpython/rev/df6c73f0e375

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

now fixed on the trunk. opened a new issue #23969 for setting the SOABI on 
MacOSX.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 948745d0d9cf by doko in branch 'default':
#22980: fix triplet configure test for powerpc-linux-gnu
https://hg.python.org/cpython/rev/948745d0d9cf

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 32c24eec035f by Ned Deily in branch 'default':
Issues #22980, 23969: For OS X, use PEP 3149-style file names for extension
https://hg.python.org/cpython/rev/32c24eec035f

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy:  -haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

updated patch and test case.

Nick's suggestion to use platform.machine() for the test is wrong. This would 
test for the environment, not for the just built binary. Try to run a 32bit 
executable on a 64bit kernel, you'll see x86_64.  Same thing with 
sysconfig.get_platform().  So if anybody is using this to determine the names 
of wheels, that can go only wrong for 32bit distributions.

--
Added file: http://bugs.python.org/file39027/ma2.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

On 04/13/2015 11:43 PM, Nick Coghlan wrote:
 
 Nick Coghlan added the comment:
 
 Maintaining the arch list can be delegated to the platform maintainers, but I 
 agree a test would be valuable.
 
 I'd suggest a test in the platform module tests that cross-checks the 
 settings in SOABI. This will also be useful in defining an algorithm that 
 setuptools can backport to Python 2.7 for naming wheel files correctly.
 
 For the new multiarch triplets, I believe the first two parts should match 
 platform.machine() and platform.system().lower(). I'm not sure what the third 
 part should match.

no, platform.machine() doesn't return any normalized machine name. At least for
ix86 (i386, i486, i586), ARM32 (armv4, armv5+, armv7, ...), ALPHA, and MIPS.

and testing for the second part only seems to be a bit odd.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 thought about it, but we only change the SOABI name, and all tests
 should behave
 as before. And because we try to load extensions without the SOABI
 name, there
 is safe fallback.

Still, we should somehow check that a non-empty triplet is included in the 
extension filename (see the test your patch removes).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

here is the more general approach encoding the triplet into the extension name. 
 This is tested with GCC and clang (clang at least on x86_64, x86, powerpc, 
powerpc64le, arm and aarch64.

--
Added file: http://bugs.python.org/file38964/ma.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 here is the more general approach encoding the triplet into the extension 
 name.

Thanks. Two comments:
- is it possible to avoid the hardcoded list? Maintaining this will be a burden
- it would be nice to have a test

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

On 04/13/2015 11:01 PM, Antoine Pitrou wrote:
 
 Antoine Pitrou added the comment:
 
 here is the more general approach encoding the triplet into the extension 
 name.
 
 Thanks. Two comments:
 - is it possible to avoid the hardcoded list? Maintaining this will be a 
 burden

Well, there are two approaches, the one I didn't choose is to derive the triplet
from the $host macro, because people like to build with abi flags as build
parameters, so I choose to entirely base these checks on compiler information.
Currently if you have a new linux architecture you fall back to the same SOABI
as before, which sounds like an acceptable solution.  Or you could fallback to
the $host macro, removing the vendor information.  But then you have two
different methods ...  or we could emit a hard error for unknown linux and
kfreebsd (Debian only) targets, and then extend the check for new architectures.

 - it would be nice to have a test

thought about it, but we only change the SOABI name, and all tests should behave
as before. And because we try to load extensions without the SOABI name, there
is safe fallback.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Nick Coghlan]

Nick Coghlan added the comment:

Maintaining the arch list can be delegated to the platform maintainers, but I 
agree a test would be valuable.

I'd suggest a test in the platform module tests that cross-checks the settings 
in SOABI. This will also be useful in defining an algorithm that setuptools can 
backport to Python 2.7 for naming wheel files correctly.

For the new multiarch triplets, I believe the first two parts should match 
platform.machine() and platform.system().lower(). I'm not sure what the third 
part should match.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

@barry, worth updating PEP 3149?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Ad hoc introduction of the bitness is at least wrong or not helpful
 for x32 and ARM ilp32.

Perhaps it's not helpful for those ABIs / architectures, but I don't see how 
it's worse than the status quo.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Matthias Klose]

Matthias Klose added the comment:

I plan to revert this fix and replace it with a more general solution, at 
least for POSIX based systems. Ad hoc introduction of the bitness is at least 
wrong or not helpful for x32 and ARM ilp32.

(currently at PyCon, and would like to discuss this at the sprint days)

--
nosy: +doko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset 25356d34b79b by Antoine Pitrou in branch 'default':
Issue #22980: Under Linux, C extensions now include bitness in the file name,
https://hg.python.org/cpython/rev/25356d34b79b

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

I've pushed the patch for Linux. I'm keeping this issue open in case other 
platforms may (want to) benefit.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

So, PEP 3149 claims EXT_SUFFIX will always contain the ABI tag, but 
configure.ac makes it Linux-specific (?!):

AC_SUBST(EXT_SUFFIX)
case $ac_sys_system in
Linux*|GNU*)
EXT_SUFFIX=.${SOABI}${SHLIB_SUFFIX};;
*)
EXT_SUFFIX=${SHLIB_SUFFIX};;
esac

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is a patch adding the bitness to the ABI tag under Linux. This is 
permitted by PEP 3149: Python implementations MAY include additional flags 
in the file name tag as appropriate.

--
Added file: http://bugs.python.org/file38155/abi_bitness.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

 Could you explain what replacing the _d suffix with a d ABI flag
 would break ?

It breaks consistency with python_d.exe and python35_d.dll, mainly, and those 
are not going to change (specifically, python.exe and python35.dll are not 
going to change, and so there's no point changing the debug versions any more 
than with _d).

It would also break consistency with untagged PYDs, which are still supported 
but still need to distinguish between debug and release builds. Currently the 
accepted suffixes are _d.cp35-{plat}.pyd and _d.pyd. While we could start 
requiring at least cp3d as the API tag, that would break a lot of development 
environments for no gain.

So basically, I see compelling reasons for allowing the platform in the name, 
good enough reasons for the version, but not good enough reasons for changing 
the existing debug tag (and no reason for the m flag at all).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.12.2014 05:54, Steve Dower wrote:
 Nobody seemed too bothered by it, so I committed a slightly simpler change 
 that only includes the most specific tag (that is, .cp35-win32.pyd or 
 .pyd). We can always add another tag easily enough if it seems useful, or 
 roll this change back if it was a mistake.

I'm not sure the format you've chosen is a good idea. We now have
a different tag for .pyc (PEP 3149) and .pyd files, in addition
to yet another tag format defined in wheels (PEP 425 and PEP 427).

The .pyd format looks similar to PEP 425 abi tag + platform tag,
but it's missing the abi flags.

IMO, we should use one of the available PEP standards instead
of creating yet another variant.

The checkin also only adds import support for the tags.

Shouldn't distutils also be changed to create such .pyd files
per default or at least via an option ?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

I justified leaving out the ABI tag in an earlier post as well as in an email 
to distutils-sig, where two of the PEPs you mention were developed, and nobody 
had any comment. Cache tags don't include platform information and are 
worthless here.

distutils uses the first extension from _imp.extension_suffixes() and so will 
automatically tag binaries. I also tested with setuptools and Cython.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 16.12.2014 14:38, Steve Dower wrote:
 
 Steve Dower added the comment:
 
 I justified leaving out the ABI tag in an earlier post as well as in an email 
 to distutils-sig, where two of the PEPs you mention were developed, and 
 nobody had any comment. Cache tags don't include platform information and are 
 worthless here.

I was referring to ABI flags 
(https://www.python.org/dev/peps/pep-0425/#abi-tag),
eg. the m when using pymalloc. If we do stick with a PEP 425 format,
we should at least stay compatible to it - even if we don't necessarily
need this on Windows.

On Linux, the extensions use the PEP 3149 tags, e.g.

readline.cpython-34m.so

Wouldn't it make sense to change those to the same PEP 425 style
format ?

Exmaple:

   readline.cp35m-linux-x86_64.so

The _d suffix on Windows would then turn into the d ABI flag.

Note that this only works nicely for Linux. For other platforms,
get_platform() returns a mess... e.g. on FreeBSD:

   freebsd-8.3-RELEASE-p3-i386   (32-bit)
   freebsd-8.3-RELEASE-p3-amd64  (64-bit)

instead of the more useful

   freebsd-i386
   freebsd-amd64

The situation is similar on other less main stream platforms,
so it may make sense to define mappings other than get_platform()
on those.

 distutils uses the first extension from _imp.extension_suffixes() and so will 
 automatically tag binaries. I also tested with setuptools and Cython.

Ah, clever. So no additional patches are needed :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

get_platform() will be difficult to reuse, for bootstrapping reasons (you need 
to know the allowed extensions as soon as the interpreter starts up, or at 
least as soon as the first extension module may be loaded).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

 get_platform() will be difficult to reuse, for bootstrapping reasons

ISTM that if you can't determine the value at compile time, then it doesn't 
matter to compilation enough to need to appear in the tag.

As far as matching PEP 425 for the sake of matching it goes, I'd rather keep 
using search path tricks than have .cp35-cp35m-win_amd64.pyd appear on every 
single one of my extension modules. Removing the _d suffix is very likely 
more disruptive than it's worth, especially since untagged pyds are still 
supported but the debug tag is still necessary. 'm' is always the case in 
Windows and is benign for a correct extension anyway, and AFAICT 'u' is totally 
unused.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: [issue22980] C extension naming doesn't take bitness into account

[M.-A. Lemburg]

On 16.12.2014 21:28, Steve Dower wrote:
 
 Steve Dower added the comment:
 
 get_platform() will be difficult to reuse, for bootstrapping reasons
 
 ISTM that if you can't determine the value at compile time, then it doesn't 
 matter to compilation enough to need to appear in the tag.

Antoine has a point there. Together with the problems I mentioned
with non-mainstream platforms, it would be better to use a compile
time definition of the platform tag as you've chosen for Windows.

PEP 425 unfortunately ignores the mentioned problems of get_platform().

 As far as matching PEP 425 for the sake of matching it goes, I'd rather keep 
 using search path tricks than have .cp35-cp35m-win_amd64.pyd appear on 
 every single one of my extension modules. Removing the _d suffix is very 
 likely more disruptive than it's worth, especially since untagged pyds are 
 still supported but the debug tag is still necessary. 'm' is always the case 
 in Windows and is benign for a correct extension anyway, and AFAICT 'u' is 
 totally unused.

You may have misread my comment. There's no need to add the Python
tag to the extension tag, only the ABI tag (with flags) and the
platform tag to determine bitness:

spam.cp35m-win_amd64.pyd

Could you explain what replacing the _d suffix with a d ABI flag
would break ?

To be clear, this would mean:

spam.cp35dm-win_amd64.pyd

instead of

spam_d.cp35m-win_amd64.pyd

-- 
Marc-Andre Lemburg
eGenix.com

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Nick Coghlan]

Nick Coghlan added the comment:

As far as PEP 425 goes, we handwaved a *lot* in order to be able to improve the 
user experience of the CPython Windows and Mac OS X binary installers by 
pairing them up with wheel files on PyPI. That handwaving is the key reason 
wheels for other platforms are still disallowed: the compatibility tags simply 
don't provide the information they need to provide for installers to make good 
decisions.

I'd love to see a clear, needs based, revision of the compatibility tagging 
specification, that covers both source tagging and binary tagging, and gets 
away from the reliance on distutils.get_platform(). It's just a rather tedious 
exercise (as this issue shows...)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Roundup Robot]

Roundup Robot added the comment:

New changeset f70c16189876 by Steve Dower in branch 'default':
#22980 Adds platform and version tags to .pyd files
https://hg.python.org/cpython/rev/f70c16189876

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

Nobody seemed too bothered by it, so I committed a slightly simpler change that 
only includes the most specific tag (that is, .cp35-win32.pyd or .pyd). We 
can always add another tag easily enough if it seems useful, or roll this 
change back if it was a mistake.

Now, let's stop talking about Windows and get back to the original discussion :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[STINNER Victor]

STINNER Victor added the comment:

Why not using on Windows the same naming scheme than Unix. It may be useful
to also add the debug flag (d) in the name for example.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

 Why not using on Windows the same naming scheme than Unix.
 It may be useful to also add the debug flag (d) in the name for example.

Windows already puts the debug flag in the name, the fact that it's CPython is 
in the .pyd extension, and the version number is in the directory for all the 
standard sys.path locations. Platform is the only variable unaccounted for.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Windows already puts the debug flag in the name, the fact that it's
 CPython is in the .pyd extension, and the version number is in the
 directory for all the standard sys.path locations.

The version number would be useful for in-place builds (i.e. when developping), 
but the patch is still a nice step forward. Could you perhaps add some tests 
(e.g. in test_distutils)?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

 The version number would be useful for in-place builds (i.e. when developping)

Ah, I see now how that would be useful (I haven't tried to deal with that 
before). I'll revise the patch to use/allow the version number.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

Added cp35 (and later cp36, etc.) to the tag, so now it looks similar to 
PEP 425 without the ABI tag (ironically, since there's fundamentally no 
difference between the Python version and the ABI). cp3 is also accepted for 
stable ABI extensions that still need a platform specifier.

distutils will always build extensions with the most specific tag, which I 
believe is best. If you want a different tag, then you currently need to do the 
build manually (unless there's a way to override the suffix 
distutils/setuptools uses? It didn't look like an obvious option)

I also added the tag into the stdlib pyds. This is kind of nice, but not really 
necessary. We don't (and probably can't) tag python.exe and python35.dll, so 
there's no opportunity to install different version/platform interpreters in 
the same directory anyway. It's a trivial change to remove, though it does 
further help test the mechanism (along with an importlib test to validate the 
tag value).

Example names with tags:

spam.cp35-win_amd64.pyd
spam_d.cp35-win32.pyd
spam.cp3-win32.pyd

--
Added file: http://bugs.python.org/file37384/22980_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

What can I do to help move this along?

It sounds like for Windows builds we could change _imp.extension_suffixes() 
from ['.pyd'] to ['.{}.pyd'.format(distutils.util.get_platform()), '.pyd'] and 
update distutils to produce the more specific name (I've got some work to do on 
distutils anyway for 3.5, so I'm happy to do this part). This would also 
include somehow hard-coding the get_platform() result into the executable 
(probably a #define in pyconfig.h)

I'm more inclined towards get_platform() than adding new architecture tags. 
Windows at least doesn't support fat binaries - the closest equivalent is 
universal apps, which use separate binaries and a naming convention. Adding a 
debug marker here would also be nice, as I've never been a huge fan of the _d 
suffix we currently have, but it's not a big deal.

I suspect any changes here would be completely separate from other platforms, 
but ISTM that we're looking at a similar change to handle the bitness/debug 
issue on Linux. I'm not volunteering to do that part :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 06/12/2014 21:11, Steve Dower a écrit :
 
 I suspect any changes here would be completely separate from other
platforms, but ISTM that we're looking at a similar change to handle the
bitness/debug issue on Linux. I'm not volunteering to do that part :)

I think committing changes on a per-platform basis is fine here. After
all the current scheme is quite platform-specific (I was unaware of this
until a few days ago :-)).

So, yes, let's get the ball rolling under Windows. I think you're the
most competent person to choose a naming scheme!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

The attached patch adds platform tags for .pyd files for win32, win-arm, 
win-amd64 and win-ia64, which are the known compilers in pyconfig.h and the 
potential return values from distutils.util.get_platform(). It also fixes a bug 
where the suffix would be incorrect if building a debug extension.

I haven't been able to think of any scenarios where this could break other than 
perhaps packaging (since distutils defaults to including the tag), and we've 
got plenty of time to sort those issues out. A quick test installing Cython and 
some packages built with Cython seemed to be fine. AIUI, MinGW/cygwin builds 
won't use PC/pyconfig.h, and so they won't see any change.

--
keywords: +patch
Added file: http://bugs.python.org/file37379/22980_windows.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

New submission from Antoine Pitrou:

Currently, C extensions are named something like _helperlib.cpython-34dm.so. 
This doesn't take into account the bitness of the interpreter (32- vs. 64-bit), 
which makes it awkward to use the same working copy with two different 
interpreters (you have to rebuild everything each time you switch bitnesses).

Worse, under Windows it seems ABI tags aren't even used, giving generic names 
such as _helperlib.pyd. Why is that?

--
messages: 232000
nosy: barry, brett.cannon, eric.snow, ncoghlan, pitrou, steve.dower, 
tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: C extension naming doesn't take bitness into account
type: enhancement
versions: Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[STINNER Victor]

STINNER Victor added the comment:

See also the PEP 3149.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

PEP 3149 says It is not currently clear that the facilities in this PEP are 
even useful for Windows. Well, it seems I have found a use for it :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

Ideally, we would use distutils.util.get_platform(). However, there are two 
cases where it relies on other modules:
- the re module under CygWin
- the sysconfig and _osx_support under OS X

Of course, ideally we should be able to hardcode this into the compiled CPython 
executable...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +ned.deily

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

As a side-note, it is interesting to note that Python currently wrongly 
identifies 32-bit builds under 64-bit Linux:

Python 3.5.0a0 (default:64a54f0c87d7, Nov  2 2014, 17:18:13) 
[GCC 4.9.1] on linux
Type help, copyright, credits or license for more information.
 import sys, os, sysconfig
 sys.maxsize
2147483647
 os.uname()
posix.uname_result(sysname='Linux', nodename='fsol', 
release='3.16.0-25-generic', version='#33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 
2014', machine='x86_64')
 sysconfig.get_platform()
'linux-x86_64'

AFAIU, sysconfig.get_platform() (or its sibling distutils.util.get_platform()) 
is used for the naming of binary distributions...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

The MULTIARCH variable can help at least under Linux:

 import sysconfig
 sysconfig.get_platform()
'linux-x86_64'
 sysconfig.get_config_var('MULTIARCH')
'i386-linux-gnu'

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[STINNER Victor]

STINNER Victor added the comment:

There is also platform.architecture(). I don't like its implementation, it 
relies on the external file program :-(

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

I'm very much in favor of adding this for .pyds on Windows.

I assume the hard part will be getting the details for Linux (doesn't bitness 
have to be compiled in there? For Windows it can be determined at 
compile-time...), but preferring an extension with the ABI tag and falling back 
on one without seems easy enough.

(Would/could this also work for .py files? So a 2.7/3.x or 
Jython/CPython/IronPython package could include tags in pure-Python code files?)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Note that there's a difference between the platform's architecture (which is 
what get_platform() returns) and the pointer size of the currently running 
Python executable.

On 64-bit Linux, it's rather rare to have an application built as 32-bit 
executable. On 64-bit Windows, it's rather common to have 32-bit applications 
running.

The best way to determine 32-bit vs. 64-bit is by using the struct module:

# Determine bitsize used by Python (not necessarily the same as
# the one used by the platform)
import struct
bits = struct.calcsize('P') * 8

This should be portable across all platforms and will always refer to the 
pointer size of the currently running Python executable.

--
nosy: +lemburg

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Note that there's a difference between the platform's architecture

Yes, that's pointed out above.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

Nothing new should be necessary for pyc files under Windows:

Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 22 2014, 11:51:45) [MSC 
v.1600 64 bit (AMD64)] on win32
Type help, copyright, credits or license for more information.
 import sys
 sys.implementation.cache_tag
'cpython-34'

The problem is with C extensions:

 import _imp
 _imp.extension_suffixes()
['.pyd']

Compare with Linux:

 import _imp
 _imp.extension_suffixes()
['.cpython-35dm.so', '.abi3.so', '.so']

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 I assume the hard part will be getting the details for Linux (doesn't bitness 
 have to be compiled in there? For Windows it can be determined at compile-
 time...), but preferring an extension with the ABI tag and falling back on
 one without seems easy enough.

Sticking to bitness should be easy (although I wonder if it would be desirable 
for platforms with fat binaries - Ned?). If we can go the extra mile and 
include platform identification all the better, of course.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

I was more interested in source file resolution than bytecode caching. If 
Python 3.5 would prefer spam.cpython-35.py or spam.cpython-3.py over 
spam.py and Python 2 preferred spam.py, then I can more easily separate the 
code that won't parse in the alternative.

Happy to be told it's unrelated and I should raise it separately, but from my 
POV resolving .pyd filenames looks very similar to resolving .py files.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 02.12.2014 19:02, Antoine Pitrou wrote:
 Sticking to bitness should be easy (although I wonder if it would be 
 desirable for platforms with fat binaries - Ned?). If we can go the extra 
 mile and include platform identification all the better, of course.

I hear the can of worms alarm ringing :-)

Seriously, I think that putting platform infos into the file name
is bound to cause more trouble than it tries to solve. Fat builds
leave the decision to the linker, which is a good method and avoids
the file name clashes.

I think we should only focus on platforms where fat builds are
uncommon, while at the same time you do have to support multiple
architectures, like e.g. Windows:

http://en.wikipedia.org/wiki/Fat_binary

Note that on Linux, 32-bit and 64-bit versions are typically placed
into different directory trees:

http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

so I'm not sure whether it's a real problem on Linux.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

Fat binaries seem to exist under:
- OS X: yes, that's why I was asking for Ned's advice
- Linux: A proof-of-concept Ubuntu 9.04 image is available... enough said
- DOS: perhaps MicroPython is interested :-)

http://en.wikipedia.org/wiki/Fat_binary

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

But since you pointed out cache-tag, should that distinguish for bitness as 
well? It seems to be 'cpython-34' for both 32-bit and 64-bit interpreters on 
Windows, which isn't really a problem now, but may become one if we start 
allowing/encouraging sharing packages between interpreters.

In fact, it probably is an issue now with user site-packages, since that path 
is the same for both 32-bit and 64-bit...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 02.12.2014 19:40, Steve Dower wrote:
 
 I was more interested in source file resolution than bytecode caching. If 
 Python 3.5 would prefer spam.cpython-35.py or spam.cpython-3.py over 
 spam.py and Python 2 preferred spam.py, then I can more easily separate 
 the code that won't parse in the alternative.
 
 Happy to be told it's unrelated and I should raise it separately, but from my 
 POV resolving .pyd filenames looks very similar to resolving .py files.

That's an interesting idea, but indeed unrelated to this ticket :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Note that on Linux, 32-bit and 64-bit versions are typically placed
 into different directory trees

By whom? Our standard installer doesn't (it uses ../lib/python-X.Y for all 
builds).

Also, one of the problems (and actually the problem which triggered this 
tracker entry) is when doing development inside a working copy (either through 
setup.py develop or setup.py build_ext --inplace - both copy C extensions 
directly into the source tree).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

@Steve: IIRC, pyc files should be portable, so there's no need to differentiate 
between various bitnesses.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Steve Dower]

Steve Dower added the comment:

@Antoine: You're right. I hereby withdraw all contributions to this thread 
after my first statement of support :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 02.12.2014 19:46, Antoine Pitrou wrote:
 
 Note that on Linux, 32-bit and 64-bit versions are typically placed
 into different directory trees
 
 By whom? Our standard installer doesn't (it uses ../lib/python-X.Y for all 
 builds).

By the system vendors. Packages (with extensions) will automatically
pick up their configuration.

 Also, one of the problems (and actually the problem which triggered this 
 tracker entry) is when doing development inside a working copy (either 
 through setup.py develop or setup.py build_ext --inplace - both copy C 
 extensions directly into the source tree).

Fair enough; it's a rare use case, but may be worth supporting.

My main point was that we shouldn't start adding tags for e.g.
PPC, Intel, ARM, etc. since platforms needing to support multiple
such architectures will typically support fat builds anyway.

How about using these flags:

b0 - 16-bit
b1 - 32-bit
b2 - 64-bit
b3 - 128-bit
and so on

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 02/12/2014 19:59, Marc-Andre Lemburg a écrit :
 
 My main point was that we shouldn't start adding tags for e.g.
 PPC, Intel, ARM, etc. since platforms needing to support multiple
 such architectures will typically support fat builds anyway.
 
 How about using these flags:
 
 b0 - 16-bit
 b1 - 32-bit
 b2 - 64-bit
 b3 - 128-bit

Fair enough, although I think we only need 32-bit and 64-bit for now,
and 32b vs. 64b would probably be more readable :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 02.12.2014 20:10, Antoine Pitrou wrote:
 
 Antoine Pitrou added the comment:
 
 Le 02/12/2014 19:59, Marc-Andre Lemburg a écrit :

 My main point was that we shouldn't start adding tags for e.g.
 PPC, Intel, ARM, etc. since platforms needing to support multiple
 such architectures will typically support fat builds anyway.

 How about using these flags:

 b0 - 16-bit
 b1 - 32-bit
 b2 - 64-bit
 b3 - 128-bit
 
 Fair enough, although I think we only need 32-bit and 64-bit for now,
 and 32b vs. 64b would probably be more readable :-)

True, I'm just not sure what the parsing requirements are and
the ABI version names are too long already, IMO. PEP 425 used
a nice short variant for the Python part.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[STINNER Victor]

STINNER Victor added the comment:

Would it be possible to add something to the sys module, computed
during the compilation, instead of having to rely on platform,
sysconfig, struct or something else?

Note: There is also the funnny x32 platform project :-)
https://sites.google.com/site/x32abi/ 32-bit pointer on 64-bit CPU.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22980] C extension naming doesn't take bitness into account

[Nick Coghlan]

Nick Coghlan added the comment:

My initial thought is to add an abitags attribute to sys.implementation
(plural so we can also indicate stable ABI support).

If we define the algorithm clearly, then setuptools  distlib could make it
available on earlier Python versions.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22980
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com