Re: Solving the multiarch triplet once and for all

2012-12-20 Thread Julien Cristau 
On Tue, Dec 18, 2012 at 10:40:56 -0500, Andrew Starr-Bochicchio wrote:

> On Tue, Dec 18, 2012 at 10:24 AM, Barry Warsaw  wrote:
> > On Dec 18, 2012, at 02:22 PM, Julien Cristau wrote:
> >>Having some explanation of what's going on would definitely help.  Is
> >>there a wiki or mailing list url that I missed and that describes what
> >>this is about?
> >
> > I don't think you missed anything specific to Python.  The Debian multiarch
> > effort is described in this wiki page (with lots of sub-links):
> 
> This is the only thing I've come across:
> 
> http://wiki.debian.org/Python/MultiArch
> 
Thanks, that helps (a little, anyway).

Cheers,
Julien
-- 
Julien Cristau  
Logilab http://www.logilab.fr/
Informatique scientifique & gestion de connaissances


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121220090108.ga23...@crater1.logilab.fr

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Jakub Wilk 

* Wookey , 2012-12-18, 17:45:
It's not just build-essential that has to be cross-compilable. It's 
everything you need to build the packages in a debootstrappable (inc 
build-essential) image.


That's actually at least 254 packages. And a quite a lot of those need 
perl or python to be multiarch aware in order to build.


Give us one example. (Of course we're interested in python, not perl.)

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218191102.ga4...@jwilk.net

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Wookey 
+++ Paul Wise [2012-12-19 01:24 +0800]:
> On Wed, Dec 19, 2012 at 12:29 AM, Dmitrijs Ledkovs wrote:
> 
> > cross-compiling the archive / cross-bootstrapping the archive for a
> > new architecture.
> 
> I suppose cross-compiling will be useful but I didn't think python was
> part of the build-essential set that must be cross-compilable, is that
> actually the case?

It's not just build-essential that has to be cross-compilable. It's
everything you need to build the packages in a debootstrappable (inc
build-essential) image.

That's actually at least 254 packages. And a quite a lot of those need
perl or python to be multiarch aware in order to build.

The set could probably be trimmed down further by adding more
DEB_BUILD_PROFILE info to make reduced builds, but there is approx no
chance of getting rid of python IMHO.

The (current best guess) at the list is on
http://wiki.debian.org/Arm64Port (that's actually derived from Quantal
- unstable does not really have enough multiarch info in packages to
get any useful answers currently)

Wookey
-- 
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218174553.gk9...@stoneboat.aleph1.co.uk

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Steve Langasek 
On Tue, Dec 18, 2012 at 11:48:56PM +0800, Paul Wise wrote:
> Seems like a lot of effort when there is no /usr/bin//python
> so there can be only one arch of python installed anyway. What are the
> use-cases for python multiarch?

Being able to cross-install packages that use libpython; and as Dmitrijs
mentions, being able to cross-build packages that use python too.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Paul Wise 
On Wed, Dec 19, 2012 at 12:29 AM, Dmitrijs Ledkovs wrote:

> cross-compiling the archive / cross-bootstrapping the archive for a
> new architecture.

I suppose cross-compiling will be useful but I didn't think python was
part of the build-essential set that must be cross-compilable, is that
actually the case?

PS: I'm subscribed.

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caktje6hkanjztu+a7ugzr-ztampx5r50aqxjtd0q5d6zbw9...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Dmitrijs Ledkovs 
On 18 December 2012 15:48, Paul Wise  wrote:
> Seems like a lot of effort when there is no /usr/bin//python
> so there can be only one arch of python installed anyway. What are the
> use-cases for python multiarch?
>

cross-compiling the archive / cross-bootstrapping the archive for a
new architecture.

Regards,

Dmitrijs.


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/canbhlujab32bcb+xptzm0z-oo-iie39jkqv2jbl17o+mnp4...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Barry Warsaw 
On Dec 18, 2012, at 10:40 AM, Andrew Starr-Bochicchio wrote:

>This is the only thing I've come across:
>
>http://wiki.debian.org/Python/MultiArch

Ah, of course.  I've added some cross-links.

-Barry


signature.asc
Description: PGP signature

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Paul Wise 
Seems like a lot of effort when there is no /usr/bin//python
so there can be only one arch of python installed anyway. What are the
use-cases for python multiarch?

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caktje6e4zf_3fu4ph5mqjf3yhcthtc9rwf+ozvbjdpwdhhw...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Jakub Wilk 

* Barry Warsaw , 2012-12-18, 10:24:
For typical (e.g. C) libraries and headers, you have paths like 
/usr/lib/ and /usr/include/.


Quoting :
"If your -dev package contains headers which vary across architectures 
then it cannot be marked as Multi-Arch: same until a policy decision is 
made about architecture-dependant headers and the toolchain is updated."


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218154729.ga3...@jwilk.net

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Andrew Starr-Bochicchio 
On Tue, Dec 18, 2012 at 10:24 AM, Barry Warsaw  wrote:
> On Dec 18, 2012, at 02:22 PM, Julien Cristau wrote:
>>Having some explanation of what's going on would definitely help.  Is
>>there a wiki or mailing list url that I missed and that describes what
>>this is about?
>
> I don't think you missed anything specific to Python.  The Debian multiarch
> effort is described in this wiki page (with lots of sub-links):

This is the only thing I've come across:

http://wiki.debian.org/Python/MultiArch

-- Andrew Starr-Bochicchio

   Ubuntu Developer 
   Debian Developer 
   PGP/GPG Key ID: D53FDCB1


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/cal6k_azayckgr8xdkwgp8-wshrcwve_wfgbywzuenkinvot...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Barry Warsaw 
On Dec 18, 2012, at 02:22 PM, Julien Cristau wrote:

>On Thu, Dec 13, 2012 at 11:34:12 +0100, Jakub Wilk wrote:
>
>> How about undoing the Python multi-arch mess until someone provides
>> evidence that the changes solve more problems than they create?
>> 
>Having some explanation of what's going on would definitely help.  Is
>there a wiki or mailing list url that I missed and that describes what
>this is about?

I don't think you missed anything specific to Python.  The Debian multiarch
effort is described in this wiki page (with lots of sub-links):

http://wiki.debian.org/Multiarch

My take on it is this: in order to support installation of packages in
multiple architectures, you need a multiarch triplet-specific directory into
which you can install architecture-dependent code.  For typical (e.g. C)
libraries and headers, you have paths like /usr/lib/ and
/usr/include/.

For Python, we need a multiarch-aware location to install extension module .so
files.  As you know, Python uses /usr/lib/pythonX.Y and
/usr/include/pythonX.Y, which aren't multiarch-aware.  We could extend
Python's ABI flag directories (PEP 3149), e.g. /usr/lib/python3.3m but I don't
think that's a good idea.  Python already defines a platform-specific
location, e.g. /usr/lib/pythonX.Y/plat-linux2, but that isn't specific enough,
so Python was changed in Debian to search for and install to
`plat-`.

So far so good, except that there are lots of other tools that just assume
`plat-linux2` (or really, `plat-${sys.platform}`) because that's what they've
always done.  No big surprise really, since *lots* of tools aren't/weren't
prepared for multiarch.  Welcome to the brave new world.

At the command line, there are two official ways to discover the multiarch
triplet:

$ dpkg-architecture -qDEB_HOST_MULTIARCH
$ gcc --print-multiarch

I don't know which one is preferred, but I'm guessing it has more to do with
whatever packages you want to depend on.

It seems silly to me to require Python programs to shell out to either of
these every time they need to know the triplet.  It's not *entirely* trivial
to do.  A good example is virtualenv's site.py, which for bootstrapping
reasons can't use subprocess.  Why force Python programs to re-invent 1000
different ways to get this information, when we can (and have to) calculate it
when Python is built.

Besides, Python programs don't necessarily want the triplet for the
architecture they're running on, but for the architecture that the Python
executable was built for.  And that's not information that shelling out will
help you with.

So my proposed solution is to calculate the triplet for the architecture that
Python is being built on at configure time, and stash that into the sys module
so that Python code has a consistent, easy, and documented way to get at it.
sys is always going to be available, so that solves the bootstrap problem that
virtualenv has.

In Python 3.3, PEP 421 describes exactly where this information should go.
sys.implementation is the place to put it, and the name must start with an
underscore, so I chose `sys.implementation._architecture`.

In Python 2.7, there is no sys.implementation, so I chose `sys._architecture`
as the place to stash this.

If your code is bilingual, then you'd use:

>>> getattr(sys, 'implementation', sys)._architecture

which doesn't seem too horrible while we still have to support Python 2
.  Once everyone agrees this is the right approach, I'll document this
in the Debian wiki, and it should be used consistently wherever it's needed.
I have opened bugs and uploaded patches to fix Python 2.7 and 3.3, and
python-virtualenv.

Cheers,
-Barry


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218102420.06347...@limelight.wooz.org

Re: Solving the multiarch triplet once and for all

2012-12-18 Thread Julien Cristau 
On Thu, Dec 13, 2012 at 11:34:12 +0100, Jakub Wilk wrote:

> How about undoing the Python multi-arch mess until someone provides
> evidence that the changes solve more problems than they create?
> 
Having some explanation of what's going on would definitely help.  Is
there a wiki or mailing list url that I missed and that describes what
this is about?

Cheers,
Julien
-- 
Julien Cristau  
Logilab http://www.logilab.fr/
Informatique scientifique & gestion de connaissances


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218132228.ga22...@crater2.logilab.fr

Re: Solving the multiarch triplet once and for all

2012-12-14 Thread Barry Warsaw 
On Dec 12, 2012, at 07:09 PM, Barry Warsaw wrote:

>Wouldn't it be nice if Python just told you what the values were?

Bugs and patches are now up on b.d.o:

2.7: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695958
3.3: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695959

This exposes the triplet as sys._architecture on Python 2.7 and
sys.implementation._architecture in Python 3.3.  Bilingual code should do
something like this:

>>> import sys
>>> getattr(sys, 'implementation', sys)._architecture

Matthias, assuming you agree with these additions, I will fix virtualenv to
use these values for experimental.  Then we can promote these on the Debian
wiki.

Cheers,
-Barry


signature.asc
Description: PGP signature

Re: Solving the multiarch triplet once and for all

2012-12-13 Thread Julian Taylor 
On 12/13/2012 11:34 AM, Jakub Wilk wrote:
> How about undoing the Python multi-arch mess until someone provides
> evidence that the changes solve more problems than they create?
> 
> (I'd like to note that the changes were never discussed on
> debian-python@, which is in violation of tech-ctte resolution.)
> 

the issue goes beyond multiarched python (for which there already are
in-python upstreamed ways to get the path, e.g.
get_python_inc(plat_specific=1)).

e.g. numpy needs it for its own of distutils
So unless you want to revert multiarch as a whole or teach all upstreams
to stop relying on stuff being in usr/lib, having something in python
itself to get the path is a good thing.


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/50ca261b.1000...@googlemail.com

Re: Solving the multiarch triplet once and for all

2012-12-13 Thread Jakub Wilk 
How about undoing the Python multi-arch mess until someone provides 
evidence that the changes solve more problems than they create?


(I'd like to note that the changes were never discussed on 
debian-python@, which is in violation of tech-ctte resolution.)


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121213103411.ga5...@jwilk.net

Solving the multiarch triplet once and for all

2012-12-12 Thread Barry Warsaw 
Today in #debian-python we discussed bug #695707, which describes a breakage
in virtualenv with Python 2.7 in experimental due to multiarch.  We explored a
few approaches for fixing this, centered around patching virtualenv itself,
but none of the solutions (including the one in PAPT svn :/ ) are entirely
satisfying.  Stefano made the observation that you really want the multiarch
triplet that Python was built with, not necessarily the triplet on the host
platform its installed on, which makes sense to my limited exposure to
multiarch.

In any case, fixing this in virtualenv is both problematic and rather
shortsighted, since this problem keeps coming up again and again in other
packages.  The other complication is that you generally have to shell-out to
get the right value, calling one of the following commands:

$ dpkg-architecture -qDEB_BUILD_MULTIARCH
$ gcc --print-multiarch

Stefano brought up the inefficiencies of doing this in virtualenv's site.py
every time you fire up your venv Python.  Doko suggested globbing sys.path
entries, but that seems error-prone, and someone else seemed to remember cases
where this was tried and didn't work very well.

Wouldn't it be nice if Python just told you what the values were?

We could calculate this once during Python's build, and now all Python code
just gets it for free with a simple import.

In Python 3.3 (and beyond) we have the perfect place to stash this
information.  PEP 421's sys.implementation allows for platform-specific
values, so we can squirrel this away as e.g. sys.implementation._architecture
for example.  (The exact name can be bikeshedded to death, but re: PEP 421
must start with an underscore.)

I now have a patch/branch that does this.

lp:~barry/ubuntu/raring/python3.3/multiarch

Yes, it's against Ubuntu Raring's Python 3.3 source branch, but that's
just a copy of experimental's 3.3.0-5, and bzr+lp was the most convenient
for this little bit of afternoon skunkworks.

$ ./configure
$ make
$ ./python -c "import sys; print(sys.implementation._architecture)"
x86_64-linux-gnu

It should only be a little more difficult to backport this to Python 2.7 (I
don't personally care about <=3.2 or <=2.7, and since this is all jessie-land
stuff, y'all shouldn't either :), since there's no PEP 421 sys.implementation.
Well, we can just hold our noses and stick the same thing in sys._architecture
for 2.7.

Notice the nice way this cleans up setup.py.  Bonus!

Feedback is most definitely welcome.  If everyone thinks this is a good thing
to pursue, I can clean up the patch, back port to 2.7, and Doko can sponsor it
into experimental.

Then I think we just hack our version of virtualenv to use sys._architecture
(since it's currently a 2.x program) and we're done .

Cheers,
-Barry


signature.asc
Description: PGP signature