Re: [Distutils] heads-up on a plot to bring linux wheels to pypi

2016-01-14 Thread Nathaniel Smith 
On Thu, Jan 14, 2016 at 2:19 AM, Nick Coghlan  wrote:
> On 14 January 2016 at 20:12, Nick Coghlan  wrote:
>> On 14 January 2016 at 15:55, Nathaniel Smith  wrote:
>>> - build some test wheels
>>> - write a proper PEP
>>> - convince pip and pypi maintainers that this is a good idea ;-)
>>
>> While I've historically advocated against the idea of defining our own
>> "Linux platform ABI" subset, the fact that Enthought and Continuum are
>> successfully distributing pre-built binaries through the simple "use
>> CentOS 5.11" approach seems promising.
>>
>> In terms of non-scientific packages, the main group I'd suggest
>> getting in touch with is pycryptography, as we'll probably want to
>> baseline a more recent version of OpenSSL than the one in CentOS 5.11.
>
> Ah, looking at https://github.com/manylinux/auditwheel, I see anything
> linking to OpenSSL would fail the platform audit, at least for the
> current draft policy. That also seems like a potentially reasonable
> approach (although it could lead to complaints about "Why doesn't
> project  offer a wheel file?")

Yeah, it's very unfortunate, that's exactly the library that you'd
like to be able to depend on, so we checked specifically. But it turns
out that openssl has broken ABI over the relevant time-period, so
there's no choice, you can't rely on the system version and have to
statically link :-(.

Though I guess this is no worse than than if you want to distribute a
wheel that needs openssl on Windows.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] heads-up on a plot to bring linux wheels to pypi

2016-01-14 Thread Nick Coghlan 
On 14 January 2016 at 20:12, Nick Coghlan  wrote:
> On 14 January 2016 at 15:55, Nathaniel Smith  wrote:
>> - build some test wheels
>> - write a proper PEP
>> - convince pip and pypi maintainers that this is a good idea ;-)
>
> While I've historically advocated against the idea of defining our own
> "Linux platform ABI" subset, the fact that Enthought and Continuum are
> successfully distributing pre-built binaries through the simple "use
> CentOS 5.11" approach seems promising.
>
> In terms of non-scientific packages, the main group I'd suggest
> getting in touch with is pycryptography, as we'll probably want to
> baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Ah, looking at https://github.com/manylinux/auditwheel, I see anything
linking to OpenSSL would fail the platform audit, at least for the
current draft policy. That also seems like a potentially reasonable
approach (although it could lead to complaints about "Why doesn't
project  offer a wheel file?")

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] heads-up on a plot to bring linux wheels to pypi

2016-01-14 Thread Alexander Walters 



On 1/14/2016 06:13, Nathaniel Smith wrote:

On Thu, Jan 14, 2016 at 2:19 AM, Nick Coghlan  wrote:

On 14 January 2016 at 20:12, Nick Coghlan  wrote:

On 14 January 2016 at 15:55, Nathaniel Smith  wrote:

- build some test wheels
- write a proper PEP
- convince pip and pypi maintainers that this is a good idea ;-)

While I've historically advocated against the idea of defining our own
"Linux platform ABI" subset, the fact that Enthought and Continuum are
successfully distributing pre-built binaries through the simple "use
CentOS 5.11" approach seems promising.

In terms of non-scientific packages, the main group I'd suggest
getting in touch with is pycryptography, as we'll probably want to
baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Ah, looking at https://github.com/manylinux/auditwheel, I see anything
linking to OpenSSL would fail the platform audit, at least for the
current draft policy. That also seems like a potentially reasonable
approach (although it could lead to complaints about "Why doesn't
project  offer a wheel file?")

Yeah, it's very unfortunate, that's exactly the library that you'd
like to be able to depend on, so we checked specifically. But it turns
out that openssl has broken ABI over the relevant time-period, so
there's no choice, you can't rely on the system version and have to
statically link :-(.

Though I guess this is no worse than than if you want to distribute a
wheel that needs openssl on Windows.

-n

Theoretically you can statically link openssl on windows, or otherwise 
include it.  An odd benefit (in this case, only) of an operating system 
that includes nothing is a culture of vendoring everything.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] heads-up on a plot to bring linux wheels to pypi

2016-01-14 Thread Glyph Lefkowitz 

> On Jan 14, 2016, at 2:12 AM, Nick Coghlan  wrote:
> 
> In terms of non-scientific packages, the main group I'd suggest
> getting in touch with is pycryptography, as we'll probably want to
> baseline a more recent version of OpenSSL than the one in CentOS 5.11.
> 

1. It's "cryptography" (or "PyCA's Cryptography", or "cryptography.io"), not 
"pycryptography".  This is an important distinction because "PyCrypto" is the 
crappy, old thing you should not use, and "cryptography" is the new hotness.

2. On every other platform where they distribute wheels, the Cryptography 
developers have statically linked both OpenSSL and libffi; I was tangentially 
involved in the effort to do this on OS X, and in the process of debugging 
that, I learned that the Linux toolchain is fairly similar.  I would imagine 
that they'd want to statically link OpenSSL the same way, for the same reasons, 
on Linux.  Cryptography does regular releases to bundle in newer OpenSSLs, 
generally more often than the underlying platforms do.  (Since Cryptography 
does not directly export OpenSSL's API as such, it's easier to do multi-verison 
compatibility with Python than with C.)

In fact I am going to go out on a limb and say that I think Cryptography could 
be ready to go with this in a few weeks if PyPI just started allowing Linux 
wheels.  We've discussed using ancient-CentOS containers for building static 
binaries the same way PyPy does.  The potentially tricky part is just building 
the static new versions of OpenSSL from scratch on old systems, I think...

-glyph___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] heads-up on a plot to bring linux wheels to pypi

2016-01-14 Thread Nathaniel Smith 
On Thu, Jan 14, 2016 at 3:57 AM, Glyph Lefkowitz
 wrote:
[...]
> In fact I am going to go out on a limb and say that I think Cryptography
> could be ready to go with this in a few weeks if PyPI just started allowing
> Linux wheels.  We've discussed using ancient-CentOS containers for building
> static binaries the same way PyPy does.  The potentially tricky part is just
> building the static new versions of OpenSSL from scratch on old systems, I
> think...

https://phusion.github.io/holy-build-box/
https://github.com/phusion/holy-build-box/blob/master/image/build.sh#L308

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Unable to build language-check

2016-01-14 Thread Sijo Jose 
Hi Erik,

I'm not able to build my project using setuptools, it fails to install
language-check library,

see my setup.py below.
--
setup(
name='name',
version='0.0.1',
description=' TA',
long_description=DESCRIPTION,
license='Proprietary License',
author='',
author_email='',
packages=find_packages(),
install_requires=[
   "language-check==0.7.5",
   "3to2==1.1.1",
],
While executing *python setup.py install*, the language-check dependency is
not getting resolved.
Is there any workaround for this...?

Thanks
Sijo Jose

On Thu, Jan 14, 2016 at 2:24 AM, Erik Bray  wrote:

> On Wed, Jan 13, 2016 at 9:30 AM, Sijo Jose 
> wrote:
> > The language-check (https://pypi.python.org/pypi/language-check)library
> has
> > the dependency of '3to2' in python 2.7,
> > In a virtualenv it works fine with pip,that is
> >
> > pip install  3to2
> > pip install language-check
> >
> >
> > But its failing with easy_install in venv
> >
> > easy_install 3to2
> > easy_install language-check (It fails)
> >
> > Due to this I'm not able to build my project using setuptools.
>
> Why do you need to install it with easy_install?  pip is the
> recommended installer for Python packages.
>
> Erik
>



-- 
*Regards*
*Sijo Jose*
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] heads-up on a plot to bring linux wheels to pypi

2016-01-14 Thread Nate Coraor 
On Thu, Jan 14, 2016 at 12:55 AM, Nathaniel Smith  wrote:

> Hi all,
>
> Just wanted give distutils-sig a heads-up that there seems to be some
> momentum gathering around a plot to bring linux wheels to pypi:
>
>   https://github.com/manylinux/manylinux
>
> Basically the idea is to define a standard baseline linux environment
> (available libraries + version numbers + ABI of these), give it a name
> (for now, "manylinux"), and then provide tools to build wheels against
> this environment, teach pip how to recognize that the system it's on
> conforms to the environment, and then convince pypi to start accepting
> wheels with this as a platform tag :-). And if we carefully define the
> baseline environment based on the experiences of the popular
> scientific python distros (Continuum's Anaconda + Enthought's Canopy)
> we can be highly confident that essentially all desktop + server linux
> distros will be compatible with these wheels.
>

Hi Nathaniel,

This is exciting news. As you probably know, I've been working in this
space as well. In fact, the January 2016 release of Galaxy[1] will be the
first to use wheels to distribute or framework dependencies, and they're
installed via pip. We also have a Docker-based build system that automates
the process of building these wheels[2].

Unfortunately, this necessitates our own modified versions of pip[3] and
wheel[4] for messing with the platform tags. My previous discussion here on
distutils-sig on the subject stalled on the discussion of these
environments, but it sounds like you're making progress where that stopped.

If there is momentum to manylinux and it will gain the favor of pip/PyPI
maintainers, I would be happy to transfer my effort there. I've written a
linux distro detection library that may be of some use if it's not possible
to base all environments off Centos 5. I, and the Galaxy project as a
whole, are very motivated to make official Linux wheels a reality.


> This strategy is orthogonal to the more ambitious efforts to define an
> interface between wheels and the platform package manager; they can
> proceed in parallel and potentially complement each other. The goal
> here is just to get linux up to feature parity with windows and osx.
>
> Status:
> - we have a draft policy
> - there's a cool tool for scanning wheels and checking whether they
> conform to the policy: https://github.com/manylinux/auditwheel
> - there's a draft docker image to make it easy to build such wheels
> (https://github.com/manylinux/manylinux/pull/2)
>
> To do:
> - bikeshed the name ("manylinux" was picked after about 2 minutes of
> discussion so as to get started. maybe "genericlinux" would be better?
> I dunno.)
>

FWIW, as an illumos proponent, this same work as is being done for Linux
will transfer over to other "distro" OSes like illumos.


> - build some test wheels
> - write a proper PEP
> - convince pip and pypi maintainers that this is a good idea ;-)
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>


Not directly related, but Galaxy is moving toward Conda as its tool
dependency distribution system, so we are likely to converge efforts in
other places as well.

--nate

[1] https://github.com/galaxyproject/galaxy/
[2] https://github.com/galaxyproject/starforge/
[3] https://github.com/natefoo/pip/tree/linux-wheels
[4] https://bitbucket.org/natefoo/wheel
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] data_files ?

2016-01-14 Thread Stuart Axon via Distutils-SIG 
Hi,   Is data_files deprecated ?   I use it to install files to a known 
location.  I had a look at some of the tickets and am still not 100% sure.
package_data won't really work, as the location isn't know.
(I have the package 'vext' which can find the files installed by vext.gi 
vext.pygtk etc) S++___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig