[Distutils] Re: failing to build a windows python38 wheel
On Tue, Oct 29, 2019 at 03:09:08PM +, Robin Becker wrote: > On 29/10/2019 12:53, Robin Becker wrote: > > Hi, I am trying to use the technique of pillow to pre-emptively > > build windows 38 wheel for x_64 and also i686. > > > > I used > > https://github.com/matthew-brett/multibuild/blob/devel/install_python.ps1 > > as a starting point, but made the version stuff explicit so my install > > ps1 looks like > > > > I am slightly mystified can anyone suggest what I am doing wrong? > > I think the problem was that the installed python needed an upgraded > pip & wheel. I seem to be building > > reportlab-3.5.32-cp38-cp38-win32.whl > > ie without the 'm' in the filename. That is correct: Python 3.8 no longer uses the 'm' flag to distinguish pymalloc builds because they're now ABI-compatible: - https://docs.python.org/3/whatsnew/3.8.html#build-and-c-api-changes - https://docs.python.org/3/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build Regards, Marius Gedminas -- In English, is it grammatically correct to use "Apple" and "relatively-inexpensive" in the same sentence? -- James Nicoll signature.asc Description: PGP signature -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/JMRYJAZA5PB5PREETLRLKGQFMD6HZQJL/
[Distutils] Re: WIll custom setup.py with (set library-prefix have post install actions) continue to be supported ?
On Sun, May 26, 2019 at 12:24:08PM +, Stuart Axon via Distutils-SIG wrote: > I'm making a python package for something python Gedit/Pluma/Xed plugin. > > Like other lots of other packages that use Gtk, I use glib-compile-schemas [1] > at install time. > > I do this by using a custom setup.py > - similar to this file this from flux: > [1]https://github.com/xflux-gui/fluxgui/blob/master/setup.py#L81 > (other app, like Meld to something similar) > > > It looks like there are a lot of changes happening to setuptools, will this > continue to work or will it break one day ? > > > This isn't the only file I'd like to generate at install time, but might be > the > hardest to work around if I can't. For gtimelog I ended up calling glib-compile-schemas at runtime, prior to `import gi`. I'm also setting os.environ['GSETTINGS_SCHEMA_DIR'] to the directory where my schema file resides (inside the virtualenv's site-packages, usually). This allows me to support 'pip install gtinelog', as well as running straight from a git checkout with no intermediate build step. Marius Gedminas -- We have enough youth, how about a fountain of SMART? -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/PES4ODIT4AN2Y53JRFJM2GCH345NRUCE/
[Distutils]Re: Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers
On Mon, Jul 16, 2018 at 06:14:52PM -0700, Nick Terrell wrote: > I collected the XAR benchmark numbers. I spent some time today > investigating what exactly is causing the difference between native > and XAR start times. The native installation I was benchmarking > against used `pkg_resources.load_entry_point()` to run the script, > while XAR called the entry point directly. Benchmarking against pkg_resources is a bit like running a race when your opponent has an iron cannonball chained to their leg: https://github.com/pypa/setuptools/issues/510 ;) Marius Gedminas -- A secret: don't tell DARPA I'm not building the sun destroying weapon they think I am. -- Michael Salib, the author of Starkiller signature.asc Description: PGP signature -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KULA7KXVBLVTZ5KW43P76QKTBTGGAMHU/
[Distutils] Re: requirements.txt or not requirements.txt?
On Mon, May 07, 2018 at 08:48:23AM +0200, Carles Sala Cladellas wrote: > Hello here! > > TL;DR: Should I use requirements.txt, or should I have my dependencies only > listed inside setup.py? Short answer: only setup.py, unless you find a reason to also use a requirements.txt. Long answer: https://packaging.python.org/discussions/install-requires-vs-requirements/ and especially https://caremad.io/posts/2013/07/setup-vs-requirement/ > But then I came across this, [1]https://stackoverflow.com/a/28842733, which > recommends using a `dev` entry in `extras_require` and then using `pip install > -e .[dev]` to install them, which could be easily extended to also have a `pip > install -e .[test]`. > > I find this a very clean option, since it reduces the amount of files the > repository and makes setup.py and makes setup.py self-contained. > > Are any known drawbacks to this approach? Is it a good way to go? I like it. It works nicely for me. It's not 100% complete approach: sometimes (when I'm working on an application rather than a tool/library) I also wish to have a list of concrete version pins with which the application has been known to work, so I use pip freeze (or pip-tools) to get a requirements.txt with all the frozen versions. HTH, Marius Gedminas -- Everyone generalizes from one example. At least, I do. -- Steven Brust signature.asc Description: PGP signature -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LQXS7TDPAMDXXR2DMH5AAYG42EGHML64/
[Distutils] Re: (Final) PyPI/Warehouse weekly report: legacy is shut down
On Tue, May 01, 2018 at 09:09:02PM -0400, Sumana Harihareswara wrote: > Ernest sunset Legacy, fixed a subsequent outage (my fault for putting a > hostname in the title of a blog post!), Surely not your fault! You merely discovered a bug (about which I would love to hear more). Marius Gedminas -- At one job, for a short time my desktop HP (running HPUX) was being used as a print server. I soon found out that having something heavy in front of the plug was a good idea, after kicking it out of the outlet a couple of times. I used my old TRS-80 4P, doubtless the slowest computer ever to perform a critical function at that company. -- David Thornley signature.asc Description: PGP signature -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/YEJKCGNAESCJXOWDIARJZM2S6XNTTA6S/
Re: [Distutils] pypi.python Error 403 Forbidden
On Tue, Feb 13, 2018 at 09:19:20PM +0100, Heiko L. wrote: > I try to install a software from pypi.python.org and receive following errmsg: > urllib2.HTTPError: HTTP Error 403: Forbidden > > and > > $ wget > http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg > HTTP request sent, awaiting response... 403 SSL is required > > After a few hours I found the following article: > https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html > ...you can no longer access /simple/ and /packages/ over HTTP and you will > have to directly go to HTTPS > > - It is true? Yes. > - Is this the right place if I have any questions? Yes. HTH, Marius Gedminas -- Any sufficiently advanced technology is indistinguishable from a rigged demo. - Andy Finkel, computer guy signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Disabling non HTTPS access to APIs on PyPI
On Sat, Oct 28, 2017 at 12:22:32AM +0300, Alex Domoradov wrote: > I got it. And what I should do with old system? For e.g. we still use ubuntu > 12.04. Is there any way to upgrade pip/setuptools? If you're using Ubuntu 12.04, then presumably you're paying Canoncial for extended support, so ask them to provide a pip/setuptools SRU. (If you're not paying Canonical, then you're not getting security updates and should upgrade ASAP.) Marius Gedminas -- Favorite MS-DOS error message: "Drive C: not ready, close door." signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Entry points: specifying and caching
On Fri, Oct 20, 2017 at 08:10:06AM -0400, Donald Stufft wrote: > Packaging tools shouldn’t be expected to know anything about it other > than the console_scripts feature Please do not forget about gui_scripts entry points! Marius Gedminas -- What can I do with Python that I can't do with C#? You can go home on time at the end of the day. -- Daniel Klein signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Getting dependecies of package from PyPiJSON
On Fri, Jul 21, 2017 at 11:39:01AM +0200, Krzysiek Płachno wrote: > Guys! Thanks a lot for all your responses and willingness to help! > > Yesterday I noticed this `requires_dist` in response for Requests package. But > actually it was the only one that had this field populated from many packages > that I asked API for. So this cannot be reliable way of getting dependecies. > > So if not using pip, the only way to get dependencies is from built packages: > .whl and .egg. > So in .egg it'd be: EGG_INFO/requires.txt and in .whl : > .dist-info/metadata.json, right? I've had some luck parsing *.egg-info/requires.txt files in sdists. This is not 100% reliable (a setup.py may dynamically-compute the install_requires list during install time depending on sys.version_info or other factors), but it was good enough for my purposes. https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py Regards, Marius Gedminas -- Hi. I'm the "I love you" .signature virus. You have been infected. Please panic immediately. signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Provisionally accepting PEP 517's declarative build system interface
On Wed, May 31, 2017 at 08:08:51PM -0400, Donald Stufft wrote: > I think this should cover the case of actually making the project pip > installable (assuming of course the setup.py or build backend doesn’t do > something silly like always sys.exit(1) instead of produce the expected > outcome) My personal favorite was PyGame doing raw_input() from its setup.py. Marius Gedminas -- After having done some test using hi-tech istruments (moving my mouse during a kernel build) [...] -- Davide Libenzi on lkml signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] reproducible builds
On Mon, Mar 20, 2017 at 11:30:59AM +, Robin Becker wrote: > thanks for this; it seems the emphasis is on security. If the intent is that > reportlab should be able to reliably reproduce the same binary output then I > think I need to do more than just fix a couple of dates. We use many > dictionary like objects to produce PDF and I am not sure all are sorted by > key during output. I'm sure the reproducible builds folks will send you patches if they find any spots that you missed. ;-) > Is there a way to excite dictionary ordering changes? I believe there was > some way to modify the hashing introduced when the dos dictionary attacks > were an issue. Would it be sufficient to generate documents with say Python > 2.7 and check against 3.6? Python 3.6 changed the dict implementation so the ordering is always stable (and matches insertion order). You'll want to test with Python 3.5, which perturbs the dict ordering randomly, as a side effect of the randomized string/bytes hashes (unless you fix it by setting the PYTHONHASHSEED environment variable[*]) [*] https://docs.python.org/3.3/using/cmdline.html#envvar-PYTHONHASHSEED Regards, Marius Gedminas -- Yes, always begin work on inherited code by removing comments. Even if they were maintained (they are not) they are natural language written by engineers who cannot be understood ordering coffee in a diner. Getting back to comments not being maintained, my saying on that one is, "Comments do not run." -- Kenny Tilton signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Module Installation Issues
On Tue, Sep 13, 2016 at 06:55:49PM -0400, Donald Stufft wrote: > >>> pip install > > And print a better error message that gives a better indication about what’s > gone wrong besides a SyntaxError? This way you could also make >>> exit work like people expect it to work, without the (). Marius Gedminas -- If your company is not involved in something called "ISO 9000" you probably have no idea what it is. If your company _is_ involved in ISO 9000 then you definitely have no idea what it is. (Scott Adams - The Dilbert principle) signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Things that are not pip-installable (was Re: moving things forward) shouldn't)
On Wed, May 04, 2016 at 11:32:33PM -0700, Nathaniel Smith wrote: > What are these things that aren't pip-installable and why isn't the > solution to fix that? Things that are not pip-installable that I've personally missed include: - pygame (there are a bunch of tickets in their bug tracker, and upstream is working slowly to fix them, just ... very slowly) - pygobject (plus you need gobject-introspection files for all the libraries you want to actually use, like GLib, Pango, Cairo, and GTK itself, and those aren't on PyPI either) > We've spent a huge amount of effort on reaching the point where pretty > much everything *can* be made pip installable. Heck, *PyQt5*, which is > my personal benchmark for a probably-totally-unpackageable package, > announced last week that they now have binary wheels on pypi for all > of Win/Mac/Linux: > > https://pypi.python.org/pypi/PyQt5/5.6 Doesn't seem to work for me, with pip 8.1.1 on a 64-bit Linux machine: $ pip install pyqt5 Collecting pyqt5 Could not find a version that satisfies the requirement pyqt5 (from versions: ) No matching distribution found for pyqt5 Marius Gedminas -- Shift happens. -- Doppler signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] 'Invalid module name' creating package with setuptools
On Thu, Apr 14, 2016 at 11:28:09AM -0400, Luís de Sousa via Distutils-SIG
wrote:
> I have a project with these contents:
>
> proj
> ├── proj
> │ ├── scriptA.py
> │ ├── scriptB.py
> │ └── __init__.py
> ├── LICENCE
> ├── README.md
> └── setup.py
>
> The setup.py file looks like:
...
> entry_points={
> 'console_scripts': [
> 'scriptA=proj:scriptA',
> 'scriptB=proj:scriptB'
I'm not sure this is right -- an entry point should point to a python
module (not a package) and a function in that module, so something like
'scriptA=proj.scriptA:main',
'scriptB=proj.scriptB:main',
I've never tried to define entry points pointing to functions defined in
the __init__.py of a package, so I don't know if you're allowed to write
scriptX=proj:fn
or if you have to explicitly say
scriptX=proj.__init__:fn
> When I try to build I get the following error:
>
> $ python setup.py bdist_wheel --universal
> error in proj setup command: ('Invalid module name', 'proj')
(That is not a good error message indeed.)
Marius Gedminas
--
This loads a GDT entry into the "Task Register": that entry points to a
structure called the Task State Segment. Some comments scattered though the
kernel code indicate that this used for task switching in ages past, along
with blood sacrifice and astrology.
-- lguest source code
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] For maximum performance, Python packages are best installed as zip files.
On Mon, Apr 11, 2016 at 01:04:55PM +0200, Thomas Güttler wrote: > From > https://pythonhosted.org/setuptools/setuptools.html#setting-the-zip-safe-flag > > > For maximum performance, Python packages are best installed as zip files. > > What kind of performance improvement is this? > > Is this improvement really measurable? > > What improvement numbers do you get? A long time ago I saw a thread (maybe on this very list) with concrete numbers. I only remember the conclusion: - everything in one zip file (including the Python stdlib) was fastest - everything unzipped was 2nd fastest - multiple zipped eggs were slowest This was probably done in the times of rotational hard disks. It would be interesting to see someone repeat that experiment. Marius Gedminas -- I've never understood the pathological fear some companies seem to have about the idea of their advertisements being seen by too many people. -- Brett Tamahori signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] abstract build system PEP update
On Thu, Feb 18, 2016 at 12:12:41PM +1300, Robert Collins wrote: > On 17 February 2016 at 20:13, Marius Gedminas <mar...@gedmin.as> wrote: > > On Tue, Feb 16, 2016 at 04:10:43PM +1300, Robert Collins wrote: > >> diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst > >> index a6e4712..56464f1 100644 > >> --- a/build-system-abstraction.rst > >> +++ b/build-system-abstraction.rst > >> @@ -68,12 +68,15 @@ modelled on pip's existing use of the setuptools > >> setup.py interface. > >> pypa.json > >> - > >> > >> -The file ``pypa.json`` acts as neutron configuration file for pip and > >> other > >> +The file ``pypa.json`` acts as neutral configuration file for pip and > >> other > >> tools that want to build source trees to consult for configuration. The > >> absence of a ``pypa.json`` file in a Python source tree implies a > >> setuptools > >> or setuptools compatible build system. > >> > >> -The JSON has the following schema. Extra keys are ignored. > >> +The JSON has the following schema. Extra keys are ignored, which permits > >> the > >> +use of ``pypa.json`` as a configuration file for other related tools. If > >> doing > >> +that the chosen keys must be namespaced - e.g. ``flit`` with keys under > >> that > >> +rather than (say) ``build`` or other generic keys. > > > > Is this going to be a file that human beings are expected to edit by > > hand? > > > > If so, can we please not use JSON? JSON is rather hostile to humans: no > > trailing commas, no possibility to add comments. > > Find another format thats ideally in the standard library, with an as > clean language-neutral schema. Yaml isn't. Toml isn't. $bikeshed. ConfigParser seems to be the only other available choice then. > Honestly - If this is the bit we bog down on, great - someone can > spend the time finding a thing to use here, but as discussed > previously, I don't care: the PEP editor that accepts the PEP can tell > me what format should be there, and I'll put it there. Until then, I > don't want to think about it because its not interesting. If it's a blob of constant metadata that describes the build system, maybe generated by the build system itself (by running some 'init-package' command, say), then I don't care either. If it's something that's going to be part of the user experience of Python packages, then I've reservations. Marius Gedminas -- Everyone has a photographic memory. Some don't have film. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] abstract build system PEP update
On Tue, Feb 16, 2016 at 04:10:43PM +1300, Robert Collins wrote: > diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst > index a6e4712..56464f1 100644 > --- a/build-system-abstraction.rst > +++ b/build-system-abstraction.rst > @@ -68,12 +68,15 @@ modelled on pip's existing use of the setuptools > setup.py interface. > pypa.json > - > > -The file ``pypa.json`` acts as neutron configuration file for pip and other > +The file ``pypa.json`` acts as neutral configuration file for pip and other > tools that want to build source trees to consult for configuration. The > absence of a ``pypa.json`` file in a Python source tree implies a setuptools > or setuptools compatible build system. > > -The JSON has the following schema. Extra keys are ignored. > +The JSON has the following schema. Extra keys are ignored, which permits the > +use of ``pypa.json`` as a configuration file for other related tools. If > doing > +that the chosen keys must be namespaced - e.g. ``flit`` with keys under that > +rather than (say) ``build`` or other generic keys. Is this going to be a file that human beings are expected to edit by hand? If so, can we please not use JSON? JSON is rather hostile to humans: no trailing commas, no possibility to add comments. Marius Gedminas -- Debugging a computer program is such an interesting activity because it's not really a matter of fixing a program. It's a matter of fixing your own understanding to the point that the cause of the bug becomes obvious. So debugging means constantly challenging your assumptions, constantly looking for the overlooked insignificant thing that turns out to be crucial. -- Joey Hess signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOABI for Unicode ABI on 2.x (was: wheel 0.27.0 released)
On Sat, Feb 06, 2016 at 09:18:39PM +1000, Nick Coghlan wrote: > On 6 February 2016 at 20:35, Marius Gedminas <mar...@gedmin.as> wrote: > > FWIW the rationale Pyenv gave when they rejected a bug asking for UCS-4 > > builds by default was "we prefer to follow upstream defaults". > > In this case, the old defaults are dubious, but the upstream fix > eliminated the relevant setting. Historically, it didn't really > matter, since very few people were building their own Python for > Linux. > > However, if that was pyenv's only reason for rejecting a switch to > wide unicode builds, it may be worth trying again, this time pointing > them to PEP 513 and the wide-build default for Python 2.7 wheels in > the manylinux build environment. Here's the issue, if you'd like to try: https://github.com/yyuu/pyenv/issues/257 (I don't use pyenv myself; all I know about this issue is from helping other people debug problems on IRC.) Marius Gedminas -- Tilton's Law of Lisp Programming: if you do not need a metaclass, do not use a metaclass. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
On Fri, Feb 05, 2016 at 12:56:05PM -0500, Nate Coraor wrote: > On Fri, Feb 5, 2016 at 12:52 PM, Nathaniel Smith <n...@pobox.com> wrote: > > > On Feb 5, 2016 9:35 AM, "Nate Coraor" <n...@bx.psu.edu> wrote: > > > Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is > > this necessary? The ABI tag should be set correctly on both the build and > > installation systems, so is including it as part of the manylinux1 ABI (and > > fixing it to wide builds only) overkill? ... > > Also, just to confirm, the new releases of pip and wheel enable this for > > 2.x; is it also available for all 3.x? > > > > -n > > > > ABI tags always worked with wheel/pip on CPython 3.2+ (it has the SOABI > config var), the new change "backports" this functionality to 2.X. What are the minimum versions of pip and wheel to have this working correctly on 2.x? Marius Gedminas -- A "critic" is a man who creates nothing and thereby feels qualified to judge the work of creative men. There is logic in this; he is unbiased -- he hates all creative people equally. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
On Sat, Jan 30, 2016 at 12:17:02PM -0800, Matthew Brett wrote: > Hi, > > > I can confirm that Debian and Anaconda builds of CPython 2.7 both have > > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode == > > 0x. Hmm. I guess they should fix that. > > > > Also the manylinux docker image currently has sys.maxunicode == > > 0x, so we should definitely fix that :-). > > A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide > unicode by default. Are there any known distributions providing UCS2 > unicode Pythons? I don't know of any. Pyenv, OTOH, deliberately uses upstream defaults and so produces narrow unicode builds. Marius Gedminas -- If you are angry with someone, you should walk a mile in their shoes... then you'll be a mile away from them, and you'll have their shoes. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Package classifiers - both major and minor Python versions?
On Thu, Jan 21, 2016 at 05:53:32PM +, Brett Cannon wrote: > On Thu, 21 Jan 2016 at 06:48 John Whitlock <john-whitl...@ieee.org> wrote: > > > A discussion about the Python language classifiers came up in a pull > > request [1], and I couldn't find a definite answer. The question is - > > should a packager specify the major Python versions, minor Python versions, > > or both? > > > > The Python Packaging User Guide's example [2] has both: > > > > # Specify the Python versions you support here. In particular, ensure > > # that you indicate whether you support Python 2, Python 3 or both. > > 'Programming Language :: Python :: 2', > > 'Programming Language :: Python :: 2.6', > > 'Programming Language :: Python :: 2.7', > > 'Programming Language :: Python :: 3', > > 'Programming Language :: Python :: 3.2', > > 'Programming Language :: Python :: 3.3', > > 'Programming Language :: Python :: 3.4', > > > > In the example, 'Programming Language :: Python :: 2' is a major version, > > and 'Programming Language :: Python :: 2.7' is a minor version. > > > > pyroma [3], which I use as a packaging linter, has insisted on both the > > major and minor versions since the first release in 2011 [4]. > > > > These were added in 2008, but the announcement on this mailing list didn't > > include guidance on usage [5]. I can't find any guidance in PEPs either. > > > > You should at least do the major versions, and if you are up for > maintaining them, then do the minor ones as well. > > You want the major ones to tell people whether you even support Python 3 or > not. Various tools like caniusepython3 rely on at least the major version > classifier existing to programmatically know about Python 3 support. Are these tools unable to realize that supporting a particular minor version implies support for the corresponding major version? Marius Gedminas -- HOST SYSTEM RESPONDING, PROBABLY UP... signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] namespace_package
On Tue, Dec 01, 2015 at 07:17:58AM -0500, KP wrote: > However, with the namespace_packages = ['foo'], the > lib\site-packages\foo\__init__.py does not get installed (even though it is > in the source tree). Instead there's just a dir with "foo/bar/__init__.py" > and "foo/blah/__init__.py". I will try to look in the "wheel" side of > things next I guess. Perhaps pip is doing something since it seems to > install even source distributables by first converting to a wheel. Can you show us your setup.py? Marius Gedminas -- I’m not big on predictions, but I do have one for 2011: HTML5 will continue to be popular, because anything popular will get labeled “HTML5.” -- Mark Pilgrim signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] namespace_package
On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: > I'm not sure where the issue is, but when I specify a namespace_package in > the setup.py file, I can indeed have multiple packages with the same base > (foo.bar, foo.blah, etc...). The files all install in to the same > directory. It drops the foo/__init__.py that would be doing the > extend_path, and instead adds a ".pth" file that is a bit over my head. > > The problem is that it does not seem to traverse the entire sys.path to > find multiple foo packages. Does every foo.x package specify namespace_packages=['foo']? Do they all ship an identical foo/__init__.py with import pkg_resources pkg_resources.declare_namespace(__name__) ? AFAIU you need both things in every package, if you want to use namespace packages. > If I do not specify namespace_packages and instead just use the > pkgutil.extend_path, then this seems to allow the packages to be in > multiple places in the sys.path. > > Is there something additional for the namespace_package that i need to > specify in order for all of the sys.path to be checked? > > I'm using 18.5 setuptoolsbut I am not sure if this somehow ties in to > wheel/pip, since I'm using that for the actual install. Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing packages using pip
On Mon, Nov 16, 2015 at 01:38:23PM +, Paul Moore wrote: > I don't know what Unix does, I suspect it retains an old > copy of the shared library for the process until the process exists, > in which case you'd see a different issue, that you do an upgrade, but > your process still uses the old code till you restart. Basically. Technically, both Linux and Windows won't let you write to a shared library you have mapped into a process's address space for execution. (You get an -ETEXT error on Linux, which one can observer if one tries to re-create virtualenv while its bin/python is currently running.) What you can do Linux that you cannot do on Windows is delete a shared library file while it's mapped into a process's address space. Then Linux lets you create a new file with the same name, while the old file stays around, nameless, until it's no longer used, at which point the disk space gets garbage-collected. (If we can call reference counting "garbage collection".) The result is as you said: existing processes keep running the old code until you restart them. There are tools (based on lsof, AFAIU) that check for this situation and remind you to restart daemons. Marius Gedminas -- We like stress testing, because we know the future will be stressful. -- Maritza Mendez signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The future of invoking pip
On Fri, Nov 06, 2015 at 02:04:38PM +1300, Robert Collins wrote: > On 6 November 2015 at 10:08, Donald Stufft <don...@stufft.io> wrote: > > * It's more to type, 10 more characters on *nix and 6 more characters on > > Windows which makes it more akward and annoying to use. This is > > particularly > > annoying inside of a virtual environment where there isn't really any > > ambiguity when one is activated. > > cat > /usr/bin/pip << EOF > python -m pip $@ > EOF > > Seriously - isn't the above entirely sufficient? It doesn't help with my usual pattern, which used to be $ virtualenv . $ bin/pip install foo but then changed[*] into $ virtualenv .venv && ln -sfn .venv/bin bin $ bin/pip install foo and is likely to change[+] into $ virtualenv .venv && mkdir -p bin && ln -sfn .venv/bin/pip bin/pip $ bin/pip install foo I am not running these by hand -- I have Makefiles to set up my app environment by creating a local virtualenv and pip installing all the tools, plus '-e .', into it. But once the basic environment is done, I'm often installing ad-hoc one-time-use extra tools with commands like $ bin/pip install runsnakerun --- [*] because virtualenv's root is becoming too cluttered with files like pip-selftest.json, and because some evil packages on PyPI install files named README.txt into the virtualenv root. [+] because if you symlink just the bin/ directory, bin/python fails to set up sys.path correctly Marius Gedminas -- I'm sure it would be possible to speed apport up a lot, after we're done making boot and login instantaneous. -- Lars Wirzenius signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPA Roadmap
On Mon, Nov 02, 2015 at 06:13:27PM -0800, Marcus Smith wrote: > Based on discussions in another thread [1], I've posted a PR to pypa.io for > a "PyPA Roadmap" > > PR: https://github.com/pypa/pypa.io/pull/7 > built version: http://pypaio.readthedocs.org/en/roadmap/roadmap/ > > To be clear, I'm not trying to dictate anything here, but rather just > trying to mirror what I think is going on for the sake of new (or old) > people, who don't have a full picture of the major todo items. > > I'm asking for help to make this as accurate as possible and to keep it > accurate as our plans change. Shouldn't Warehouse be mentioned there? Marius Gedminas -- Initially, there were few or no postal regulations governing packages mailed parcel post. To construct a bank in Vernal, Utah in 1916, a Salt Lake City Company figured out that the cheapest way to send 40 tons of bricks to the building was by Parcel Post. Each brick was individually wrapped & mailed. Postal rules were promptly rewritten. -- http://en.wikipedia.org/wiki/United_States_Postal_Service signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A smaller step towards de-specializing setuptools/distutils
On Thu, Oct 29, 2015 at 05:11:52PM -0400, Donald Stufft wrote: > On October 29, 2015 at 5:09:45 PM, Marcus Smith (qwc...@gmail.com) wrote: > > help me out here... how can we dynamically construct dependencies as we're > > building wheels today? > > I’m not sure I understand the confusion… since a wheel is created by > executing setup.py, you’d just have your build tool dynamically output > different wheels based on the system you’re building on (or whatever > axis is causing the dynamic dependencies). An example is like: > > https://github.com/pypa/twine/blob/a0c87357d9d5d588082c9a59f6efc6f6bc3d3498/setup.py#L28-L31 install_requires = [ "pkginfo >= 1.0", "requests >= 2.3.0", "requests-toolbelt >= 0.4.0", "setuptools >= 0.7.0", ] if sys.version_info[:2] < (2, 7): install_requires += [ "argparse", ] setup( ... install_requires=install_requires, ) But code like this doesn't work! You build a wheel on Python 2.7, you get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel cache, and when you try to install it on Python 2.6, pip tries to use the same wheel, with install_requires computed for Python 2.7 instead of 2.6. Unless you override the wheel dependencies completely in setup.cfg[+]. [*] https://github.com/pypa/twine/blob/master/setup.cfg#L2: [wheel] universal = 1 [+] https://github.com/pypa/twine/blob/master/setup.cfg#L9-L15 [metadata] requires-dist = requests >= 2.3.0 requests-toolbelt >= 0.4.0 pkginfo >= 1.0 setuptools >= 0.7.0 argparse; python_version == '2.6' Marius Gedminas -- Since this protocol deals with Firewalls there are no real security considerations. -- RFC 3093 signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Environment markers: ready for prime time?
Hi!
pip 7 enables wheel caching by default, which is good.
Wheel caching means you can't compute dynamic dependencies any more --
i.e. things like
setup(
...
install_requires=[...] + (['enum34'] if sys.version_info[0] < 3 else []),
...)
will cause problems.
As far as I understand, you're supposed to use environment markers
instead.
Problem 1: where's the documentation? E.g.
https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/
has no mention of the word "marker".
Try to google "setuptools environment marker" (and how is a user going
to discover the magical keyword they need to google is "environment
marker")? Try to find anything resembling documentation on the 1st
page. The best I could find was
http://docs.openstack.org/developer/pbr/#environment-markers which only
works if you use pbr.
Even the spec (https://www.python.org/dev/peps/pep-0426/#environment-markers)
only shows how the markers are supposed to appear in the JSON metadata.
No clue is provided for poor writers of setup.py files.
I somehow discovered the syntax once (I don't remember how -- most
likely kind people in #pypa spoon-fed me), but now I'm cargo-culting my
existing setup.py files that already use environment markers.
Problem 2: what are the minimum versions of the tools that your users
must have before you can rely on environment markers?
- setuptools >= 0.7 ("Added experimental environment marker support")
- wheel >= 0.24 (if you have wheel 0.23 or older, environment markers are
silently broken and have fun figuring out why:
https://github.com/pypa/pip/issues/2870).
- does the pip version matter at all? I think not; please correct me if
I'm wrong.
Some official answers from the hard-working PyPA visionaries would be welcome.
Marius Gedminas
--
I once held a little hand
That made my sad heart sing.
Twas the loveliest hand I'd ever held,
Four Aces and a King
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Where should I put tests when packaging python modules?
On Tue, Oct 06, 2015 at 05:21:27PM -0400, Barry Warsaw wrote: > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, > >so it apparently doesn't find it useful either. > > Oh how I wish there was a standard way to *declare* how to run the test suite, > such that all our automated tools (or the humans :) didn't have to guess. I have hopes for 'tox.ini' becoming the standard way to test a Python project. Marius Gedminas -- "Actually, the Singularity seems rather useful in the entire work avoidance field. "I _could_ write up that report now but if I put it off, I may well become a weakly godlike entity, at which point not only will I be able to type faster but my comments will be more on-target."- James Nicoll signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 503 - Simple Repository API
On Fri, Sep 04, 2015 at 09:17:18PM -0400, Donald Stufft wrote:
> You can see this PEP online at https://www.python.org/dev/peps/pep-0503/ or I
> have reproduced it inline below.
>
...
> Below the root URL is another URL for each individual project contained within
> a repository. The format of this URL is ``//`` where the
>
> is replaced by the normalized name for that project, so a project named
> "HolyGrail" would have an URL like ``/holygrail/``. This URL must response
> with
Typo: "must response" -> "must respond"
> a valid HTML5 page with a single anchor element per file for the project. The
> text of the anchor tag **MUST** be the filename of the file and the href
> attribute **MUST** be an URL that links to the location of the file for
> download. The URL **SHOULD** include a hash in the form of an URL fragment
> with
> the following syntax: ``#=``, where is the
> lowercase name of the hash function (such as ``sha256``) and
> is
> the hex encoded digest.
>
> In addition to the above, the following constraints are placed on the API:
>
> * All URLs **MUST** end with a ``/`` and the repository **SHOULD** redirect
> the
> URLs without a ``/`` to add a ``/`` to the end.
(except for URLs pointing to downloadable files, I presume?)
> * There is no constraints on where the files must be hosted relative to the
> repository.
> Normalized Names
>
>
> This PEP references the concept of a "normalized" project name. As per PEP 426
> the only valid characters in a name are the ASCII alphabet, ASCII numbers,
> ``.``, ``-``, and ``_``.
For a second I thought this referred to a normalized name and got
confused ("I thought _ and - were interchangeable?").
Maybe it's worth clarifying with s/in a name/in a ("unnormalized")
name/. Or maybe not -- the rest of the paragraph clears away the
confusion quickly enough.
> The name should be lowercased with all runs of the
> characters ``.``, ``-``, or ``_`` replaced with a single ``-`` character. This
> can be implemented in Python with the ``re`` module::
>
> import re
>
> def normalize(name):
> return re.sub(r"[-_.]+", "-", name).lower()
Oh, excellent! Having this documented briefly and clearly is most welcome!
Marius Gedminas
--
yeah, i'm reading the answers, currently
but what I see is that there is no real procedure to rebuild initfs
the common way seems to use a loop device with a jffs2 filesystem, put
original files there, and add other files, then umount, flash and pray
Corsac: You forgot "ritual sacrifice of a medium sized rodent".
Without that, it'll never work.
And if it doesn't work the first time, re-adjust towel ordering in the
restroom and try again
-- #maemo
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Virtualenv bug breaking tox runs
On Wed, Jul 15, 2015 at 12:55:50AM -0400, Randy Syring wrote: I'm trying to use tox from python2 to test a python3 environment. But I get an exception like the following: rsyring@loftex:~/projects/blaze/blazeutils-src$ python -m virtualenv --python python3 ~/tmp/testvenv Running virtualenv with interpreter /home/rsyring/bin/python3 Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/virtualenv.py, line 14, in module import shutil File /opt/python34/lib/python3.4/shutil.py, line 11, in module import fnmatch File /opt/python34/lib/python3.4/fnmatch.py, line 15, in module import functools File /opt/python34/lib/python3.4/functools.py, line 21, in module from collections import namedtuple File /opt/python34/lib/python3.4/collections/__init__.py, line 17, in module from reprlib import recursive_repr as _recursive_repr File /usr/local/lib/python2.7/dist-packages/reprlib.py, line 3, in module from repr import * ImportError: No module named 'repr' There is a bug for this in virtualenv: https://github.com/pypa/virtualenv/issues/625 So, I realize I could use `python3 -m virtualenv` if I was running the command myself. But, tox is making the call to virtualenv. This was working the last time I tested it but has now broken. I'm guessing that is because I upgraded tox or virtualenv? It broke now because you 'sudo pip install'ed something that depends on something that ships /usr/local/lib/python2.7/dist-packages/reprlib.py (which probably comes from pies2overrides) Any pointers welcome as to the best way to resolve this issue now? I would recommend sudo pip uninstall pies2overrides (and whatever depended on it). A good way to avoid pain is to never ever 'sudo pip install' stuff. A different workaround would be to create a clean virtualenv, install tox into it, then run tox from that virtualenv. Marius Gedminas -- Un*x admins know what they are doing by definition. -- Bernd Petrovitsch signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to implement ‘setup.py’ functionality that itself needs third-party distributions (was: Module from install breaks subsequent install of different distribution)
On Tue, Jan 20, 2015 at 12:50:14PM +1100, Ben Finney wrote: Tres Seaver tsea...@palladion.com writes: setuptools itself is extensible by means of entry points. Both entry points in your example (as cited by Marius) demonstrate that: one adds support for a new keyword argument to 'setup()'[1], and the other defines a new writer for 'egg-info'[2]. By design, both of those are supposed to be loaded / available for any invocation of 'setup()' in a Python where the are installed (not merely for packages which mention them). What recourse do I have, then? I'm using entry points because it seems to be the only way I can declare functionality that resides in a module alongside the ‘setup.py’ which itself needs third-party packages. Are you aware that those entry points will be used for every single package the user tries to install after installing python-daemon? * During the distribution build stage (‘./setup.py build’ or earlier), I want to parse a reST document and derive some of the distribution metadata from that. And presumably you don't want to use regexps for that. ;) * The ‘setup.py’ is the wrong place for this; it's complex and deserves its own module which can be imported for unit testing. * This also is totally unrelated to the functionality this distribution installs, so it doesn't belong in any of the packages to distribute for installation. * So I place it in a separate top-level module, ‘version’, only for use within ‘setup.py’. These three items sound very reasonable and shouldn't be causing problems. * That module itself needs a third-party distribution (‘docutils’) from PyPI. So I need that distribution to be installed *before* the ‘version’ module can be used. Apparently the ‘setup_requires’ option is the only way to do that. This is the point where my experience with setup_requires would tempt me to go hey, regexes aren't _that_ bad. ;) * Then, in order to declare how that functionality is to be used for commands like ‘./setup.py egg_info’, I have no other way than declaring a Setuptools entry point. I think there's also the option of providing custom command classes to the setup() function. For completeness' sake there's also the option of monkey-patching distutils/setuptools internals (not recommended). * Declaring a Setuptools entry point makes Setuptools think the distribution-specific module must be imported by every other distribution in the same run. *And in every subsequent run.* This is important. Entry points live in the installed package metadata. You shouldn't ever define an entry point that points to a package or module that won't be installed. Of course it's not available there so those distributions fail to install. * It even thwarts the installation of ‘docutils’, the very distribution that is needed for ending this circular dependency. What am I missing? How can I implement complex functionality specific to packaging this distribution, without making an utter mess? I'll attempt some suggestions: 1. Simplify metadata extraction so it doesn't rely on docutils. 2. Implement metadata extraction using custom command classes[*] and setup_requires. (Sorry about the lack of specifics: I've never done this. I'm not even 100% certain the things you want can be done this way.) [*] https://docs.python.org/2/distutils/extending.html#integrating-new-commands, the bits about setup(..., cmdclass=...); not the bits about command_packages. 3. Duplicate the metadata in your setup.py; have a unit test that extracts it from your ReST and from your setup.py and compares the two. Since your release process should involve a run all tests step, this ought to prevent you from making releases with mismatched metadata. (Not a big fan of this one, but it's better than some alternatives, like having duplicated metadata that must be checked for consistency by hand.) 4. Move your entry points into an installed module that checks the name of the package it's working on and does nothing if it's not python-daemon. (Not a fan of this one, it adds a moving part that can break any subsequent Python package installation.) 5. Use some replacement of setup_requires as suggested elsewhere in this thread, so you could import docutils at the top level of a setup.py. (I've never tried this approach and have no idea how reliable it is.) HTH, Marius Gedminas -- Since this protocol deals with Firewalls there are no real security considerations. -- RFC 3093 signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Module from install breaks subsequent install of different distribution (was: Python module for use in ‘setup.py’ but not to install)
installing docutils
(in the same process!), and your code makes assumptions about the
package it's used with ('there's a version.py and a ChangeLog in the
current working directory') that are false when it's used with other
packages, installed via setup_requires.
HTH,
Marius Gedminas
--
I think one of the enduring tragedies of the 22nd century will be that during
the 20th and 21st centuries we persistently treat nuclear reactors as if
they're nuclear weapons, and nuclear weapons as if they're nuclear reactors.
-- Charlie Stoss
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools 8 changes are great, but ...
://jenkins.simplistix.co.uk/job/checker-buildout/ Marius Gedminas -- To express oneself In seventeen syllables Is very diffic -- John Cooper Clark. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools 8 changes are great, but ...
On Mon, Dec 15, 2014 at 01:45:28PM -0500, Donald Stufft wrote: On Dec 15, 2014, at 1:27 PM, Jim Fulton j...@zope.com wrote: I think the changes in version management in setuptools 8 are a great step forward, but I think the transition is going to hurt a lot. For buildout, I'm thinking of of releasing 2.3.1 that reverts the changes in 2.3 and adds a requirement for setuptools 8 to give more time to respond to these changes. Is there something I’m not aware that is broken currently? I thought the transition was going pretty smoothly overall considering that a core piece of code inside of setuptools was touched. ~80 zope.* test builds are currently failing, for mysterious reasons due to setuptools 8.0.x not interacting well with zc.buildout: https://mail.zope.org/pipermail/zope-dev/2014-December/046509.html Here's a summary of the various errors: https://mail.zope.org/pipermail/zope-dev/2014-December/046508.html Newer builds also added a bunch of warnings of the form /var/lib/jenkins/jobs/zopetoolkit_trunk/workspace/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'zc.recipe.testrunner-1.0.5 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions. Recent threads on distutils-sig@ help explain some of what's happening. E.g. the 'zope.app.wsgi3.11,4.0dev,=3.12' probably used to be interpreted as an 3.11 or =3.12 and just needs to be hunted down and replaced with !=3.11.*, or something like that. (It's painful when you get requirement conflict errors with no indication about the source of those requirements.) Marius Gedminas -- The difference between Microsoft and 'Jurassic Parc': In one, a mad businessman makes a lot of money with beasts that should be extinct. The other is a film. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Cookie-related PyPI 503 errors
I keep getting 503 errors from PyPI that go away after I clear my cookies. The entire cycle is as follows: 1. Visit PyPI, log in (because I need to do some maintenance or something of my packages). 2. Things continue to work for about a week. 3. Visit PyPI, get a 503 error from Varnish after a noticeable timeout: Error 503 backend read error backend read error Guru Mediation: Details: cache-iad2132-IAD 1418630124 4121731453 Varnish cache server 4. Clear the PyPI cookie, reload page, things work again. 5. Go back to step 1. Having to clear my cookies about once a week is getting annoying. Can this be fixed please? Marius Gedminas -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What binary formats should we be distributing?
On Thu, Nov 27, 2014 at 11:25:41PM +, Paul Moore wrote: On 27 November 2014 at 17:47, Randy Syring randy.syr...@level12.io wrote: pymssql uses Cython and the FreeTDS C library to support connections to Microsoft SQL Server. We would like to distribute binary packages to ease installation for users. Our current build matrix looks like this: https://github.com/pymssql/pymssql/wiki/Build-Matrix However, we are trying to cut that down some since, by the time we support 5 versions of python (2.6,2.7,3.2,3.3,3.4), we are having to release 30 and now, since we are considering .msi files, maybe 40 files per release. I'm trying to figure out if there are any downsides to: - replacing our .exe with .msi (lxml uses .exe, but we noticed Python itself is shipping .msi files now) - only shipping wheels and no longer shipping eggs I would also appreciate any other thoughts regarding what kind of binary formats a responsible package owner would ship. I can only really comment on Windows, but I would say that wheel is sufficient there. The longer term intent is that wheel supersedes all of the other binary formats as a universal standard. There's a way to go yet, so there may be reasons to still ship other formats, but you should be looking to phase them out once your users no longer need them. I would strongly recommend against msi, as they are not typically useful for Python packages. Sure, the Python distribution itself uses msi, but the reasons for that are not really applicable to Python packages. The bdist_msi format is very rarely used in practice, and that is likely to decrease rather than increase. The one proviso that I would make is that this is from a hobbyist/open source perspective. I'd guess from the fact that you're distributing a SQL Server library that you have a larger than average proportion of corporate users. In that sort of environment, MSI installers may be more useful, as they integrate better with corporate deployment tools. But that's not something I can really comment on. The wininst (exe) format is probably of limited value, It is of use for systems without pip where integration with add/remove programs is useful, but such systems are probably rare. But if you want to distribute a standalone installer format, exe is probably more familiar to Python package users than msi. On the downside, it doesn't work with virtualenvs. As regards eggs, they are essentially the precursor to wheels as a binary distribution format. Users still using easy_install will need eggs (but why aren't they using pip these days?). It's possible that older systems such as buildout may still need eggs, but I'm not familiar with these systems and don't know if they can use wheels instead these days. Buildout is based on easy_install and doesn't support wheels. It supports .egg or, I believe, .exe installers. I'm not sure about .msi. If setuptools were to support wheels, buildout would get that support automatically, AFAIU. (Why doesn't setuptools support wheels natively?) Wheel files can be created automatically from .egg/.exe files using 'wheel convert', so supporting both shouldn't be a big hassle and can be automated. Marius Gedminas -- 2B OR NOT 2B == FF signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files
On Fri, Nov 28, 2014 at 04:03:59PM +1000, Nick Coghlan wrote: Here's my proposed change: = The default platform tag is distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _ . If /etc/os-release [N] exists on the system, then the values in the 'ID' and 'VERSION_ID' fields are read, all hyphens - and periods . replaced with underscore _ , and the results appended to the default tag after a separating underscore. Examples: * win32 * macosx_10_6_intel * linux_x86_64_fedora_20 * linux_x86_64_rhel_7_0 * linux_x86_64_debian_7_0 * linux_x86_64_ubuntu_14_04 = The [N] reference would then be a reference to http://www.freedesktop.org/software/systemd/man/os-release.html for the definition of the format of os-release. (Note that while the format originated with systemd, plenty of distros have also started providing it regardless of which init system they use) ... This also won't help with older Linux distros that don't offer /etc/os-release, but I'm OK with that - those can just continue to show up as linux_x86_64, and PyPI can continue to disallow those uploads. I was curious about what older distros meant in the context of Ubuntu, so I looked it up: /etc/os-release exists in Ubuntu 12.04 LTS (and newer) but didn't exist in Ubuntu 10.04 LTS. Support for Ubuntu 10.04 LTS ends in April 2015. Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?
On Fri, Nov 07, 2014 at 03:46:36PM +, Paul Moore wrote: I'm in the process of developing an automated solution to allow users to quickly set up a Windows box so that it can be used to compile Python extensions and build wheels. While it can obviously be used by Windows developers who want to quickly set up a box, my main target is Unix developers who want to provide wheels for Windows users. To that end, I'd like to get an idea of what sort of access to Windows a typical Unix developer would have. I'm particularly interested in whether Windows XP/Vista is still in use, and whether you're likely to already have Python and/or any development tools installed. Ideally, a clean Windows 7 or later virtual machine is the best environment, but I don't know if it's reasonable to assume that. Another alternative is to have an Amazon EC2 AMI prebuilt, and users can just create an instance based on it. That seems pretty easy to do from my perspective but I don't know if the connectivity process (remote desktop) is a problem for Unix developers. Any feedback would be extremely useful. I don't maintain any Python packages with C extensions, so I'm not sure my feedback is useful. Nevertheless: I have a cloud VM running Windows Server WhicheverWasTheHighestNumberAtTheTime with no C compilers on it. (I don't think the bits of Mingw that come with Git for Windows include gcc.) I have all the relevant versions of Python installed on it, with added setuptools and pip in each. IIRC I also installed tox and virtualenv into their site packages. I use this VM as a Jenkins slave to run tests of various Python packages. Some of them need binaries, and I rely on package maintainers to upload Windows wheels to PyPI. Since *of course* they don't all do that, so I have to maintain a set of wheels automatically converted from .exe and .egg installers with https://github.com/mgedmin/wheelwright. Anything that makes it easier for package maintainers build Windows wheels would be very welcome! Marius Gedminas -- For every complex problem, there is a solution that is simple, neat, and wrong signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a package with is a module
On Mon, Oct 27, 2014 at 06:47:45AM -0700, Ethan Furman wrote: If there is an answer, tutorial, readme, or docs that would help, a link would be greatly appreciated. I have finally converted my dbf library to Python3, but I want to also continue supporting at least 2.7, and I see no reason to remove the existing library which also supports back to 2.4. My objgraph is a single-file Python module that supports Pythons from 2.4 to 3.4. [*] [*] Well, it _probably_ supports 2.4 and 2.5; I don't actually test that because tox no longer supports 2.4 or 2.5 (and tox dropped support for those Pythons because virtualenv dropped support for them, IIRC). So I have multiple problems: - how do I tell PyPI that this file is for Python2.4 - 2.6, this other file is for 2.7, and this other other file is for 3.3+ ? Maintaining three separate files is a pain. Most people who support multiple Python versions would recommend sticking to a single source written in the common language subset. It's not that hard. http://python3porting.com/noconv.html has helpful suggestions. If you don't want to rely on the `six` module from PyPI, you can always copy the tricks it performs and do them yourself. - if I have to stick it all in one archive, how do I tell setup.py which to install? Even if that's possible, I don't think it's a good idea. - is it possible to have all three source files as, say, .txt files, and then have some Python code before the setup() call which renames the correct one to dbf.py? If you go this route I suggest copying instead of renaming. How do I know where the .txt files are at to rename them? Many setup.py files fail if you run them when your working directory is not the one where the setup.py resides itself. I think you can safely rely on that implementation detail. Marius Gedminas -- If the facts don't fit the theory, change the facts. -- Albert Einstein signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] advice re: packaging tasks
On Thu, Oct 02, 2014 at 09:15:54AM -0700, Chris Jerdonek wrote: On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas mar...@pov.lt wrote: I wrote restview for a different purpose, but found it rather useful for discovering ReStructuredText problems that would make PyPI's fall back to plaintext rendering. If you run restview --long-description in a directory containing a setup.py, it'll invoke `python setup.py --long-description' and interpret it using the same docutils settings as PyPI, highlighting any errors. For a more automated (but perhaps less accurate) solution I pipe python setup.py --long-description | rst2html /dev/null Yes, I had been familiar with this latter approach. It is documented here: https://docs.python.org/2/distutils/packageindex.html#pypi-package-display It did not work for me in my case though. I eventually found that my use of URL fragments to link to sections elsewhere on the same page, and using relative links (supported by GitHub, https://github.com/blog/1395-relative-links-in-markup-files ) are what broke things. Incidentally, this gotcha is why restview now has a --pypi-strict mode: https://github.com/mgedmin/restview/issues/18#issuecomment-54122181 (This is enabled by default for restview --long-description.) Here is a link to a PyPI bug report regarding making it easier to find such issues: https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way If people are interested, I wrote a pandoc filter to convert such links to things that will continue to work once uploaded to PyPI: https://gist.github.com/cjerdonek/76608610df43fd5b0fc3 ... Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? I'm not sure what you mean. Do you have any examples? I mean that if you are working on project MyProject with package myproject inside the repo, you might find yourself adding ad hoc custom code to setup.py. If this setup.py code starts to grow (a bit like your Makefile may grow), it might make sense to move some of the setup code to a package called something like myproject_setup alongside myproject (which would be used only for setup tasks). And if this setup code was sufficiently complicated, you might find yourself wanting to add unit tests for some of it, so you might have myproject_setup/tests (and even testing it as part of your automated tests, etc). Right. I wanted to see some concrete examples of that code you find putting in your setup.py files. All I've seen before were bits that concatenate README.rst + CHANGES.rst into a long_description, or parse the version number from some source file in the name of DRY. Marius Gedminas -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet? signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting
On Fri, Oct 03, 2014 at 02:05:50AM -0400, Donald Stufft wrote: Using this additional location within pip is also simple and can be included on a per invocation, per shell, or per user basis. The pip 6.0 will also include the ability to configure this on a per virtual environment or per machine basis as well. This can be as simple as: :: $ # As a CLI argument $ pip install --extra-index-url https://index.example.com/ myproject $ # As an environment variable $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject $ # With a configuration file $ echo [global]\nextra-index-url = https://pypi.example.com/; ~/.pip/pip.conf $ pip install myproject This is where I get a question: what do I do if package X wants an extra repository FOO, and package Y wants an extra repository BAR, and my project relies on both X and Y? I assume the --extra-index-url=URL argument to pip install can be repeated multiple times. It's less clear what to do about environment variables or config file settings. Do I specify space-separated URLs? Newline separated? An example would be good. Marius Gedminas -- Jim's Three Laws of Engineering: 1. F = ma 2. You can't solve a problem unless you know the answer 3. You can't push a rope signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] advice re: packaging tasks
On Tue, Sep 30, 2014 at 04:44:09PM -0700, Chris Jerdonek wrote: I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end. 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run python setup.py sdist. Are there better solutions for this out there so I don't have to rely on my own hack? I wrote check-manifest for this. Prior to that I used to have horrific Makefile rules that did a similar check: https://github.com/mgedmin/objgraph/blob/master/Makefile#L69 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? I wrote restview for a different purpose, but found it rather useful for discovering ReStructuredText problems that would make PyPI's fall back to plaintext rendering. If you run restview --long-description in a directory containing a setup.py, it'll invoke `python setup.py --long-description' and interpret it using the same docutils settings as PyPI, highlighting any errors. For a more automated (but perhaps less accurate) solution I pipe python setup.py --long-description | rst2html /dev/null in my Makefile: https://github.com/mgedmin/objgraph/blob/master/Makefile#L99 Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo? I'm really happy with being able to make releases of any of my packages with make release This runs the test suite (tox/detox FTW), checks for uncommitted changes, makes sure MANIFEST.in is correct, makes sure the version number in setup.py matches the version number in CHANGES.rst (and release date is today), makes sure there are no RST errors in long_descrption, and reminds me the commands I need to run to creates a version control system tag and package + upload the sdist to PyPI. (I'm too paranoid to let automated tools perform externally-visible actions like uploading to PyPI or pushing git tags, so I just copy paste the commands once I'm sure everything's ship-shape.) There are other excellent tools for this (e.g. zest.releaser). One day I'll get tired of copy-pasting my Makefiles from project to project and I'll switch to zest.releaser. Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? I'm not sure what you mean. Do you have any examples? Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sun, Sep 28, 2014 at 02:21:11PM -0700, Chris Jerdonek wrote: Would this also affect the ability to update the readme information for a version on PyPI (i.e. the information displayed on the default home page generated by PyPI for the release) once the version has already been uploaded to PyPI? There are sometimes issues encountered with the display of that information that can't easily be checked without doing an actual version upload. restview has a --pypi-strict mode for this use-case #shamelessplug https://pypi.python.org/pypi/restview I haven't recently tried reuploading the metadata for a version, mainly because of uncertainty around how PyPI would handle it. In the past when I needed to fix stupid mistakes in my long_description, I'd edit it using the web interface. arius Gedminas -- if (DefRel.empty() == false) -- apt-pkg/policy.cc (apt 0.5.23) signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels or system packages for pip on ubuntu
On Wed, Sep 03, 2014 at 12:24:10PM +0200, Reinout van Rees wrote: I'm investigating some options for making our servers a bit more neat. Basic problem: lots of what we do needs mapnik, numpy, gdal, psycopg2 and so. Python libraries with C code and system dependencies. All of them have ubuntu packages, but especially for gdal we sometimes need a newer version. A PPA can help here, but I thought a wheel could be nice, too. System packages? Yes, we use buildout with syseggrecipe. You pass syseggrecipe a bunch of packages (mapnik, gdal), it looks up those packages in the OS and installs them in buildout's develop-eggs/ directory. Works quite well. Isolation + selective use of system packages. Do you use buildout 1.x then? buildout 2.x doesn't support isolation, so all system packages are available (unless you wrap it with a virtualenv). Two questions: a) If I use ubuntu packages, I'll have to run pip/virtualenv with --system-site-packages. pip install numpy will find the global install just fine. But pip freeze will give me all site packages' versions, which is not what I want. = is there a way to *selectively* use such a system package in an otherwise-isolated virtualenv? You can symlink selected directories (the Python package directory and sometimes .so files from other subdirs) into the virtualenv. This is hacky and I know of no tool to automate it. b) Making a bunch of wheels seems like a nice solution. Then you can just use a virtualenv and pip install numpy gdal psycopg2 But how do you differentiate between ubuntu versions? Not every wheel will work with both 12.04 and 14.04, I'd say. But both ubuntu versions will have the same linux_x86_64 tag. = What is the best way to differentiate here? Separate company-internal wheelhouse per ubuntu version? Custom tags? My gut feeling says go with a company-internal wheelhouse. A simple directory with a bunch of *.whl files exposed via Apache's mod_autoindex suffices. export PIP_FIND_LINKS=https://intranet.server/wheels/trusty/ to make use of it automatically. $DISTRIB_CODENAME, defined by sourcing /etc/lsb-release, can be helpful here. Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Accepting PEP 440: Version Identification and Dependency Specification
On Fri, Aug 22, 2014 at 10:34:39PM +1000, Nick Coghlan wrote: I just pushed Donald's final round of edits in response to the feedback on the last PEP 440 thread, and as such I'm happy to announce that I am accepting PEP 440 as the recommended approach to identifying versions and specifying dependencies when distributing Python software. The PEP is available in the usual place at http://www.python.org/dev/peps/pep-0440/ Awesome! Minor nit: http://legacy.python.org/dev/peps/pep-0440/#final-releases still uses the older N[.N]+ spelling, which perhaps should be changed to N(.N)* to be consistent with http://legacy.python.org/dev/peps/pep-0440/#public-version-identifiers and to make it more apparent at a glance that one number with no dots is also a valid version identifier. Marius Gedminas -- NT 5.0 is the last nail in the Unix coffin. Interestingly, Unix isn't in the coffin... It's wondering what the heck is sealing itself into a wooden box 6 feet underground... -- Jason McMullan signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP470, backward compat is a ...
On Fri, May 16, 2014 at 07:12:01PM +, holger krekel wrote: On Fri, May 16, 2014 at 13:38 -0500, Carl Meyer wrote: One option is Holger's solution: scraping the current links and populating them as verified external links. We both don't like this because it involves PyPI misleading users about the status of those links, and you also don't like it because you want to deprecate verified external links too. Regarding the last bit, i indeed don't want to see the possibility removed to register external links-with-checksum. PEP438 just introduced it and i think it's sufficient and efficient for allowing maintainers to host files externally while making it easy for end users choose to accept it along with a slight loss of reliability that --allow-externals might entail. Integrity guruantees of such external links and pypi-uploaded files are strictly the same. Well, not with MD5. Marius Gedminas -- It's my understanding that although in principle TCP can handle huge throughputs in practice many stacks haven't been optimized for that case, so you have to either use a utility which opens multiple TCP sessions in parallel or do something really radical like upgrade to the latest version of the linux kernel. -- Bram Cohen signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip install -e . vs. python setup.py develop
On Sun, Apr 13, 2014 at 04:08:42PM -0400, Noah Kantrowitz wrote: On Apr 11, 2014, at 1:29 PM, Chris Withers ch...@simplistix.co.uk wrote: On 07/04/2014 04:05, Noah Kantrowitz wrote: You should recommend using pip for it, mostly because as you said that will work even with packages that don't use setuptools :-) It also is required when doing a develop install with extras, though that requires a slightly more verbose syntax due to a bug in pip. What's the syntax? pip install -e file:///path/to/thing[one,two] What's the bug? https://github.com/pypa/pip/pull/500 was merged two years ago and was supposed to enable pip install -e .[one,two] Marius Gedminas -- What can I do with Python that I can't do with C#? You can go home on time at the end of the day. -- Daniel Klein signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse XMLRPC
On Fri, Mar 07, 2014 at 01:03:23AM -0500, Donald Stufft wrote: The JSON API is functional again and is available for testing! Excellent! I can observe two differences with the current PyPI code: First, the Content-Type of the response differs: $ curl -sI https://pypi.python.org/pypi/setuptools/json | grep Content-Type Content-Type: application/json; charset=UTF-8 $ curl -sI https://pypi-preview.a.ssl.fastly.net/pypi/setuptools/json | grep Content-Type Content-Type: application/json (I don't know which is correct, or whether the difference actually matters.) Second, the current PyPI codebase returns absolute URLs, while Warehouse returns paths relative to the site root: $ curl -s https://pypi.python.org/pypi/setuptools/json | grep 'url:' url: https://pypi.python.org/packages/3.4/s/setuptools/setuptools-2.2-py2.py3-none-any.whl;, url: https://pypi.python.org/packages/source/s/setuptools/setuptools-2.2.tar.gz;, $ curl -s https://pypi-preview.a.ssl.fastly.net/pypi/setuptools/json | python -m json.tool | grep 'url:' url: /packages/3.4/s/setuptools/setuptools-2.2-py2.py3-none-any.whl url: /packages/source/s/setuptools/setuptools-2.2.tar.gz I can adapt my script to cope. Marius Gedminas -- Key emulation: [ ] Intuitive [*] Emacs(Seen in an MCEdit dialog) signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] recommended setuptools version
On Fri, Mar 07, 2014 at 02:02:59AM -0800, Benedek Zoltan wrote: I'm a bit confused with the setuptools versioning. When I install it by ez_setup.py it installs the 3.0.2 version, but on pypi the latest version is 2.2. Is there a difference between the two? Which is the recommended one? 3.0 was released last night, broke a bunch of popular software, and was hurriedly pulled from PyPI. The current version of ez_setup.py available at https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py also installs 2.2. I suggest you stick with 2.2 until the dust settles. Marius Gedminas -- 2B OR NOT 2B == FF signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse XMLRPC
On Wed, Mar 05, 2014 at 07:01:49AM -0500, Donald Stufft wrote: On Mar 5, 2014, at 5:48 AM, anatoly techtonik techto...@gmail.com wrote: JSON-RPC is a better choice for recommended external API, because it doesn't rely on memory hungry and potentially unsafe XML libraries. On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft don...@stufft.io wrote: Just a quick FYI that the last missing piece (search) for XMLRPC was merged in Warehouse today. If you have an application that depends on XMLRPC on PyPI and you’d like to test it, please try against the url: https://pypi-preview.a.ssl.fastly.net/pypi Let me know if you find any issues, or you can file a bug at: https://github.com/pypa/warehouse XMLRPC is used in order to maintain compatibility with what is already there. Something better will replace it eventually and XMLRPC will be deprecated. This is a bit confusing to me now. You're reacting as if anatoly suggested a new JSON interface to replace the legacy XML-RPC one. But the old PyPI codebase already had a JSON API[1]. I'm using it to keep track of Python 3 support status of about 800 packages maintained by the Zope Foundation: http://zope3.pov.lt/py3/ [1] https://wiki.python.org/moin/PyPIJSON I assume at some point Warehouse will add support for the JSON API and you'll issue a call for testing? Marius Gedminas -- The typewriter was invented by Hungarian immigrant Qwert Yuiop, who left his signature on the keyboard. -- Kim signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI Rate Limiting
On Sat, Feb 08, 2014 at 05:15:21PM -0500, Ernest W. Durbin III wrote: The initial rates will be limited to 5 req/s per IP with bursts of 10 requests allowed. Client requests up to the burst limit will be delayed to maintain a 5 req/s maximum. Any requests past the 10 request burst will receive an HTTP 429 response code per RFC 6585. On Mon, Feb 10, 2014 at 01:49:53AM -0800, Noah Kantrowitz wrote: On Feb 10, 2014, at 1:48 AM, Chris Jerdonek chris.jerdo...@gmail.com wrote: Also, if the server isn't artificially slowing down requests, what does, Client requests up to the burst limit [of 10 requests] will be delayed to maintain a 5 req/s maximum mean? Any requests beyond the rate limits will get an HTTP 503 with an empty body. So is it 429 or 503? And is there a delay or not? Marius Gedminas -- Linux is a fast moving project, with very fast evolving components. If you're using an older distribution, older than 4 to 6 months (and anything with Enterprise in the name is by definition old), please consider going to a newer distribution. -- http://www.linuxpowertop.org/known.php signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP-459 feedback from openSUSE packaging
On Tue, Feb 04, 2014 at 12:43:40PM +0100, Sascha Peilicke wrote: The 'license' metadata tag is causing the most issues for us. A perceived 50% just put GPL in there. Which GPL version? GPL-only or actually LGPL?. FWIW you can sometimes get more detailed information from the classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers E.g. setup(... licence='GPL', classifiers=[ ..., 'License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)', ... ]) Marius Gedminas -- Unix gives you enough rope to shoot yourself in the foot. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python Packaging User Guide has moved
On Thu, Jan 16, 2014 at 09:20:43AM -0500, Daniel Holth wrote: Has anyone ever written a setup.py that was *not* copy-and-pasted from somewhere else? Presumably the 1st setup.py had to have been written by hand somehow. Marius Gedminas -- Five words to strike fear into the heart of any software developer: 'How long will it take?' -- Sean McGrath ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI download issues from Rackspace Cloud
On Mon, Nov 25, 2013 at 10:38:09AM +0200, Marius Gedminas wrote: My experimental code: https://gist.github.com/mgedmin/7637559 I now have two Wireshark pcap files of two SSL conversations: one failed, one successful. It's a bit beyond my skills to analyze them. I think the server did send everything (I see a TLSv1 Application Data record of size 5654, which looks like the final chunk), but Wireshark marks it as [TCP Out-Of-Order], and then there are some desperate-looking [TCP Retransmission] packets at the end. (Incidentally, the captured download was truncated at 1303669 bytes -- i.e. it was missing the last three TLS application data chunks of 8000, 8000 and 5634 bytes.) Given that Firefox/curl seem to be able to download PyPI packages over HTTPS without problems, I'd be inclined to blame it on a bug in the CPython's ssl module, maybe. But doesn't Requests use it too? I never got to the bottom of this. I upgraded pip to 1.5 (painfully -- had to download the .tar.gz with curl, then use python -m pip install -U pip*.tar.gz to upgrade) and these SSL download errors disappeared. Marius Gedminas -- Actually, the Singularity seems rather useful in the entire work avoidance field. I _could_ write up that report now but if I put it off, I may well become a weakly godlike entity, at which point not only will I be able to type faster but my comments will be more on-target.- James Nicoll signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI download issues from Rackspace Cloud
Digging deeper: - SSLSocket.read() returns a premature EOF, truncating the downloaded file, which makes the md5 hash to be different. - if I fish out the SSLSocket object and set suppress_ragged_eofs = False, then I get ssl.SSLError: [Errno 8] _ssl.c:1426: EOF occurred in violation of protocol - SSLSocket.read(8192) returns chunks of 8000 bytes except the very first one (7669 bytes) and, in cases when the download does _not_ fail, the last one (5634 bytes). When the download fails, it's missing one or two last chunks (it varies). - SSLSocket.read(16384) does the same; SSLSocket.read(4096) returns pairs of chunks of 4096 and 3904 bytes, hinting that the underlying SSL communication works in multiples of 8000. My experimental code: https://gist.github.com/mgedmin/7637559 I now have two Wireshark pcap files of two SSL conversations: one failed, one successful. It's a bit beyond my skills to analyze them. I think the server did send everything (I see a TLSv1 Application Data record of size 5654, which looks like the final chunk), but Wireshark marks it as [TCP Out-Of-Order], and then there are some desperate-looking [TCP Retransmission] packets at the end. (Incidentally, the captured download was truncated at 1303669 bytes -- i.e. it was missing the last three TLS application data chunks of 8000, 8000 and 5634 bytes.) Given that Firefox/curl seem to be able to download PyPI packages over HTTPS without problems, I'd be inclined to blame it on a bug in the CPython's ssl module, maybe. But doesn't Requests use it too? Confused, Marius Gedminas -- MCSE == Marginal Computer Software Enthusiast signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI download issues from Rackspace Cloud
On Sat, Nov 23, 2013 at 08:59:35AM -0500, Donald Stufft wrote: Can you try with 1.5rc1? This was trickier than I thought, because pip appears to be incapable of upgrading itself on Windows: $ git clone https://github.com/pypa/pip $ virtualenv env $ env/scripts/pip install -U ./pip WindowsError: [Error 5] Access is denied: 'c:\\users\\mg\\...\\scripts\\pip.exe' but the advice in https://github.com/pypa/pip/issues/1299 helped: $ env/scripts/python -m pip -U ./pip We switched to requests in that version and perhaps it side steps the issue? Looks like. 10 out of 10 successful downloads with pip git master using: $ unset PIP_DOWNLOAD_CACHE $ rm -f virtualenv-1.10.1.tar.gz env/scripts/pip install -d . virtualenv As a control, I tested again with pip 1.4.1, and 2 out of 4 downloads failed with a bad hash error. Interesting. Marius Gedminas -- Those parts of the system that you can hit with a hammer (not advised) are called hardware; those program instructions that you can only curse at are called software. -- Levitating Trains and Kamikaze Genes: Technological Literacy for the 1990's. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PyPI download issues from Rackspace Cloud
I recently spin up a Windows VM on Rackspace Cloud. I'm seeing a very weird problem: downloads from PyPI fail with checksum errors, nondeterministically. Sometimes it's a md5 hash mismatch error from pip[1]. Sometimes the error is a CRC error deep in the gzip module. It's not only pip -- I've seen this error from virtualenv[2], I've seen it from get-pip.py. [1] https://jenkins.gedmin.as/job/restview-on-windows/1/console https://jenkins.gedmin.as/job/restview-on-windows/2/console https://jenkins.gedmin.as/job/restview-on-windows/3/console https://jenkins.gedmin.as/job/check-manifest-on-windows/11/console [2] https://jenkins.gedmin.as/job/check-manifest-on-windows/7/console https://jenkins.gedmin.as/job/check-manifest-on-windows/6/console The error is sporadic and tends to go away (and come back) on retries (look at the restview-on-windows jobs: first pip install mock fails, then succeeds, then fails again on next retry). I haven't encountered any download troubles with Firefox. In fact that's how I got 'pip install tox' to work in the end -- pypi downloads of virtualenv/py libraries kept failing, so I downloaded the tarballs with Firefox and pip installed them from local files. I've ran curl https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.gz | md5sum - like 10 times and got the same (correct) md5 hash each time. Meanwhile unset PIP_DOWNLOAD_CACHE rm -f virtualenv-1.10.1.tar.gz pip install -d . virtualenv fails with a bad md5hash error 9 times out of 10. Curiously, it's the same bad hash every time: 20cf17baaf2126f99d6f652831f82d75. I used http://www.python.org/ftp/python/2.7.6/python-2.7.6.msi to install Python on this Windows 2012 server (64-bit OS, 32-bit Python). Any ideas? Marius Gedminas -- IBM motto: We found five vowels hiding in a corner, and we used them _all_ for the 'eieio' instruction so that we wouldn't have to use them anywhere else -- Linus Torvalds signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse and XMLRPC
On Mon, Nov 18, 2013 at 12:19:36PM -0500, Donald Stufft wrote:
On Nov 18, 2013, at 11:54 AM, Barry Warsaw ba...@python.org wrote:
Are you planning on putting a REST API on Warehouse?
Eventually, but the primary goal right now is feature parity with PyPI legacy.
JSON API is part of the PyPI legacy: http://wiki.python.org/moin/PyPiJson
I use it to build http://zope3.pov.lt/py3/ with
https://github.com/mgedmin/ztk-py3-status/blob/master/get_pypi_status.py
(I tried pointing that script to preview-pypi.python.org, but all the
package/json pages returned {}, which, I guess, simply means the JSON
API is not implemented yet.)
I don't know whether REST API is the same thing, or if we're talking
about two different things here.
Marius Gedminas
--
Voodoo Programming: Things programmers do that they know shouldn't work but
they try anyway, and which sometimes actually work, such as recompiling
everything.
-- Karl Lehenbauer
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI v2 (Was: PyPI pull request #7)
On Sat, Nov 16, 2013 at 12:00:59AM -0500, Donald Stufft wrote: As I said, running a local copy of PostgreSQLis very easy. There are installers for Windows and OSX, and pretty much every *nix distribution will have it packaged. My ex-coworker Ignas came up with a Makefile snippet that sets up a local PostgreSQL sandbox inside the working tree: https://gist.github.com/Ignas/1676353 It's incredibly handy for Jenkins: you can clone jobs as you like without having to manually assign database names to avoid conflicts from jobs running simultaneously. All you have to do is use the Port Allocator Plugin to set unique PGPORTs for the jobs. I also like that I can git clone a project and run 'make test' without having to worry about setting up PostgreSQL databases. It probably only works on Debian/Ubuntu, and it requires that you have PostgreSQL installed, but it doesn't need a system-wide PostgreSQL to be running. Marius Gedminas -- Those who can't write, write manuals. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools and use_2to3
On Mon, Nov 11, 2013 at 04:29:32PM +, Paul Moore wrote: I'm trying to convert a setup.py to use setuptools (specifically the docutils build, FWIW). The existing setup.py has some custom command stuff to run 2to3 on build. Aside: I believe there's consensus now that 2to3 is not the best way to support Python 2.x and 3.x. It's not that hard to have a single source tree in a portable subset of Python 2.x and 3.x. See python3porting.com and https://pypi.python.org/pypi/six. As I understand it, setuptools should do this automatically by using use_2to3=True. So, I've changed the code to just do: setup( ... packages=find_packages(), use_2to3=True, ... ) However, when I do python setup.py bdist_wheel, I find that docutils/__init__.py within the wheel still contains (for example) class ApplicationError(StandardError): ... So the StandardError - Exception fixer isn't being run. Am I doing something wrong here? The setuptools documentation isn't very clear on what I should do, or what to expect. I wouldn't be surprised to learn that bdist_wheel doesn't support use_2to3, but I don't actually *know* that for a fact, since I never used 2to3. Marius Gedminas -- What goes up, must come down. Ask any system administrator. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Removing dependency_links
On Sun, Oct 27, 2013 at 03:52:04PM -0700, Marcus Smith wrote: So I asked the person I know, and this is what he said, Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library. Is there another approach or work-around he can be using? What is the right way for him to do it? e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in your app install) fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then package it, and place it in /my/forks and then pip install --find-links=/my/forks/ my_app This is a useful feature and losing it would cause some measure of pain. I tend to use this via find-links in buildout.cfg files. Not just for forks, but also for private packages not released to PyPI. Specifying dependency_links in random packages' setup.py's is a nuisance and I would rather it went away. I always turn it off by specifying allow-hosts = *.python.org in buildout.cfg So, two different things here. One is specified by the user who runs pip (or some other installation tool). The other is specified by the package builder. One is fine, the other is not. Marius Gedminas -- Never assume the reader has read the subject line. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecate and Block requires/provides
On Thu, Oct 17, 2013 at 09:59:58AM -0400, Donald Stufft wrote: On Oct 17, 2013, at 9:53 AM, Nick Coghlan ncogh...@gmail.com wrote: Yes, but breaking these users' uploads is not the way to do that. That's an incredibly user hostile step to take, and the problem doesn't even come close to justifying breaking even a not-really-working process (since you can work around the current breakage by manually installing the dependencies - you can't work around an inability to upload without making changes to your code). If you really wanted to push them towards fixing their releases, you could always have PyPI send them an email saying something like: ... This could be part of the deprecation process, but unless there's a plan in place to deprecate them I don't know how useful this email would be. It's a reactive warning to something that confuses people while they are creating a package. In other words by the time the email goes out they've already been confused. Is it possible to make 'python setup.py sdist upload' emit a warning message about using these deprecated fields, without rejecting the upload? Marius Gedminas -- Q: How many IBM CPU's does it take to execute a job? A: Four; three to hold it down, and one to rip its head off. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout bootstrap.py v1 died /?python-distribute.org down
On Mon, Oct 14, 2013 at 10:14:15PM +, Alex Clark wrote: Adam GROSZER agroszer.ll at gmail.com writes: python-distribute.org seems to be down, that breaks zc.buildout v1 bootstrap.py when used with -d to use distribute. Distribute is dead, long live Setuptools (don't use -d anymore) Are you saying that python-distribute.org was shut down on purpose and will not come back up? Marius Gedminas -- We have enough youth, how about a fountain of SMART? signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] dependency_links format changed?
On Fri, Oct 11, 2013 at 11:31:46AM -0700, Thatcher Peskens wrote: I've been looking for information on this topic, but cannot really find anything about it.. It looks like setuptools has changed something that breaks a bunch of my installs (last deployed months ago) that were working before. I don't believe anything major changed in setuptools itself, but pip 1.4 changed behavior and stopped installing unreleased package versions by default. I have some dependencies on a python package A* which is not available in PyPi, which itself has dependencies not available on PyPi. A has a setup.py containing: install_requires=[ 'django-inline-edit', ] and dependency_links = [ 'http://github.com/gabrielgrant/django-inline-edit/tarball/master#egg=django-inline-edit', ] Now... This doesn't work anymore, where it used to work before. I have found that if I specify the install_requires with an explicit version e.g. django-inline-edit==dev AND the dependency_link with #egg=django-inline-edit-dev it does work. Sounds like the pip change. My previous experience shows that it was working without the explicit version before, and what I find on the internet also seem to suggest it should work. Has this been changed on purpose, or should it be reported as a bug? It was on purpose. Marius Gedminas -- Gates' Law: Every 18 months, the speed of software halves. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] URL Structure of Packages URLs
On Tue, Oct 08, 2013 at 09:19:37AM -0400, Donald Stufft wrote: I was wondering if anyone was aware of anything that relies on the current structure of package urls, namely: /packages/source|any|etc/D/Django-1.0.tar.gz? I would like to change this but before I work up a concrete plan for people to comment on and discuss I'm trying to figure out what, if anything, depends on that current structure. I've told Launchpad that it can look for new releases at URLs like http://pypi.python.org/packages/source/g/gtimelog/gtimelog-*.tar.gz (for a few projects, not just GTimeLog). I'm not sure how Launchpad crawls those pages looking for releases; all the documentation I could find in 5 minutes was this blog post: http://blog.launchpad.net/cool-new-stuff/automatically-import-files-to-launchpad-using-product-release-finder Some Debian packages have debian/watch files using similar URL patterns to watch for new upstream releases showing up on PyPI. This mechanism is documented at https://wiki.debian.org/debian/watch/. Here's python-requests: $ apt-get source python-requests $ cat requests-1.1.0/debian/watch version=3 http://pypi.python.org/packages/source/r/requests/requests-(.*)\.tar\.gz HTH, Marius Gedminas -- Some people have told me they don't think a fat penguin really embodies the grace of Linux, which just tells me they have never seen a angry penguin charging at them in excess of 100mph. They'd be a lot more careful about what they say if they had. -- Linus Torvalds, announcing Linux v2.0 signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: On 20 September 2013 00:02, Benjamin Root ben.v.r...@gmail.com wrote: I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. shameless_plug There are tools that make this process somewhat less painful: https://pypi.python.org/pypi/check-manifest /shameless_plug Regards, Marius Gedminas -- Nobody will ever need more than 640k RAM! -- Bill Gates, 1981 Windows 95 needs at least 8 MB RAM. -- Bill Gates, 1996 Nobody will ever need Windows 95. -- logical conclusion signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 08:15:43AM -0400, Jim Fulton wrote: On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas mar...@pov.lt wrote: On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: On 20 September 2013 00:02, Benjamin Root ben.v.r...@gmail.com wrote: I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Although there's so much magic in this particular facet of setuptools that I never have much confidence in my understanding. This is why check-manifest runs 'python setup.py sdist' in a clean exported directory with no VCS metadata for verification. Did setuptools recently learn about git? For a while, an advantage of git was that setuptools ignored it. I don't believe the setuptools-git plugin was ever merged into the core. I would love an option to ignore VCS and let me specify what I want *explicitly*. Marius Gedminas -- If it ain't broke, don't fix it. - Bert Lantz signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 10:16:28AM -0400, Tres Seaver wrote: On 09/20/2013 09:11 AM, Sebastien Douche wrote: On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton j...@zope.com wrote: I would love an option to ignore VCS and let me specify what I want *explicitly*. Or remove the handling of VCS? -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. Heh. I wrote check-manifest because relying on setuptools's handling of VCS was an unacceptable bug magnet to me. (New svn working tree format _or_ you forget to install setuptools-git on this one machine? No error, just a broken sdist.) BTW if you use zest.releaser, check-manifest plugs into it and stops the release (well, asks if you REALLY want to continue, defaulting to No) if your MANIFEST.in doesn't match your VCS. Marius Gedminas -- It was the kind of night on which bad novels began. -- 1634: The Galileo Affair by Eric Flint and Andrew Dennis signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bootstrap 1.5.2, python2.6 and DistributionNotFound
On Tue, Sep 10, 2013 at 07:44:05AM +0200, Alessandro Dentella wrote:
Hi,
I'm having problems using buildout on squeeze (that uses python 2.6). On
ubuntu 12.10 (that uses python2.7) I have no problems.
I'm still usig buildout 1.5.2 as I need a recipe for virtual env that is
not yet available for buildout 2.+ (tl.buildout_virtual_python).
BTW buildout 1.7.1 exists, and should be compatible with 1.5.x.
Be sure you're using the latest version of pre-2.0 bootstrap.py by
downloading it from http:/downloads.buildout.org/1/bootstrap.py
When executing python bootstrap I'm getting:
pkg_resources.DistributionNotFound: distribute
This problem occurs also with a very simple conf that I found on
the net when I started lunar.tar.gz, I uploaded the version I'm using here
[1] after substituting bootstrap.py with a recent one.
Here what I get:
root@fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py
Downloading
http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
Creating directory '/tmp/lunar/lunar/bin'.
Creating directory '/tmp/lunar/lunar/parts'.
Creating directory '/tmp/lunar/lunar/eggs'.
Creating directory '/tmp/lunar/lunar/develop-eggs'.
Getting distribution for 'setuptools'.
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown
distribution option: 'src_root'
warnings.warn(msg)
Got setuptools 1.1.4.
Generated script '/tmp/lunar/lunar/bin/buildout'.
root@fw1-vpn-saa:/tmp/lunar/lunar# bin/buildout
Traceback (most recent call last):
File bin/buildout, line 17, in module
import zc.buildout.buildout
File
/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py,
line 40, in module
import zc.buildout.download
File
/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/download.py,
line 20, in module
from zc.buildout.easy_install import realpath
File
/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/easy_install.py,
line 31, in module
import setuptools.package_index
File
/usr/local/lib/python2.6/dist-packages/distribute-0.6.24-py2.6.egg/setuptools/package_index.py,
line 157, in module
sys.version[:3], require('distribute')[0].version
File build/bdist.linux-i686/egg/pkg_resources.py, line 673, in require
File build/bdist.linux-i686/egg/pkg_resources.py, line 576, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
pkg_resources.DistributionNotFound: distribute
If I use 'python bootstrap -d' that uses distribute, it will raise this
error even on ubuntu 12.10/python2.7:
root@fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py -d
Downloading
http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz
Extracting in /tmp/tmpIsQPGq
Now working in /tmp/tmpIsQPGq/distribute-0.6.49
Building a Distribute egg in /tmp/tmpkqSNaa
/tmp/tmpkqSNaa/distribute-0.6.49-py2.6.egg
Getting distribution for 'distribute'.
warning: install_lib: 'build/lib.linux-i686-2.6' does not exist -- no
Python modules to install
Got distribute 0.7.3.
While:
Bootstrapping.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File
/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py, line
1866, in main
getattr(buildout, command)(args)
File
/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py, line
399, in bootstrap
ws.require('zc.buildout')
File build/bdist.linux-i686/egg/pkg_resources.py, line 698, in require
needed = self.resolve(parse_requirements(requirements))
File build/bdist.linux-i686/egg/pkg_resources.py, line 596, in resolve
raise DistributionNotFound(req)
DistributionNotFound: setuptools=0.7
Any help is appreciated
Use a virtualenv to give both distribute and setuptools = 0.7 to
buildout:
virtualenv env
env/bin/pip install -U setuptools
env/bin/python bootstrap.py
bin/buildout
This works with both buildout 1.x and 2.x.
Marius Gedminas
--
Writing like a l33t script kiddie hax0r is the absolute kiss of death and
guarantees you will receive nothing but stony silence (or, at best, a heaping
helping of scorn and sarcasm) in return.
-- ESR (http://www.tuxedo.org/~esr/faqs/smart-questions.html)
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] (no subject)
On Wed, Sep 04, 2013 at 04:38:32PM -0400, Donald Stufft wrote: On Sep 4, 2013, at 3:20 PM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 09/04/2013 04:59 PM, Antoine Pitrou wrote: Then please add helpful guidelines as to how people can choose a safe and easy to remember password /or passphrase/. Most people aren't password experts, and the current one-line message isn't useful. A link here should do the trick (which succinctly sums up this entire thread): https://xkcd.com/936/ I hate that comic :| Why? Marius Gedminas -- You can't have megalomania. *I* have megalomania. -- Joe Bednorz signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout bootstrap.py doesn't work on Sabayon Linux with system python
On Thu, Aug 22, 2013 at 09:07:52PM -0700, Benedek Zoltan wrote:
I've downloaded bootstrap.py and tried to initialize with system python:
sabd1@sab /home/buildout $ wget
http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
FWIW that's an old version. You should be using one of
wget http://downloads.buildout.org/1/bootstrap.py
wget http://downloads.buildout.org/2/bootstrap.py
to get a bootstrap for zc.buildout 1.7.x or 2.2, respectively.
If you don't know which one you want, use 2.
sabd1@sab /home/buildout $ python bootstrap.py
Downloading
http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Traceback (most recent call last):
...
pkg_resources.VersionConflict: (setuptools 0.6c11
(/tmp/tmpzJN6Tt/setuptools-0.6c11-py2.7.egg),
Requirement.parse('setuptools=0.7'))
I know, it works with virtualenv, but with system python is this expected
behavior?
Basically, yes. At least it's what I've come to expect.
Here's my fool-proof method of setting up buildouts on the brave new
post-setuptools-0.7 world:
virtualenv python
python/bin/pip install -U setuptools
python/bin/python bootstrap.py
bin/buildout
You'll notice that I upgrade setuptools in the virtualenv I created.
That's because I'm using python-virtualenv from Ubuntu, and it installs
an old copy of distribute in the virtualenv by default. Then bootstrap
tries to upgrade it to the newest distribute, which is a shim that
depends on new setuptools, but bootstrap's upgrader isn't smart enough
to go and fetch dependencies so it all breaks down in the same error you've
seen. Upgrading setuptools with pip avoids this failure.
Marius Gedminas
--
Beware of bugs in the above code; I have only proved it correct, not tried it.
-- Donald Knuth
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Preemptive Apology for Volume of Mail
On Tue, Jun 04, 2013 at 02:49:58AM -0400, Donald Stufft wrote: I *was* smart enough to make sure it only emailed for packages that it actually needed too. Just not smart enough not to send people one per package :/ Heh, for some packages I actually got two emails. I'm guessing I'm listed twice in the roles table. Stephan Richter had a script that granted access to all of zope.* to somebody; perhaps I was manually to a couple of packages before that script ran. Or something. PyPI could use something like GitHub organization accounts. Marius Gedminas -- C is a language that combines all the elegance and power of assembly language with all the readability and maintainability of assembly language. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Buildout can't talk to PyPI any more?
Could it be that recent PyPI CDN changes broke zc.buildout somehow? Witness this: $ wget http://downloads.buildout.org/2/bootstrap.py $ cat buildout.cfg [buildout] parts = zodbbrowser [zodbbrowser] recipe = zc.recipe.egg $ python bootstrap.py $ bin/buildout Error: Couldn't find a distribution for 'zodbbrowser'. Meanwhile both $ virtualenv env env/bin/pip install zodbbrowser and $ virtualenv env env/bin/easy_install zodbbrowser work fine. Marius Gedminas -- Those who can't write, write manuals. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout can't talk to PyPI any more?
On Wed, May 29, 2013 at 11:03:39AM +0200, Lennart Regebro wrote: On Wed, May 29, 2013 at 10:19 AM, Marius Gedminas mar...@pov.lt wrote: Could it be that recent PyPI CDN changes broke zc.buildout somehow? Witness this: $ wget http://downloads.buildout.org/2/bootstrap.py $ cat buildout.cfg [buildout] parts = zodbbrowser [zodbbrowser] recipe = zc.recipe.egg $ python bootstrap.py $ bin/buildout Error: Couldn't find a distribution for 'zodbbrowser'. Works for me. Works for me too, now. Interesting detail: when buildout was failing, and I tried curl http://pypi.python.org/simple/zodbbrowser/ all I got was a bunch of binary garbage. To be fair, curl -I showed Content-encoding: gzip, and piping curl to zcat showed the HTML just fine. But the interesting thing is that when I try this now, I get raw HTML. Can it be that buildout doesn't handle Content-Encoding: gzip? I was also looking at raw HTTP requests with ngrep, and saw that buildout issued something like GET /simple/zodbbrowser HTTP/forgotwhichversionsadly Host: ... Accept-Encoding: identity User-Agent: blah blah distribute 0.6.44 blah blah Python/urllib That Accept-Encoding header kind of hints that distribute can't handle gzipped index pages. So maybe Varnish or nginx or something on the CDN side was misconfigured and ignored the Accept-Encoding? But why would pip/easy_install work then -- they use distribute too!? (Or was I accidentally using a virtualenv that had setuptools instead of distribute? Python packaging aargh.) Another possibly interesting detail: I made a release of zodbbrowser 0.11.0 this morning. Buildout was unable to get it right after that, and this condition lasted for several hours (from 07:40 UTC until some point between 08:20 and 09:00 UTC). The Zope Windows buildbot had a couple of similar intermittent errors (buildout was unable to find a distribution of some package) that went away when the build job was retried after 24 hours. Could it be that something about the PyPI CDN causes new package releases to be gzipped for an hour or so, and then revert to plain HTML? I'm tempted to do a test, something like curl -I http://pypi.python.org/simple/somepackage/ cd ~/src/somepackage fullrelease # zest.releaser FTW curl -I http://pypi.python.org/simple/somepackage/ if I can find some package ready for a new release. Marius Gedminas -- We'll show a small example here with one user calling another. (Under international treaties describing technical papers these users must be called Alice and Bob.) -- Anthony Baxter signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout can't talk to PyPI any more?
On Wed, May 29, 2013 at 02:30:55PM +0200, Maurits van Rees wrote: Op 29-05-13 14:16, Donald Stufft schreef: Can you get me outputs of curl with the -i flag of it both gzipped and not gzipped? I can't seem to reproduce it. When it works I get this: $ curl -i http://pypi.python.org/simple/pep8/ HTTP/1.1 200 OK ... Content-Length: 3232 ... Vary: Accept-Encoding When I get a gzipped version I get this: $ curl -i http://pypi.python.org/simple/pep8/ HTTP/1.1 200 OK ... Content-encoding: gzip ... There's a discussion about this in #buildout on irc.freenode.net It looks like the upstream server is misconfigured and doesn't emit Vary: Accept-Encoding when a client does a GET with Accept-Encoding: gzip and gets back a gzipped response (which then gets cached and served to everyone for one hour). Donald Stufft is aware of this and is trying to fix it from his phone while sitting in an OR waiting room. I'm in awe. Marius Gedminas -- IBM motto: TEN vowels? Don't you know vowels are scrd? -- Linus Torvalds signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?
On Fri, May 24, 2013 at 08:05:45PM +, holger krekel wrote: On Fri, May 24, 2013 at 17:51 +0300, Marius Gedminas wrote: On Fri, May 24, 2013 at 02:01:16PM +, holger krekel wrote: On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: Most packages also have an egg-info inside of them you can parse. Of course the issue is that you're only going to get the requirements of the system that ran setup.py, either the authors or the servers. Which doesn't accurately represent all of the dependencies all of the time. True but maybe it would go a long way for most packages. In that case you might find https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py useful. looks interesting but a bit of a strange UI it seems. Why not accept a tar archive? Or am i missing something? It's part of a pipeline that builds http://zope3.pov.lt/py3/ At that point I don't have a single tar archive; I have a list of package names, versions and sdist URLs. Marius Gedminas -- Photons have energy, and trying to cram too many into too small of a space can cause a black hole to form, which is, needless to say, not a desirable trait for an optical computer. -- http://scottaaronson.com/blog/?p=261#comment-13693 signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?
On Fri, May 24, 2013 at 02:01:16PM +, holger krekel wrote: On Fri, May 24, 2013 at 09:55 -0400, Donald Stufft wrote: Most packages also have an egg-info inside of them you can parse. Of course the issue is that you're only going to get the requirements of the system that ran setup.py, either the authors or the servers. Which doesn't accurately represent all of the dependencies all of the time. True but maybe it would go a long way for most packages. In that case you might find https://github.com/mgedmin/ztk-py3-status/blob/master/get_deps.py useful. Marius Gedminas -- HOST SYSTEM RESPONDING, PROBABLY UP... signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438 - Transition Phase 1
On Mon, May 20, 2013 at 02:21:05AM -0400, Donald Stufft wrote: On May 20, 2013, at 2:18 AM, Lennart Regebro rege...@gmail.com wrote: On Sun, May 19, 2013 at 10:20 PM, Donald Stufft don...@stufft.io wrote: Hrm, ZPT doesn't seem to be stripping the CDATA or unescaping the strings? https://gist.github.com/dstufft/5608838 is what i have in the template file and that appears verbatim in the output? Yes? It will escape *data* inserted into the template (unless told not to), but what is in the template will appear in the output unescaped. I'm not sure how any template system can work otherwise, but perhaps I've been using Zope too long. :-) //Lennart Maybe you can tell me what I'm doing wrong? Using zope.pagetemplate. ;) More seriously, zope.pagetemplate has two parsing modes: HTML and XML. Nobody actually uses the XML mode (pt files start with an ?xml? declaration, all tal/metal namespaces must be explicitly defined using xmlns:tal=url-that-nobody-can-remember). The HTML mode allows you to write Javascript just like you would do it in a browser, with no extra XML-quoting: script type=text/javascript if (1 2) alert(it works!); /script Does this not work for you? I'm currently looking at a Zope3 app that does precisely this in its working page templates. I need to insert a script tag with Javascript in it. Tres told me to put the contents of the script tag in CDATA blocks which I did, and then when the template was rendered it still had the CDATA blocks so it was invalid javascript. I seem to recall hacks of the form script ... // ![CDATA[ ... // ]] /script but I haven't seen one in a really long time. He also said to just put the javascript in the body of the script but xml escape it. Which I did, and when the template was rendered the data was still xml escaped and again invalid javascript. I think scripts in XHTML were supposed to be XML-escaped. AFAIU zope.pagetemplate was designed back when XHTML was supposed to be The Bright Future of the Web. Marius Gedminas -- Always proofread carefully to see if you any words out. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?
On Wed, May 15, 2013 at 04:10:47PM -0400, Donald Stufft wrote: PyPI XMLRPC? Doesn't that require *two* HTTP requests per package? One to get a list of package versions, and one to get the metadata for a specified version number? I studied http://wiki.python.org/moin/PyPIXmlRpc as best as I could before deciding I couldn't do any better than one HTTP request per package (to get JSON data for the latest release) for https://github.com/mgedmin/ztk-py3-status/blob/master/get_pypi_status.py Marius Gedminas -- Never trust a smiling Gates. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout 2 more craziness: still using site-packages from another python install
On Fri, Apr 05, 2013 at 12:11:18PM -0300, Chris Withers wrote: On 05/04/2013 11:52, Chris Withers wrote: Hi All, This is driving me nuts, I hope someone can help. Okay, so using my cunning buildout knowledge (huh!) I tried this: [buildout] develop = . ../xlrd ../xlwt parts = test py docs versions = versions [versions] distribute=0.6.35 snip buzzkill:xlutils chris$ bin/buildout Develop: '/Users/chris/LocalGIT/xlutils/.' Develop: '/Users/chris/LocalGIT/xlutils/../xlrd' Develop: '/Users/chris/LocalGIT/xlutils/../xlwt' Unused options for buildout: 'unzip'. Updating test. Updating py. Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'. Updating docs. Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'. YAY! (or so I thought) buzzkill:xlutils chris$ bin/test snip Ran 327 tests with 2 failures and 27 errors in 0.478 seconds. Tearing down left over layers: Tear down zope.testrunner.layer.UnitTests in 0.000 seconds. buzzkill:xlutils chris$ cat bin/test #!/usr/local/bin/python2.6 import sys sys.path[0:0] = [ '/Users/chris/LocalGIT/xlutils', '/Users/chris/buildout-eggs/zope.testrunner-4.3.3-py2.6.egg', '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages', '/Users/chris/buildout-eggs/zope.exceptions-4.0.6-py2.6.egg', snip Oh FFS :-( Why is the python2.6 buildout using xlrd from site-packages in a python2.7 installation?! Check develop-eggs/, there may be a distribute.egg-link file that puts /Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages in your sys.path. When in doubt, rm -rf develop-eggs before retrying. Marius Gedminas -- 1 + 1 = 3 -- from a Microsoft advertisement signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout 2 craziness: upgrading to distribute 0.6.26 and using site-packages from another python install
On Fri, Apr 05, 2013 at 11:52:17AM -0300, Chris Withers wrote: This is driving me nuts, I hope someone can help. So, pre-amble, I'm on a Mac, I have a few python installations, the ones in play here are: snip buzzkill:xlutils chris$ cat bin/buildout #!/usr/local/bin/python2.6 import sys sys.path[0:0] = [ '/Users/chris/buildout-eggs/distribute-0.6.35-py2.6.egg', '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg', ] ... buzzkill:xlutils chris$ bin/buildout Upgraded: distribute version 0.6.26; restarting. Generated script '/Users/chris/LocalGIT/xlutils/bin/buildout'. ... buzzkill:xlutils chris$ cat bin/buildout #!/usr/local/bin/python2.6 import sys sys.path[0:0] = [ '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg', '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages', ] I've seen this on Ubuntu. IIRC a Python 3-bootstrapped buildout ended up with /usr/lib/python2.7/dist-packages in sys.path, in my case. Come again?! Why is site-packages from a *different python installation* ending up in bin/buildout? Any help here would be very gratefully received... Workaround #1: run 'bin/buildout -N' so it won't try to upgrade distribute. Workaround #2: always always always wrap your buildouts in a virtualenv. I started using a 'virtual-bootstrap' script that does #!/bin/sh rm -rf develop-eggs .installed.cfg python virtualenv --distribute python python/bin/python bootstrap.py bin/buildout $@ and haven't encountered any issues yet. Marius Gedminas -- When in trouble or in doubt, run in circles, scream and shout. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distribute does not install with python3 on Ubuntu machine
On Mon, Mar 18, 2013 at 02:14:12PM -0700, Lennart Regebro wrote: On Fri, Mar 15, 2013 at 2:46 PM, Giulio Genovese giulio.genov...@gmail.com wrote: sudo pip-3.2 install --upgrade distribute I get this: File setuptools/dist.py, line 103 except ValueError, e: You can't upgrade distribute with pip under Python 3. This is a known problem. https://github.com/pypa/pip/issues/650 I think the problem is that with someone recently forgot to put the parentheses (i.e. except ValueError, e: should be except (ValueError, e):) and therefore this does not work anymore with python3. No, the syntax under Python 3 is except ValueError as 3, but that doesn't work with Python 2.4 and Python 2.5 and we are still supporting them. It should be easy to fix. It isn't. The solution is to not try to upgrade distribute with pip under Python 3. Uninstall it and install it again instead. Wouldn't merging Vinay's distribute3 branch fix this? http://mail.python.org/pipermail/distutils-sig/2013-February/019990.html Marius Gedminas -- * philiKON wonders what niemeyer is committing :) *** benji_york is now known as benji benji murder? -- #zope3-dev signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [venv] distribute-0.6.35 fails on python 3.3?
On Tue, Feb 26, 2013 at 06:52:41PM +, Vinay Sajip wrote: How hard would it be to rewrite distribute to use a common language subset instead of relying on 2to3? I did this months ago, when doing PEP 405 venv development was getting old fast because of the 2to3 delay: https://bitbucket.org/vinay.sajip/distribute3 It's been mentioned on this list, but none of the distribute maintainers appear to have picked up on it. I filed an upstream issue to bring it to their attention: https://bitbucket.org/tarek/distribute/issue/356/installation-of-distribute-is-slow-on It's been a short while (10 Jan 2013) since I synchronised with the main repo, but it typically doesn't take all that long to do. All the tests pass whenever I merge upstream changes, but I'm not sure how much confidence the maintainers have about test coverage and whether all tests passing means enough for them to release into the wild. Marius Gedminas -- C is quirky, flawed, and an enormous success. -- Dennis M. Ritchie signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [venv] distribute-0.6.35 fails on python 3.3?
On Tue, Feb 26, 2013 at 09:17:29AM +, Paul Moore wrote: On 26 February 2013 06:21, Chris Withers ch...@simplistix.co.uk wrote: File ./setuptools/dist.py, line 103 except ValueError, e: ^ SyntaxError: invalid syntax Can other people reproduce this? When was that Python 3 incompatible except clause introduced? I quite often see this, it's usually because something that pip is doing (building the metadata/dependency stuff, the actual code build is fine) is failing to run 2to3. I can't offer anything in the way of a fix, I'm afraid... How hard would it be to rewrite distribute to use a common language subset instead of relying on 2to3? Marius Gedminas -- C++ is a loaded machine gun helpfully pointed at your feet with the safety off. -- ChaosDiscord on Slashdot signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bug in z3c.recipe.scripts
On Thu, Feb 14, 2013 at 09:55:37PM -0500, Gary Poster wrote: Hi all. happy to grant pypi access to whomever. My PyPI username is 'mgedmin'. Thanks! Marius Gedminas -- Apologies for taking up the bandwidth with the apology. Anything else I can apologise for .. er no can't think of anything, sorry about that. Andy Hunt (Member of British Olympic Apology Squad) signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bug in z3c.recipe.scripts
On Thu, Feb 14, 2013 at 09:20:05AM -0500, Jim Fulton wrote: z3c.recipe.scripts doesn't work with buildout 2. It never will. There ought to be a z3c.recipe.scripts 1.0.2 release that install_requires zc.buildout 2.0.0a1. Gary Poster is the only person who has PyPI access. Does anyone remember his email address? (I can probably fish it out, but got no time right now.) Marius Gedminas -- When we say we want readable code, we don't mean we want to sit in a comfortable chair and page through a Java-saga. --- http://www.wordaligned.org/articles/pitching-python-in-three-syllables signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout/distribute (?) issue with Python 3 on Windows
On Wed, Feb 13, 2013 at 11:07:05AM +, Chris Withers wrote:
Hi All,
Anyone seen anything like this?
No.
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File
c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py,
line 1808, in main
getattr(buildout, command)(args)
File
c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py,
line 606, in install
installed_files = self[part]._call(recipe.install)
File
c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py,
line 1333, in _call
return f()
File
c:\users\jenkins\.buildout\eggs\zc.recipe.egg-2.0.0a3-py3.3.egg\zc\recipe\egg\egg.py,
line 162, in install
relative_paths=self._relative_paths,
File
c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\easy_install.py,
line 916, in scripts
contents = dist.get_metadata('scripts/' + name)
File
c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py,
line 1217, in get_metadata
return self._get(self._fn(self.egg_info,name)).decode(utf-8)
File
c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py,
line 1327, in _get
stream = open(path, 'rb')
PermissionError: [Errno 13] Permission denied:
'c:\\users\\jenkins\\.buildout\\eggs\\coverage-3.6-py3.3-win32.egg\\EGG-INFO\\scripts\\__pycache__'
Heh.
Full log here:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=windows/175/console
Ideas welcome...
I think something like this
diff --git a/src/zc/buildout/easy_install.py b/src/zc/buildout/easy_install.py
index 1c81593..60fab6e 100644
--- a/src/zc/buildout/easy_install.py
+++ b/src/zc/buildout/easy_install.py
@@ -913,6 +913,8 @@ def scripts(reqs, working_set, executable, dest=None,
# /EGG-INFO/scripts/.
if dist.metadata_isdir('scripts'):
for name in dist.metadata_listdir('scripts'):
+if dist.metadata_isdir(name):
+continue
contents = dist.get_metadata('scripts/' + name)
distutils_scripts.append((name, contents))
else:
IOW bug in buildout, please file.
Marius Gedminas
--
Always proofread carefully to see if you any words out.
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Problems installing otrace - unexpanded @libdir@ string
On Sun, Feb 10, 2013 at 09:09:12AM -0600, Skip Montanaro wrote:
At work we use encap to create packages to distribute to our desktops
and server. Creating a package involves installing it into an
isolated directory, then creating an encap package from the
directory's contents. Packages which expect to use distutils are a
bit problematic, but with some magic sauce on the command line, these
have become tractable. I'm having trouble installing otrace
(http://pypi.python.org/pypi/otrace). It appears to go well enough:
% python setup.py install --single-version-externally-managed --root /
--install-lib=/var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages
--prefix=/var/tmp/python27_otrace-0.30.9
...
When I look at the generated otrace script I see something that
doesn't look right though:
% cat /var/tmp/python27_otrace-0.30.9/bin/otrace
#!/opt/local/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'otrace==0.30.9','console_scripts','otrace'
__requires__ = 'otrace==0.30.9'
import sys
sys.path.append('@libdir@')
...
What is @libdir@, and why isn't it being expanded to something
useful? This script appears to be generated by Distutils.
Not distutils, but setuptools (or its fork distribute).
The EASY-INSTALL-ENTRY-SCRIPT comment is a hint.
Here's the source code to the latest Distribute:
https://bitbucket.org/tarek/distribute/src/a286137eb40d/setuptools/command/easy_install.py?at=default#cl-1815
It's not what you're running because it doesn't do the
'sys.path.append()' bit.
It would help if you could figure out which version of setuptools (or
distribute) you're using.
python -c 'import setuptools; print(setuptools.__file__)'
could give you a hint.
Marius Gedminas
--
If C gives you enough rope to hang yourself, C++ gives you enough rope to bind
and gag your neighborhood, rig the sails on a small ship, and still have enough
rope left over to hang yourself from the yardarm.
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] I wonder what is the best practice to clean a buildout project ?
On Thu, Jan 31, 2013 at 03:37:21PM +0100, Stéphane Klein wrote: I wonder what is the best practice to clean a buildout project ? I would like a command to remove : * bin/ * develop-eggs * parts * .installed.cfg I can create a Makefile clean, or clean.sh script. What do you do in your project ? I usually create a Makefile with rules to 'make run', 'make test' and so on, as appropriate (and a plain 'make' to build a local virtualenv for isolation, bootstrap bin/buildout with it, and run bin/buildout whenever buildout.cfg or versions.cfg or my setup.py changes). I've never found the need to 'make clean' a buildout project, but if I did, I'd add a makefile rule. Marius Gedminas -- WARN_(accel)(msg null; should hang here to be win compatible\n); -- WINE source code signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip and distutils
On Tue, Jan 29, 2013 at 12:40:17PM -0500, Donald Stufft wrote: On Monday, January 28, 2013 at 5:46 PM, Vraj Mohan wrote: -- Forwarded message -- From: Vraj Mohan r.vrajmo...@gmail.com (mailto:r.vrajmo...@gmail.com) Date: Mon, Jan 28, 2013 at 4:31 PM Subject: pip and distutils To: python-l...@python.org (mailto:python-l...@python.org) I have created a source distribution using distutils which specifies external packages using: setup( ..., requires = ['Foo (= 0.7)', 'Bar (= 2.4.5)'], ... ) When I use pip to install this distribution, I find that it does not automatically install the packages Foo and Bar. What extra step do I need to perform to have pip automatically install the required packages? I am using Python 3.2.3. You're using the distutils2 Syntax but pip is built ontop of setuptools. You want requires = [Foo=0.7, Bar=2.4.5] I think you meant install_requires=[Foo = 0.7, Bar = 2.4.5], (This also requires that you import setup from setuptools and not distutils.) Marius Gedminas -- Look! Before our very eyes, the future is becoming the past. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [buildout] supported python versions for buildout 2?
On Wed, Jan 23, 2013 at 09:47:05AM -0500, Jim Fulton wrote: On Wed, Jan 23, 2013 at 3:24 AM, Chris Withers ch...@simplistix.co.uk wrote: Hi Jim, What are the supported python versions for Buildout 2? http://pypi.python.org/pypi/zc.buildout/2.0.0b1 This version of buildout supports Python 2.6, 2.7, 3.2 and 3.3. And trove classifiers at the bottom of that page say Programming Language :: Python :: 2.4 Programming Language :: Python :: 2.5 Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.2 Programming Language :: Python :: 3.3 I'll submit a pull request to remove 2.4 and 2.5. (later) https://github.com/buildout/buildout/pull/45 Marius Gedminas -- Veni, Vidi, VISA: I came, I saw, I did a little shopping. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions))
On Sat, Jan 12, 2013 at 11:18:36AM -0500, Jim Fulton wrote: On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton j...@zope.com wrote: ... I propose that buildout-versions get incorporated into buildout in the following way: OK, proposal 1 wasn't accepted. Here's another stab: Proposal 2 -- 1. The ``allow-picked-versions`` option gets a new allowed value of ``show``. if there are unpicked versions and this option is set to ``show``, then picked/unpinned versions are reported in a way suitable for copying into a versions section, presumably with the same format used by buildout-versions today. +0.5 (the missing 0.5 is for aesthetic reasons, but I have no better suggestions at the moment, and besides bikeshedding on this would be counter-productive) 2. New buildout option: ``update-versions-file``. This takes a path (relative to buildout directory) of a file to update with any unpinned versions (in a manner roughly the same as buildout-versions does today). +1 3. New buildout option: ``python-version`` that restricts the Python version, with the same semantics as buildout-version provides now. +1 4. Change: develop eggs found in the buildout's develop-eggs directory will be used even if their version conflicts with a pinned version. +1 for the intent, -1 for the implementation. To me develop-eggs was always some kind of mystical implementation detail that sometimes broke things (leftover egg-link files even after I removed those names from my 'develop =' list and re-ran bin/buildout; which always occurred at a point in time when my internal stack was full and I couldn't investigate/file bugs and just rm-rf'ed develop-eggs to be able to continue). I suggest this instead: develop eggs explicitly listed in the [buildout] 'develop' options will be used even if their version conflicts with a pinned version. 5. In buildout 2, The default value of the versions option will be versions, rather than being unset. This will allow users to omit:: version = versions from their buildout section. +1, assuming that was a misspelling of 'versions = versions'. (Corner case analysis: I expect that 'versions = ' will turn off version pinning. I've no intention of ever making use of that corner case) 6. To make it a little easier to supply buildout versions on the command line, make buildout the default section for command-line options, so:: update-versions-file=versions.cfg or:: allow-picked-versions=show would be allowed. (They are rejected now.) +0.5 (gut feeling: I like this. I haven't had time to think about the possible consequences, so I'm withholding the other 0.5 for now.) (Aside: I always wanted buildout to have --newer and --not-newer, because I cannot for the life of me remember which is -n and which is -N. Being able to say bin/buildout newer=off would relieve that need considerably.) Marius Gedminas -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions))
On Sat, Jan 12, 2013 at 03:42:48PM -0500, Alex Clark wrote: On 2013-01-12 16:18:36 +, Jim Fulton said: On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton j...@zope.com wrote: ... I propose that buildout-versions get incorporated into buildout in the following way: OK, proposal 1 wasn't accepted. Here's another stab: Proposal 2 -- 1. The ``allow-picked-versions`` option gets a new allowed value of ``show``. if there are unpicked versions and this option is set to ``show``, then picked/unpinned versions are reported in a way suitable for copying into a versions section, presumably with the same format used by buildout-versions today. So the possible values becomes: true, false, show (which is true verbose) 2. New buildout option: ``update-versions-file``. This takes a path (relative to buildout directory) of a file to update with any unpinned versions (in a manner roughly the same as buildout-versions does today). 3. New buildout option: ``python-version`` that restricts the Python version, with the same semantics as buildout-version provides now. 4. Change: develop eggs found in the buildout's develop-eggs directory will be used even if their version conflicts with a pinned version. Did somebody ask for this? I believe I mentioned this. I used to trip on this gotcha practically every time: - work on package foo that depends on bar - discover a bug in bar that manifests when I use it from foo - check out bar from svn trunk - add a 'mg.cfg' in foo's source tree with [buildout] extends = buildout.cfg develop = ../bar - bin/buildout -c mg.cfg - try some import pdb; pdb.set_trace() or debug prints in ../bar/src/..., run a project in foo, wonder why the breakpoints/debug prints won't work, check bin/runfoo, see ~/.buildout/eggs/bar-1.2.3.egg in there, realize what's the matter - edit mg.cfg again, add [versions] bar = - run bin/buildout -c mg.cfg again, continue debugging. It's an unnecessary speedbump. 6. To make it a little easier to supply buildout versions on the command line, make buildout the default section for command-line options, so:: update-versions-file=versions.cfg or:: allow-picked-versions=show would be allowed. (They are rejected now.) So that means I can pass in foo=bar and it will be applied to the buildout section? How about allowing parameter/values to be applied to any section I believe that's already allowed: $ buildout foo:bar=baz To set parameter 'bar' with value 'baz' in section 'foo'. Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Better version pinning in buildout (buildout-versions)
On Tue, Jan 08, 2013 at 08:02:56AM -0500, Jim Fulton wrote: On Tue, Jan 8, 2013 at 7:45 AM, Marius Gedminas mar...@pov.lt wrote: ... If backwards-compatibility weren't an consideration, I'd be tempted to have buildout 2.0 hardcode the versions section name to [versions] and have the append-new-versions code append a \n\n[versions]\n line if necessary. That way you could say [buildout] append-picked-versions = buildout.cfg and have it Just Work with no extra bootstrapping. It would work until someone added a new section at the end. That's why I said have the append-new-versions code append a [versions] line if necessary or If append-new-versions added a versions section if the last section wasn't a versions section, at which point you could end up with multiple versions sections spread over the file. Yes. Better than appending version pins to an unrelated section, where they would be silently ignored as unknown option values. Having buildout modify a hand-edited file just seems like a Bad Idea to me. I think of it as a tool that helps me update a hand-edited file. I haven't decided whether it's a good idea or a Bad Idea to have a buildout.cfg do that updating automatically. Pros: - whenever you add a new dependency, you'll get a pin for it, so you never commit a version of a project with floating dependencies Cons: - if you decide that you want a certain dependency to float (pinning distribute caused weird failures on my buildbot, so I kept it floating), your buildbot keeps updating versions.cfg all the time and you need explicit svn revert steps to avoid merge conflicts on the next svn up. Marius Gedminas -- 31337 is a prime number, go figure... signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Better version pinning in buildout (buildout-versions)
On Mon, Jan 07, 2013 at 09:45:58AM -0500, Jim Fulton wrote: No. The versions-file can be used with the existing mechanism. I tried, but apparently failed, to make this clear in the proposal. If both a versions file and a versions section is used, the versions section behaves as it does now and versions in the versions file override versions specified in the versions section. This seems backwards to me. Consider this example $ cat buildout.cfg [buildout] I-forgot-the-suggested-new-spelling-for-a-versions-file = versions.txt parts = ... ... $ cat mg.cfg [buildout] extends = buildout.cfg versions = versions [versions] SomePackage = overridden_version I would expect bin/buildout -c mg.cfg to use my overridden version from mg.cfg, not the one from versions-file.txt. Also, having two similar but slightly distinct mechanisms for version pinning? I'm -1 on that. Marius Gedminas -- lg_PC.gigacharset (lg = little green men language, PC = proxima centauri) -- Markus Kuhn provides an example of a locale signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout.dumppickedversions and buildout_versions
On Fri, Jan 04, 2013 at 01:15:05PM +, Chris Withers wrote: On 04/01/2013 13:03, Jim Fulton wrote: On Fri, Jan 4, 2013 at 2:42 AM, Chris Withersch...@simplistix.co.uk wrote: - if a python version is specified, blow up if a different version is used to run buildout. That's an interesting one no one mentioned. :) Yeah, buildout-versions is a project I mainly develop 'cos I need it not because I want to make the community happy ;-) I hit issues with a customer where it turned out the wrong python distribution was being used, so I added this to buildout-versions... it is in the docs, but I didn't make a big song and dance about it. It's a useful feature. We migrated from 2.6 to 2.7 in one project, which meant everyone had to re-run bootstrap.py with the right Python. *Of course* some people/CI bots did not do it, which ended up wasting time chasing strange errors. I wish I'd noticed buildout-versions had this check before I hacked up an ad-hoc Makefile snippet of my own. Marius Gedminas -- It was the kind of night on which bad novels began. -- 1634: The Galileo Affair by Eric Flint and Andrew Dennis signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout.dumppickedversions and buildout_versions
On Thu, Jan 03, 2013 at 07:35:20AM -0500, Jim Fulton wrote: On Sat, Dec 15, 2012 at 11:56 AM, Chris Withers ch...@simplistix.co.uk wrote: Yeah, the stuff that buildout.dumppickedversions and buildout_versions provide should just be in the core. Jim, if I worked up a pull request would you merge it? Maybe :) I tried to look at buildout.dumppickedversions and buildout_versions and I couldn't quite tell what they were for or how they were different. Are they different? Or are they 2 versions of the same thing? buildout-versions is a clone of buildout.dumppickedversions with a rather better choice of name. I don't remember Chris's rationale for rewriting it instead of contributing to buildout.dumppickedversions. The problem is this: people tend to release new package versions to PyPI, and they tend to sometimes break backwards compatibility (intentionally or unintentionally), and then your existing buildouts start to fail. Buildout has version pinning to combat this, but you have to manage version pins by hand. It's easy to miss a dependency or two when you add a new package to your buildout, or upgrade to a newer version that introduces an indirect dependency. Buildout-versions solves the problem of missing version pins in two ways: it can warn you when your buildout uses packages without a version pin, and it can automatically update your version pins with those packages. It's enough to add [buildout] extensions = buildout-versions to your buildout.cfg to get a warning if any packages that are installed by buildout do not have a version pin. I imagine it could be a buildout core feature named, e.g., 'warn-about-picked-versions = true' (a non-fatal version of 'allow-picked-versions = false'). If you also add buildout_versions_file = versions.cfg it'll append new version pins to versions.cfg with the versions that are picked while you run bin/buildout. It's not smart -- you have to ensure the file you're writing to ends with a [versions] section, and that your mail buildout.cfg specifies `extends = versions.cfg` and that you have `[buildout] versions = versions` in either of the config files. You can append directly to buildout.cfg if you're careful enough to keep the [versions] section at the bottom, but I prefer a separate versions.cfg file to reduce clutter and make it easier to grep. As far as I can tell, buildout.dumppickedversions always overwrites the versions.cfg file, while buildout-versions only appends new version pins. The append-only behaviour makes it easier to add meaningful comments like # zc.somepackage 1.3.7 fixes this important bug about X above version pins when you update them manually. This could be a builtin feature called, e.g., 'append-picked-versions-to = versions.cfg'. I consider buildout-versions to be essential for my use of buildout. Marius Gedminas -- IBM motto: If you can't read our assembly language, you must be borderline dyslexic, and we don't want you to mess with it anyway -- Linus Torvalds signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout.org A record
On Fri, Sep 28, 2012 at 03:18:06PM +0200, Christian Theune wrote: On Sep 28, 2012, at 3:16 PM, Jim Fulton j...@zope.com wrote: Status quo for now. I like the idea of using github pages to host buildout.org. When someone sets that up, we'll switch over. Ack. Um, status quo is www.buildout.org is broken, or am I missing something? $ links buildout.org Error loading http://www.buildout.org/: Host not found $ host -t ns buildout.org buildout.org name server ns1.gocept.com. buildout.org name server ns2.gocept.com. $ host buildout.org ns1.gocept.com. buildout.org has address 176.9.22.59 $ host www.buildout.org ns1.gocept.com. Host www.buildout.org.lan not found: 5(REFUSED) No idea where that '.lan' comes from $ host www.buildout.org. ns1.gocept.com. Host www.buildout.org. not found: 3(NXDOMAIN) Marius Gedminas -- BYTE editors are people who separate the wheat from the chaff, and then carefully print the chaff. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout 1.6 release tomorrow
On Mon, Aug 13, 2012 at 05:31:28PM -0400, Alex Clark wrote: Please weigh in now if you have any serious concerns re: a Buildout 1.6 release, as described here: - http://blog.aclark.net/2012/08/13/bootstrapping-a-buildout-1-6-release/ Sounds excellent! My favourite upcoming changes are: - https://bugs.launchpad.net/bugs/697913 : Buildout doesn't honor exit code from scripts. Fixed. - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Marius Gedminas -- Lisp-style macros [...] are to C-style macros what Emacs is to cat. -- Jacek Generowicz signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
