Re: [Distutils] zc.buildout and System Python
On Mon, Oct 25, 2010 at 4:53 AM, Eric Lemoine eric.lemo...@camptocamp.com wrote: Hi I'm currently switching to zc.buildout 1.5. I've been looking at having a Python environment that is fully isolated from the main, system-wide Python environment. Can you confirm that zc.buildout 1.5 supports that? That depends on what exactly that is. The easiest way to have a Python environment that is fully isolated from the main, system-wide Python environment is to have a separate build. Does this imply working with a System Python? buildout 1.5 *tries* to coexist with system Python installations. The term System Python confuses me a bit, as what I want is an environment that is isolated from the system-wide Python environment? So could someone please explain what a System Python refers to exactly? Maybe we should use a different term, but System Python is a euphemism for a Python that has been packaged by an OS packager. For better or worse, these packagers tend to be innovative and deliver Python installations that differ from a from-source build in unpredictable ways. This could involve additions, omissions, or specialized build options. Buildout 1.5 tries to coexist with system Pythons by hiding additions in site-packages (or the innovative equivalent). Virtualenv does something similar. I use the term clean python to refer to a Python installation based on a straightforward build from source, and without any site-local packages in it's site-packages directory. (Where straightforward is a euphemism for a build after making sure that certain system libraries, such as zlib, are available where Python can find them.) We here at ZC package these installs as RPMs that are available system wide along side the system Python. I tell all my friends to use clean Python's for anything important. Also, the doc says: Using a system Python is inherently fragile. Yes. Using a clean, freshly-installed Python without customization in site-packages is more robust and repeatable. Does this mean using a System Python isn't recommended for production? Yes. That's what it means. And how can use a clean, freshly-installed Python? What's a clean, freshly-installed Python anyway? freshly installed is irrelevent, otherwise see above. Again, what I'm after is a Python environment that is fully isolated from the system-wide Python, just like virtualenv --no-site-packages provides? The only way to get a Python that is fully isolated from another Python install is to build one yourself from source. Note: - System-wide isn't really relevent. What's important is a predictable install, which I choose to define as one you'd get from a source build, without special additions or omissions. You can have a clean Python that is installed system wide. Theoretically, system packages could provide clean Pythons, although I've never seen one. It occurs to me that it would be nice if we made clean Python packages available for some of the popular Unix platforms. I'm not sure what would be involved in doing that, from a distribution point of view. - virtualenv --no-site-packages doesn't give you complete isolation. In particular, it doesn't protect you from omissions. Jim -- Jim Fulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] howto python setup.py develop in distutils2
how can i get python setup.py develop or something similar with distutils2? i'm just looking for a simple way to include a package into current python environment while working on that package. $ pip freeze|grep -i dist Distutils2==1.0a3 any tips appreciated. Aljosa Mohorovic ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] howto python setup.py develop in distutils2
On Thu, Oct 28, 2010 at 8:14 AM, Aljoša Mohorović aljosa.mohoro...@gmail.com wrote: how can i get python setup.py develop or something similar with distutils2? i'm just looking for a simple way to include a package into current python environment while working on that package The feature is not existing yet in distutils2. In the meantime you can change your PYTHONPATH to include the package $ pip freeze|grep -i dist Distutils2==1.0a3 any tips appreciated. Aljosa Mohorovic ___ Distutils-SIG maillist - distutils-...@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout and System Python
On Thu, Oct 28, 2010 at 10:22:58AM -0400, Jim Fulton wrote: It occurs to me that it would be nice if we made clean Python packages available for some of the popular Unix platforms. I'm not sure what would be involved in doing that, from a distribution point of view. If you're talking about a python that is carried by the OS in their package sets, updatable using the OS tools, etc catch me on IRC (abadger1999 on irc.freenode.net) and we could talk about this. Off the top of my head, I think it would be possible with a few compromises but not easy in the decision department. For instance, distributions have rules such as don't bundle libraries that are available on the system that would apply to things like libffi which are built from within python by default. Or the use of wide-unicode which isn't the default in a vanilla upstream build but is the default on the Linux distributions that I know of. Or the use of multilib which makes for a split directory layout for libraries instead of a single location. The biggest issue I see is that it wouldn't be possible to fix bugs in these packages. Perhaps it would be possible to compromise and fix bugs but only when the patches are backports from the upstream repository but we presently do that in Fedora for firefox/xulrunner/thunderbird because of mozilla's trademark agreement and it causes no end of conflicts between contributors. -Toshio pgp5s6xOBkCEd.pgp Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] howto python setup.py develop in distutils2
2010/10/28 Tarek Ziadé ziade.ta...@gmail.com: The feature is not existing yet in distutils2. In the meantime you can change your PYTHONPATH to include the package is somebody working on this or it's not a priority for next milestone? Aljosa ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout and System Python
On Thu, Oct 28, 2010 at 11:47 AM, Toshio Kuratomi a.bad...@gmail.com wrote: On Thu, Oct 28, 2010 at 10:22:58AM -0400, Jim Fulton wrote: It occurs to me that it would be nice if we made clean Python packages available for some of the popular Unix platforms. I'm not sure what would be involved in doing that, from a distribution point of view. If you're talking about a python that is carried by the OS in their package sets, updatable using the OS tools, etc That would be great. It might be enough to post pre-built packages. shug catch me on IRC (abadger1999 on irc.freenode.net) and we could talk about this. Off the top of my head, I think it would be possible with a few compromises but not easy in the decision department. Which makes it unattractive. I'm really not interested in getting embroiled in a political process. BTW, I really don't care about certain types of innovation (e.g. file locations, wide unicode) as long as I as a developer don't feel them. It occurs to me that it would be useful if there was a definition of a standard Python that provided a baseline that developers could count on. Today, the closest thing to a standard is the Python distribution. I suppose that doesn't have to be the standard. Of course, defining such a standard might be really painful, especially via email. It might be a good PyCon discussion/sprint topic. The biggest issue I see is that it wouldn't be possible to fix bugs in these packages. Perhaps it would be possible to compromise and fix bugs but only when the patches are backports from the upstream repository I'm not sure what you mean. Bugs are fixed via Python distributions. Is this not fast enough? but we presently do that in Fedora for firefox/xulrunner/thunderbird because of mozilla's trademark agreement and it causes no end of conflicts between contributors. I assume that wouldn't be a problem for Python, assuming I have a clue what that is. :) Jim -- Jim Fulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout and System Python
On Oct 28, 2010, at 12:08 PM, Jim Fulton wrote: BTW, I really don't care about certain types of innovation (e.g. file locations, wide unicode) as long as I as a developer don't feel them. It occurs to me that it would be useful if there was a definition of a standard Python that provided a baseline that developers could count on. Today, the closest thing to a standard is the Python distribution. I suppose that doesn't have to be the standard. Of course, defining such a standard might be really painful, especially via email. It might be a good PyCon discussion/sprint topic. We should do this. The System Python has too many competing OS-specific constraints that pretty much ensure it will always be idiosyncratic. FWIW, in Debian/Ubuntu, we're at least trying to *document* some of those issues: http://wiki.debian.org/Python I'm supportive of an effort to define Clean Python as a separate thing that you can install in parallel and use as a better base for your third party application deployment. To be most useful, I think this should be as similar and predictable as possible across distributions. Of course, it gets harder still when you want to extend that to cross-OS/platform. But maybe there's still something we can do here. We should put this on the agenda for Pycon. Would the language summit be an appropriate forum (at least as a start)? -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] howto python setup.py develop in distutils2
2010/10/28 Aljoša Mohorović aljosa.mohoro...@gmail.com: 2010/10/28 Tarek Ziadé ziade.ta...@gmail.com: The feature is not existing yet in distutils2. In the meantime you can change your PYTHONPATH to include the package is somebody working on this or it's not a priority for next milestone? I'd love to have it in a4 or b1, but I do lack of time so I am not sure when it will be included. But that's definitely a feature we would like to have. -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] An observation on how system packagers and developers can be friends
Periodically, in various venues, we discuss the challenges of deploying applications with or in spite of system packaging of Python and system packaging philosophies. (Note that I'm mainly talking g about deploying applications, as opposed to individual Python packages.) In my experience, the conversation usually unfolds with developers (like me:) snarkely saying how annoyed we are and systems packagers (and administrators) explaining all of the rules they have to follow and that we should follow too. In practice, I think the result is usually that developers make an end-run around system packaging system, either building in production, or by building system packages that bundle dependencies to get needed control. The result is that at least some of the system packagers goals are subverted. In my experience, both sides have legitimate goals, but the problem is rarely, if ever, approached from the point of view of recognizing both sets of goals coming up with a (probably new) solution that addresses both sets of goals. (Note that doing things the way I've always done them is *not* a valid goal. :) I believe that solutions that address both sets of goals are possible and even practical, but it has to start with a consideration of the underlying goals. Jim -- Jim Fulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout and System Python
On Thu, Oct 28, 2010 at 12:08:30PM -0400, Jim Fulton wrote: On Thu, Oct 28, 2010 at 11:47 AM, Toshio Kuratomi a.bad...@gmail.com wrote: On Thu, Oct 28, 2010 at 10:22:58AM -0400, Jim Fulton wrote: It occurs to me that it would be nice if we made clean Python packages available for some of the popular Unix platforms. I'm not sure what would be involved in doing that, from a distribution point of view. If you're talking about a python that is carried by the OS in their package sets, updatable using the OS tools, etc That would be great. It might be enough to post pre-built packages. shug nod There's a few ways to achieve that as well and it would be a lot simpler. There's the opensuse build system which lets you build packages for a variety of distributions. There's ubuntu ppas and fedora personal repos that let you host within the distributions namespace but are marked as being separate... Lots of different options there that might be suitable. catch me on IRC (abadger1999 on irc.freenode.net) and we could talk about this. Off the top of my head, I think it would be possible with a few compromises but not easy in the decision department. Which makes it unattractive. I'm really not interested in getting embroiled in a political process. nod If going the route of getting a clean python into the distributions themselvs, there's going to be a good deal of politics as there are a lot of hard questions to answer there and a lot of contrary goals to reconcile. The idea of simply hosting package repositories for each distribution would be a lot easier in this area. BTW, I really don't care about certain types of innovation (e.g. file locations, wide unicode) as long as I as a developer don't feel them. It occurs to me that it would be useful if there was a definition of a standard Python that provided a baseline that developers could count on. Today, the closest thing to a standard is the Python distribution. I suppose that doesn't have to be the standard. Of course, defining such a standard might be really painful, especially via email. It might be a good PyCon discussion/sprint topic. nod That could be a productive definition. The biggest issue I see is that it wouldn't be possible to fix bugs in these packages. Perhaps it would be possible to compromise and fix bugs but only when the patches are backports from the upstream repository I'm not sure what you mean. Bugs are fixed via Python distributions. Is this not fast enough? Correct, it's not fast enoguh. Many distributions move imuch faster than python releases. Even slow moving distributions can simply be releasing/releasing updates out of sync with when python itself releases. As an example, Fedora releases a new version every six months. Each release has a 13 month lifetime. During the 13 month lifetime, Fedora releases updated packages almost daily. So if someone filed a bug that python-2.7.1-1 had a segfault or a UnicodeError or some other bug that the maintainer felt was worthwhile to fix in the released Fedora, they would ship an updated python package (perhaps with a backport from python-2.x's tree or perhaps by coding a fix and then submitting the fix upstream afterwards) and make the update as soon as they felt they had a workable solution. but we presently do that in Fedora for firefox/xulrunner/thunderbird because of mozilla's trademark agreement and it causes no end of conflicts between contributors. I assume that wouldn't be a problem for Python, assuming I have a clue what that is. :) Well -- the causitive agent is different but the results are similar. In mozilla's case, the issue is adding code that mozilla doesn't endorse as with their permission you have to abandon the trademarks. In a clean python's case, it would be that we want to enforce on ourselves to only ship what's in upstream. In both cases, it prevents fixing bugs and making other changes ahead of an external (to the distribution) schedule. -Toshio pgp8OpvpQEaG2.pgp Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] An observation on how system packagers and developers can be friends
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/28/2010 12:44 PM, Jim Fulton wrote: Periodically, in various venues, we discuss the challenges of deploying applications with or in spite of system packaging of Python and system packaging philosophies. (Note that I'm mainly talking g about deploying applications, as opposed to individual Python packages.) In my experience, the conversation usually unfolds with developers (like me:) snarkely saying how annoyed we are and systems packagers (and administrators) explaining all of the rules they have to follow and that we should follow too. In practice, I think the result is usually that developers make an end-run around system packaging system, either building in production, or by building system packages that bundle dependencies to get needed control. The result is that at least some of the system packagers goals are subverted. In my experience, both sides have legitimate goals, but the problem is rarely, if ever, approached from the point of view of recognizing both sets of goals coming up with a (probably new) solution that addresses both sets of goals. (Note that doing things the way I've always done them is *not* a valid goal. :) I believe that solutions that address both sets of goals are possible and even practical, but it has to start with a consideration of the underlying goals. I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2, but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzJusQACgkQ+gerLs4ltQ6GcQCgkLKlUnbb85KFaYVMlJpBBE5F B1MAn0hDSjU7GJ24/zHExDFqg9kkmTXz =Lztf -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout and System Python
On Thu, Oct 28, 2010 at 12:26 PM, Barry Warsaw ba...@python.org wrote: On Oct 28, 2010, at 12:08 PM, Jim Fulton wrote: BTW, I really don't care about certain types of innovation (e.g. file locations, wide unicode) as long as I as a developer don't feel them. It occurs to me that it would be useful if there was a definition of a standard Python that provided a baseline that developers could count on. Today, the closest thing to a standard is the Python distribution. I suppose that doesn't have to be the standard. Of course, defining such a standard might be really painful, especially via email. It might be a good PyCon discussion/sprint topic. We should do this. Let's do it! :) ... But maybe there's still something we can do here. We should put this on the agenda for Pycon. Not sure what that involves. Would the language summit be an appropriate forum (at least as a start)? I think it would be a good idea to at least mention it there. I'm not sure all of the interested people will be there. Jim -- Jim Fulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] An observation on how system packagers and developers can be friends
On Oct 28, 2010, at 02:02 PM, Tres Seaver wrote: I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2, but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2). We're getting closer to being able to support this more easily upstream with all the work for exposing abi build flags in shared library and executable names. But that won't help you much because it's Python 3. ;) But I think to the extent that something like this could be backported into Python 2, it's probably the general approach to take. All that aside, understanding specifically what a clean Python means is a great first step. ;) -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] An observation on how system packagers and developers can be friends
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/28/2010 04:50 PM, Barry Warsaw wrote: On Oct 28, 2010, at 02:02 PM, Tres Seaver wrote: I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2, but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2). We're getting closer to being able to support this more easily upstream with all the work for exposing abi build flags in shared library and executable names. But that won't help you much because it's Python 3. ;) Doesn't matter to me anyway, as I don't have the issue about trying to share hierarchies between multiple slightly-differing builds. ;) But I think to the extent that something like this could be backported into Python 2, it's probably the general approach to take. All that aside, understanding specifically what a clean Python means is a great first step. ;) Heh, untar + CMMI into a non-system prefix works for me. ;) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzJ7q0ACgkQ+gerLs4ltQ4T3QCggKh9FSRDXdVqP3l6xmVthLaJ YOwAn3CSP3oF0tL3c7oweK3Ecz8QDg8j =ZNIn -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] An observation on how system packagers and developers can be friends
On Thu, Oct 28, 2010 at 5:44 PM, Tres Seaver tsea...@palladion.com wrote: Heh, untar + CMMI into a non-system prefix works for me. ;) +1 with the small addition of after making sure the dev dependencies Python sniffs out to build modules for (zlib, crypto bits, etc.) are available. -- Benji York ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] An observation on how system packagers and developers can be friends
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/28/2010 05:54 PM, Benji York wrote: On Thu, Oct 28, 2010 at 5:44 PM, Tres Seaver tsea...@palladion.com wrote: Heh, untar + CMMI into a non-system prefix works for me. ;) +1 with the small addition of after making sure the dev dependencies Python sniffs out to build modules for (zlib, crypto bits, etc.) are available. Heh, agreed. That bites me on about every third machine I set up for the first time. The Usual Suspects (TM) are whatever the local packaging system calls the following (and their -dev or -devel packages, if split out): - - zlib - - ncurses - - bz2 - - readline - - openssl Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzJ9HQACgkQ+gerLs4ltQ4BawCfWxmQP+HB9atlynYrnMCTt2ka oJQAn3rk7OskTRP6vzo7kI4KjoGMhYp3 =R7tK -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] sh execution of setuptools egg not working
I'll probably realize what stupid thing I'm doing as soon as I send this message, but oh well if that's what it takes ... ;) I want to install setuptools on a centos 4.3 system, for which the system python is 2.4 (ugh) so I've compiled and installed python 2.6 into /usr/local. When I try running the setuptools egg as a shell script, the following happens: --- $ type python2.6 python2.6 is /usr/local/bin/python2.6 $ sudo sh setuptools-0.6c11-py2.6.egg [sudo] password for waterbug: setuptools-0.6c11-py2.6.egg: line 3: exec: python2.6: not found --- ... which seems odd. Just as an experiment, I tried running it without sudo -- it then finds python2.6 but conks out with a different error: --- $ sh setuptools-0.6c11-py2.6.egg prefix=~ Traceback (most recent call last): File string, line 1, in module ImportError: No module named setuptools.command.easy_install --- Any suggestions appreciated! Hope this isn't a faq that I missed ... Thanks, Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] An observation on how system packagers and developers can be friends
On Oct 28, 2010, at 6:08 PM, Tres Seaver wrote: Heh, agreed. That bites me on about every third machine I set up for the first time. The Usual Suspects (TM) are whatever the local packaging system calls the following (and their -dev or -devel packages, if split out): - - zlib - - ncurses - - bz2 - - readline - - openssl On Debian, there's a handy shortcut: 'apt-get build-dep', which will install the build dependencies for any given source package. So 'apt-get build-dep python' will get you all set to build Python. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources: Loading resources in a uniform fashion
At 06:18 PM 10/28/2010 +0200, Alan Franzoni wrote: Hello, lately I was thinking about writing a kind of factory method for parsing an url and returning a resource filename or stream, something like: fs_resource = load_resource_filename(file:///etc/software/config.conf) pkg_resource = load_resource_filename(pkg://package.something) I'd like the first one to just return the file system path /etc/software/config.conf, while I'd like the second one one to do something roughly like from pkg_resources import resource_filename return resource_filename(package, something) This way it would be pretty easy to get a consistent way to describe a resource from a stream. I've got some questions: 1) is there anything around that already does something like that? PEAK does, but it's even less frequently updated than setuptools. ;-) There may be other things out there as well, I don't know. 2) would you think it to be a good addition for pkg_resources, or would it go beyond its scope? Way beyond scope. It makes more sense for it to be part of a url management package. 3) can you see some obvious issues? 4) how would you handle requirements string? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] sh execution of setuptools egg not working
At 06:10 PM 10/28/2010 -0400, Stephen Waterbury wrote: Any suggestions appreciated! Hope this isn't a faq that I missed ... I would suggest trying it with the local Python 2.4; if that works, then your 2.6 build is likely broken in some way. (Another thing to check would be your PYTHONPATH.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
