[Distutils] Script-specific .pth files (or baking paths into scripts)
I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. Have I missed anything? If so, what? :) If not, I'll probably write something myself and share it. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Jim Fulton wrote: I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. Have I missed anything? If so, what? :) If not, I'll probably write something myself and share it. workingenv (http://svn.colorstudy.com/home/ianb/workingenv) doesn't use .pth files, but does put in script-specific paths. As I note in my previous email (setuptools: hardcoding the path in a script and site.py), I think this would probably be cleanest to do through a new installation option, that could also be placed into distutils.cfg. Right now what workingenv does is very specific to workingenv, and doesn't work off any public setuptools API. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
At 11:50 AM 5/17/2006 -0400, Jim Fulton wrote: I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. I haven't done it yet, mainly because I haven't decided how it should work, exactly. Mostly I'm concerned that it seems to need a bunch of options in order to support all the use cases that have been requested. For example, you've asked mainly that all the requirements be pre-baked, and Ian has asked also that nothing leak into the current environment that isn't pre-baked. (Which is also a reasonable requirement for high-security scripts.) It also seems like some of these options should be controllable by the script author to some extent. I just haven't come up with a good design for what the options should be. Mostly the issue is that the script-writing code is already too complicated for my taste, and this is going to make it worse, and I haven't had any brilliant ideas yet on how to fix that. :) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Ian Bicking wrote: Jim Fulton wrote: I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. Have I missed anything? If so, what? :) If not, I'll probably write something myself and share it. workingenv (http://svn.colorstudy.com/home/ianb/workingenv) doesn't use .pth files, but does put in script-specific paths. As I understand it, it is really putting in environment-specific paths. I think I want script-specific paths. As I note in my previous email (setuptools: hardcoding the path in a script and site.py), I think this would probably be cleanest to do through a new installation option, that could also be placed into distutils.cfg. Right now what workingenv does is very specific to workingenv, and doesn't work off any public setuptools API. I have an intuition that we're thinking about this incorrectly. We seem to be going through a lot of effort to fake out something, athough I'm not sure what. I have a growing suspiction that the traditional model of a large namespace of modules that is used by many applications serves simple scripting very well but doessn't work well for applications. I think this is an area where more prototyping is warrented, at least for my needs. I'll let y'all know what, if anything I come up with. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Phillip J. Eby wrote: At 11:50 AM 5/17/2006 -0400, Jim Fulton wrote: I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. I haven't done it yet, mainly because I haven't decided how it should work, exactly. Mostly I'm concerned that it seems to need a bunch of options in order to support all the use cases that have been requested. For example, you've asked mainly that all the requirements be pre-baked, and Ian has asked also that nothing leak into the current environment that isn't pre-baked. (Which is also a reasonable requirement for high-security scripts.) It's also a reasonable requirement for scripts you don't want breaking randomly. These continue to be my bane. What are the use cases? My use cases can be done by inserting something into the beginning of the path before site is imported. The something can be absolute or a script-relative path. Except for the script-relative part, it's equivalent to setting PYTHONPATH. If you don't have a custom site.py, then it's equivalent to just appending it to the path after site.py is imported. If this is done through an installation option, then this covers me (where I have a distutils.cfg controlling this). It can cover script customization (with setup.cfg), maybe that can be improved by doing some variable interpolation as well. What variable interpolation I don't know, as I'm unsure what the actual use case is. I can imagine a use case where a script represents a complete application, and wants specific control of all its libraries. But I'm not sure how that fits in with eggs -- more the basket idea where you get a bundle of things, and you install the bundle and include it (and only it?) on the path. I'm all for that too, but there's more to it than just the path issue. I see the working environment idea as complementary -- the working environment is what you develop in, and more attention is placed on making tools like easy_install work in that environment. But if you want to package it up, you get something more like a basket. So I can imagine implementing it the same way as the working environment -- you add a specific path before importing site.py, and the environment/basket gets to take over there. So... I guess custom site.py's seem to give script authors a great deal of flexibility. Except that isn't really a per-script kind of control, it's more per-basket. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Jim Fulton wrote:
Ian Bicking wrote:
Jim Fulton wrote:
I seem to remember a threat from from Phillip to make a proposal
for installing script-specific ,pth files with scripts, but I don't
think I every saw anything. Have I missed anything? If so, what? :)
If not, I'll probably write something myself and share it.
workingenv (http://svn.colorstudy.com/home/ianb/workingenv) doesn't
use .pth files, but does put in script-specific paths.
As I understand it, it is really putting in environment-specific paths.
I think I want script-specific paths.
I don't think I understand the distinction? Specifically, this is what
gets put in:
#!/usr/bin/python2.4 -S
import sys, os
join, dirname = os.path.join, os.path.dirname
lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' %
tuple(sys.version_info[:2]))
sys.path.insert(0, lib_dir)
import site
# EASY-INSTALL-SCRIPT: 'Cheetah==1.0','cheetah'
__requires__ = 'Cheetah==1.0'
import pkg_resources
pkg_resources.run_script('Cheetah==1.0', 'cheetah')
I.e., it's hardcoding ../lib/python2.4 into the path. The relative
directory is right for workingenv (it makes it movable), but an absolute
location might be what you want. That's not very different, really.
The environment is really built off this path override, not the other
way around. The path override is in turn picked up from the environment
during installation, but that's just a matter of self-consistency.
As I note in my previous email (setuptools: hardcoding the path in a
script and site.py), I think this would probably be cleanest to do
through a new installation option, that could also be placed into
distutils.cfg. Right now what workingenv does is very specific to
workingenv, and doesn't work off any public setuptools API.
I have an intuition that we're thinking about this incorrectly.
We seem to be going through a lot of effort to fake out something,
athough I'm not sure what.
Primarily I'm trying to fake out site.py, which is the source of most of
my problems. It's a reasonable implementation of one packaging
scenario, but it's hard to replace, and the standard implementation
isn't what I want.
I also want to fake out distutils slightly, mostly to provide my own
distutils.cfg, which is again a good idea that is far harder to use than
it should be.
And then I fake out setuptools, which I'd really rather not do, in part
because my faking out keeps breaking.
Oh, and faking out of #!, which if it wasn't so obnoxiously useless
would be sufficient to handle the path stuff on its own.
I have a growing suspiction that
the traditional model of a large namespace of modules
that is used by many applications serves simple scripting very well
but doessn't work well for applications.
Absolutely, that's what I feel as well. I want to opt-in to modules,
not get some huge cloud. At this point, setuptools/easy_install
actually makes that cloud expand much more quickly with multi-version
installations, and that's caused real problems for me.
--
Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Jim Fulton wrote: It's also a reasonable requirement for scripts you don't want breaking randomly. These continue to be my bane. What are the use cases? Well, here's how workingenv deals with some of these right now... I want to be able to install multiple programs (scripts), each with their own dependencies. Note that different programs might depend on different versions of the same package. Inconsistent requirements in a single working environment rely on setuptool's multiversion install, which should work fine on its own when used in limited places. I want to be able to update a package for one program without implicitly affecting the others. (This is a restatement of your desire not to get implicit updates) This is fine. I want everything to be installed in a self-contained project, ala your working environment. I don't want to touch a Python install. Yup. I to be able to use a simple install script, like easy_install. Except for frequent breakage, this works. I hope that breakage is transitional. Also, the python console works fine and represents the working environment, not the larger environment. With workingenv, I also am playing around with --requirements, which is kind of a complete installation plan. I think an external list of requirements is much easier to maintain than packages that carry their own requirements; at least for a certain set of problems, like co-developed packages, and building consistent development environments from code that is actively developed. Or just building a development environment at all, as when you are building a web application. I don't want to have to set environment variables or provide funny options when I run programs. I'd like to minimize this when installing things. When the setuptools monkeypatch works, this is fine. I want the system to be simple and understandable. I'm willing to sacrifice a lot of automation for this. Explicit is better than implicit. :) Except for some (IMHO necessary) faking out of bits and pieces, workingenv is really just a matter of setting the path and keeping the path from being implicitly extended. So I think it's quite simple. It would be nice if Python proper was so simple to work with, but I'm not 100% sure what I really want yet, and I really don't feel like the grief of proposing anything on python-dev. And anyway, I need this to work with old Python versions. Perhaps one personal motivation is that I don't like to give lots of options to tools, I would rather they just work correctly, because any option I give is something I might give incorrectly. That said, having to use /path/to/env/bin/easy_install doesn't bother me, even though easy_install --prefix=/path/to/env *does* bother me, so maybe workingenv can be a little simpler if I give up completely on integrating with any larger system of packages. It should be possible to move or link scripts around. It is not a requirement for me to be able to move an environment without running some program to adjust paths, as long as such a program exists. Scripts can't be moved at this time. The entire environ can be moved, though the activation scripts have to be adjusted, because I haven't figured out a way to do script-relative paths in a shell script that is sourced. If you don't use the activation script, then that's fine. And maybe the only thing that really needs the activation script is the python interactive interpreter. -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Ian Bicking wrote:
Jim Fulton wrote:
Ian Bicking wrote:
Jim Fulton wrote:
I seem to remember a threat from from Phillip to make a proposal
for installing script-specific ,pth files with scripts, but I don't
think I every saw anything. Have I missed anything? If so, what? :)
If not, I'll probably write something myself and share it.
workingenv (http://svn.colorstudy.com/home/ianb/workingenv) doesn't
use .pth files, but does put in script-specific paths.
As I understand it, it is really putting in environment-specific paths.
I think I want script-specific paths.
I don't think I understand the distinction? Specifically, this is what
gets put in:
#!/usr/bin/python2.4 -S
import sys, os
join, dirname = os.path.join, os.path.dirname
lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' %
tuple(sys.version_info[:2]))
sys.path.insert(0, lib_dir)
import site
# EASY-INSTALL-SCRIPT: 'Cheetah==1.0','cheetah'
__requires__ = 'Cheetah==1.0'
import pkg_resources
pkg_resources.run_script('Cheetah==1.0', 'cheetah')
I.e., it's hardcoding ../lib/python2.4 into the path. The relative
directory is right for workingenv (it makes it movable), but an absolute
location might be what you want. That's not very different, really.
Right. I understood this from the links you gave. Every program (script)
installed
into the environment has the same path. If two programs use the same package,
then upgrading the package for one program updates them all. I want to avoid
that.
...
I have an intuition that we're thinking about this incorrectly.
We seem to be going through a lot of effort to fake out something,
athough I'm not sure what.
Primarily I'm trying to fake out site.py,
That's one of the things we're trying to fake out. :)
...
I also want to fake out distutils slightly, mostly to provide my own
distutils.cfg, which is again a good idea that is far harder to use than
it should be.
Yup, that's another example.
And then I fake out setuptools, which I'd really rather not do, in part
because my faking out keeps breaking.
Faking isn't fun.
Oh, and faking out of #!, which if it wasn't so obnoxiously useless
would be sufficient to handle the path stuff on its own.
:)
I suspect this is a case of too much automation in the wrong place.
We are tiptoeing around trying to trick setuptools/distutils/site
into doing the right thing automatically.
I have a growing suspiction that
the traditional model of a large namespace of modules
that is used by many applications serves simple scripting very well
but doessn't work well for applications.
Absolutely, that's what I feel as well. I want to opt-in to modules,
not get some huge cloud. At this point, setuptools/easy_install
actually makes that cloud expand much more quickly with multi-version
installations, and that's caused real problems for me.
Yup.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Script-specific .pth files?
Here's a nutty idea I thought of when thinking about some of the recent back and forth. I realized, when thinking about plugins versus scripts is that a key use case is getting a script installed so that it will run. A script comes from an egg, which depends on other eggs, and so on. (Of course, this gets more exciting for complex script, like, say, Zope. :) It occurs to me that, when a script is installed, easy_install could generate a .pth file that named exactly those eggs the script needed. A wrapper script could be generated that used this .pth file. There wouldn't be any need for run-time analysis. The script wouldn't need to manipulate PYTHONPATH to find the needed eggs, as all of the needed eggs would be captured in the script-specific .pth file. Rather than determining the working set at run time, the working set would be determined at install time. I see a number of advantages to this approach: - No complicated run-time path gymnastics - Faster start-up, as there is no run-time searching for eggs or analysis of which eggs to use - More determinstic behavior. An installed script will always use the same eggs. The human who installed the script can inspect the .pth file to see exactly what's being used. They can even modify it if the feel so bold. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files?
At 04:19 PM 2/11/2006 -0500, Jim Fulton wrote: Here's a nutty idea I thought of when thinking about some of the recent back and forth. I realized, when thinking about plugins versus scripts is that a key use case is getting a script installed so that it will run. A script comes from an egg, which depends on other eggs, and so on. (Of course, this gets more exciting for complex script, like, say, Zope. :) It occurs to me that, when a script is installed, easy_install could generate a .pth file that named exactly those eggs the script needed. A wrapper script could be generated that used this .pth file. It'd have to be some other extension, to avoid confusing a normally-started Python that happens to be processing .pth files in that directory. The code could be as simple as: sys.path[:0]=[line.strip() for line in open(__file__+'.eggs','r')] There wouldn't be any need for run-time analysis. The script wouldn't need to manipulate PYTHONPATH to find the needed eggs, as all of the needed eggs would be captured in the script-specific .pth file. Rather than determining the working set at run time, the working set would be determined at install time. I see a number of advantages to this approach: - No complicated run-time path gymnastics - Faster start-up, as there is no run-time searching for eggs or analysis of which eggs to use It's not actually going to eliminate the processing, unless the importing of the entry point gets hardcoded into the script, such that it no longer uses the entry point mechanism to locate the entry point at runtime. I'm not sure I want to do that, in part because this data is more fragile if you move things around. Right now, you can upgrade a dependency of a package without needing to reinstall the depender, because the paths get fixed up at runtime. Skipping the dynamic resolution would break if you remove an older version of something on upgrade. - More determinstic behavior. An installed script will always use the same eggs. The human who installed the script can inspect the .pth file to see exactly what's being used. They can even modify it if the feel so bold. You lose the ability to override PYTHONPATH-supplied paths this way, so it's a bit less backward-compatible. On the other hand, scripts are hardwired to a specific version of the eggs and will barf if you use PYTHONPATH to override them anyway. Hm. This just might work. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
