Re: [Zope3-Users] Buildout and custom python
Jim Fulton wrote:
On Dec 20, 2007, at 8:03 PM, David Pratt wrote:
Hey Darryl. Our minds were in a similar place today. I finally had
decided to move away from my system python starting on my mac and also
having a spot of trouble. I put something simple together below to
illustrate what happened in my instance.
Here the eggs built with the python would not install. I initially
though it was the lxml I was compiling, but it was just happened to be
the first part requiring the custom-python. Eggs were made, and I
could see the temp folder also, but when was finalizing the install,
it threw the error and without much verbosity, the tempfolder it was
installing into disappeared.
Note, I did not include the custom-python as a part in the buildout. I
tend to like the python24:location type of notation since it is fairly
explicit when you have a number of pieces of software you are building.
Regards
David
[buildout]
develop = .
parts = python24
interpreter
index = http://download.zope.org/ppix
log-level = DEBUG
[python24]
recipe = zc.recipe:cmmi
url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
[custom-python]
executable = ${python24:location}/bin/python
[interpreter]
recipe = zc.recipe.egg
interpreter = interpreter
eggs = Cheetah==1.0
python = custom-python
Here are the highlights:
Having *just* highlights makes it difficult to follow what's going on.
Some verification it is using the right python...
Getting distribution for 'Cheetah==1.0'.
We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = 0.6c7
Running easy_install:
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/parts/python24/bin/python
"-c" "from setuptools.command.easy_install import main; main()"
"-mUNxd"
"/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH" "-v"
"/Users/davidpratt/Desktop/Buildouts/download-cache/dist/Cheetah-1.0.tar.gz"
path=/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/setuptools-0.6c7-py2.4.egg
Processing Cheetah-1.0.tar.gz
Last few lines of output.. some time later.
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/top_level.txt
to 755
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah
to 755
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah-compile
to 755
Installed
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg
Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:
pkg_resources.require("Cheetah") # latest installed version
pkg_resources.require("Cheetah==1.0") # this exact version
pkg_resources.require("Cheetah>=1.0") # this version or higher
Note also that the installation directory must be on sys.path at
runtime for
this to work. (e.g. by being the application's script directory, by
being on
PYTHONPATH, or by being added to sys.path by your code.)
While:
Installing interpreter.
Getting distribution for 'Cheetah==1.0'.
Error: Couldn't install: Cheetah 1.0
[mac-pg:~/Desktop/Buildouts/osxdeploy2] davidpra%
That's very odd. It feels like something is missing in the output.
Hi Jim. Yeah, I know. Weird huh ... the folder was there, then
disappeared and no other clue why it bailed when finalizing. I did
buildout -vvv to give me some good verbosity for debugging also. Thing
is when run with system python, it does the right stuff. Anything more
you could suggest I check out?
I am glad I decided to stick with 10.4 for a little while longer. I am
still on a PPC platform. I have had my own trouble nasty experience with
a 10.4.9 --> 10.4.10 upgrade a couple of months ago and ended up
reinstalling the os and leaving it at 10.4.9 until I get a new machine.
Did not need the os interfering with my work. Odd for apple to have this
issue, but I was not alone at the time.
Regards,
David
I tried reproducing this. I didn't have the contents of your directory,
so I had to comment out "develop = .". You have a typo in the recipe
specification for the python24 part. After fixing that, I ran the
buildout above using the system Python on Mac OS X 10.5.1. Worked
great. The interpreter script uses the custom Python.
The interpreter runs, but when I try to import cheeta, I get a bus
error.. Unfortunately, there is a problem with using cmmi to install
Python on Mac OS X 10.5. There is a bug in Python's configure that
causes Python to be built incorrectly. The generated pyconfig.h file
currently requires some tweaking. :(
Jim
--
Jim Fulton
Zope Corporation
Re: [Zope3-Users] Buildout and custom python
Benji York wrote: David Pratt wrote: Hi Benji. This is exactly what I have been doing up till now and has been working well for quick work on a local development machine. My current thinking though is to take control of as much of the software as possible so that development == deployment on my local machine to mitigate the risk of breaking things even if it means more disk. I am doing this in conjunction with stripping the deployment server to it's barest bones and bringing as much of the software into the buildout as possible. I really would like to see a two stage buildout that does the python construction with a python.cfg and then the main buildout with buildout.cfg file as part of the standard fare. I'm trying a few things today to see if a simple event class and callback can be used to create the python first and have the callback's handler run the main buildout as an experiment. Where you draw the boundary line between the "environment" and the "application" has a big impact on how you make these types of decisions. You want to find the optimum place to draw that line so you end up getting the most benefit from the least amount of work (it's kind of analogous to the max-flow min-cut theorem from graph theory). Hi Benji. You are absolutely right about this. Also I have to say the amount of software going into things was sort of freaking me out over the last year or so, particulary when all the eggified packages were just rolling out. What I am coming to learn is that the set of software that I am using, while relatively large is also reasonably finite. And I mean the essential server software starts looking small in relation to two or three hundred eggs after a while :-) Secondly, a buildout.cfg is pretty readable - I like knowing I don't have to guess about how something was built, its all in black and white with setting I no longer need to look up or remember. In this way packaging systems like ports and yum make you a little lax about a distribution and its settings. I have a buildout recipe for gcc also, how ironic :-) Crazy thing was that pyLucene up until recently would only work on certain platforms with gcc-3.4.6, so I have had to go this route to construct a compiler that would allow pyLucene to be built also. I get your point though, and ports and yum are nice and easy to work with. The way I am attempting to mitigate issues with buildouts and servers is specialization. The fact that a server in as much as possible is reliably the same is what I want for administration. Regards, David Let me use an example to illustrate. Say you decided to build Python with your buildout. After all, your app uses Python, so to have good reproducibility you want to make sure Python is perfectly clean and built repeatably so you don't get any surprises. Makes sense. Your app is enjoying increased success and one day you need to add some new servers to your cluster. You buy a few new machines, put your OS on them, and build your app. You then run your tests and they fail. Darn. You investigate and find that your Python was built without support for zip compression because the zlib development libraries aren't included in your base OS install. Now you have a decision to make, do you add zlib to your buildout, or do you add the zlib development package to your OS? If Python is part of the environment you add the zlib development package to your OS. Of course that means you need a good way of controlling what is in the OS (environment). Conveniently, there are ways to do that (RPM, APT, ports, etc.). At this point if you draw the app/environment boundary to include Python in your app, you should add zlib to your buildout, right? What happens if you hit a bug in GCC when compiling Python? Do you include GCC in your buildout to make sure you get the right version? It gets worse the deeper you go. :) ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Hi Jim. I like this idea. I then makes it trivial to identify impacts of
using different pythons on your application also since you can just
change a single line in your buildout. For example, below, just change
the ${python24:location}/bin/python to whatever python you want to
evaluate another python including pointing to your system python if you
wish.
Regards,
David
[python24]
recipe = zc.recipe:cmmi
url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
... more stuff
[python25]
recipe = zc.recipe:cmmi
url = http://python.org/ftp/python/2.5.1/Python-2.5.1.tgz
... more stuff
[active-python]
executable = ${python24:location}/bin/python
[interpreter]
recipe = zc.recipe.egg
interpreter = interpreter
eggs = zc.buildout
Cheetah==1.0
python = active-python
Jim Fulton wrote:
On Dec 20, 2007, at 8:58 PM, David Pratt wrote:
Right. Thanks for this, there are couple of solutions here and shell
script is a reasonable solution but these methods seems counter to the
simplicity of buildout. I had also seen a rebootstrap package in svn.
I wonder if something similar could not be accomplished this way using
python with a means of producing a callback. I really don't want
anything too messy to accomplish this. Many thanks.
I'll note that much of the pain here (or at least in the distutils-sig
thread) arises from the fact that the system
Python and the custom Python have the same version but are built in
incompatible ways. :( Unfortunately, binary egg names don't capture
enough information about the ways the eggs were built.
I'm pondering an option to provide different egg directories for
different Python interpreters. So, maybe something like:
[buildout]
parts = python24 ...
[python24]
recipe = zc.recipe.cmmi
url = ...
executable = ${buildout:parts-directory}/bin/python
eggs-directory = ${buildout:directory}/eggs
develop-eggs-directory = ${buildout:parts-directory}/develop-eggs
The idea is that egg-aware recipes would use the egg locations specified
in the Python section. I think this would provide a fairly clean way to
build and use a custom Python.
Of course, it would be nice to have a recipe to build python that set
the various variables.
I can see the benefit of building a custom Python as part of the
buildout when you don't want users of a buildout to build a clean Python
of their own. However, if you can require your users to do this, for
example, if you are your user, then life is much simpler when a clean
Python is managed separately.
Jim
--
Jim Fulton
Zope Corporation
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
On Dec 20, 2007, at 8:58 PM, David Pratt wrote:
Right. Thanks for this, there are couple of solutions here and shell
script is a reasonable solution but these methods seems counter to
the simplicity of buildout. I had also seen a rebootstrap package in
svn. I wonder if something similar could not be accomplished this
way using python with a means of producing a callback. I really
don't want anything too messy to accomplish this. Many thanks.
I'll note that much of the pain here (or at least in the distutils-sig
thread) arises from the fact that the system
Python and the custom Python have the same version but are built in
incompatible ways. :( Unfortunately, binary egg names don't capture
enough information about the ways the eggs were built.
I'm pondering an option to provide different egg directories for
different Python interpreters. So, maybe something like:
[buildout]
parts = python24 ...
[python24]
recipe = zc.recipe.cmmi
url = ...
executable = ${buildout:parts-directory}/bin/python
eggs-directory = ${buildout:directory}/eggs
develop-eggs-directory = ${buildout:parts-directory}/develop-eggs
The idea is that egg-aware recipes would use the egg locations
specified in the Python section. I think this would provide a fairly
clean way to build and use a custom Python.
Of course, it would be nice to have a recipe to build python that set
the various variables.
I can see the benefit of building a custom Python as part of the
buildout when you don't want users of a buildout to build a clean
Python of their own. However, if you can require your users to do
this, for example, if you are your user, then life is much simpler
when a clean Python is managed separately.
Jim
--
Jim Fulton
Zope Corporation
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
David Pratt wrote: Hi Benji. This is exactly what I have been doing up till now and has been working well for quick work on a local development machine. My current thinking though is to take control of as much of the software as possible so that development == deployment on my local machine to mitigate the risk of breaking things even if it means more disk. I am doing this in conjunction with stripping the deployment server to it's barest bones and bringing as much of the software into the buildout as possible. I really would like to see a two stage buildout that does the python construction with a python.cfg and then the main buildout with buildout.cfg file as part of the standard fare. I'm trying a few things today to see if a simple event class and callback can be used to create the python first and have the callback's handler run the main buildout as an experiment. Where you draw the boundary line between the "environment" and the "application" has a big impact on how you make these types of decisions. You want to find the optimum place to draw that line so you end up getting the most benefit from the least amount of work (it's kind of analogous to the max-flow min-cut theorem from graph theory). Let me use an example to illustrate. Say you decided to build Python with your buildout. After all, your app uses Python, so to have good reproducibility you want to make sure Python is perfectly clean and built repeatably so you don't get any surprises. Makes sense. Your app is enjoying increased success and one day you need to add some new servers to your cluster. You buy a few new machines, put your OS on them, and build your app. You then run your tests and they fail. Darn. You investigate and find that your Python was built without support for zip compression because the zlib development libraries aren't included in your base OS install. Now you have a decision to make, do you add zlib to your buildout, or do you add the zlib development package to your OS? If Python is part of the environment you add the zlib development package to your OS. Of course that means you need a good way of controlling what is in the OS (environment). Conveniently, there are ways to do that (RPM, APT, ports, etc.). At this point if you draw the app/environment boundary to include Python in your app, you should add zlib to your buildout, right? What happens if you hit a bug in GCC when compiling Python? Do you include GCC in your buildout to make sure you get the right version? It gets worse the deeper you go. :) -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
On Dec 21, 2007, at 10:18 AM, David Pratt wrote: I really would like to see a two stage buildout that does the python construction with a python.cfg and then the main buildout with buildout.cfg file as part of the standard fare. I'm trying a few things today to see if a simple event class and callback can be used to create the python first and have the callback's handler run the main buildout as an experiment. In other words you want to build two products with one buildout. Python is a language your product depends on. It does not need to be a part of your product as the C language and libc are not part of the Python. As I mentioned in the previous post in this thread one can get the same development and production environments by properly packing clean python and their product(s). The added advantage is that N products can use the same clean python instead of having it deployed N times as a part of their buildout. -- Zvezdan Petkovic <[EMAIL PROTECTED]> ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
On Dec 20, 2007, at 8:03 PM, David Pratt wrote:
Hey Darryl. Our minds were in a similar place today. I finally had
decided to move away from my system python starting on my mac and
also having a spot of trouble. I put something simple together below
to illustrate what happened in my instance.
Here the eggs built with the python would not install. I initially
though it was the lxml I was compiling, but it was just happened to
be the first part requiring the custom-python. Eggs were made, and I
could see the temp folder also, but when was finalizing the install,
it threw the error and without much verbosity, the tempfolder it was
installing into disappeared.
Note, I did not include the custom-python as a part in the buildout.
I tend to like the python24:location type of notation since it is
fairly explicit when you have a number of pieces of software you are
building.
Regards
David
[buildout]
develop = .
parts = python24
interpreter
index = http://download.zope.org/ppix
log-level = DEBUG
[python24]
recipe = zc.recipe:cmmi
url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
[custom-python]
executable = ${python24:location}/bin/python
[interpreter]
recipe = zc.recipe.egg
interpreter = interpreter
eggs = Cheetah==1.0
python = custom-python
Here are the highlights:
Having *just* highlights makes it difficult to follow what's going on.
Some verification it is using the right python...
Getting distribution for 'Cheetah==1.0'.
We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = 0.6c7
Running easy_install:
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/parts/python24/bin/
python "-c" "from setuptools.command.easy_install import main;
main()" "-mUNxd" "/Users/davidpratt/Desktop/Buildouts/osxdeploy2/
eggs/tmprQq-MH" "-v" "/Users/davidpratt/Desktop/Buildouts/download-
cache/dist/Cheetah-1.0.tar.gz"
path=/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/
setuptools-0.6c7-py2.4.egg
Processing Cheetah-1.0.tar.gz
Last few lines of output.. some time later.
changing mode of /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/
tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/
top_level.txt to 755
changing mode of /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/
tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/
cheetah to 755
changing mode of /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/
tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/
cheetah-compile to 755
Installed /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-
MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg
Because this distribution was installed --multi-version, before you
can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to
one of
these examples, in order to select the desired version:
pkg_resources.require("Cheetah") # latest installed version
pkg_resources.require("Cheetah==1.0") # this exact version
pkg_resources.require("Cheetah>=1.0") # this version or higher
Note also that the installation directory must be on sys.path at
runtime for
this to work. (e.g. by being the application's script directory, by
being on
PYTHONPATH, or by being added to sys.path by your code.)
While:
Installing interpreter.
Getting distribution for 'Cheetah==1.0'.
Error: Couldn't install: Cheetah 1.0
[mac-pg:~/Desktop/Buildouts/osxdeploy2] davidpra%
That's very odd. It feels like something is missing in the output.
I tried reproducing this. I didn't have the contents of your
directory, so I had to comment out "develop = .". You have a typo in
the recipe specification for the python24 part. After fixing that, I
ran the buildout above using the system Python on Mac OS X 10.5.1.
Worked great. The interpreter script uses the custom Python.
The interpreter runs, but when I try to import cheeta, I get a bus
error.. Unfortunately, there is a problem with using cmmi to install
Python on Mac OS X 10.5. There is a bug in Python's configure that
causes Python to be built incorrectly. The generated pyconfig.h file
currently requires some tweaking. :(
Jim
--
Jim Fulton
Zope Corporation
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
On Dec 21, 2007, at 10:18 AM, David Pratt wrote: Hi Benji. This is exactly what I have been doing up till now and has been working well for quick work on a local development machine. My current thinking though is to take control of as much of the software as possible so that development == deployment on my local machine to mitigate the risk of breaking things even if it means more disk. I am doing this in conjunction with stripping the deployment server to it's barest bones and bringing as much of the software into the buildout as possible. I really would like to see a two stage buildout that does the python construction with a python.cfg and then the main buildout with buildout.cfg file as part of the standard fare. I'm trying a few things today to see if a simple event class and callback can be used to create the python first and have the callback's handler run the main buildout as an experiment. IMO, it is much more practical to create a clean system package for Python. By a clean system package, I mean one that is equivalent to buildoing from source and has nothing added. This is equivalent to what you would build yourself but makes deployment much easier because you don't have to rebuild Python every time you make a new release. You can use this in development as well. Jim -- Jim Fulton Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Hi Benji. This is exactly what I have been doing up till now and has been working well for quick work on a local development machine. My current thinking though is to take control of as much of the software as possible so that development == deployment on my local machine to mitigate the risk of breaking things even if it means more disk. I am doing this in conjunction with stripping the deployment server to it's barest bones and bringing as much of the software into the buildout as possible. I really would like to see a two stage buildout that does the python construction with a python.cfg and then the main buildout with buildout.cfg file as part of the standard fare. I'm trying a few things today to see if a simple event class and callback can be used to create the python first and have the callback's handler run the main buildout as an experiment. Regards, David Benji York wrote: Darryl Cousins wrote: I'm trying to created an isolated environment using buildout. I had understood that defining a custom build python would ensure that all eggs are installed and compiled with the custom python. An alternative to building Python with buildout is to build a single "clean" Python and use it for all their buildouts. Some advantages of this approach are that your buildouts take less time to build from scratch (no building Python every time), and are a bit smaller (by about 115MB on my system). I keep lots of buildouts around (a few dozen), so those advantages add up. Of course it also lets you stay away from the generally "dirty" system Python (just as the Python-per-buildout strategy does as well). ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
On Dec 21, 2007, at 9:28 AM, Christian Theune wrote: Am Freitag, den 21.12.2007, 09:15 -0500 schrieb Benji York: Christian Theune wrote: And you can mix-and match: have buildout profiles that use your personal `good` Python for development and production profiles that build it themselves. OTOH this is somewhat controversial regarding test what you fly, fly what you test ... ;) As long as you also run your tests in production, it seems fine to me. True. OTOH you really don't want your development environment to be far off your production environment. Joel (Spolsky) wrote a good article about leaking abstractions. It can be rather nasty to find a failing test when wanting to put something into production. (Of course this is way better than going in production and not seeing failing tests.) If that clean python is built as a package (e.g., an rpm) and the product is also built as a package (see zc.sourcerelease) which declares to depend on the clean python package, the development and production environments will be the same. It'll fly what it tested. :-) -- Zvezdan Petkovic <[EMAIL PROTECTED]> ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Am Freitag, den 21.12.2007, 09:15 -0500 schrieb Benji York: > Christian Theune wrote: > > And you can mix-and match: have buildout profiles that use your personal > > `good` Python for development and production profiles that build it > > themselves. OTOH this is somewhat controversial regarding test what you > > fly, fly what you test ... ;) > > As long as you also run your tests in production, it seems fine to me. True. OTOH you really don't want your development environment to be far off your production environment. Joel (Spolsky) wrote a good article about leaking abstractions. It can be rather nasty to find a failing test when wanting to put something into production. (Of course this is way better than going in production and not seeing failing tests.) Christian -- gocept gmbh & co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Christian Theune wrote: And you can mix-and match: have buildout profiles that use your personal `good` Python for development and production profiles that build it themselves. OTOH this is somewhat controversial regarding test what you fly, fly what you test ... ;) As long as you also run your tests in production, it seems fine to me. -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Am Freitag, den 21.12.2007, 08:30 -0500 schrieb Benji York: > Darryl Cousins wrote: > > I'm trying to created an isolated environment using buildout. I had > > understood that defining a custom build python would ensure that all > > eggs are installed and compiled with the custom python. > > An alternative to building Python with buildout is to build a single > "clean" Python and use it for all their buildouts. > > Some advantages of this approach are that your buildouts take less time > to build from scratch (no building Python every time), and are a bit > smaller (by about 115MB on my system). I keep lots of buildouts around > (a few dozen), so those advantages add up. > > Of course it also lets you stay away from the generally "dirty" system > Python (just as the Python-per-buildout strategy does as well). And you can mix-and match: have buildout profiles that use your personal `good` Python for development and production profiles that build it themselves. OTOH this is somewhat controversial regarding test what you fly, fly what you test ... ;) Christian -- gocept gmbh & co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Darryl Cousins wrote: I'm trying to created an isolated environment using buildout. I had understood that defining a custom build python would ensure that all eggs are installed and compiled with the custom python. An alternative to building Python with buildout is to build a single "clean" Python and use it for all their buildouts. Some advantages of this approach are that your buildouts take less time to build from scratch (no building Python every time), and are a bit smaller (by about 115MB on my system). I keep lots of buildouts around (a few dozen), so those advantages add up. Of course it also lets you stay away from the generally "dirty" system Python (just as the Python-per-buildout strategy does as well). -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
Hi,
On Thu, 2007-12-20 at 21:58 -0400, David Pratt wrote:
> Right. Thanks for this, there are couple of solutions here and shell
> script is a reasonable solution but these methods seems counter to the
> simplicity of buildout. I had also seen a rebootstrap package in svn. I
> wonder if something similar could not be accomplished this way using
> python with a means of producing a callback. I really don't want
> anything too messy to accomplish this. Many thanks.
Yes ... I somehow felt that it should be possible with buildout alone.
But that said I'm content with the shell script solution.
Regards,
Darryl
>
> Regards,
> David
>
>
> Darryl Cousins wrote:
> > Hi David,
> >
> > I've tried out the methods described here:
> > http://mail.python.org/pipermail/distutils-sig/2007-November/008489.html
> >
> > and have plumped for the buildout.sh shell script which builds python
> > and then uses that to bootstrap and re-continue the buildout. It is
> > working well and I see that all eggs are then compiled with the compiled
> > python.
> >
> > I had already begun using a Makefile to build/test/run the project so I
> > find it convenient to use a shell script in the mix.
> >
> > Regards,
> > Darryl
> >
> > On Thu, 2007-12-20 at 21:03 -0400, David Pratt wrote:
> >> Hey Darryl. Our minds were in a similar place today. I finally had
> >> decided to move away from my system python starting on my mac and also
> >> having a spot of trouble. I put something simple together below to
> >> illustrate what happened in my instance.
> >>
> >> Here the eggs built with the python would not install. I initially
> >> though it was the lxml I was compiling, but it was just happened to be
> >> the first part requiring the custom-python. Eggs were made, and I could
> >> see the temp folder also, but when was finalizing the install, it threw
> >> the error and without much verbosity, the tempfolder it was installing
> >> into disappeared.
> >>
> >> Note, I did not include the custom-python as a part in the buildout. I
> >> tend to like the python24:location type of notation since it is fairly
> >> explicit when you have a number of pieces of software you are building.
> >>
> >> Regards
> >> David
> >>
> >> [buildout]
> >> develop = .
> >> parts = python24
> >> interpreter
> >> index = http://download.zope.org/ppix
> >> log-level = DEBUG
> >>
> >> [python24]
> >> recipe = zc.recipe:cmmi
> >> url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
> >>
> >> [custom-python]
> >> executable = ${python24:location}/bin/python
> >>
> >> [interpreter]
> >> recipe = zc.recipe.egg
> >> interpreter = interpreter
> >> eggs = Cheetah==1.0
> >> python = custom-python
> >>
> >>
> >> Here are the highlights:
> >>
> >> Some verification it is using the right python...
> >>
> >> Getting distribution for 'Cheetah==1.0'.
> >> We have the best distribution that satisfies 'setuptools'.
> >> Picked: setuptools = 0.6c7
> >> Running easy_install:
> >> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/parts/python24/bin/python
> >> "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd"
> >> "/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH" "-v"
> >> "/Users/davidpratt/Desktop/Buildouts/download-cache/dist/Cheetah-1.0.tar.gz"
> >> path=/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/setuptools-0.6c7-py2.4.egg
> >>
> >> Processing Cheetah-1.0.tar.gz
> >>
> >> Last few lines of output.. some time later.
> >>
> >> changing mode of
> >> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/top_level.txt
> >>
> >> to 755
> >> changing mode of
> >> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah
> >>
> >> to 755
> >> changing mode of
> >> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah-compile
> >>
> >> to 755
> >>
> >> Installed
> >> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg
> >>
> >> Because this distribution was installed --multi-version, before you can
> >> import modules from this package in an application, you will need to
> >> 'import pkg_resources' and then use a 'require()' call similar to one of
> >> these examples, in order to select the desired version:
> >>
> >> pkg_resources.require("Cheetah") # latest installed version
> >> pkg_resources.require("Cheetah==1.0") # this exact version
> >> pkg_resources.require("Cheetah>=1.0") # this version or higher
> >>
> >>
> >> Note also that the installation directory must be on sys.path at runtime
> >> for
> >> this to work. (e.g. by being the application's script directory, by
> >> being on
> >> PYTHONPATH, or by being added to sys.path by your code.)
> >>
> >> While:
> >>Installing interpreter.
> >>Getting distribution for 'Cheetah==1.0'.
> >
Re: [Zope3-Users] Buildout and custom python
Right. Thanks for this, there are couple of solutions here and shell
script is a reasonable solution but these methods seems counter to the
simplicity of buildout. I had also seen a rebootstrap package in svn. I
wonder if something similar could not be accomplished this way using
python with a means of producing a callback. I really don't want
anything too messy to accomplish this. Many thanks.
Regards,
David
Darryl Cousins wrote:
Hi David,
I've tried out the methods described here:
http://mail.python.org/pipermail/distutils-sig/2007-November/008489.html
and have plumped for the buildout.sh shell script which builds python
and then uses that to bootstrap and re-continue the buildout. It is
working well and I see that all eggs are then compiled with the compiled
python.
I had already begun using a Makefile to build/test/run the project so I
find it convenient to use a shell script in the mix.
Regards,
Darryl
On Thu, 2007-12-20 at 21:03 -0400, David Pratt wrote:
Hey Darryl. Our minds were in a similar place today. I finally had
decided to move away from my system python starting on my mac and also
having a spot of trouble. I put something simple together below to
illustrate what happened in my instance.
Here the eggs built with the python would not install. I initially
though it was the lxml I was compiling, but it was just happened to be
the first part requiring the custom-python. Eggs were made, and I could
see the temp folder also, but when was finalizing the install, it threw
the error and without much verbosity, the tempfolder it was installing
into disappeared.
Note, I did not include the custom-python as a part in the buildout. I
tend to like the python24:location type of notation since it is fairly
explicit when you have a number of pieces of software you are building.
Regards
David
[buildout]
develop = .
parts = python24
interpreter
index = http://download.zope.org/ppix
log-level = DEBUG
[python24]
recipe = zc.recipe:cmmi
url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
[custom-python]
executable = ${python24:location}/bin/python
[interpreter]
recipe = zc.recipe.egg
interpreter = interpreter
eggs = Cheetah==1.0
python = custom-python
Here are the highlights:
Some verification it is using the right python...
Getting distribution for 'Cheetah==1.0'.
We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = 0.6c7
Running easy_install:
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/parts/python24/bin/python
"-c" "from setuptools.command.easy_install import main; main()" "-mUNxd"
"/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH" "-v"
"/Users/davidpratt/Desktop/Buildouts/download-cache/dist/Cheetah-1.0.tar.gz"
path=/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/setuptools-0.6c7-py2.4.egg
Processing Cheetah-1.0.tar.gz
Last few lines of output.. some time later.
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/top_level.txt
to 755
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah
to 755
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah-compile
to 755
Installed
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg
Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:
pkg_resources.require("Cheetah") # latest installed version
pkg_resources.require("Cheetah==1.0") # this exact version
pkg_resources.require("Cheetah>=1.0") # this version or higher
Note also that the installation directory must be on sys.path at runtime for
this to work. (e.g. by being the application's script directory, by
being on
PYTHONPATH, or by being added to sys.path by your code.)
While:
Installing interpreter.
Getting distribution for 'Cheetah==1.0'.
Error: Couldn't install: Cheetah 1.0
[mac-pg:~/Desktop/Buildouts/osxdeploy2] davidpra%
Jim Fulton wrote:
On Dec 20, 2007, at 6:03 PM, Darryl Cousins wrote:
I'm trying to created an isolated environment using buildout. I had
understood that defining a custom build python would ensure that all
eggs are installed and compiled with the custom python.
But when the buildout comes to install ZODB it is being compiled agains
the system python headers. (-I/usr/include/python2.4). Which is not what
I want, and at that point the custom python has not even been built.
Any thoughts.
[buildout]
python = custom-python
parts = python
custom-python
[python]
recipe = zc.recipe.cmmi
url = http://www.python.org/ftp/python/2.4.4/Python-2.4.4
Re: [Zope3-Users] Buildout and custom python
Hi David,
I've tried out the methods described here:
http://mail.python.org/pipermail/distutils-sig/2007-November/008489.html
and have plumped for the buildout.sh shell script which builds python
and then uses that to bootstrap and re-continue the buildout. It is
working well and I see that all eggs are then compiled with the compiled
python.
I had already begun using a Makefile to build/test/run the project so I
find it convenient to use a shell script in the mix.
Regards,
Darryl
On Thu, 2007-12-20 at 21:03 -0400, David Pratt wrote:
> Hey Darryl. Our minds were in a similar place today. I finally had
> decided to move away from my system python starting on my mac and also
> having a spot of trouble. I put something simple together below to
> illustrate what happened in my instance.
>
> Here the eggs built with the python would not install. I initially
> though it was the lxml I was compiling, but it was just happened to be
> the first part requiring the custom-python. Eggs were made, and I could
> see the temp folder also, but when was finalizing the install, it threw
> the error and without much verbosity, the tempfolder it was installing
> into disappeared.
>
> Note, I did not include the custom-python as a part in the buildout. I
> tend to like the python24:location type of notation since it is fairly
> explicit when you have a number of pieces of software you are building.
>
> Regards
> David
>
> [buildout]
> develop = .
> parts = python24
> interpreter
> index = http://download.zope.org/ppix
> log-level = DEBUG
>
> [python24]
> recipe = zc.recipe:cmmi
> url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
>
> [custom-python]
> executable = ${python24:location}/bin/python
>
> [interpreter]
> recipe = zc.recipe.egg
> interpreter = interpreter
> eggs = Cheetah==1.0
> python = custom-python
>
>
> Here are the highlights:
>
> Some verification it is using the right python...
>
> Getting distribution for 'Cheetah==1.0'.
> We have the best distribution that satisfies 'setuptools'.
> Picked: setuptools = 0.6c7
> Running easy_install:
> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/parts/python24/bin/python
> "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd"
> "/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH" "-v"
> "/Users/davidpratt/Desktop/Buildouts/download-cache/dist/Cheetah-1.0.tar.gz"
> path=/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/setuptools-0.6c7-py2.4.egg
>
> Processing Cheetah-1.0.tar.gz
>
> Last few lines of output.. some time later.
>
> changing mode of
> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/top_level.txt
>
> to 755
> changing mode of
> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah
>
> to 755
> changing mode of
> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah-compile
>
> to 755
>
> Installed
> /Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg
>
> Because this distribution was installed --multi-version, before you can
> import modules from this package in an application, you will need to
> 'import pkg_resources' and then use a 'require()' call similar to one of
> these examples, in order to select the desired version:
>
> pkg_resources.require("Cheetah") # latest installed version
> pkg_resources.require("Cheetah==1.0") # this exact version
> pkg_resources.require("Cheetah>=1.0") # this version or higher
>
>
> Note also that the installation directory must be on sys.path at runtime for
> this to work. (e.g. by being the application's script directory, by
> being on
> PYTHONPATH, or by being added to sys.path by your code.)
>
> While:
>Installing interpreter.
>Getting distribution for 'Cheetah==1.0'.
> Error: Couldn't install: Cheetah 1.0
> [mac-pg:~/Desktop/Buildouts/osxdeploy2] davidpra%
>
> Jim Fulton wrote:
> >
> > On Dec 20, 2007, at 6:03 PM, Darryl Cousins wrote:
> >> I'm trying to created an isolated environment using buildout. I had
> >> understood that defining a custom build python would ensure that all
> >> eggs are installed and compiled with the custom python.
> >>
> >> But when the buildout comes to install ZODB it is being compiled agains
> >> the system python headers. (-I/usr/include/python2.4). Which is not what
> >> I want, and at that point the custom python has not even been built.
> >>
> >> Any thoughts.
> >>
> >> [buildout]
> >> python = custom-python
> >> parts = python
> >> custom-python
> >>
> >>
> >> [python]
> >> recipe = zc.recipe.cmmi
> >> url = http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tar.bz2
> >> extra_options = --with-threads
> >>--with-readline
> >>--enable-unicode=ucs2
> >>
> >> [custom-python]
> >> execut
Re: [Zope3-Users] Buildout and custom python
Hey Darryl. Our minds were in a similar place today. I finally had
decided to move away from my system python starting on my mac and also
having a spot of trouble. I put something simple together below to
illustrate what happened in my instance.
Here the eggs built with the python would not install. I initially
though it was the lxml I was compiling, but it was just happened to be
the first part requiring the custom-python. Eggs were made, and I could
see the temp folder also, but when was finalizing the install, it threw
the error and without much verbosity, the tempfolder it was installing
into disappeared.
Note, I did not include the custom-python as a part in the buildout. I
tend to like the python24:location type of notation since it is fairly
explicit when you have a number of pieces of software you are building.
Regards
David
[buildout]
develop = .
parts = python24
interpreter
index = http://download.zope.org/ppix
log-level = DEBUG
[python24]
recipe = zc.recipe:cmmi
url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz
[custom-python]
executable = ${python24:location}/bin/python
[interpreter]
recipe = zc.recipe.egg
interpreter = interpreter
eggs = Cheetah==1.0
python = custom-python
Here are the highlights:
Some verification it is using the right python...
Getting distribution for 'Cheetah==1.0'.
We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = 0.6c7
Running easy_install:
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/parts/python24/bin/python
"-c" "from setuptools.command.easy_install import main; main()" "-mUNxd"
"/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH" "-v"
"/Users/davidpratt/Desktop/Buildouts/download-cache/dist/Cheetah-1.0.tar.gz"
path=/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/setuptools-0.6c7-py2.4.egg
Processing Cheetah-1.0.tar.gz
Last few lines of output.. some time later.
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/top_level.txt
to 755
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah
to 755
changing mode of
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg/EGG-INFO/scripts/cheetah-compile
to 755
Installed
/Users/davidpratt/Desktop/Buildouts/osxdeploy2/eggs/tmprQq-MH/Cheetah-1.0-py2.4-macosx-10.3-ppc.egg
Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:
pkg_resources.require("Cheetah") # latest installed version
pkg_resources.require("Cheetah==1.0") # this exact version
pkg_resources.require("Cheetah>=1.0") # this version or higher
Note also that the installation directory must be on sys.path at runtime for
this to work. (e.g. by being the application's script directory, by
being on
PYTHONPATH, or by being added to sys.path by your code.)
While:
Installing interpreter.
Getting distribution for 'Cheetah==1.0'.
Error: Couldn't install: Cheetah 1.0
[mac-pg:~/Desktop/Buildouts/osxdeploy2] davidpra%
Jim Fulton wrote:
On Dec 20, 2007, at 6:03 PM, Darryl Cousins wrote:
I'm trying to created an isolated environment using buildout. I had
understood that defining a custom build python would ensure that all
eggs are installed and compiled with the custom python.
But when the buildout comes to install ZODB it is being compiled agains
the system python headers. (-I/usr/include/python2.4). Which is not what
I want, and at that point the custom python has not even been built.
Any thoughts.
[buildout]
python = custom-python
parts = python
custom-python
[python]
recipe = zc.recipe.cmmi
url = http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tar.bz2
extra_options = --with-threads
--with-readline
--enable-unicode=ucs2
[custom-python]
executable = ${buildout:parts-directory}/python/bin/python
Your buildout doesn't mention ZODB3 or anything that uses it, so it is
hard to comment.
Jim
--
Jim Fulton
Zope Corporation
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Buildout and custom python
On Dec 20, 2007, at 6:03 PM, Darryl Cousins wrote:
I'm trying to created an isolated environment using buildout. I had
understood that defining a custom build python would ensure that all
eggs are installed and compiled with the custom python.
But when the buildout comes to install ZODB it is being compiled
agains
the system python headers. (-I/usr/include/python2.4). Which is not
what
I want, and at that point the custom python has not even been built.
Any thoughts.
[buildout]
python = custom-python
parts = python
custom-python
[python]
recipe = zc.recipe.cmmi
url = http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tar.bz2
extra_options = --with-threads
--with-readline
--enable-unicode=ucs2
[custom-python]
executable = ${buildout:parts-directory}/python/bin/python
Your buildout doesn't mention ZODB3 or anything that uses it, so it is
hard to comment.
Jim
--
Jim Fulton
Zope Corporation
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
