[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote:
Jim Fulton wrote:
[foo]
recipe=zc.recipe.egg
eggs = egg1 egg2 ...
interpreter = mypy
extra-paths = path-to-your-instance/lib/python
scripts = mypy
This is great :) I used eggs = ${instance:eggs} to make sure it has the
same eggs as our Zope instance, seems to work very well.
I don't suppose there's any way to make old-style products (e.g
Products.CMFCore) available for import as well?
If they're deployed as eggs
(http://wiki.zope.org/zope2/EggifyingZopesExtensionMechanismQuotProductsQuot)
then they would be importable. Of course, you'd also need Zope 2 on your
PYTHONPATH.
--
http://worldcookery.com -- Professional Zope documentation and training
Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: Unless Utopia really exists I think developers all have their own thoughts about setting up their development environment. Maybe. Except if we (the plone core developers) use ploneout then we are all using the same environment, and we duplicate less work. Same reasoning I had with instancemanager, so I agree. I explicitly made it reasonable easy to set some defaults to have it fit everyone's favourite way of ordering his stuff. Instance in ~/instances/philips by default, but Jean-Paul can set it to a default of ~/Sites/philips, for instance. But even with that level of adaptability-to-individual-taste, not everyone uses it for our zest projects. Rocky not, iirc, for instance. Neither nouri. So: it duplicates less work and we all would have the same environment are great reasons, but they don't always work :-) Again most developers are pretty demanding about setting up a development environment. Here at Zest most of us are comfortable with instancemanager. That's possibly something to do with the fact that you wrote it :) I'm not saying it isn't great, and there's no reason why people shouldn't use it if it works for them. But zc.buildout predates it, seems to be adopted more and more in the Zope 3 world (including people like grok), seems more flexible to me... and I like it. :) Well, predates it... Theoretically, yes. But in practice it was some internal zope corp tool. I only first noticed it when Jim gave a talk about the new improved egg-using version at 2006's Europython. Later that same day I gave a lightning talk about instancemanager. So at least to my viewpoint, it has the same age. http://vanrees.org/weblog/archive/2006/07/04/jim-fulton-zc-buildout http://vanrees.org/weblog/archive/2006/07/04/europython-lightning-talks Ultimately, though, it doesn't really matter what people use. However, at the upcoming Amsterdam sprint, I intend to ask people to do this: $ svn co https://svn.plone.org/svn/plone/ploneout/trunk plone3 $ cd plone3 $ python bootstrap.py $ ./bin/buildout and they will have a Plone 3 instance to develop from. I won't be helping anyone set up Plone in another way, simply because from previous sprints I know that we waste an awful lot of time on something so simple as getting people to symlinks eggs around (yipes). During Plone 3 development, dealing with two bundles (which meant each review bundle needed two bundles as well) that explicitly check out various eggs was a pain in the ass as well. Something that automates this is very useful. I started instancemanager for the same reason: to do away with hour-long bughunts just because someone had an old product lying around. From what I understand, Jean-Paul has an instancemanager configuration that builds a nice plone 3.0 instance from the normal, unmodified bundle with just a few lines of instancemanager configuration. Just keep the bundle sane and everything should be OK. Either buildout/ploneout or instancemanager can help. The goal is the same. Reinout -- Reinout van Rees r.van.rees @ zestsoftware.nl http://vanrees.org/weblog/ http://zestsoftware.nl/ Military engineers build missiles. Civil engineers build targets. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: Ian Bicking wrote: What I would *like* the distinction between workingenv and buildout to be is that workingenv is interactive (i.e., install with easy_install) and buildout is declarative (i.e., specify your environment with buildout.cfg). Well said. I was looking for a way to bring this up in the discussion. This is a key difference. When you use workingenv, how to you reproduce an existing workingenv? Is there something you can check in and later check out to get the same workingenv? Or is this just not in scope? Maybe it should be out of scope, but I have built something into workingenv for it. You specify a file or URL with --requirements, a text file of things to install. You can also indicate --find-links and --editable in the file, as well as refer to another requirements file. The files look like: Pylons==0.8.2 # Adds several eggs and links that aren't on PyPI: -f http://pylonshq.com/download/0.8.2/ ZPTKit SQLObject==0.7.2 Then it will install all these packages. You might then put this at http://svn.myrepo.org/svn/projects/pylons_project.txt and then create another like: -r http://svn.myrepo.org/svn/projects/pylons_project.txt -f http://svn.myrepo.org/svn/projects/project_index.html -e MyProject This would set up the basic Pylons environment, but also add an editable version of MyProject into src/, including activating the file. (Notably MyProject *should* require Pylons and the other packages, seemingly rendering the pylons_project.txt useless, but probably *shouldn't* require exactly Pylons==0.8.2 because then you'd have to break that requirement to even test if such an exacting requirement was actually true.) workingenv also saves some of these settings (like the requirements file) so you can rerun it on a directory. If the requirements file had changed you'd get the updates, or if workingenv had changed it'd rewrite its files. Ideally you'll switch things around, rerun your tests, do some development, and once you've reached a stable state push those changes back into the requirements file. Another motivation for this was to make a simple command for people to try out a new framework, like: wget http://svn.colorstudy.com/home/ianb/workingenv/workingenv.py # You might also have several files attached to specific tutorials or # use cases: python workingenv.py \ -r http://some-framework.org/requirements.txt MyProject cd MyProject source bin/activate cd src paster create -t some_framework MyCode cd MyCode paster serve development.ini And they'd have an editable hello world at that point pretty reliably, without any system dependencies that might make the installation finicky. I had this bootstrapping idea in mind when I added --requirements, I wasn't really thinking much about deployment. It doesn't work well for things like database drivers (which are often needed for tutorials), which are often hard to install with easy_install; this is probably why I should make --site-packages default to on, and then just tell developers to use --no-site-packages to tests their requirements files. But anyway, with the goal of bootstrapping new developers without having to explain the tedious details of Python's packaging conventions, I think workingenv does reasonably well. -- Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: Similarly, say I had two applications, one in Zope and one in Pylons, part of the same deployment (possibly interwoven using wsgi). If I installed elementtree as an egg in buildout.cfg, I'd expect it to be available to both. If that means patching some of pylon's confg files or startup scripts to explicitly reference eggs that buildout is managing, it would mean that I'd need a very specific recipe for Pylons development that would need to know a lot of detail about how that sets up its environment Yes, depending on how complex Pylon's setup was. For example, if Pylons exposed it's scripts using setuptools entrypoints, you you could install them with buildout. Pylon's doesn't have its own scripts. Generally an application will require Pylons, Pylons requires PasteScript, and PasteScript provides a paster script. A quirk in buildout was that asking for the paster script to go in bin/ didn't seem to work unless I specifically was installing PasteScript. Anyway, they all use entry points, so that part is fine. I've considered encouraging each WSGI/PasteDeploy application to ship its own script that hides the existence of paster (or any specific framework). But that doesn't seem quite right either, since a WSGI application can be (and in production usually is) used as a library in addition to a stand-alone application. At that point it becomes similar to a Zope product, or Plone content type, or... well, I guess there's limits to the similarity as I try to find analogs. I guess that's the style differences between Zope and... all the other frameworks. I'm not sure what name really describes the difference in pattern. (in the same way, the z2c.recipe.zope2instance recipe knows about how the zopectl and runzope scripts set their PYTHONPATH and patches them). Zope startup scripts are sort of interesting because they are instance specific -- combining instance-specific configuration with software definition. This is something that setuptools entry points don't handle very will by themselves. That's why buildout has custom script generation facilities that alllow definition of extra initialization code and entry point arguments. I wouldn't be surprised if Pylons had similar needs. You just have to give the config file as an argument when you start the server, there's nothing setup that binds a script with a config file. I played with a #! setup where the config file became a script, but I found it rather hard to work with (there's a lot of problems with how #! works across platforms). So for now it's just always explicit, and if you want something else you have to set up your own shell scripts. -- Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
On Jan 29, 2007, at 2:03 PM, Ian Bicking wrote: Jim Fulton wrote: Ian Bicking wrote: What I would *like* the distinction between workingenv and buildout to be is that workingenv is interactive (i.e., install with easy_install) and buildout is declarative (i.e., specify your environment with buildout.cfg). Well said. I was looking for a way to bring this up in the discussion. This is a key difference. When you use workingenv, how to you reproduce an existing workingenv? Is there something you can check in and later check out to get the same workingenv? Or is this just not in scope? Maybe it should be out of scope, but I have built something into workingenv for it. You specify a file or URL with --requirements, a text file of things to install. You can also indicate --find-links and --editable in the file, as well as refer to another requirements file. The files look like: Pylons==0.8.2 # Adds several eggs and links that aren't on PyPI: -f http://pylonshq.com/download/0.8.2/ ZPTKit SQLObject==0.7.2 Then it will install all these packages. You might then put this at http://svn.myrepo.org/svn/projects/pylons_project.txt and then create another like: -r http://svn.myrepo.org/svn/projects/pylons_project.txt -f http://svn.myrepo.org/svn/projects/project_index.html -e MyProject This would set up the basic Pylons environment, but also add an editable version of MyProject into src/, including activating the file. (Notably MyProject *should* require Pylons and the other packages, seemingly rendering the pylons_project.txt useless, but probably *shouldn't* require exactly Pylons==0.8.2 because then you'd have to break that requirement to even test if such an exacting requirement was actually true.) So this is a bit like the buildout configuration file. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
On Jan 29, 2007, at 2:25 PM, Ian Bicking wrote: Jim Fulton wrote: Similarly, say I had two applications, one in Zope and one in Pylons, part of the same deployment (possibly interwoven using wsgi). If I installed elementtree as an egg in buildout.cfg, I'd expect it to be available to both. If that means patching some of pylon's confg files or startup scripts to explicitly reference eggs that buildout is managing, it would mean that I'd need a very specific recipe for Pylons development that would need to know a lot of detail about how that sets up its environment Yes, depending on how complex Pylon's setup was. For example, if Pylons exposed it's scripts using setuptools entrypoints, you you could install them with buildout. Pylon's doesn't have its own scripts. Generally an application will require Pylons, Pylons requires PasteScript, and PasteScript provides a paster script. A quirk in buildout was that asking for the paster script to go in bin/ didn't seem to work unless I specifically was installing PasteScript. Anyway, they all use entry points, so that part is fine. Right. The eggs recipe only installs scripts for the named eggs, not for dependencies. I've considered encouraging each WSGI/PasteDeploy application to ship its own script that hides the existence of paster (or any specific framework). But that doesn't seem quite right either, since a WSGI application can be (and in production usually is) used as a library in addition to a stand-alone application. At that point it becomes similar to a Zope product, or Plone content type, or... well, I guess there's limits to the similarity as I try to find analogs. I guess that's the style differences between Zope and... all the other frameworks. I'm not sure what name really describes the difference in pattern. I think I do. :) Traditionally, Zope was a pluggable *application*, rather than a library. Typically eggs are meant to be used as libraries. We're moving in the direction of having most of Zope available as eggs that can be used as libraries. Zope 3 is already being widely used that way. In the future there will still be pluggable applications (e.g. Plone) built on libraries provided by the Zope, Plone, and other projects. (in the same way, the z2c.recipe.zope2instance recipe knows about how the zopectl and runzope scripts set their PYTHONPATH and patches them). Zope startup scripts are sort of interesting because they are instance specific -- combining instance-specific configuration with software definition. This is something that setuptools entry points don't handle very will by themselves. That's why buildout has custom script generation facilities that alllow definition of extra initialization code and entry point arguments. I wouldn't be surprised if Pylons had similar needs. You just have to give the config file as an argument when you start the server, there's nothing setup that binds a script with a config file. Right. And for convenience, you often want that. I played with a #! setup where the config file became a script, but I found it rather hard to work with (there's a lot of problems with how #! works across platforms). yup So for now it's just always explicit, and if you want something else you have to set up your own shell scripts. I think that in practice, you'll want to automate this. At least we did. In many cases, you can avoid generating shell scripts and generate Python scripts instead. I like this much better and the techniques that Phillip Eby worked out can be used to make this work nicely on both Unix-based systems and Windows. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: Ian Bicking wrote: Martin Aspeli wrote: whit wrote: actually, in my current workplace, workingenv is the standard way to set up one's dev environment. but in the context of the previous statement, familar is perhaps a better word. I'm still not clear how widely used workingenv is? Is it officially endorsed anywhere else? It steps more lightly than buildout does. What does that mean? It implements less and relies more on what other tools already do. It's also mostly equivalent in mechanism to virtual-python, which is used quite widely. Both use setuptools conventions more closely than buildout does. It would be nice if I could say then you get access to all the setuptools-related management tools, except there are almost none :( But if they *did* exist you'd get access to them ;) I suggest that workingenv and buildout are both such tools. Build stuff seems surprisingly contentious. The debate around setuptools itself has always been far more difficult than it should be; there's a lot of stop energy. So the Python community as a whole is moving very slowly on this stuff. I suggest that, other than contention, this is somewhat healthy. People with different goals will often need different tools and make different tradeoffs. Sure, but I'd also like if there was a clear story for people doing this sort of stuff. I hate the difficulty describing how to do this general stuff, especially when describing it to people who don't even know what their goals are and just need *one* good solution rather than a choice of solutions. Which is to say, I'd rather we try to figure out... something, rather than just chock it up to different goals and go our separate ways. I haven't yet figured out what that something is, and probably that's the hard part. Maybe part of it is a clear and simple scaling from something fairly ad hoc (like a workingenv that you've manually installed stuff into) into something more formalized (like a buildout). Casual and beginning users work best with something they can directly touch and inspect. In a more formalized system it's better to have a central place that described intention and the full system -- the casual user probably hasn't figured out their intention or the full system until well after they've completed their task. Of course, some of buildout's scope is outside of workingenv's -- like building non-Python libraries, and setting up specific application instances. I think it's fine if non-Python libraries require a more advanced setup, but application instances are something I'd like to have a better story for. (Indirectly I'm still trying to figure out the best way to create application instances for PasteDeploy apps too -- I have some stuff in there, but I'm not terribly happy with it, and I haven't figured out what instance creation should be attached to.) -- Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
whit wrote: Jim Fulton wrote: whit wrote: ... Specific use cases would help to guide this. the main usecase for me is the following... hanno writes a recipe for plone, and I want to use that recipe as part of setting up a openplans development environment (for example inside my workingenv that I've been developing w/out plone). Does this recipe create something in particular? A Plone instance? A Plone software install? (correct me if I'm wrong hanno) currently it would create a zope instance w/ all the packages needed to create a plone instance(via the zmi.) I have written a couple of recipes that used together can get you an environment where you can create a Plone instance in the ZMI. So far there are no Plone specific recipes. The only Plone specific thing at the moment is the buildout.cfg found in ploneout. Currently there's: z2c.recipe.zope2install -- It will download either a tarball or checkout a SVN branch, put it into parts and run 'setup.py build_ext -i' on it. z2c.recipe.zope2instance -- This creates a Zope2 instance for you. You give it a path to a Zope2 version, which it will call mkzopeinstance.py from. There's a bunch of options that allow you to configure everything from various zope.conf options, multiple products directories, ... and it will create a zopectl like starter script in the buildout root bin folder that'll let you run tests from it and start/stop the instance. This recipe reuses zc.recipe.egg and is the only controversial in regard to workingenv. z2c.recipe.bundlecheckout -- You give this recipe an url and tell it which vcs system it is (defaulting to svn, with support for cvs as well) and it will checkout the url and put it into /parts. z2c.recipe.distros -- And another one which you give an url to and it will download an unpack a tarball into /parts. You can tell the recipe that the tarball has a nested structure (so it will go down one level and unpack that, this is useful for the CMF tarball for instance) or that the folder inside the tarball has a version suffix (like PluginRegistry does it for example). Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote:
Eggs exist to pkg_resources (the runtime portion of setuptools) simply
by being available on a path. E.g., if you have lib/python/ on the
path, and lib/python/Foo-1.0.egg/ exists, then if you do
pkg_resources.require('Foo') that will add lib/python/Foo-1.0.egg/ to
the path, and you can import from it. When you can actually import from
the egg it is active. You can also see its entry points at that time,
but not if it is just available.
If lib/python/Foo-1.0.egg/ is on the path to start with you can import
from it directly.
This is what zc.reipe.egg does I believe.
It activates (i.e., adds eggs to the path) in the scripts. I think
setuptools' egg activation will be superfluous at that point, because a
complete and consistent set of eggs is activated up-front. I'm not even
sure if you can activate any extra eggs (that might be available on
eggs/ or something) at that point, because the containing directory
isn't on the path. But since buildout controls those directories that
would be fine, I guess, assuming you don't put anything in those
locations on your own.
In that way I suppose lib/pythonX.Y/ would be a misnomer of sorts, as
eggs/ is just a storage area for eggs that are managed elsewhere (in
buildout.cfg).
OTOH, easy_install and setuptools generally has no problem dealing with
eggs that *aren't* on the path, if you use a .egg-link or .pth file to
point to the egg (preferably both). So while Jim hopes to have a
centrally located repository of eggs (where buildouts only draw on a
subset), this is already possible using setuptools' conventions. Then
lib/python could become
(BTW, existing Zope 2 stuff often seems to prefer lib/python/ over
lib/pythonX.Y/, for reasons I don't really understand -- it basically
prefers the distutils --home option over the --prefix option. So using
workingenv with Zope I use workingenv.py --home, which sets up the
environment slightly differently)
Secondly, have you seen this proposal? I'm sure you'd have something to
add.
http://wiki.zope.org/zope2/EggifyingZopesExtensionMechanismQuotProductsQuot
zc.buildout also uses entry points for its recipes, and several people
have talked about entry points instead of ZCML slugs, so I think things
are moving in this direction.
I left some comments over here a while back:
http://wiki.zope.org/zope3/ZopeConfigurationEggSupport
But that's more about ZCMLy things. Philipp's proposal looks fine to
me, though it doesn't seem to address how products would be activated or
not (it seems to mention that eggs wouldn't be installed as products by
default, but doesn't propose a mechanism to select the products).
Because of the difficulty scanning for entry points if the egg isn't
activated, I'd advocate sticking to something like enumerating (or
blacklisting) distribution names. Then if you have:
[zope2.products]
CMFDefault = Products.CMFDefault
the entry point name CMFDefault wouldn't be significant. Or you could
select not just distributions, but distributions+entry_point name. In
Paste I use egg:Package#entry_point_name (where entry_point_name
defaults to main). I believe buildout uses Package:entry_point_name,
where entry_point_name defaults to default. If we could all agree on
a convention, that'd be nice.
I strongly prefer main, because that's really what I think the unnamed
entry point is, not just default -- default just refers to the
parsing/locating strategy, not the intent. I also would like something
that is unambiguously different from a Python object specifier, so that
you could take either an explicit object reference or an entry point in
the same place and be able to clearly tell the difference. But if we
can agree on something and document it, I'm willing to update Paste
Deploy to support the new/standard syntax.
Ian
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: This thread is getting rather long... :) But except for the interaction of workingenv and buildout, which I'm not smart enough to say anything useful about, we are almost finished ;) - There is a 2.5 branch of ploneout (http://svn.plone.org/svn/ploneout/branches/2.5) which pulls down Plone 2.5's tarballs. This is possibly less useful at the moment, more as an experiment, because it doesn't really set you up for developing Plone 2.5's code (for that, we should use svn:externals similarly to ploneout trunk). I changed the branch to correspond to the usual Plone 2.5 core development setup. For other projects, you can re-use the z2c.recipe.* recipes or any other recipes. Recipes are just eggs being loaded with entry points. They are also fairly easy to write once you've seen an example. Meanwhile all the recipes have releases on the cheeseshop, so you can play around with them. They lack any documentation expect the example used in ploneout so far, but I'll fix that soon ;) Among the new features that I implemented this weekend is one inspired by Gocept's zope3instance recipe. The zope2instance recipe will create a script in the buildout root bin folder now, that acts as a simple wrapper around zopectl. So you can run everything from ./bin/instance test -s Products.MyProduct to ./bin/instance start from the buildout root without having to go down into parts. I changed the test command a bit in that script though. It will now automatically put all activated eggs and Products on the test path, so you can run tests in eggs (managed by buildout) without having to specify --test-path manually, which I found extremely annoying as of late. Just as a side note, I found the recipes to be stable and feature-rich enough to dump my old development setup and use the ploneout trunk and 2.5 branch instead ;) Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: whit wrote: actually, in my current workplace, workingenv is the standard way to set up one's dev environment. but in the context of the previous statement, familar is perhaps a better word. I'm still not clear how widely used workingenv is? Is it officially endorsed anywhere else? It steps more lightly than buildout does. It's also mostly equivalent in mechanism to virtual-python, which is used quite widely. Both use setuptools conventions more closely than buildout does. It would be nice if I could say then you get access to all the setuptools-related management tools, except there are almost none :( But if they *did* exist you'd get access to them ;) Build stuff seems surprisingly contentious. The debate around setuptools itself has always been far more difficult than it should be; there's a lot of stop energy. So the Python community as a whole is moving very slowly on this stuff. Ian ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Ian Bicking wrote: ... I would assume that buildout is specifically disabling easy_install's updating of easy-install.pth WRT egg installation, buildout follows easy_install's multi-version model. It installs eggs in such a way that multiple versions can be installed at the same time. As with easy_install in multi-version mode, no easy_install.pth file is updated. -- buildout is still installing packages with easy_install, Not nearly as much as it used to. In fact, buildout only uses easy_install in one or 2 special cases. As I've learned more about setuptools, I've been able to use lower-level interfaces, which are much cleaner. but with some monkeypatches of the command. workingenv also monkeypatches easy_install, but only to change script creation (as noted in another email) and to keep it from rejecting workingenv's site.py. Anyway, if easy_install was allowed to do its thing then they'd be more compatible. I don't think this has much to do with compatibility. The first step to compatibility is deciding what it means. :) I'm all in favor of workingenv/buildout compatibility. I'd like to see some specifics of how people would like to use workingenv amd buildout together. I have some guesses, but I'd rather hear people say what they want to do. I think this would be much more useful than a discussion of implementation details at this point. Once we know what we want the end result to be, I'm sure you and I can work out some implementation that makes sense. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: The first step to compatibility is deciding what it means. :) I'm all in favor of workingenv/buildout compatibility. I'd like to see some specifics of how people would like to use workingenv amd buildout together. I have some guesses, but I'd rather hear people say what they want to do. I think this would be much more useful than a discussion of implementation details at this point. Once we know what we want the end result to be, I'm sure you and I can work out some implementation that makes sense. I agree, and I find myself a bit confused by this orientation as well. The main use case I could imagine wanting to solve would be that I'd like to run 'python' inside zc.buildout and have an interactive prompt that had all the eggs that zc.buildout knew about available. That is, I'd like to be able to do from zope.interface import ... and so on. Similarly, say I had two applications, one in Zope and one in Pylons, part of the same deployment (possibly interwoven using wsgi). If I installed elementtree as an egg in buildout.cfg, I'd expect it to be available to both. If that means patching some of pylon's confg files or startup scripts to explicitly reference eggs that buildout is managing, it would mean that I'd need a very specific recipe for Pylons development that would need to know a lot of detail about how that sets up its environment (in the same way, the z2c.recipe.zope2instance recipe knows about how the zopectl and runzope scripts set their PYTHONPATH and patches them). In both these cases, I wouldn't care much about workingenv per se, only that I had a sensible way of managing this environment. Martin ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
whit wrote: ... Specific use cases would help to guide this. the main usecase for me is the following... hanno writes a recipe for plone, and I want to use that recipe as part of setting up a openplans development environment (for example inside my workingenv that I've been developing w/out plone). Does this recipe create something in particular? A Plone instance? A Plone software install? I'd like with minimal effort to have build in my setup without having to dig too deep into the existing buildout or the framework driving it. By have build in my setup, you mean, use buildout in your setup to leverage Hanno's recipe? This might be as simple as having buildout(at my own risk), do less with creating special pythonpaths in scripts(or perhaps just appending the existing one as the zope2 zopectl script does now). The eggs recipe let's you add paths not provided by scripts. Other recipes that extend the eggs recipe typically let you do this too. I'm not sure if this is what you are getting at.. If you could get a bit more specific, then so could I. :) I'd also like to have the development eggs and normal eggs go where I currently have my eggs(lib/python, /src). as I believe you've pointed out that can be done by simple modification of the cfg file and that'd be fine. Yup. I'll repeat that allowing for shared eggs across buildouts can be a major win. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
whit wrote: Jim Fulton wrote: ... zc.buildout is in no way zope specific. Can a Zope developer not develop a tool without it being stamped as zope specific? maybe... maybe not. When a developer struggles with more than one tool from the same general source, it matters little to them whether one depends on the other or not. That's true for languages, companies, frameworks and everything else. I think it really depends whether something transcends it's immediate community(and maybe here whether a z floats in front of it's name somewhere). So lets see. To contribute to the wider Python community, I should change the name of my Company or change jobs. Hm. for packages from zope or plone people, if they rides roughly with existing python techs In what way does buildout ride roughly with existing python techs. It builds on setuptools. Do you mean that if there is another community package that does something somewhat similar, we aren't allow to create a similar package just because we are associated with the Zope project. and come from a zopeish repository, Oh, I have to use a different repository. sadly they are liable to get branded w/ a legacy reputation that haunts the name. Has it occurred to you that the problem here doesn't lie with the Zope project or Zope developers? inversely, those of us zope and plonistas may be a bit uncritical of projects coming from close to home Oh, as someone who gets lots of criticism, I really don't think that is much of a problem. and allow such technologies to stay inaccessible to wider audiences. How are we doing this? The primary forum for discussing buildout is the distutils sig. It is released through PyPI. It doesn't depend on any other technology that we use -- other than Python. What should I do to make it more accessible? jim, this is not a personal attack on zope corp or the good work you guys have done. It is merely an observation about our entire community concerning some of social factors pertinent to this end of the python pond stated as an answer to what was obviously a rhetorical question by you. I apologize if it seemed otherwise. You should read what you said above carefully, if you haven't already. I won't address your new set rhetorical questions/statements, except to say that we, zope community in toto, are the only people who can take action on our side of our relationship with the rest the world. That's right, we are the only ones who can take action on our side. So, what have we done: - We've adopted standard technologies that let us work with others, like setuptools and wsgi. - We are actively looking at ways to leverage other technologies, (e.g. templating technologies and paste deploy). - We designed Zope 3 specifically to more easily leverage and be leveraged by other systems, - We are splitting up our software and distributing it as eggs to the wider Python community. - We actively participate in the Python community, through conferences and sigs. in making zc.buildout a widely useful minimal dependency technology and by taking part in this conversation(as acrimonious as it may seem at this point), I think you are positively effecting this relationship. I hope so, That's why I'm participating. But, is there any reason why discussions like this have to be acrimonious? Here are some suggestions (these are for all of the participants here): - Stick to technical content. Try to resist attacking people's attitudes and motivations, which you might not really know. - On technology, please try to stick to specifics that people can actually respond to, rather than generalities. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Ian Bicking wrote: Martin Aspeli wrote: ... If lib/python/Foo-1.0.egg/ is on the path to start with you can import from it directly. This is what zc.reipe.egg does I believe. It activates (i.e., adds eggs to the path) in the scripts. I think setuptools' egg activation will be superfluous at that point, because a complete and consistent set of eggs is activated up-front. Yup I'm not even sure if you can activate any extra eggs (that might be available on eggs/ or something) at that point, because the containing directory isn't on the path. Right. If you had the need to do that, you would need to list it or some other location on the path, which the relevant recipes let you do. But since buildout controls those directories that would be fine, I guess, assuming you don't put anything in those locations on your own. It doesn't exert that much control. That is why the eggs directory can be shared. If buildout adds an egg link, then it will remove it when the corresponding directory is removed from the develop option. Side note: Unlike setuptools, buildout prefers develop eggs over non-develop eggs. If there is a develop egg that matches a requirement, it will use it. This is to make development easier. A common pattern for me is that I use a regular egg and find a problem with it. I temporarily check its source out into my buildout and add it to the develop option. When I've made necessary changes, I can make a new egg and switch to that. At that point, it is important to remove the develop egg link. This happens automatically when I remove the project check out from my develop option. In that way I suppose lib/pythonX.Y/ would be a misnomer of sorts, as eggs/ is just a storage area for eggs that are managed elsewhere (in buildout.cfg). Yes OTOH, easy_install and setuptools generally has no problem dealing with eggs that *aren't* on the path, if you use a .egg-link or .pth file to point to the egg (preferably both). So while Jim hopes to have a centrally located repository of eggs (where buildouts only draw on a subset), this is already possible using setuptools' conventions. Except with setuptools conventions, this repository has to be in your Python install, or be in some directory that you specify in some inconvenient way that setuptools is willing to pay attention to. workingenv overcomes this in various ways. Then lib/python could become ? (BTW, existing Zope 2 stuff often seems to prefer lib/python/ over lib/pythonX.Y/, for reasons I don't really understand -- it basically prefers the distutils --home option over the --prefix option. So using workingenv with Zope I use workingenv.py --home, which sets up the environment slightly differently) This is an example of Zope trying to follow conventions. This goes back to early versions of Python that used lib/python rather than lib/pythonX.Y. This is one reason that I don't find following these conventions to be all that useful. There's also the fact that this convention is Unix specific and leads to annoying nested directories where the nesting provides no real value. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Ian Bicking wrote: Martin Aspeli wrote: whit wrote: actually, in my current workplace, workingenv is the standard way to set up one's dev environment. but in the context of the previous statement, familar is perhaps a better word. I'm still not clear how widely used workingenv is? Is it officially endorsed anywhere else? It steps more lightly than buildout does. What does that mean? It's also mostly equivalent in mechanism to virtual-python, which is used quite widely. Both use setuptools conventions more closely than buildout does. It would be nice if I could say then you get access to all the setuptools-related management tools, except there are almost none :( But if they *did* exist you'd get access to them ;) I suggest that workingenv and buildout are both such tools. Build stuff seems surprisingly contentious. The debate around setuptools itself has always been far more difficult than it should be; there's a lot of stop energy. So the Python community as a whole is moving very slowly on this stuff. I suggest that, other than contention, this is somewhat healthy. People with different goals will often need different tools and make different tradeoffs. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: Jim Fulton wrote: The first step to compatibility is deciding what it means. :) I'm all in favor of workingenv/buildout compatibility. I'd like to see some specifics of how people would like to use workingenv amd buildout together. I have some guesses, but I'd rather hear people say what they want to do. I think this would be much more useful than a discussion of implementation details at this point. Once we know what we want the end result to be, I'm sure you and I can work out some implementation that makes sense. I agree, and I find myself a bit confused by this orientation as well. The main use case I could imagine wanting to solve would be that I'd like to run 'python' inside zc.buildout and have an interactive prompt that had all the eggs that zc.buildout knew about available. That is, I'd like to be able to do from zope.interface import ... and so on. As I mentioned before, this is already supported: [buildout] parts = ... foo ... [foo] recipe=zc.recipe.egg eggs = egg1 egg2 ... interpreter = mypy After running the buildout, you will have bin/mypy. If you run bin/mypy, then you'll get a Python prompt and you can import and use anything provided by the listed eggs and their dependencies. You can also do: bin/mypy somescript args and run the script somescript with the given arguments and it will have the path set up for it. One wart is that any scripts defined by the eggs will be installed too when you run the buildout, which you may or may not want. There's a way to suppress this but I should probably add a dedicated interpreter recipe that just creates an interpreter. Similarly, say I had two applications, one in Zope and one in Pylons, part of the same deployment (possibly interwoven using wsgi). If I installed elementtree as an egg in buildout.cfg, I'd expect it to be available to both. If that means patching some of pylon's confg files or startup scripts to explicitly reference eggs that buildout is managing, it would mean that I'd need a very specific recipe for Pylons development that would need to know a lot of detail about how that sets up its environment Yes, depending on how complex Pylon's setup was. For example, if Pylons exposed it's scripts using setuptools entrypoints, you you could install them with buildout. (in the same way, the z2c.recipe.zope2instance recipe knows about how the zopectl and runzope scripts set their PYTHONPATH and patches them). Zope startup scripts are sort of interesting because they are instance specific -- combining instance-specific configuration with software definition. This is something that setuptools entry points don't handle very will by themselves. That's why buildout has custom script generation facilities that alllow definition of extra initialization code and entry point arguments. I wouldn't be surprised if Pylons had similar needs. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote:
Martin Aspeli wrote:
Jim Fulton wrote:
The first step to compatibility is deciding what it means. :)
I'm all in favor of workingenv/buildout compatibility.
I'd like to see some specifics of how people would like
to use workingenv amd buildout together. I have some guesses,
but I'd rather hear people say what they want to do.
I think this would be much more useful than a discussion
of implementation details at this point. Once we know what
we want the end result to be, I'm sure you and I can work out
some implementation that makes sense.
I agree, and I find myself a bit confused by this orientation as well.
The main use case I could imagine wanting to solve would be that I'd
like to run 'python' inside zc.buildout and have an interactive prompt
that had all the eggs that zc.buildout knew about available. That is,
I'd like to be able to do from zope.interface import ... and so on.
As I mentioned before, this is already supported:
[buildout]
parts = ... foo ...
[foo]
recipe=zc.recipe.egg
eggs = egg1 egg2 ...
interpreter = mypy
After running the buildout, you will have bin/mypy.
Oh, didn't see that. That's cool! :)
If you run bin/mypy, then you'll get a Python prompt and you can import
and use anything provided by the listed eggs and their dependencies.
I guess this wouldn't work for Zope's packages in
${SOFTWARE_HOME}/lib/python because they're not eggs known to buildout,
but I could add that to my path manually (and when all of Zope is eggs...)
You can also do:
bin/mypy somescript args
and run the script somescript with the given arguments and it
will have the path set up for it.
One wart is that any scripts defined by the eggs will be installed
too when you run the buildout, which you may or may not want. There's
a way to suppress this but I should probably add a dedicated interpreter
recipe that just creates an interpreter.
At least if it was optional to install the eggs' script, that'd be nice.
I guess it won't do much harm if the eggs install their scripts, though.
Martin
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
- ploneout (http://svn.plone.org/svn/plone/ploneout/trunk) is really
an environment that Plone 3 core developers could (should?) use as a
consistent way of setting up a Zope 2.10 instance with Plone 3 and all
dependencies. It uses svn:externals quite extensively to pull in Plone's
source code into src/ (the eggs) and products/ (the products) with the
intention that this is the code being developed. Ploneout's README.txt
has more information about this scenario.
To through in an extra angle in this discussion, I would like to contribute the
following thoughts.
Unless Utopia really exists I think developers all have their own thoughts
about setting up their development environment. In my opinion buildout's target
should be deployment of complex production systems.
I had to read the buildout documentation twice to even get a grip of it and
yesterday I wondered around in a forest of recipes in ploneout. Although I see
the value of buildout, copying over the entire buildout to setup a second
sandbox, seems like a waste of disk space to me.
We work on quite a number of different projects at once and want to be able to
reproduce a working instance (running the production Data.fs preferably). To
service our environment we (mainly the van Rees brothers) wrote instancemanager.
We expect a minimal set of requirements:
- Python installed
- Zope libraries installed next to each other, e.g.
/usr/local/zope/2.10.2, each version in a separate folder.
Using instancemanager you can:
* Create / Start / Stop / Restart instances
* Backup / Restore / Pack instance databases
* Install Products
* Run tests
* Generate Apache VirtualHost configs
* Easy upgrade Zope versions
During the Snowsprint we'll finish Windows compatibility and creation of Z3
instances.
Minimal configuration in ~/.instancemanager/userdefaults.py to show
instancemanager which zope's are available.
zope_location_template = '/usr/local/zope/%(zope_verion)'
Each instance is then configured using a project.py.
~/.instancemanager/demo.py
Inside this project file we can define all needed instance configuration
options:
* Initial user and password
* Portnumbers
* Zope / Python version used
* Whether to use zeo, and how many zeo clients
* Sources to install in the Products or lib/python folder
- symlink_sources : Allow svn checkouts of products to be linked.
- symlinkbundle_sources : Allow for svn checkouts of bundles (3.0)
- archive_sources: Allow for official release tar files to be
extracted.
- archivebundle_sources: Allow for official release tar bundles to be
extracted.
Sources are defined as a list of dicts:
archive_sources = [
{'url':'http://www.zope.org/Members/shh/DocFinderTab/1.0.1/DocFinderTab-1.0.1.tar.gz'},
{'url':'http://plone.org/products/clouseau/releases/0.7.1/Clouseau.0.7.1.zip'},
]
Here I defined two developers tools that need to be downloaded and extracted to
the Products folder of my instance.
symlinkbundle_sources = [
{'url':'https://svn.plone.org/svn/plone/bundles/3.0',},
{'url':'https://svn.plone.org/svn/plone/bundles/3.0-lib',
'pylib':True,},
]
Here I define the 3.0 bundle of Zope Products that will be symlinked and for
the second we added the key 'pylib' a value of True to indicate this set of
sources
need to be linked in the lib/python folder.
Sources are downloaded to a configurable location, instancemanager has set the
following defaults:
archive_basedir_template = /home/myuser/downloads
archivebundle_basedir_template = /home/myuser/downloads
symlink_basedir_template = /home/myuser/svn
symlinkbundle_basedir_template = /home/myuser/svn
Downloading happens only once, to limit bandwidth.
Again most developers are pretty demanding about setting up a development
environment.
Here at Zest most of us are comfortable with instancemanager.
There's an egg available in the cheeseshop, but for the latest features you can
also checkout
https://svn.plone.org/svn/collective/instancemanager/trunk
Kind regards,
Jean-Paul
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote:
Jim Fulton wrote:
Martin Aspeli wrote:
Jim Fulton wrote:
The first step to compatibility is deciding what it means. :)
I'm all in favor of workingenv/buildout compatibility.
I'd like to see some specifics of how people would like
to use workingenv amd buildout together. I have some guesses,
but I'd rather hear people say what they want to do.
I think this would be much more useful than a discussion
of implementation details at this point. Once we know what
we want the end result to be, I'm sure you and I can work out
some implementation that makes sense.
I agree, and I find myself a bit confused by this orientation as well.
The main use case I could imagine wanting to solve would be that I'd
like to run 'python' inside zc.buildout and have an interactive
prompt that had all the eggs that zc.buildout knew about available.
That is, I'd like to be able to do from zope.interface import ...
and so on.
As I mentioned before, this is already supported:
[buildout]
parts = ... foo ...
[foo]
recipe=zc.recipe.egg
eggs = egg1 egg2 ...
interpreter = mypy
After running the buildout, you will have bin/mypy.
Oh, didn't see that. That's cool! :)
If you run bin/mypy, then you'll get a Python prompt and you can import
and use anything provided by the listed eggs and their dependencies.
I guess this wouldn't work for Zope's packages in
${SOFTWARE_HOME}/lib/python because they're not eggs known to buildout,
but I could add that to my path manually (and when all of Zope is eggs...)
[foo]
recipe=zc.recipe.egg
eggs = egg1 egg2 ...
interpreter = mypy
extra-paths = path-to-your-instance/lib/python
You can also do:
bin/mypy somescript args
and run the script somescript with the given arguments and it
will have the path set up for it.
One wart is that any scripts defined by the eggs will be installed
too when you run the buildout, which you may or may not want. There's
a way to suppress this but I should probably add a dedicated interpreter
recipe that just creates an interpreter.
At least if it was optional to install the eggs' script, that'd be nice.
I guess it won't do much harm if the eggs install their scripts, though.
It is optional:
[foo]
recipe=zc.recipe.egg
eggs = egg1 egg2 ...
interpreter = mypy
extra-paths = path-to-your-instance/lib/python
scripts = mypy
But more work than I think it should be.
The scripts option allows you to select which scripts to include
or to rename them.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote:
[foo]
recipe=zc.recipe.egg
eggs = egg1 egg2 ...
interpreter = mypy
extra-paths = path-to-your-instance/lib/python
scripts = mypy
This is great :) I used eggs = ${instance:eggs} to make sure it has the
same eggs as our Zope instance, seems to work very well.
I don't suppose there's any way to make old-style products (e.g
Products.CMFCore) available for import as well?
Martin
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: I don't have a usecase for executing the scripts with any python interpeter other than the one which ran setuptools to generate them, and therefore don't care for the hard-wired path manipulation I would agree that having to mangle multiple scripts is annoying. On the other hand, I find the activate/deactivate dance annoying. I rather think that the scripts that buildout has created should not require any special activation or introduction to a particular environment. workingenv doesn't require that you activate before running scripts, each script contains in effect what active/deactive does. Basically it munges script creation to do: #!/usr/bin/python -S import sys, os sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib/python%s.%s' % (sys.version_info[0], sys.version_info[1])) import site virtual-python -- which might be the better/simpler solution, if you just ignore the tree of symlinks -- selects the environment through the interpreter itself (i.e., #!/path/to/virtual-python/bin/python). Ian ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote:
I don't have a usecase for executing the scripts with any python
interpeter other than the one which ran setuptools to generate them, and
therefore don't care for the hard-wired path manipulation
I would agree that having to mangle multiple scripts is annoying. On
the other hand, I find the activate/deactivate dance annoying. I
rather think that the scripts that buildout has created should not
require any special activation or introduction to a particular
environment.
Perhaps a solution is to let buildout put things in standard places
(as I'm typing this, we now have all the ploneout .egg and .egg-link
files/directories in ${buildout}/lib/python) and then have an optional
workingenv at the root of the buildout that could be activated when
you want that, but let the buildout-managed scripts (e.g. zope
start/stop and test runner) not require this?
I have a feeling there's more to this than agreeing on directory
layout though. But I hope it oughtn't be *that much more*.
I did some experimentation with this. Basically, I did:
- make egg-directory and develop-egg-directory in buildout.cfg be
lib/python2.4 (which is what workingenv uses)
- run the buildout again
- create standard workingenv (python workingenv.py .) in the root
buildout directory
The results are fairly encouraging:
- buildout-installed eggs don't work in the workingenv jail...
- ...but they do if I manually add them to lib/python2.4/easy_install.pth
- easy_installed packages (i.e. installed with workingenv's
easy_install) are available to zope
This was only a brief test, but seems to me that if buildout was
managing the easy_install.pth file we wouldn't be a million miles off
something where the source bin/activate dance was optional and would
work as a generic solution, but would not be required for zope's startup
and other buildout-managed scripts.
I would assume that buildout is specifically disabling easy_install's
updating of easy-install.pth -- buildout is still installing packages
with easy_install, but with some monkeypatches of the command.
workingenv also monkeypatches easy_install, but only to change script
creation (as noted in another email) and to keep it from rejecting
workingenv's site.py.
Anyway, if easy_install was allowed to do its thing then they'd be more
compatible.
However, I'm not sure whether and how buildout should manage
easy_install.pth in a non-hacky way. A hacky way would be to have a
buildout recipie that, if included, would just patch that file up based
on buildout['buildout']['eggs'] (i.e. the list of eggs listed in
buildout.cfg under the main [buildout] section).
I don't really understand the purpose of easy_install.pth and why it
matters here, though, so suggestions welcome. :)
Eggs exist to pkg_resources (the runtime portion of setuptools) simply
by being available on a path. E.g., if you have lib/python/ on the
path, and lib/python/Foo-1.0.egg/ exists, then if you do
pkg_resources.require('Foo') that will add lib/python/Foo-1.0.egg/ to
the path, and you can import from it. When you can actually import from
the egg it is active. You can also see its entry points at that time,
but not if it is just available.
If lib/python/Foo-1.0.egg/ is on the path to start with you can import
from it directly. The eggs listed in easy-install.pth are put on the
path, and hence are activated by default. You can still require a
different egg version (e.g., Foo==1.1), and I believe setuptools will
(if it can) switch the path to remove the default activated egg and put
on the new egg.
If you install a package with easy_install -m (multiversion) then none
of the versions go on the path, and eggs will only be added when they
are specifically required. Any egg listed in install_requires in
setup.py will get added as soon as you require the original egg, and
when you run a script from an egg it'll activate that egg as well, so if
you are thorough with your requirements then none of the eggs need to be
activated.
One problem if you are using entry points is that if your egg isn't
activated and you are searching entry points by name (not by egg) then
your egg has to be activated to work (Buffet and paster create templates
are both searched by name, for example). I have expected this to cause
more problems than it has, but since people have stopped leaning too
heavily on setuptools versioning (thanks to virtual-python, workingenv,
and buildout) I think it hasn't come up much. Anyway, if Zope was to
start using entry points to replace some of its ZCML (which I personally
think would be a good idea) this is something to be aware of.
Ian
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: whit wrote: I'm not clear on what the advantage would be. I'm probably missing some use cases. I think they are both valid approaches to the problem. my main usecase is to be able to use buildouts in a workingenv without having to rewrite the recipes... right now, I have to do one or the other. Sure. I still don't know what that means. :) *Maybe* you'd like to use workingenv to install eggs and use buildout to do other things like set up zope instances. If that's the case, then it's just a matter of writing recipes that take into account how you install eggs and scripts. But maybe that's not what you mean. ... If people really see value in having buildout and workingenv work together, perhaps Ian and I could explore this a bit at PyCon next month. +1. my general feeling is that there is a enough overlap that this would be beneficial. Specific use cases would help to guide this. the main usecase for me is the following... hanno writes a recipe for plone, and I want to use that recipe as part of setting up a openplans development environment (for example inside my workingenv that I've been developing w/out plone). I'd like with minimal effort to have build in my setup without having to dig too deep into the existing buildout or the framework driving it. This might be as simple as having buildout(at my own risk), do less with creating special pythonpaths in scripts(or perhaps just appending the existing one as the zope2 zopectl script does now). I'd also like to have the development eggs and normal eggs go where I currently have my eggs(lib/python, /src). as I believe you've pointed out that can be done by simple modification of the cfg file and that'd be fine. try to justify why a zope hammer is better and why zope folk keep building new hammers that don't play nice with everyone else's hammer loses us cred. You seem to be implying that workingenv is somehow a standard tool and that buildout is a zope-specific tool that needlessly duplicates a standard tool. I was perhaps offbase in my comment about egg handling. workingenv of course is not a standard tool. what I'm presenting here is a perspective I face on a regular basis; for a developer using workingenv already, it's far more standard than something else he is not using...and vice versa. No, it's more familiar -- to these people. actually, in my current workplace, workingenv is the standard way to set up one's dev environment. but in the context of the previous statement, familar is perhaps a better word. in the case of tools that enable a developer to start quickly, developers are very difficult to ween from habits that are currently working. when a large and complex task such as building plone becomes codified in a technology that is not familar or easily compatible with the the developers current environment, it's an issue. The alternative seems to be a custom shell script. I can see how this would be familiar to the people who wrote it. When there are well established and documented buildout recipes for doing common tasks, then there will be familiar tools, for some definition of familiar. this is where rob started with our deployment strategy. I like buildit and buildout's codification of what happens in a build in cfg file since it creates a fairly auditable place to see what is happening(as well as extend it). we are going to re-write the shell scripts using buildit and if plone uses zc.buildout, would like not to have to replicate that part in our own system. zc.buildout is in no way zope specific. Can a Zope developer not develop a tool without it being stamped as zope specific? maybe... maybe not. When a developer struggles with more than one tool from the same general source, it matters little to them whether one depends on the other or not. That's true for languages, companies, frameworks and everything else. I think it really depends whether something transcends it's immediate community(and maybe here whether a z floats in front of it's name somewhere). So lets see. To contribute to the wider Python community, I should change the name of my Company or change jobs. Hm. for packages from zope or plone people, if they rides roughly with existing python techs In what way does buildout ride roughly with existing python techs. It builds on setuptools. Do you mean that if there is another community package that does something somewhat similar, we aren't allow to create a similar package just because we are associated with the Zope project. and come from a zopeish repository, Oh, I have to use a different repository. sadly they are liable to get branded w/ a legacy reputation that haunts the name. Has it occurred to you that the problem here doesn't lie with the Zope project or Zope developers? inversely, those of us zope and plonistas may be a bit uncritical of projects coming from close to home Oh, as someone
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Ian Bicking wrote: It would be a concern if, for instance, Plone started depending on buildout recipes for installation, without plain distutils recipes. Of course right now there are no distutils recipes for old-style Products. So actually it's an active issue -- if buildout enables Plone to keep from moving to distutils/setuptools/egg-style package deployment (because buildout is flexible enough to support the old patterns) then that would make it harder for workingenv. But I don't think anyone is proposing that buildout means that Plone (and Zope 2/Five) shouldn't continue its move towards traditional Python packaging. No, quite the contrary in fact. The other day I wrote a proposal that Zope's old-style Products will turn into regular python packages using pkg_resource's namespace package support and eggs' entry points (http://wiki.zope.org/zope2/EggifyingZopesExtensionMechanismQuotProductsQuot). It will allow us to deploy old-style products as eggs (using setup.py). The proposal explicitly mentions workingenv and zc.buildout as two mechanisms that allow such deployment w/ Zope. -- http://worldcookery.com -- Professional Zope documentation and training Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Rob Miller wrote: honestly, it seems to me that buildout tries to do too much. That's ok. I often don't need the big hammer that buildout is. That's when I tend to use workingenv (even if it's' just for trying out whether something's easy_install'able) it's trying to handle both repeatable deployment recipes AND providing a sandbox within which to run things. there may not be a point to having an extra layer on top of buildout, but buildout sure does seem to me a bit heavy if all i want is a sandbox. Buildout is actually quite simple, it took me a while to grok it, too. I started writing a tutorial for it but got interupted by other stuff. I sure hope to finish it. so now i have to learn the workingenv way if i just need a sandbox, but i have to learn the buildout way if i also want repeatable deployments? I think buildout can serve well for sandboxes if you work in a team, because everybody uses the same recipe for sandboxes. Workingenv made it more complex than it needed to be (or buildout as stated before, I don't mind using zc.buildout, but I don't want to have to learn zc.buildout to use it meaningfully in my existing setup. If a buildout recipes could be executed by themselves(like buildout-zope2, buildout-deliverance, buildout-squid) as well as by aggregated recipes. This would make buildout a killer tool inside my workingenv rather than a choice I had to make between two technologies. As said already, I think once you've got buildout, there's no need for workingen, except if you think that Zope stinks ;) this is shortsighted, IMO. i know zope quite well, but i bounced off of buildout b/c it required too much knowledge to even get started. There's nothing Zope specific about buildout. It just happens to be developed by Zope people. The egg and cmmi recipes allow it to be used for the majority of other Python software out there, though (and if that's not enough, you can alway simplement a custom recipe). Note that I bounced off of it too, until Martijn and Christian explained it to me and it was forced on me via grok. It's purely a documentation issue, I think. personally, i like chris mcdonough's approach with his buildit package. it does two things: - it retrieves files from anywhere on the 'net (cvs, svn, tarball, whatever) and puts them where you want them on your target machine(s) - it launches external scripts that then perform whatever final configuration you may want to perform. buildit is also recipe driven, and it's smart enough to pick up where it left off on a previous run, a'la make. i'd guess that you could use buildit and workingenv to accomplish pretty much everything you can do w/ zc.buildout, and you wouldn't have to bend your brain so much to do so. Sure. My point has always been that everybody should choose the tools they understand best. I think there's a place for both workingenv and buildout out there. I don't think that's short-sighted. My only assertion was that using workingenv and buildout together seems like overcomplicating things, as they're both trying to solve a lot of similar issues. -- http://worldcookery.com -- Professional Zope documentation and training Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
whit wrote: I'm not clear on what the advantage would be. I'm probably missing some use cases. I think they are both valid approaches to the problem. my main usecase is to be able to use buildouts in a workingenv without having to rewrite the recipes... right now, I have to do one or the other. Sure. I still don't know what that means. :) *Maybe* you'd like to use workingenv to install eggs and use buildout to do other things like set up zope instances. If that's the case, then it's just a matter of writing recipes that take into account how you install eggs and scripts. But maybe that's not what you mean. ... If people really see value in having buildout and workingenv work together, perhaps Ian and I could explore this a bit at PyCon next month. +1. my general feeling is that there is a enough overlap that this would be beneficial. Specific use cases would help to guide this. try to justify why a zope hammer is better and why zope folk keep building new hammers that don't play nice with everyone else's hammer loses us cred. You seem to be implying that workingenv is somehow a standard tool and that buildout is a zope-specific tool that needlessly duplicates a standard tool. I was perhaps offbase in my comment about egg handling. workingenv of course is not a standard tool. what I'm presenting here is a perspective I face on a regular basis; for a developer using workingenv already, it's far more standard than something else he is not using...and vice versa. No, it's more familiar -- to these people. in the case of tools that enable a developer to start quickly, developers are very difficult to ween from habits that are currently working. when a large and complex task such as building plone becomes codified in a technology that is not familar or easily compatible with the the developers current environment, it's an issue. The alternative seems to be a custom shell script. I can see how this would be familiar to the people who wrote it. When there are well established and documented buildout recipes for doing common tasks, then there will be familiar tools, for some definition of familiar. zc.buildout is in no way zope specific. Can a Zope developer not develop a tool without it being stamped as zope specific? maybe... maybe not. When a developer struggles with more than one tool from the same general source, it matters little to them whether one depends on the other or not. That's true for languages, companies, frameworks and everything else. I think it really depends whether something transcends it's immediate community(and maybe here whether a z floats in front of it's name somewhere). So lets see. To contribute to the wider Python community, I should change the name of my Company or change jobs. Hm. for packages from zope or plone people, if they rides roughly with existing python techs In what way does buildout ride roughly with existing python techs. It builds on setuptools. Do you mean that if there is another community package that does something somewhat similar, we aren't allow to create a similar package just because we are associated with the Zope project. and come from a zopeish repository, Oh, I have to use a different repository. sadly they are liable to get branded w/ a legacy reputation that haunts the name. Has it occurred to you that the problem here doesn't lie with the Zope project or Zope developers? inversely, those of us zope and plonistas may be a bit uncritical of projects coming from close to home Oh, as someone who gets lots of criticism, I really don't think that is much of a problem. and allow such technologies to stay inaccessible to wider audiences. How are we doing this? The primary forum for discussing buildout is the distutils sig. It is released through PyPI. It doesn't depend on any other technology that we use -- other than Python. What should I do to make it more accessible? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Ian Bicking wrote: Jim Fulton wrote: I actually tried to do this once before with zc.buildout, but I didn't get far -- probably a result of lack of effort and lack of familiarity with the overall stack. But I also recognize lots of the questions about stuff like the zope.conf file and Data.fs that still seem unresolved. Certainly when you tried this, buildout was very young and we hadn't written recipes to deal with these issues. We've made a lot of progress since then. Well, the last time I really used it was early December, and it still felt slow and awkward to me at the time, with several funny quirks. Hm, It's a bit hard to respond to awkward and quirks. I'll respond to performance issues a bit below. And frankly I like easy_install. It's probably 10x faster than buildout. I doubt that that is true now. Although that probably depends on what you are doing. Early versions of buildout did a lot of things inefficiently as I was still learning setuptools. Because of the way that buildout caches index information, I expect that creating a buildout from scratch that used a lot of eggs would be much faster than using easy_install. One difference though is that buildout checks for the most recent compatible versions of all of the eggs it's using every time you run it, whereas, as I understand it, with workingenv, you'd just run easy_install manually when you want a new egg. Correct. The basic process with workingenv is: 1. Set it up. 2. Start installing stuff. 3. Try running stuff. 4. Realize you got it wrong, missed something, want to do more development, return to 2. I actually find myself doing the 2-4 loop pretty often, both in development and when first deploying something. Just the amount of time to do bin/buildout -h was substantial (though I don't really understand why, except that buildout seemed to be working way too hard to update itself). Ah yes. This is a good point. By default, buildout checks for newer versions of distributions for which there are open-ended requirements. This can take frustratingly long -- especially because pypi is so darn slow. One advantage of buildout over easy_install (and I assume workingenv) is that the eggs you get are deterministic by default. They are always the newest versions that satisfy your requirements. With easy_install, you get the most recently installed eggs that satisfy your requirements. This means that the eggs you have depend a lot on when you installed them. To achieve this, buildout looks for newer distributions when a requirement doesn't have an upper bound or when the upper bound isn't satisfied by an already-installed egg. I really should add a quick mode that skips looking for newer versions of requirements are met by what's already installed. This would make the iterative style you describe go much faster. I would certainly appreciate this myself. I will do this soon. You can bypass the checks by running in offline mode. Then buildout runs very fast. Because of the ability to share eggs accross buildouts, it is often possible to run a buidout using lots of eggs in offline mode. It has been suggested that there should be a mode for buildout that only talks to the network when there isn't a local egg that satisfied a requirement. This would make buildout work more like workingenv when few if any eggs are actually needed. Yes; more like easy_install does as well, actually. Though the way easy_install works is hardly intuitive; I find myself frequently saying yes, you installed it, but did you -U install it? In particular, upgrading a distribution doesn't upgrade it's dependencies. This makes it harder to control which distributions are used in an environment. With easy_install, even through distributions are automatically included by virtue of being dependencies, they aren't automatically updated. There's no way to say I want the most recent version of everything. I wanted to make it easier to get the most recent version of the distributions used, which is why buildout has a different policy for looking up distributions. ... Plus buildout's desire to own everything and destroy everything it does not own ;) I'm not aware that it destroys anything. Could you be more specific? Well, it owns parts, and the recipes control that. Doesn't it also delete and reinstall there? Yes. Buildout tried to make a buildout reflect it's specification. This is an important feature. It uninstalls as well as installs. But it isn't controlling anything it wasn't asked to control. How it treats each area of the buildout I'm unclear. I can't help that. I've documented how this works in great detail. Simply making the file layout a bit more conventional, and describing anything non-obvious, would make buildout feel a lot more comfortable to the new user. What is conventional? Python uses different layouts on different systems. The Unix layout and Windows layout are quite
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Ian Bicking wrote: Jim Fulton wrote: If *Plone* becomes incompatible with workingenv that'd be bothersome I agree. But if a buildout is incompatible, eh... who knows, I would hope that buildout would not have to be compatible with workingenv, whatever that means, in order for Plone to be compatible. Then again, I'm not sure what compatibility means in this context. It would be a concern if, for instance, Plone started depending on buildout recipes for installation, without plain distutils recipes. Of course right now there are no distutils recipes for old-style Products. So actually it's an active issue -- if buildout enables Plone to keep from moving to distutils/setuptools/egg-style package deployment (because buildout is flexible enough to support the old patterns) then that would make it harder for workingenv. But I don't think anyone is proposing that buildout means that Plone (and Zope 2/Five) shouldn't continue its move towards traditional Python packaging. So basically I would just hope that Plone doesn't lean to heavily on buildout. Certainly buildout encourages the use of eggs. I think we're all moving toward the use of eggs pretty aggressively. Of course, a Zope installation, most notably instance creation, has requirements beyond the scope of eggs. Buildout addresses those requirements. it might even make sense to create something like a freeze this workingenv as a buildout script. That one directory structure can't be both at the same time isn't a huge problem in my mind. Deployment involves far more than getting the software installed. Yes; in a workingenv model you start with an environment, then you actually make your deployment using tools of your choice. workingenv doesn't deploy for you. Admittedly your choice isn't always good, because maybe you didn't want to make a choice ;) And buildout is one such tool. Path names aren't really the problem. We got a little guerrilla war going on over the setuptools' script-generating monkey patches. We'd both probably prefer a proper way to change how setuptools generates scripts, but it's not clear that we could really be compatible -- enumeration vs. changing the path is a totally different strategy. Um, we're both changing the path -- just in different ways. Maybe enumeration vs adding a directory of eggs is a better description. Setuptools controls the directory of eggs (via easy-install.pth), while buildout controls the scripts and doesn't give setuptools that control. No, the user controls the directory of eggs using easy_install, workingenv, or buildout. easy_install goes further by creating .pth files. A .pth contains a single static path. This is really no-different than a single static path baked into a script by buildout. In both cases, the user has some control over how this path is baked. I chose to include the path in the script because it makes the script more independent of it's environment. Once created, the script can be moved or linked anywhere and run from anywhere without any brittle rules for finding the .pth file. Phillip Eby suggested that script- dependent .pth files might be put alongside the script, but I think just including the path in the script is cleaner. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jim Fulton wrote:
Ian Bicking wrote:
Jim Fulton wrote:
If *Plone* becomes incompatible with workingenv that'd be bothersome
I agree.
But if a buildout is incompatible, eh... who knows,
I would hope that buildout would not have to be compatible with
workingenv, whatever that means, in order for Plone to be compatible.
Then again, I'm not sure what compatibility means in this context.
It would be a concern if, for instance, Plone started depending on
buildout recipes for installation, without plain distutils recipes. Of
course right now there are no distutils recipes for old-style Products.
So actually it's an active issue -- if buildout enables Plone to keep
from moving to distutils/setuptools/egg-style package deployment
(because buildout is flexible enough to support the old patterns) then
that would make it harder for workingenv. But I don't think anyone is
proposing that buildout means that Plone (and Zope 2/Five) shouldn't
continue its move towards traditional Python packaging.
So basically I would just hope that Plone doesn't lean to heavily on
buildout.
Certainly buildout encourages the use of eggs. I think we're all moving
toward the use of eggs pretty aggressively.
Of course, a Zope installation, most notably instance creation, has
requirements beyond the scope of eggs. Buildout addresses those
requirements.
it might even make sense to create something like a freeze this
workingenv as a buildout script. That one directory structure can't
be both at the same time isn't a huge problem in my mind.
Deployment involves far more than getting the software installed.
Yes; in a workingenv model you start with an environment, then you
actually make your deployment using tools of your choice. workingenv
doesn't deploy for you. Admittedly your choice isn't always good,
because maybe you didn't want to make a choice ;)
And buildout is one such tool.
Path names aren't really the problem. We got a little guerrilla war
going on over the setuptools' script-generating monkey patches. We'd
both probably prefer a proper way to change how setuptools generates
scripts, but it's not clear that we could really be compatible --
enumeration vs. changing the path is a totally different strategy.
Um, we're both changing the path -- just in different ways.
Maybe enumeration vs adding a directory of eggs is a better
description. Setuptools controls the directory of eggs (via
easy-install.pth), while buildout controls the scripts and doesn't give
setuptools that control.
No, the user controls the directory of eggs using easy_install, workingenv, or
buildout.
I don't think buildout's default locations would be called sensible by
anybody except the folks who wrote it. Here is some of what I find
puzzling:
- Why don't binary eggs go in one of the standard location
(lib/python or lib/python2.x)?
- Why not put development eggs in a directory which matches existing
conventions ('src')?
- Why are pieces of the buildout squirreled away under 'parts',
instead of putting them in a location which signals their purpose
('lib', 'libexec', 'var', etc.)?
- Why do I have to cd down into 'parts/instances/foo' to run the
application which is the point of the buildout, rather than running
scripts from 'bin'?
The perplexing filesystem layout was one of the biggest strikes against
zpkg: it feels as though buildout has some of the same heritage.
easy_install goes further by creating .pth files. A .pth contains
a single static path. This is really no-different than a single static path
baked into a script by buildout.
Except that *any* script running in that enviroment has access to the
paths set up by *all* the egg installations: adding a new egg doesn't
require touching the scripts.
In both cases, the user has some control
over how this path is baked. I chose to include the path in the script
because
it makes the script more independent of it's environment. Once created, the
script can be moved or linked anywhere and run from anywhere without any
brittle rules for finding the .pth file. Phillip Eby suggested that script-
dependent .pth files might be put alongside the script, but I think just
including the path in the script is cleaner.
For deployment, perhaps, but you have to munge *each* script when you
add a new egg in development, rather than letting setuptools tweak the
.pth files (which are the standard Python path extension mechanism).
It is therefore *much* harder to try out a new egg in a buildout
environment, using easy_install / setuptools, than in a workingenv (or
the environment created by 'virtual_python', which is what I use on a
day-to-day basis).
I don't have a usecase for executing the scripts with any python
interpeter other than the one which ran setuptools to generate them, and
therefore don't care for the hard-wired path
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Tres Seaver wrote:
I don't think buildout's default locations would be called sensible by
anybody except the folks who wrote it.
I think a lot of this may have to do with sensible defaults; most (all?)
of this is settable via options in buildout.cfg, which is reassuring at
least.
Here is some of what I find
puzzling:
- Why don't binary eggs go in one of the standard location
(lib/python or lib/python2.x)?
I'd tend to agree that that'd make sense. I think for ploneout's
buildout.cfg, we may set the option accordingly.
- Why not put development eggs in a directory which matches existing
conventions ('src')?
At least as it's used in ploneout, the development eggs are in src/ and
when they're installed, you get .egg-link files in development-eggs. We
may again put that into lib/python for ploneout's purposes.
- Why are pieces of the buildout squirreled away under 'parts',
instead of putting them in a location which signals their purpose
('lib', 'libexec', 'var', etc.)?
Where would a Zope 2 instance or a Zope 2 checkout sit? I thinks parts/
is as good as anything here.
For the z2c.recipe.zope2filestorage (which manages the location of
Data.fs) recipe I've just made it so that it uses 'var' instead of parts.
- Why do I have to cd down into 'parts/instances/foo' to run the
application which is the point of the buildout, rather than running
scripts from 'bin'?
Hanno is working on a recipe to let us get the zope instance start/stop
script in the top level bin for ploneout. There's already a recipe for
running tests from there. Philipp was of the opinion that you shouldn't
have to care about anything inside parts (it's buildout's playground)
and I'd like to make it so that this is the case.
In both cases, the user has some control
over how this path is baked. I chose to include the path in the script because
it makes the script more independent of it's environment. Once created, the
script can be moved or linked anywhere and run from anywhere without any
brittle rules for finding the .pth file. Phillip Eby suggested that script-
dependent .pth files might be put alongside the script, but I think just
including the path in the script is cleaner.
For deployment, perhaps, but you have to munge *each* script when you
add a new egg in development, rather than letting setuptools tweak the
.pth files (which are the standard Python path extension mechanism).
It is therefore *much* harder to try out a new egg in a buildout
environment, using easy_install / setuptools, than in a workingenv (or
the environment created by 'virtual_python', which is what I use on a
day-to-day basis).
I don't have a usecase for executing the scripts with any python
interpeter other than the one which ran setuptools to generate them, and
therefore don't care for the hard-wired path manipulation
I would agree that having to mangle multiple scripts is annoying. On the
other hand, I find the activate/deactivate dance annoying. I rather
think that the scripts that buildout has created should not require any
special activation or introduction to a particular environment.
Perhaps a solution is to let buildout put things in standard places (as
I'm typing this, we now have all the ploneout .egg and .egg-link
files/directories in ${buildout}/lib/python) and then have an optional
workingenv at the root of the buildout that could be activated when you
want that, but let the buildout-managed scripts (e.g. zope start/stop
and test runner) not require this?
I have a feeling there's more to this than agreeing on directory layout
though. But I hope it oughtn't be *that much more*.
Martin
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
I don't have a usecase for executing the scripts with any python
interpeter other than the one which ran setuptools to generate them, and
therefore don't care for the hard-wired path manipulation
I would agree that having to mangle multiple scripts is annoying. On the
other hand, I find the activate/deactivate dance annoying. I rather
think that the scripts that buildout has created should not require any
special activation or introduction to a particular environment.
Perhaps a solution is to let buildout put things in standard places (as
I'm typing this, we now have all the ploneout .egg and .egg-link
files/directories in ${buildout}/lib/python) and then have an optional
workingenv at the root of the buildout that could be activated when you
want that, but let the buildout-managed scripts (e.g. zope start/stop
and test runner) not require this?
I have a feeling there's more to this than agreeing on directory layout
though. But I hope it oughtn't be *that much more*.
I did some experimentation with this. Basically, I did:
- make egg-directory and develop-egg-directory in buildout.cfg be
lib/python2.4 (which is what workingenv uses)
- run the buildout again
- create standard workingenv (python workingenv.py .) in the root
buildout directory
The results are fairly encouraging:
- buildout-installed eggs don't work in the workingenv jail...
- ...but they do if I manually add them to lib/python2.4/easy_install.pth
- easy_installed packages (i.e. installed with workingenv's
easy_install) are available to zope
This was only a brief test, but seems to me that if buildout was
managing the easy_install.pth file we wouldn't be a million miles off
something where the source bin/activate dance was optional and would
work as a generic solution, but would not be required for zope's startup
and other buildout-managed scripts.
However, I'm not sure whether and how buildout should manage
easy_install.pth in a non-hacky way. A hacky way would be to have a
buildout recipie that, if included, would just patch that file up based
on buildout['buildout']['eggs'] (i.e. the list of eggs listed in
buildout.cfg under the main [buildout] section).
I don't really understand the purpose of easy_install.pth and why it
matters here, though, so suggestions welcome. :)
Martin
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
whit wrote: Martin Aspeli wrote: Philipp von Weitershausen wrote: This is awesome, and by that I don't mean the fact that we have a plone buildout, but that we actually have Zope 2 recipes for buildout. I hope they can be moved to svn.zope.org for further development to benefit the whole Zope 2 community. I believe this is just a matter of contrib agreements being sorted out (Hanno?). I guess I need to get mine sorted out as well if I'm going to keep working on this when it moves... :-/ I also see that workingenv was abandoned. That's very good to hear because buildout has a lot of machinery for installing eggs already, it would just've been duplicated with workingenv... is there some advantage to the way that buildout handles eggs over workingenv. as I understand it, workingenv *only* handles python setup and does that well and transparently. The point is that buildout *already* handles eggs. There's really no point for having an extra layer on top of buildout. The zc.recipe.egg recipe can install any egg (as a development one or not) in an automated fashion, which is exactly what you'd want from a buildout. the source bin/activate dance is the only thing I see being a detriment here(and with the latest workingenv, your shell prompt lets you know you are in an env). Not everybody likes the activate dance. With buildout, you don't need it. The recipes make sure that the scripts that get installed into the buildout's 'bin' directory have the right PYTHONPATH set and have access to the eggs you requested for the buildout. Workingenv made it more complex than it needed to be (or buildout did, depending on which perspective you look at it from). I believe Hanno wanted to rescue the recipe in case others found it useful, but it's not used for now. what about if I'm already using workingenv... and want to use zope or plone in my workingenv? I see no problem with that. zc.buildout is one way of deploying Python software like Zope. As long as you've got eggs, you could install them manually into your workingenv just fine. buildout just does it an automated fashion (and, of course, it can do more than just installing eggs). currently, I don't see an easy way to use buildouts inside a workingenv, whereas the rest of python world works great. I will have alot of trouble explaining to my developer who already think zope smells that they have to change the way they work to use zc.buildout recipes. for example, I can't use the deliverance or lxml buildout with an existing topp.deploy workingenv because of buildout's arbitrary egg handling scheme. If zc.buildout didn't try to do so much, the python would be installed transparently like everything else I easy_install. I'm not too fond of zc.buildout's directory scheme eiher. In particular, I wish that it would use 'lib/python' instead of 'eggs' and 'src' instead of 'develop-eggs'. I don't know enough zc.buildout details to be able to say whether that can be chagned by remimplementing the egg installation recipe. I would sure hope it is. as stated before, I don't mind using zc.buildout, but I don't want to have to learn zc.buildout to use it meaningfully in my existing setup. If a buildout recipes could be executed by themselves(like buildout-zope2, buildout-deliverance, buildout-squid) as well as by aggregated recipes. This would make buildout a killer tool inside my workingenv rather than a choice I had to make between two technologies. As said already, I think once you've got buildout, there's no need for workingen, except if you think that Zope stinks ;) -- http://worldcookery.com -- Professional Zope documentation and training Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Philipp von Weitershausen wrote:
The point is that buildout *already* handles eggs. There's really no
point for having an extra layer on top of buildout. The zc.recipe.egg
recipe can install any egg (as a development one or not) in an automated
fashion, which is exactly what you'd want from a buildout.
At least it's what I want. That is, easy_install may as well put things
in the ether as far as I'm concerned, and I'm more comfortable with a
single file (buildout.cfg) that gives me an overview of which eggs my
application consists of.
Very open to be persuaded otherwise, though. ;-)
the source bin/activate dance is the only thing I see being a
detriment here(and with the latest workingenv, your shell prompt lets
you know you are in an env).
Not everybody likes the activate dance. With buildout, you don't need
it. The recipes make sure that the scripts that get installed into the
buildout's 'bin' directory have the right PYTHONPATH set and have access
to the eggs you requested for the buildout.
On the one hand, this patching is a bit odd, but probably just a result
of the fact that Zope itself isn't terribly egg/pythonpath friendly. On
the other hand, I don't like the stateful nature of the activate dance
to point where it feels hacky to me. I know others disagree, though.
I see no problem with that. zc.buildout is one way of deploying Python
software like Zope. As long as you've got eggs, you could install them
manually into your workingenv just fine. buildout just does it an
automated fashion (and, of course, it can do more than just installing
eggs).
... and I've come to like its approach to stringing together recipes and
passing options. It fits my brain. :)
I'm not too fond of zc.buildout's directory scheme eiher. In particular,
I wish that it would use 'lib/python' instead of 'eggs' and 'src'
instead of 'develop-eggs'. I don't know enough zc.buildout details to be
able to say whether that can be chagned by remimplementing the egg
installation recipe. I would sure hope it is.
in buildout.cfg, I did:
[buildout]
eggs-directory = lib/python
develop-eggs-directory = lib/python
Eggs are now in ${buildout}/lib/python, and it seems to work fine (but I
had to stop short of testing in detail). I imagine that if workingenv
was told of the same directory, the two would co-exist.
I want to play with the zope 2 recipies to make filesystem layout a bit
more flexible in these.
as stated before, I don't mind using zc.buildout, but I don't want to
have to learn zc.buildout to use it meaningfully in my existing setup.
If a buildout recipes could be executed by themselves(like
buildout-zope2, buildout-deliverance, buildout-squid) as well as by
aggregated recipes. This would make buildout a killer tool inside my
workingenv rather than a choice I had to make between two technologies.
As said already, I think once you've got buildout, there's no need for
workingen, except if you think that Zope stinks ;)
Plone stinks!
Martin
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Philipp von Weitershausen wrote: whit wrote: Martin Aspeli wrote: Philipp von Weitershausen wrote: This is awesome, and by that I don't mean the fact that we have a plone buildout, but that we actually have Zope 2 recipes for buildout. I hope they can be moved to svn.zope.org for further development to benefit the whole Zope 2 community. I believe this is just a matter of contrib agreements being sorted out (Hanno?). I guess I need to get mine sorted out as well if I'm going to keep working on this when it moves... :-/ I also see that workingenv was abandoned. That's very good to hear because buildout has a lot of machinery for installing eggs already, it would just've been duplicated with workingenv... is there some advantage to the way that buildout handles eggs over workingenv. as I understand it, workingenv *only* handles python setup and does that well and transparently. The point is that buildout *already* handles eggs. There's really no point for having an extra layer on top of buildout. The zc.recipe.egg recipe can install any egg (as a development one or not) in an automated fashion, which is exactly what you'd want from a buildout. honestly, it seems to me that buildout tries to do too much. it's trying to handle both repeatable deployment recipes AND providing a sandbox within which to run things. there may not be a point to having an extra layer on top of buildout, but buildout sure does seem to me a bit heavy if all i want is a sandbox. so now i have to learn the workingenv way if i just need a sandbox, but i have to learn the buildout way if i also want repeatable deployments? Workingenv made it more complex than it needed to be (or buildout as stated before, I don't mind using zc.buildout, but I don't want to have to learn zc.buildout to use it meaningfully in my existing setup. If a buildout recipes could be executed by themselves(like buildout-zope2, buildout-deliverance, buildout-squid) as well as by aggregated recipes. This would make buildout a killer tool inside my workingenv rather than a choice I had to make between two technologies. As said already, I think once you've got buildout, there's no need for workingen, except if you think that Zope stinks ;) this is shortsighted, IMO. i know zope quite well, but i bounced off of buildout b/c it required too much knowledge to even get started. i think it's much more likely that people from the greater python community will pick up and start using workingenv than they will zc.buildout. personally, i like chris mcdonough's approach with his buildit package. it does two things: - it retrieves files from anywhere on the 'net (cvs, svn, tarball, whatever) and puts them where you want them on your target machine(s) - it launches external scripts that then perform whatever final configuration you may want to perform. buildit is also recipe driven, and it's smart enough to pick up where it left off on a previous run, a'la make. i'd guess that you could use buildit and workingenv to accomplish pretty much everything you can do w/ zc.buildout, and you wouldn't have to bend your brain so much to do so. -r ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
On Thu, 25 Jan 2007 12:45:26 -0800, Martin Aspeli [EMAIL PROTECTED] wrote: Plone stinks! It's like a fine cheese. -- Alexander Limi · http://limi.net ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Philipp von Weitershausen wrote: whit wrote: Martin Aspeli wrote: Philipp von Weitershausen wrote: This is awesome, and by that I don't mean the fact that we have a plone buildout, but that we actually have Zope 2 recipes for buildout. I hope they can be moved to svn.zope.org for further development to benefit the whole Zope 2 community. I believe this is just a matter of contrib agreements being sorted out (Hanno?). I guess I need to get mine sorted out as well if I'm going to keep working on this when it moves... :-/ I also see that workingenv was abandoned. That's very good to hear because buildout has a lot of machinery for installing eggs already, it would just've been duplicated with workingenv... is there some advantage to the way that buildout handles eggs over workingenv. as I understand it, workingenv *only* handles python setup and does that well and transparently. The point is that buildout *already* handles eggs. There's really no point for having an extra layer on top of buildout. The zc.recipe.egg recipe can install any egg (as a development one or not) in an automated fashion, which is exactly what you'd want from a buildout. well that's awesome that buildout has duplicated this effort the source bin/activate dance is the only thing I see being a detriment here(and with the latest workingenv, your shell prompt lets you know you are in an env). Not everybody likes the activate dance. With buildout, you don't need it. The recipes make sure that the scripts that get installed into the buildout's 'bin' directory have the right PYTHONPATH set and have access to the eggs you requested for the buildout. is there really a difference between zopectl and source bin/activate? I guess the main difference here is where PYTHONPATH gets set and how people work with it. for example, if you just start python and want to play with code sounds like with zc.buildout you are out of luck for things installed in the buildout. In our situation, we might have multiple versions of software running on different processes started in different workingenv (often not even using zope).being able to contextualize the python path is a useful development tool; this is understandable for buildout to avoid(as a deployment tool), but if we are considering using buildout as a prescribed way for people to manage their development environments, it seems lacking. Workingenv made it more complex than it needed to be (or buildout did, depending on which perspective you look at it from). I believe Hanno wanted to rescue the recipe in case others found it useful, but it's not used for now. what about if I'm already using workingenv... and want to use zope or plone in my workingenv? I see no problem with that. zc.buildout is one way of deploying Python software like Zope. As long as you've got eggs, you could install them manually into your workingenv just fine. buildout just does it an automated fashion (and, of course, it can do more than just installing eggs). which I could automate in my env (workingenv takes recipes also). for that matter, eggs automate the installation of eggs... I'm not sure what it would take to make buildout just automate install eggs into my env... but not putting them in special directories would be a start. currently, I don't see an easy way to use buildouts inside a workingenv, whereas the rest of python world works great. I will have alot of trouble explaining to my developer who already think zope smells that they have to change the way they work to use zc.buildout recipes. for example, I can't use the deliverance or lxml buildout with an existing topp.deploy workingenv because of buildout's arbitrary egg handling scheme. If zc.buildout didn't try to do so much, the python would be installed transparently like everything else I easy_install. I'm not too fond of zc.buildout's directory scheme eiher. In particular, I wish that it would use 'lib/python' instead of 'eggs' and 'src' instead of 'develop-eggs'. I don't know enough zc.buildout details to be able to say whether that can be chagned by remimplementing the egg installation recipe. I would sure hope it is. +1 this may be all that is required to make the two play well together(or have buildout respect an existing workingenv when doing it's local install of eggs). Maybe it's just a matter of zc.buildout seeing workingenv is in effect and not composing special pythonpaths. harmony is my interest here. workingenv is pretty low-level and works hard to work well with python. there is no reason that zc.buildout should have a problem with that.. it just needs to do less when appropriate. as stated before, I don't mind using zc.buildout, but I don't want to have to learn zc.buildout to use it meaningfully in my existing setup. If a buildout recipes could be executed by themselves(like buildout-zope2, buildout-deliverance, buildout-squid) as
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
whit wrote: Not everybody likes the activate dance. With buildout, you don't need it. The recipes make sure that the scripts that get installed into the buildout's 'bin' directory have the right PYTHONPATH set and have access to the eggs you requested for the buildout. is there really a difference between zopectl and source bin/activate? I guess the main difference here is where PYTHONPATH gets set and how people work with it. for example, if you just start python and want to play with code sounds like with zc.buildout you are out of luck for things installed in the buildout. In our situation, we might have multiple versions of software running on different processes started in different workingenv (often not even using zope).being able to contextualize the python path is a useful development tool; this is understandable for buildout to avoid(as a deployment tool), but if we are considering using buildout as a prescribed way for people to manage their development environments, it seems lacking. I think this is the basic difference -- workingenv is development-centric, while buildout is deployment-centric. This does not necessarily mean the best tool for the job, because focusing on development and ignore deployment isn't a good job, nor the other way around. At TOPP we know how to address both stories with workingenv. We haven't figured it out with buildout, despite trying. And we definitely aren't satisfied with just a deployment tool that doesn't address our development needs. But there's a lot of use cases for Plone specifically where carefully developing just a deployment solution makes sense. That doesn't make sense for us, because we're not in the kind of consulting business where the relative scale of development and deployment works like that. But eh; the presence of a buildout for Plone doesn't hurt our position any. Anything that gets rid of another ad hoc crappy deployment is good by me. And if other people are working on it, all the better! If *Plone* becomes incompatible with workingenv that'd be bothersome (though it was not compatible with workingenv without some work in the first place). But if a buildout is incompatible, eh... who knows, it might even make sense to create something like a freeze this workingenv as a buildout script. That one directory structure can't be both at the same time isn't a huge problem in my mind. I'm not too fond of zc.buildout's directory scheme eiher. In particular, I wish that it would use 'lib/python' instead of 'eggs' and 'src' instead of 'develop-eggs'. I don't know enough zc.buildout details to be able to say whether that can be chagned by remimplementing the egg installation recipe. I would sure hope it is. +1 this may be all that is required to make the two play well together(or have buildout respect an existing workingenv when doing it's local install of eggs). Maybe it's just a matter of zc.buildout seeing workingenv is in effect and not composing special pythonpaths. harmony is my interest here. workingenv is pretty low-level and works hard to work well with python. there is no reason that zc.buildout should have a problem with that.. it just needs to do less when appropriate. Path names aren't really the problem. We got a little guerrilla war going on over the setuptools' script-generating monkey patches. We'd both probably prefer a proper way to change how setuptools generates scripts, but it's not clear that we could really be compatible -- enumeration vs. changing the path is a totally different strategy. I'd rather see easy-install.pth become a better package database, or some other strategy of external requirement specifications, instead of building it into scripts. I understand the issue Jim is trying to solve here, but putting everything into the buildout.cfg and then imperatively setting up the files from there bothers me and does not make development easy IMHO. That's mostly the problem. Then workingenv would do its part by monkeypatching distutils and setuptools to install things locally, and changing site.py to not automatically pick up things globally. And buildout could do its stuff that applies to other more complicated setups than just setting up some libraries, which is about where workingenv stops. Well, what I'd *really* like is an idea of a working environment that applies more widely than Python, because other languages (including but not limited to the dynamic languages) have all these same problems, and we all have half-assed solutions. But I don't even know how to start that conversation. And don't get me started on how the distributions are going to look at all this! Ian ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists -
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
I hate to jump into this thread but I'll make a few comments. On Jan 25, 2007, at 5:13 PM, whit wrote: Philipp von Weitershausen wrote: whit wrote: Martin Aspeli wrote: Philipp von Weitershausen wrote: This is awesome, and by that I don't mean the fact that we have a plone buildout, but that we actually have Zope 2 recipes for buildout. I hope they can be moved to svn.zope.org for further development to benefit the whole Zope 2 community. I believe this is just a matter of contrib agreements being sorted out (Hanno?). I guess I need to get mine sorted out as well if I'm going to keep working on this when it moves... :-/ I also see that workingenv was abandoned. That's very good to hear because buildout has a lot of machinery for installing eggs already, it would just've been duplicated with workingenv... is there some advantage to the way that buildout handles eggs over workingenv. as I understand it, workingenv *only* handles python setup and does that well and transparently. Right, buildout handles a lot more. For example, we use buildout to build C libraries and to generate various sorts of application artifacts like configuration files, custom startup scripts, and so on, in addition to Python scripts. The point is that buildout *already* handles eggs. There's really no point for having an extra layer on top of buildout. The zc.recipe.egg recipe can install any egg (as a development one or not) in an automated fashion, which is exactly what you'd want from a buildout. well that's awesome that buildout has duplicated this effort I'm not sure what effort you think buildout is duplicating. WRT eggs, buildout, like workinenv/easy_install builds on setuptools. setuptools does the heavy lifting with regard to building eggs. Buildout does have some nice features for building eggs with extensions that need custom setup options. Like workenv, buildout supports installing eggs in development environments rather than the system Python. workingenv bends easy_install to it's will, while buildout goes below easy_install. This allows buildout to have more control, which was important to me. Buildout gives me greater control over egg revisions used and the way that scripts get generated. (Maybe workingenv would give me more control than I think it does -- I haven't looked at it in a while.) the source bin/activate dance is the only thing I see being a detriment here(and with the latest workingenv, your shell prompt lets you know you are in an env). Not everybody likes the activate dance. With buildout, you don't need it. The recipes make sure that the scripts that get installed into the buildout's 'bin' directory have the right PYTHONPATH set and have access to the eggs you requested for the buildout. is there really a difference between zopectl and source bin/ activate? I guess the main difference here is where PYTHONPATH gets set and how people work with it. for example, if you just start python and want to play with code sounds like with zc.buildout you are out of luck for things installed in the buildout. I'm not sure what you mean here. I'll note that buildout lets you easily create interpreter scripts. When run without arguments, you get a Python prompt. When run with a script name and arguments, then they will run the script with the arguments. You define an interpreter by specifying on or more eggs and those eggs and their dependencies (and additional paths you can optionally set) will be included in the interpreter's path. This makes playing with code pretty easy. In our situation, we might have multiple versions of software running on different processes started in different workingenv (often not even using zope).being able to contextualize the python path is a useful development tool; this is understandable for buildout to avoid(as a deployment tool), but if we are considering using buildout as a prescribed way for people to manage their development environments, it seems lacking. I don't know what you mean by contextualize the python path. An important feature of buildout (actualy the egg recipe) is that it lets you create scripts with exactly the eggs they need. This was a core motivation of buildout. And, as I have mentioned above, you can also create interpreter scripts for experimentation and running other scripts. Perhaps these interpreter scripts are like your workingenvs. Workingenv made it more complex than it needed to be (or buildout did, depending on which perspective you look at it from). I believe Hanno wanted to rescue the recipe in case others found it useful, but it's not used for now. what about if I'm already using workingenv... and want to use zope or plone in my workingenv? I see no problem with that. zc.buildout is one way of deploying Python software like Zope. As long as you've got eggs,
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
On Jan 25, 2007, at 5:09 PM, Ian Bicking wrote: Whit pointed me to this thread. Yeah, someone pointed me to it too. :) I won't reply to specifics, but maybe just describe what we're doing (and planning to do), and how workingenv differs from zc.buildout. I'll avoid responding to general qualitative statements. ... I actually tried to do this once before with zc.buildout, but I didn't get far -- probably a result of lack of effort and lack of familiarity with the overall stack. But I also recognize lots of the questions about stuff like the zope.conf file and Data.fs that still seem unresolved. Certainly when you tried this, buildout was very young and we hadn't written recipes to deal with these issues. We've made a lot of progress since then. The thing that frustrated me with zc.buildout is that I knew how to do what I wanted to do, but I still felt like I was a long way from being able to do it. And little things, like unhelpful error messages Yeah, buildout still needs to do a lot better with error messages, although it has probably made some progress since you tried it. and frustratingly slow performance really killed my motivation. That has improved quite a bit. ... And frankly I like easy_install. It's probably 10x faster than buildout. I doubt that that is true now. Although that probably depends on what you are doing. Early versions of buildout did a lot of things inefficiently as I was still learning setuptools. Because of the way that buildout caches index information, I expect that creating a buildout from scratch that used a lot of eggs would be much faster than using easy_install. One difference though is that buildout checks for the most recent compatible versions of all of the eggs it's using every time you run it, whereas, as I understand it, with workingenv, you'd just run easy_install manually when you want a new egg. You can bypass the checks by running in offline mode. Then buildout runs very fast. Because of the ability to share eggs accross buildouts, it is often possible to run a buidout using lots of eggs in offline mode. It has been suggested that there should be a mode for buildout that only talks to the network when there isn't a local egg that satisfied a requirement. This would make buildout work more like workingenv when few if any eggs are actually needed. easy_install is what people use in documentation, and its conventions are the ones people know (why does buildout not use Pkg==version, for instance?). It does. When specifying eggs, you use standard setuptools requirement syntax. As for the technical reasons they don't work together: * workingenv allows and leaves it to setuptools to maintain the package installation database (basically easy-install.pth). This is not a very good database, but eh. buildout doesn't really have a database, but instead just enforces what buildout.cfg indicates. buildout uses the buildout configuration file to store what you want. It uses .installed.cfg to capture what you have. These are both databases of sorts. * workingenv relies on that database to give default versions and to setup the Python path. The fixup it does of installed scripts is fairly minimal, just setting up sys.path enough to force its site.py to get called. buildout enumerates all the activated packages, and ignores easy-install.pth. This is basically what makes it easy_install-incompatible. Yup. I wanted something far more static and predictable for scripts generated by buildout. Plus buildout's desire to own everything and destroy everything it does not own ;) I'm not aware that it destroys anything. Could you be more specific? * As a result buildout supports multiple things in the same buildout that have conflicting version requirements, but where the packages themselves don't realize this (but the deployer does). If the packages know their requirements then setuptools' native machinery allows things to work fine. Yes. I expect that usually, packages won't be very specific. The buildout configuration file provides a place to be specific. * Some see bin/activate as a jail. Both workingenv and buildout are deliberately jail-like. Both Jim and I loathe the non- repeatability of system-wide installations (at least I think I can speak for him on that one point ;). bin/activate lets you into that jail, and lets you work there. There is no way into a buildout. I'm not familiar with bin/activate, but it sounds like an interpreter script created with buildout. Frankly this weirds me out, and is a big part of my past frustration with it. Maybe that's because I'm in the relatively uncommon situation that I actually know what's going on under the hood of Python imports and packaging, and so it bothers me that I can't debug things directly. Anyway, neither requires activation when using scripts generated
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
On Jan 25, 2007, at 5:44 PM, Ian Bicking wrote: workingenv is development-centric, while buildout is deployment- centric. This does not necessarily mean the best tool for the job, because focusing on development and ignore deployment isn't a good job, nor the other way around. buildout focusses on both. ... If *Plone* becomes incompatible with workingenv that'd be bothersome I agree. But if a buildout is incompatible, eh... who knows, I would hope that buildout would not have to be compatible with workingenv, whatever that means, in order for Plone to be compatible. Then again, I'm not sure what compatibility means in this context. it might even make sense to create something like a freeze this workingenv as a buildout script. That one directory structure can't be both at the same time isn't a huge problem in my mind. Deployment involves far more than getting the software installed. ... Path names aren't really the problem. We got a little guerrilla war going on over the setuptools' script-generating monkey patches. We'd both probably prefer a proper way to change how setuptools generates scripts, but it's not clear that we could really be compatible -- enumeration vs. changing the path is a totally different strategy. Um, we're both changing the path -- just in different ways. I'd rather see easy-install.pth become a better package database, or some other strategy of external requirement specifications, instead of building it into scripts. We certainly disagree there. OTOH, I wouldn't be opposed to having a recipe for generating scripts that used the .pth files created by workingenv. I understand the issue Jim is trying to solve here, but putting everything into the buildout.cfg and then imperatively setting up the files from there bothers me and does not make development easy IMHO. This seems to be a matter of taste. I like developing with buildout. It makes development easier for me. :) To each his own. That's mostly the problem. Then workingenv would do its part by monkeypatching distutils and setuptools to install things locally, and changing site.py to not automatically pick up things globally. I applaud you for this effort. I chose to work below easy_install and site.py rather that trying to bend/monkey-path it to my will. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: If *Plone* becomes incompatible with workingenv that'd be bothersome I agree. But if a buildout is incompatible, eh... who knows, I would hope that buildout would not have to be compatible with workingenv, whatever that means, in order for Plone to be compatible. Then again, I'm not sure what compatibility means in this context. It would be a concern if, for instance, Plone started depending on buildout recipes for installation, without plain distutils recipes. Of course right now there are no distutils recipes for old-style Products. So actually it's an active issue -- if buildout enables Plone to keep from moving to distutils/setuptools/egg-style package deployment (because buildout is flexible enough to support the old patterns) then that would make it harder for workingenv. But I don't think anyone is proposing that buildout means that Plone (and Zope 2/Five) shouldn't continue its move towards traditional Python packaging. So basically I would just hope that Plone doesn't lean to heavily on buildout. it might even make sense to create something like a freeze this workingenv as a buildout script. That one directory structure can't be both at the same time isn't a huge problem in my mind. Deployment involves far more than getting the software installed. Yes; in a workingenv model you start with an environment, then you actually make your deployment using tools of your choice. workingenv doesn't deploy for you. Admittedly your choice isn't always good, because maybe you didn't want to make a choice ;) Path names aren't really the problem. We got a little guerrilla war going on over the setuptools' script-generating monkey patches. We'd both probably prefer a proper way to change how setuptools generates scripts, but it's not clear that we could really be compatible -- enumeration vs. changing the path is a totally different strategy. Um, we're both changing the path -- just in different ways. Maybe enumeration vs adding a directory of eggs is a better description. Setuptools controls the directory of eggs (via easy-install.pth), while buildout controls the scripts and doesn't give setuptools that control. I'd rather see easy-install.pth become a better package database, or some other strategy of external requirement specifications, instead of building it into scripts. We certainly disagree there. OTOH, I wouldn't be opposed to having a recipe for generating scripts that used the .pth files created by workingenv. I'm not sure what the problem would be? I appreciate the desire for a set of requirements that is different from the requirements of the package, and workingenv has some support for that (with the --requirements option), but it's a different style from buildout. easy-install.pth would almost be okay, except for the fact that it is constantly pointing to things that don't exist, has setuptools' hacks in it (to work around the atrocious Python standard site.py), and doesn't describe intent, it's just an enumeration of what is available and activated (i.e., available without specifically requiring something). Besides all that, it's still a workable foundation IMHO. -- Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Rob Miller wrote: honestly, it seems to me that buildout tries to do too much. it's trying to handle both repeatable deployment recipes AND providing a sandbox within which to run things. there may not be a point to having an extra layer on top of buildout, but buildout sure does seem to me a bit heavy if all i want is a sandbox. so now i have to learn the workingenv way if i just need a sandbox, but i have to learn the buildout way if i also want repeatable deployments? Potentially. But I find it kind of reassuring to have a well-defined list of which eggs are part of my special environment i.e. the one tied to my instance. As said already, I think once you've got buildout, there's no need for workingen, except if you think that Zope stinks ;) this is shortsighted, IMO. i know zope quite well, but i bounced off of buildout b/c it required too much knowledge to even get started. i think it's much more likely that people from the greater python community will pick up and start using workingenv than they will zc.buildout. Again, I think the two are orthogonal. And honestly, I found zc.buildout pretty easy to understand. I resisted it for a while, it seems liked it *should* be complex, and I won't pretend to understand how it manages eggs in any detail, but I don't think it matters. Look at http://dev.plone.org/plone/browser/ploneout/trunk/buildout.cfg - I find that pretty self-explanatory. I tried to document how it works at a high level and how you may use it here http://dev.plone.org/plone/browser/ploneout/trunk/README.txt. And writing a new recipe is as simple as this: http://dev.plone.org/plone/browser/ploneout/trunk/src/z2c.recipe.zope2checkout/src/z2c/recipe/zope2checkout/__init__.py All that is plain python code, the only thing you need to understand about buildout is that it has a dict-like object called 'options' which reflects the options in the current part's section in buildout.cfg, and a higher level dict-like object called 'buildout' which has the options for all the parts (so buildout['foo'] are the options for part [foo] in buidout.cfg). Each part is associated with a recipie. Recipies are ordered. personally, i like chris mcdonough's approach with his buildit package. it does two things: - it retrieves files from anywhere on the 'net (cvs, svn, tarball, whatever) and puts them where you want them on your target machine(s) You can do that quite easily with buildout as well. I would like to make a more generic recipe for retrieving tarballs for e.g. zope installation, and I think it'd be as hard as figuring out which python function to use to download something. - it launches external scripts that then perform whatever final configuration you may want to perform. Again, if you want a recipe to do that, it's trivial to write (10 lines of code?). Instead of an external script, though, I would probably find it easier to write that as a buildout recipe in python. buildit is also recipe driven, and it's smart enough to pick up where it left off on a previous run, a'la make. i'd guess that you could use buildit and workingenv to accomplish pretty much everything you can do w/ zc.buildout, and you wouldn't have to bend your brain so much to do so. I'm just struggling to see why it's so hard to figure out how buildout works. Perhaps it just fits my brain. But honestly, once Hanno showed me his initial steps with ploneout and I'd taken ten minutes with pdb inside the __init__() and install() functions (the only two...) in his recipe the pieces fitted together in my head almost instantly. I don't greatly care how the standard zc.recipe.egg mechanism installs eggs because it works and because I can see clearly where they come from, how I create new ones and I'm satisfied that they are available ot my python interpreter. If I did care, I'm sure I wouldn't find it that hard to trace, though. Martin ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Jim Fulton wrote: I actually tried to do this once before with zc.buildout, but I didn't get far -- probably a result of lack of effort and lack of familiarity with the overall stack. But I also recognize lots of the questions about stuff like the zope.conf file and Data.fs that still seem unresolved. Certainly when you tried this, buildout was very young and we hadn't written recipes to deal with these issues. We've made a lot of progress since then. Well, the last time I really used it was early December, and it still felt slow and awkward to me at the time, with several funny quirks. And frankly I like easy_install. It's probably 10x faster than buildout. I doubt that that is true now. Although that probably depends on what you are doing. Early versions of buildout did a lot of things inefficiently as I was still learning setuptools. Because of the way that buildout caches index information, I expect that creating a buildout from scratch that used a lot of eggs would be much faster than using easy_install. One difference though is that buildout checks for the most recent compatible versions of all of the eggs it's using every time you run it, whereas, as I understand it, with workingenv, you'd just run easy_install manually when you want a new egg. Correct. The basic process with workingenv is: 1. Set it up. 2. Start installing stuff. 3. Try running stuff. 4. Realize you got it wrong, missed something, want to do more development, return to 2. I actually find myself doing the 2-4 loop pretty often, both in development and when first deploying something. Just the amount of time to do bin/buildout -h was substantial (though I don't really understand why, except that buildout seemed to be working way too hard to update itself). You can bypass the checks by running in offline mode. Then buildout runs very fast. Because of the ability to share eggs accross buildouts, it is often possible to run a buidout using lots of eggs in offline mode. It has been suggested that there should be a mode for buildout that only talks to the network when there isn't a local egg that satisfied a requirement. This would make buildout work more like workingenv when few if any eggs are actually needed. Yes; more like easy_install does as well, actually. Though the way easy_install works is hardly intuitive; I find myself frequently saying yes, you installed it, but did you -U install it? As for the technical reasons they don't work together: * workingenv allows and leaves it to setuptools to maintain the package installation database (basically easy-install.pth). This is not a very good database, but eh. buildout doesn't really have a database, but instead just enforces what buildout.cfg indicates. buildout uses the buildout configuration file to store what you want. It uses .installed.cfg to capture what you have. These are both databases of sorts. * workingenv relies on that database to give default versions and to setup the Python path. The fixup it does of installed scripts is fairly minimal, just setting up sys.path enough to force its site.py to get called. buildout enumerates all the activated packages, and ignores easy-install.pth. This is basically what makes it easy_install-incompatible. Yup. I wanted something far more static and predictable for scripts generated by buildout. Plus buildout's desire to own everything and destroy everything it does not own ;) I'm not aware that it destroys anything. Could you be more specific? Well, it owns parts, and the recipes control that. Doesn't it also delete and reinstall there? How it treats each area of the buildout I'm unclear. Simply making the file layout a bit more conventional, and describing anything non-obvious, would make buildout feel a lot more comfortable to the new user. * As a result buildout supports multiple things in the same buildout that have conflicting version requirements, but where the packages themselves don't realize this (but the deployer does). If the packages know their requirements then setuptools' native machinery allows things to work fine. Yes. I expect that usually, packages won't be very specific. The buildout configuration file provides a place to be specific. workingenv allows this, insofar as you can be specific while installing things, and with the requirements file. But it doesn't make the individual scripts very specific, if for instance appfoo requires libX1.0, and appbar requires libX1.1, but you actually want appfoo to use libX==1.0 and appbar to use libX==1.1 and install them in the same buildout. That's the only case where buildout seems to be able to express something workingenv can't. * Some see bin/activate as a jail. Both workingenv and buildout are deliberately jail-like. Both Jim and I loathe the non-repeatability of system-wide installations (at least I think I can speak for him on that one point ;).
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: Rob Miller wrote: honestly, it seems to me that buildout tries to do too much. it's trying to handle both repeatable deployment recipes AND providing a sandbox within which to run things. there may not be a point to having an extra layer on top of buildout, but buildout sure does seem to me a bit heavy if all i want is a sandbox. so now i have to learn the workingenv way if i just need a sandbox, but i have to learn the buildout way if i also want repeatable deployments? Potentially. But I find it kind of reassuring to have a well-defined list of which eggs are part of my special environment i.e. the one tied to my instance. to find this i simply look in /path/to/workingenv/lib/python2.4. As said already, I think once you've got buildout, there's no need for workingen, except if you think that Zope stinks ;) this is shortsighted, IMO. i know zope quite well, but i bounced off of buildout b/c it required too much knowledge to even get started. i think it's much more likely that people from the greater python community will pick up and start using workingenv than they will zc.buildout. Again, I think the two are orthogonal. orthogonal, but with overlap. And honestly, I found zc.buildout pretty easy to understand. I resisted it for a while, it seems liked it *should* be complex, and I won't pretend to understand how it manages eggs in any detail, but I don't think it matters. Look at http://dev.plone.org/plone/browser/ploneout/trunk/buildout.cfg - I find that pretty self-explanatory. I tried to document how it works at a high level and how you may use it here http://dev.plone.org/plone/browser/ploneout/trunk/README.txt. And writing a new recipe is as simple as this: http://dev.plone.org/plone/browser/ploneout/trunk/src/z2c.recipe.zope2checkout/src/z2c/recipe/zope2checkout/__init__.py All that is plain python code, the only thing you need to understand about buildout is that it has a dict-like object called 'options' which reflects the options in the current part's section in buildout.cfg, and a higher level dict-like object called 'buildout' which has the options for all the parts (so buildout['foo'] are the options for part [foo] in buidout.cfg). Each part is associated with a recipie. Recipies are ordered. personally, i like chris mcdonough's approach with his buildit package. it does two things: - it retrieves files from anywhere on the 'net (cvs, svn, tarball, whatever) and puts them where you want them on your target machine(s) You can do that quite easily with buildout as well. I would like to make a more generic recipe for retrieving tarballs for e.g. zope installation, and I think it'd be as hard as figuring out which python function to use to download something. i have no doubt that zc.buildout will do everything that buildit will do, and probably much more. but i like simple. and for me, having something light like workingenv to manage my sandbox, and a library like buildit for putting files into those sandboxes feels simple. - it launches external scripts that then perform whatever final configuration you may want to perform. Again, if you want a recipe to do that, it's trivial to write (10 lines of code?). Instead of an external script, though, I would probably find it easier to write that as a buildout recipe in python. the script could of course be python as well. buildit is also recipe driven, and it's smart enough to pick up where it left off on a previous run, a'la make. i'd guess that you could use buildit and workingenv to accomplish pretty much everything you can do w/ zc.buildout, and you wouldn't have to bend your brain so much to do so. I'm just struggling to see why it's so hard to figure out how buildout works. Perhaps it just fits my brain. But honestly, once Hanno showed me his initial steps with ploneout and I'd taken ten minutes with pdb inside the __init__() and install() functions (the only two...) in his recipe the pieces fitted together in my head almost instantly. your efforts to figure something out and then document will have a major impact on people understanding this, as per usual. but still, something about it just feels heavy. the idea of having two separate tools, each of which does one thing well, working together to solve a problem suits my sensibilities more than having one more monolithic tool. luckily, we don't really have to convince each other of anything, here. it's entirely possible to have zc.buildout recipes for installing Zope and Plone as well as buildit / workingenv recipes for the same purpose. -r ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martijn Faassen wrote: Hey, Just some feedback in case you haven't gotten this already: I have problem running bin/buildout as it seems to fail getting workingenv.py from the cheeseshop: zc.buildout.easy_install: Getting new distribution for workingenv.py=0.3 Page at http://www.python.org/pypi/workingenv.py/ links to .py file(s) without version info; an index scan is required. error: Couldn't find a setup script in /tmp/tmpDRKwglget_dist/workingenv.py Traceback (most recent call last): Ah I had that problem with buildout as well. I could install it using my global easy_install though. Probably we need to append some '#egg=project-version' here according to http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi as workingenv is distributed as a single Python script rather than a package. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martin Aspeli wrote: I just emailed Hanno a few questions, but I thought I'd post them here as well for further discussion: - Is it so that I should check out ploneout, run boostrap.py, then ../bin/buildout.sh for each project? Or can I somehow use the same checkout of ploneout for multiple projects? This works against the way buildout works. You could create a seperate instance (just copy the instance part in the buildout.cfg and rename it to instance2) and specify a different list of products or eggs that instance should include. What we really would want here instead, is being able to reuse the same Zope checkout or Zope installation in general and potentially also the bundle checkouts. For the later part this is already possible, as the instance recipes takes any local folder and symlinks the products found in there into the instance. - How does that offline I-already-have-a-bundle-checkout-thank-you thing work? It checks whether or not the offline option is either set for the part or globally. If one is set and the part directory is in place, the recipe won't do anything. - I notice that the recipie puts slugs into etc/package-includes; I don't think they hurt (once we fix KSS, which ree is doing right now!), but they're not really necessary as far as Plone is concerned because Products/CMFPlone pulls them in. I'm guessing we should make it possible to disable this behaviour? The slugs are not necessary, I just put them in there to show how the Zope3instance recipe handles these things and show us an alternative way of managing those. - I had to easy_install elementtree for openid to be happy; guess we should add that as a dependency somewhere? Yep, as noted in another mail, even Archetypes depends on it. - In my workingenv (i.e. when I've done source bin/activate) I had some trouble using 'paster', because it couldn't find various eggs, e.g. ZopeSkel, Paste, PasteScript, PasteDeploy and Cheetah. What's annoying is that I had to do each one of these manually, i.e. I ran 'easy_install ZopeSkel' and it found the egg (without any trip to the cheese shop), but then it complained about PasteScript, so I did 'easy_install PasteScript', but then it complained about PasteDeploy etc. This is while I'm running 'paster create'. Did I do something wrong? I think I read somewhere that workingenv can re-use my system PYTHONPATH which may or may not be useful I did not include the global Python site packages, we might want to change that as Daniel pointed out. I have no idea why easy_install behaves in those strange ways though. - Once I got it up, I used 'paster create -t plone' to create a new project in parts/instance/src/my.package. Is this good practice? Nope, as Martijn and Daniel already pointed out, don't ever do anything manually in parts. Everything in there should be managed by recipes and buildout.cfg. Put those things in the ploneout/src folder instead and include them from there (probably using buildout's develop option as said in another mail as well). - I did 'python setup.py develop' in my new package to create a development egg. I could then run tests with './bin/zopectl test -m my.package --test-path src/my.package'. Is this good practice? We really need to get the testrunner recipe back into ploneout (I tried it, but ended up having problems finding the right combination of paths to be able to run tests). This should give you a test.py in ploneout/bin which you could use to run all tests and it should include all needed paths in order for you not having to specify the test path yourself. Look at http://dev.plone.org/plone/browser/ploneout/trunk/buildout.cfg?rev=11950 for my last attempt of defining this test script. Something similar should probably work for all eggs that are managed by buildout. But this of course doesn't play nicely with workingenv. - Am I supposed to add my own eggs (even if they're not in svn anywhere) to the buildout for my project? If they are not in SVN they should have a release somewhere ;) If you are in an early stage you could put them into ploneout/src or directly under ploneout. In the end both the buildout way as well as my workingenv recipe only need a local path to include them, so you could manage them outside ploneout completely as well. - How am I meant to update my plone bundles? I tried running ../bin/buildout -v again, but that wiped my parts/instance directory! It came back, but lacking things like Products/ ... That's odd. I had no troubles running ./bin/buildout again. This is what currently should update all Plone specific bundles. Even if the instance would be recreated it should contain all those products :( Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
I only know the answer for a few of these... Martin Aspeli wrote: - In my workingenv (i.e. when I've done source bin/activate) I had some trouble using 'paster', because it couldn't find various eggs, e.g. ZopeSkel, Paste, PasteScript, PasteDeploy and Cheetah. What's annoying is that I had to do each one of these manually, i.e. I ran 'easy_install ZopeSkel' and it found the egg (without any trip to the cheese shop), but then it complained about PasteScript, so I did 'easy_install PasteScript', but then it complained about PasteDeploy etc. This is while I'm running 'paster create'. Did I do something wrong? I think I read somewhere that workingenv can re-use my system PYTHONPATH which may or may not be useful workingenv has the --site-packages option which includes all site packages available at install time. I'm not sure what's going on in your case though, because those dependencies should certainly be tracked down normally, no? - Once I got it up, I used 'paster create -t plone' to create a new project in parts/instance/src/my.package. Is this good practice? I believe you're not supposed to put anything into parts/. parts/ is buildout's playground. - How am I meant to update my plone bundles? I tried running ./bin/buildout -v again, but that wiped my parts/instance directory! It came back, but lacking things like Products/ ... Welcome to the wonderful world of buildout. *scnr* ;-) Daniel ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martijn Faassen wrote: Hey, I only caught this message earlier today, but this is really cool! It's really nice to see some zope 2 recipes and I hope they indeed will end up on svn.zope.org soon! Your workingenv recipe sounds very interesting and I should try this soon. Does it allow easy_install to be used as well? If so, what does that mean (do paths get updated?) I believe so. I'll be playing with these recipes first excuse I get. :) I just emailed Hanno a few questions, but I thought I'd post them here as well for further discussion: - Is it so that I should check out ploneout, run boostrap.py, then ./bin/buildout.sh for each project? Or can I somehow use the same checkout of ploneout for multiple projects? - How does that offline I-already-have-a-bundle-checkout-thank-you thing work? - I notice that the recipie puts slugs into etc/package-includes; I don't think they hurt (once we fix KSS, which ree is doing right now!), but they're not really necessary as far as Plone is concerned because Products/CMFPlone pulls them in. I'm guessing we should make it possible to disable this behaviour? - I had to easy_install elementtree for openid to be happy; guess we should add that as a dependency somewhere? - In my workingenv (i.e. when I've done source bin/activate) I had some trouble using 'paster', because it couldn't find various eggs, e.g. ZopeSkel, Paste, PasteScript, PasteDeploy and Cheetah. What's annoying is that I had to do each one of these manually, i.e. I ran 'easy_install ZopeSkel' and it found the egg (without any trip to the cheese shop), but then it complained about PasteScript, so I did 'easy_install PasteScript', but then it complained about PasteDeploy etc. This is while I'm running 'paster create'. Did I do something wrong? I think I read somewhere that workingenv can re-use my system PYTHONPATH which may or may not be useful - Once I got it up, I used 'paster create -t plone' to create a new project in parts/instance/src/my.package. Is this good practice? - I did 'python setup.py develop' in my new package to create a development egg. I could then run tests with './bin/zopectl test -m my.package --test-path src/my.package'. Is this good practice? - Am I supposed to add my own eggs (even if they're not in svn anywhere) to the buildout for my project? - How am I meant to update my plone bundles? I tried running ./bin/buildout -v again, but that wiped my parts/instance directory! It came back, but lacking things like Products/ ... Martin ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Hi, Martijn Faassen wrote: Hey, I only caught this message earlier today, but this is really cool! It's really nice to see some zope 2 recipes and I hope they indeed will end up on svn.zope.org soon! I have sent my contributer agreement per snail-mail last week to Zope Corp. so it might turn up on somebody's desk in the next days. Then I'll move those recipes... Your workingenv recipe sounds very interesting and I should try this soon. Does it allow easy_install to be used as well? If so, what does that mean (do paths get updated?) All it does is to call workingenv itself on the created Zope instance (any folder really). You get the usual activate script you need to run to enable the workingenv and let easy_install install into the workingenv. If this should turn out to be a good way we would want to activate the workingenv in some scripts automatically, like zopectl for instance. In addition to supporting workingenv's requirements option (a file with a list of dependencies) I also implemented a simple option to let it call setup.py develop on a number of specified packages, so these get installed into the workingenv as well. Note however that I did not understand how to effectively use buildout's system of eggs and develop-eggs, so this might be a duplicated effort. My use case here was to get 19 Python packages from SVN (which aren't released in any way) and use those as a SVN checkout, but include them into the Zope instance in development mode, so I can still work with my SVN checkout as I do now, but have those packages available in the instance with all their egg information like entrypoints. I'll be playing with these recipes first excuse I get. :) That would be great. Those recipes need a definitive review by someone who knows more about buildout. I felt like I was fighting against buildout in some places. All I wanted to get is a running Plone development environment which includes real eggs in development mode, rather then just dumping in the packages without any egg information like we currently do. These recipes are currently more of a (working) prototype right now, to let us explore buildouts capabilities in a real complex situation and to get some interested (Plone) folks to try it and provide feedback. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like
Martijn Faassen wrote: Hey, I only caught this message earlier today, but this is really cool! It's really nice to see some zope 2 recipes and I hope they indeed will end up on svn.zope.org soon! Your workingenv recipe sounds very interesting and I should try this soon. Does it allow easy_install to be used as well? If so, what does that mean (do paths get updated?) I believe so. I'll be playing with these recipes first excuse I get. :) I just emailed Hanno a few questions, but I thought I'd post them here as well for further discussion: - Is it so that I should check out ploneout, run boostrap.py, then ./bin/buildout.sh for each project? Or can I somehow use the same checkout of ploneout for multiple projects? - How does that offline I-already-have-a-bundle-checkout-thank-you thing work? - I notice that the recipie puts slugs into etc/package-includes; I don't think they hurt (once we fix KSS, which ree is doing right now!), but they're not really necessary as far as Plone is concerned because Products/CMFPlone pulls them in. I'm guessing we should make it possible to disable this behaviour? - I had to easy_install elementtree for openid to be happy; guess we should add that as a dependency somewhere? - In my workingenv (i.e. when I've done source bin/activate) I had some trouble using 'paster', because it couldn't find various eggs, e.g. ZopeSkel, Paste, PasteScript, PasteDeploy and Cheetah. What's annoying is that I had to do each one of these manually, i.e. I ran 'easy_install ZopeSkel' and it found the egg (without any trip to the cheese shop), but then it complained about PasteScript, so I did 'easy_install PasteScript', but then it complained about PasteDeploy etc. This is while I'm running 'paster create'. Did I do something wrong? I think I read somewhere that workingenv can re-use my system PYTHONPATH which may or may not be useful - Once I got it up, I used 'paster create -t plone' to create a new project in parts/instance/src/my.package. Is this good practice? - I did 'python setup.py develop' in my new package to create a development egg. I could then run tests with './bin/zopectl test -m my.package --test-path src/my.package'. Is this good practice? - Am I supposed to add my own eggs (even if they're not in svn anywhere) to the buildout for my project? - How am I meant to update my plone bundles? I tried running ./bin/buildout -v again, but that wiped my parts/instance directory! It came back, but lacking things like Products/ ... Martin ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
