[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
Hi Tres! Tres Seaver wrote: yuppie wrote: So the use cases for these DeltaProfiles are very limited. Using XSLT would allow us to unify DeltaProfiles and ExtensionProfiles, providing an automated way for creating ExtensionProfiles. The major use case for deltas (no matter whether they do XSLT transforms or apply patches) is to permit re-applying local changes after an upgrade. They aren't likely to be satisfactory as a mode for distributing add-ons, I think. I agree that creating ExtensionProfiles for add-ons will never be a completely automated process. But it would be easier to polish a generated profile than writing ExtensionProfiles from scratch. More important is that ExtensionProfiles can be applied to a profile, not just to a site. If that is not the case your use case (re-applying local changes after an upgrade) will not work. I hope this example illustrates the problem: Step 1: A Site is created using a base profile and some extensions. Step 2: The site is customized TTW. Step 3: An additional ExtensionProfile is applied. Step 4: The site is further customized TTW. Step 5: Now we want to upgrade the used profiles. How would you create the necessary DeltaProfile? AFAICS this only works if we are able to create an uncustomized version of the profile combination used in that site. At no point you can create this uncustomized profile with a snapshot. And right now the only tool we have for combing profiles is the site itself. So we either have to implement a machinery that allows to apply the existing ExtensionProfile format to base profiles directly or we have to switch to a new format for ExtensionProfiles. That new format should use a generic format like XSLT or diff. Diffs are not made for manual editing and break easily if the base profiles are changed / updated. I would definitely like to break the requirement that extensions have to displace (even temporarily) the tool's import profile. I would also like to work out how we might safely revert the application of an extension, as well as tracking dependencies among them. I am *not* sanguine about using XSLT as the primary mechanism for describing an extension. While I still believe XSLT would be the most powerful solution it is not a necessary part of my plan. My initial posting described an alternative approach: If we split up the profile files in small pieces we can use layers and just override the pieces we want to change. This is not as fine grained as diffs or XSLT, but in exchange much simpler. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
Regarding CMFQuickInstaller, I wanted to remind the list that CPS has subclassed portal_setup to add a number of functionalities related to upgrades (but not really uninstall): cps:upgradeStep. I also revamped part of the ZMI to provide easier ways to apply one import step for one profile (in non-purge mode). People are welcome to take that code and fold it back into CMF. (I'm very pressed by internal project at the moment so won't do it myself.) Installing a CPS 3.4.1 site to check the features out wouldn't take you much time :) Florent yuppie wrote: GenericSetup uses a completely different approach than CMFQuickInstaller. It is focused on states, not on changes. The early versions of GenericSetup (FKA CMFSetup) didn't even have extension profiles and importVarious handlers. I invented those hacks to give GenericSetup more momentum, but especially importVarious was never meant as a permanent solution. The GenericSetup UI is counter-intuitive because it was built for complete profiles/snapshots. importVarious is dangerous because it is a hack. I know that so far GenericSetup can't replace CMFQuickInstaller. But the fact people are missing CMFQuickInstaller functionality doesn't mean it has to be implemented following the old patterns. Install/uninstall code for CMFQuickInstaller is hard to write, usually only add-on programmers do that. Just replacing python code by XML-files will not make it much easier. And now regarding your concerns: - I don't think we should use the same machinery for configuration data and content. (I know the distinction is fuzzy, but the big mass is pure content.) AFAICS it is not very hard to specify areas that contain content and should not be touched if profiles are reapplied. - The procedure I have in mind depends on the ability to create customization snapshots. As a first step the setup tool would create this snapshot. In the next step it would combine all dependencies of that snapshot minus uninstalled extensions plus new extensions. The result is a profile that contains the latest data of all selected profiles plus the customizations from the snapshot. This profile would be used to reset the site. - Speed is not that important. It doesn't hurt if it takes a few seconds until a profile is installed or uninstalled. There are a few very expensive tasks like creating indexes. The handler can make sure indexes are not removed if the new profile needs the same index. - I don't know if we really need a way to reset specific products. AFAICT the more common use case is to reset specific objects like tools. I'd prefer a tab on the object that allows to load preconfigured settings instead of using the setup tool for that. - While I hope we can get rid of importVarious it would be less dangerous to use if only complete profiles are applied to the site. Cheers, Yuppie ___ Zope-CMF maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
Hi Tres! Tres Seaver wrote: I'll note that my original architecture document[1] contemplated two kinds of add-on profiles: - ExtensionProfiles could register *new* kinds of steps, as well as making non-destructive insertions to the configuration created by a baseline profile. - DeltaProfiles essentially captured line-level diffs to a baseline profile. I think there is room for both, where we get away from the need to make EPs the current profile in order to use them. DeltaProfiles could be applied by patching a snapshot, and then installing the snapshot as a new baseline. Because not all the files being patched are XML, I don't think we can or should rely on XSLT: diff might be enough. I don't know if we have the resources to implement XSLT diffs and of course XSLT makes only sense for XML (we can still use diff for other files). But for XML XSLT has big advantages over normal diffs: - normal diffs for XML files are often very hard to read and edit - small changes in the XML file often make it impossible to apply a normal diff So the use cases for these DeltaProfiles are very limited. Using XSLT would allow us to unify DeltaProfiles and ExtensionProfiles, providing an automated way for creating ExtensionProfiles. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 yuppie wrote: Hi Tres! Tres Seaver wrote: I'll note that my original architecture document[1] contemplated two kinds of add-on profiles: - ExtensionProfiles could register *new* kinds of steps, as well as making non-destructive insertions to the configuration created by a baseline profile. - DeltaProfiles essentially captured line-level diffs to a baseline profile. I think there is room for both, where we get away from the need to make EPs the current profile in order to use them. DeltaProfiles could be applied by patching a snapshot, and then installing the snapshot as a new baseline. Because not all the files being patched are XML, I don't think we can or should rely on XSLT: diff might be enough. I don't know if we have the resources to implement XSLT diffs and of course XSLT makes only sense for XML (we can still use diff for other files). But for XML XSLT has big advantages over normal diffs: - normal diffs for XML files are often very hard to read and edit Any diff is hard to edit, so much so that I never do more than delete entire unwanted regions. The problem of using diff against XML is only mitigated by the fact that we go to a fair amount of trouble to generate the XML files in such a way as to minimize the diffs. I would be OK with some kind of optional dependency on something like ElementTree for applying XSLT-based changes. I am *not* OK with having to maintain any such transforms myself. - small changes in the XML file often make it impossible to apply a normal diff Hand-editing profile XML files is tricky, anyway, at least for the ones stored in version control. I always do a round-trip after the edit to canonicalize the XML in the profile. So the use cases for these DeltaProfiles are very limited. Using XSLT would allow us to unify DeltaProfiles and ExtensionProfiles, providing an automated way for creating ExtensionProfiles. The major use case for deltas (no matter whether they do XSLT transforms or apply patches) is to permit re-applying local changes after an upgrade. They aren't likely to be satisfactory as a mode for distributing add-ons, I think. I would definitely like to break the requirement that extensions have to displace (even temporarily) the tool's import profile. I would also like to work out how we might safely revert the application of an extension, as well as tracking dependencies among them. I am *not* sanguine about using XSLT as the primary mechanism for describing an extension. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEyNPB+gerLs4ltQ4RApfxAJ4yQNxmj4sgczUUbAr+xXsrpeemLgCeMrlj NLIF7Qm6NYHtQVAeh36tTz8= =1NNR -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
Hi Lennart! Lennart Regebro wrote: I don't exactly understand what makes a customization snapshot different from an extension profile, You can't understand what is different because there is no difference ;) except that you note what profile it is diffed from. Even this small difference doesn't exist. The profile it is diffed from is just a normal dependency. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lennart Regebro wrote: I don't exactly understand what makes a customization snapshot different from an extension profile, except that you note what profile it is diffed from. Other than that, I think this proposal sounds good. I'll note that my original architecture document[1] contemplated two kinds of add-on profiles: - ExtensionProfiles could register *new* kinds of steps, as well as making non-destructive insertions to the configuration created by a baseline profile. - DeltaProfiles essentially captured line-level diffs to a baseline profile. I think there is room for both, where we get away from the need to make EPs the current profile in order to use them. DeltaProfiles could be applied by patching a snapshot, and then installing the snapshot as a new baseline. Because not all the files being patched are XML, I don't think we can or should rely on XSLT: diff might be enough. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEx8uQ+gerLs4ltQ4RAitUAJsFILLQ56ricDH0qSd5TpwaUF0RBgCdFLhX roGXOpMmDUAesd8Kt1VvuHg= =iDkb -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
yuppie wrote: Since CMF 1.5.1 CMFSetup/GenericSetup supports extension profiles. They are quite useful, but their current format has some fundamental issues: I posted something in a similar vein to this on plone-developers last night, about using GenericSetup profiles for product installation and un-installation. The basic premise is that people install Plone (which has a base profile) and the install and un-install products, trying them out. They do this on sites they intend to keep (even if they're just test instance) and may not want to re-create the site from scratch (including content!) each time they decide they don't like some simple product. See http://www.nabble.com/Installing-products-using-GenericSetup-tf1995512.html 1.) Extension profiles are hard to create and maintain +1 - I've just been learning to do this and intend to document it, but there are some conceptual barriers if nothing else. The thing that gets to me is that when you want to apply an extension profile (i.e. get the extra functionality it installs) you click run all import steps after applying the profile. It seems very counter-intuitive (and potentially dangerous) to re-run e.g. all workflow installation for all products when I only wanted the new workflows provided by my new product that registered that new extension profile. You need a lot of knowledge about the import handlers. Only if you know how their non-purging mode is implemented and which update directives are available you can write good extension profiles. There are no tools that help you to create extension profiles. Bearing in mind I don't understand purging and non-purging modes, take that as a yes :) 2.) Sites using extension profiles are hard to create and maintain You have to know which profiles depend on other profiles. This was another problem I hit. It seemed to me profiles were being applied alphabetically by the Plone ZMI setup form, but who knows ;-) It's hard to control the order in which profiles are applied. The Comparison tab becomes useless if the site uses several extensions that are not part of the 'initial_configuration' snapshot. (Sometimes I create a new site to get a clean snapshot of the applied profiles and delete that site again. A quite expensive procedure.) A snapshot contains no information which original profiles it 'contains'. The tool has no control on the applied profiles. There is no way to uninstall profiles. This is the other big one - we can't reasonably tell people to make extension profiles for their add-on products if users won't be able to remove them again cleanly. The irony is that the information contained in the GS XML files should make it way easier to figure out how to do clean un-installs compared to the Extensions.Install.uninstall() method idiom. And now forget everything you know about extension profiles. Imagine they were like this: Extension profiles describe how to transform base profiles. They know which profiles they can transform (AKA dependencies). They are applied to an other profile, *not* to the site. That sounds sensible. Only complete profiles (base profiles or extended base profiles) are applied to the site. The handlers only need to understand complete configuration files. That sounds easier, but also a bit scary. If I want to just install some product after a site has been populated with content, applying a complete profile implies that it may stomp on changes I've made manually afterwards (people click around the ZMI after reading some how-to all the time). And what about being able to re-install (i.e. reset to original state as defined by the extension profile) and un-install (remove/reset the state that existed before the profile) a single product only? The setup tool knows how to merge profiles without modifying the site. You can compare your snapshot with any valid combination of profiles you like. There might even be a function that searches for the profile combination that has the smallest diff to your snapshot and finds this way the best dependencies list. And now imagine we have a tool that can *create* extension profiles based on the current customizations. This would allow to provide many new features: - customization snapshots: Snapshots would just contain the customizations and a list of dependencies, not a complete base profile. - install/uninstall: Basically any setup change is applied by recreating the site from scratch. At least for some expensive to create items like indexes we have to make sure we don't delete/recreate them if they were not modified. Uninstall would be no different procedure, we just don't reapply a specific profile. This is the bit that worries me. Recreating the site from scratch sounds like it'd be a nightmare. What if some bug causes portal_setup to effectively wipe people's sites or people's settings? What about the cases where the XML vocabulary known to
Re: [Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
yuppie-2 wrote: GenericSetup uses a completely different approach than CMFQuickInstaller. It is focused on states, not on changes. Indeed. From having used it recently, it just seems to be an easier way of working, so I'm trying to find out how we can meet the use cases that CMFQI meets now based on this technology - if at all. The early versions of GenericSetup (FKA CMFSetup) didn't even have extension profiles and importVarious handlers. I invented those hacks to give GenericSetup more momentum, but especially importVarious was never meant as a permanent solution. The GenericSetup UI is counter-intuitive because it was built for complete profiles/snapshots. importVarious is dangerous because it is a hack. That was my feeling too. I can see that it will be a necessary hack, though, until we've covered everyhing anyone ever wanted to do with some import/export handler. I know that so far GenericSetup can't replace CMFQuickInstaller. But the fact people are missing CMFQuickInstaller functionality doesn't mean it has to be implemented following the old patterns. /quiote No, definitely. What I was trying to get back to was the original use cases for CMFQI - people download some product, want to try it out, install it, decide they don't like it, uninstall it, or re-install it for an upgrade, or leave it in place. Incidentally, one of the biggest problem peoplems people have with Plone is that they configure a site a certain way (including third party products) and then want to deploy that to a production server, leaving content intact. GS is obviously the answer to that, so I'm keen on trying to promote third party products that integrate with it as well as possible. Install/uninstall code for CMFQuickInstaller is hard to write, usually only add-on programmers do that. Just replacing python code by XML-files will not make it much easier. But 90% of what we do in an installer is boilerplate - add a workflow, register a new content type, install a skin - all stuff that's done easier with GS than with python APIs. And now regarding your concerns: - I don't think we should use the same machinery for configuration data and content. (I know the distinction is fuzzy, but the big mass is pure content.) AFAICS it is not very hard to specify areas that contain content and should not be touched if profiles are reapplied. +1 - The procedure I have in mind depends on the ability to create customization snapshots. As a first step the setup tool would create this snapshot. In the next step it would combine all dependencies of that snapshot minus uninstalled extensions plus new extensions. The result is a profile that contains the latest data of all selected profiles plus the customizations from the snapshot. This profile would be used to reset the site. Bearing in mind that my understanding is not completely solid, the option I keep thinking we need is to say run all import steps, but only for this one extension profile. I have a workflows.xml and a types.xml and whatever else in my extension profile for my product. I'd like to uninstall or re-install or install workflows and types *for my product*, but I'm not interested in evaluating every workflows.xml and types.xml in every active profile everywhere (e.g. those that came with Plone or CMF or some other product), lest they'd changed or I'd changed them and didn't want my changes stomped on. Even though the actual import step is defined in another product (e.g. CMFCore) I assume it'd be possible to simply not include steps for other profiles? - Speed is not that important. It doesn't hurt if it takes a few seconds until a profile is installed or uninstalled. There are a few very expensive tasks like creating indexes. The handler can make sure indexes are not removed if the new profile needs the same index. Probably true. On plone.org we've had cases where things result in reindexes that take 45 minutes or more to reindex, causing Squid timeouts and other nastiness. We just to be a bit careful, I guess. - I don't know if we really need a way to reset specific products. AFAICT the more common use case is to reset specific objects like tools. I'd prefer a tab on the object that allows to load preconfigured settings instead of using the setup tool for that. This doesn't resonate with my experience from Plone, if I'm understanding you correctly. I agree that some way of resetting individual tools may be useful, but that's still very low level. If Joe Average downloads CMFCrazyPollingProduct, he may install it, and need to un-install it cleanly. He may also need to re-set it if he messed up its configuration or maybe he's the developer and wants to test the effects of a clean re-install. Or, if the product is updated, he may need to re-run its installation. - While I hope we can get rid of importVarious it would be less dangerous to use if only complete profiles are
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
Hi Martin! Martin Aspeli wrote: - The procedure I have in mind depends on the ability to create customization snapshots. As a first step the setup tool would create this snapshot. In the next step it would combine all dependencies of that snapshot minus uninstalled extensions plus new extensions. The result is a profile that contains the latest data of all selected profiles plus the customizations from the snapshot. This profile would be used to reset the site. Bearing in mind that my understanding is not completely solid, the option I keep thinking we need is to say run all import steps, but only for this one extension profile. I have a workflows.xml and a types.xml and whatever else in my extension profile for my product. I'd like to uninstall or re-install or install workflows and types *for my product*, but I'm not interested in evaluating every workflows.xml and types.xml in every active profile everywhere (e.g. those that came with Plone or CMF or some other product), lest they'd changed or I'd changed them and didn't want my changes stomped on. Even though the actual import step is defined in another product (e.g. CMFCore) I assume it'd be possible to simply not include steps for other profiles? The concept of import steps is confusing because it was invented before extension profile support was added. Originally an import step represented a piece of the profile. The handler property just provided the information which handler to use for this piece of the profile. That did no longer work with extension profiles. Now an import step just registers an handler. Only the last selected profile is active. Normal import steps depend on configuration files and do nothing if the active profile does not contain related files. importVarious steps have to simulate that behavior by testing if they are already applied. If the import handlers follow those rules (CMF handlers do) only the profile *for your product* is applied. Running all steps just means that all handlers try to find related files in the active profile and apply them *if* they exist. So your example should work right now. My proposal would mean that everything is reset, but without loosing customizations. So while the procedure would be much more expensive the result should be similar. - I don't know if we really need a way to reset specific products. AFAICT the more common use case is to reset specific objects like tools. I'd prefer a tab on the object that allows to load preconfigured settings instead of using the setup tool for that. This doesn't resonate with my experience from Plone, if I'm understanding you correctly. I agree that some way of resetting individual tools may be useful, but that's still very low level. If Joe Average downloads CMFCrazyPollingProduct, he may install it, and need to un-install it cleanly. This is the normal uninstall procedure I described in my last mail. He may also need to re-set it if he messed up its configuration or maybe he's the developer and wants to test the effects of a clean re-install. Well. You can't always tell which customizations belong to which product. But after uninstalling a product customizations of related objects will fail if they no longer exist. So we if we uninstall and reinstall a product the effect should be similar to a reset. Or, if the product is updated, he may need to re-run its installation. Do you mean you want to switch to the latest version of a specific profile but not of all profiles? - While I hope we can get rid of importVarious it would be less dangerous to use if only complete profiles are applied to the site. I suppose, except if I'm trying to just install product X, running some (potentially destructive) importVarious from some other unrelated product still seems risky. It's still a hack. But I think we should focus on making writing handlers easier instead on making it easier to work around writing handlers. The changes I propose make it easier to write handlers. Writing handlers for PropertyManagers is already quite easy and generic support for Zope 3 schemas could be added. Perhaps its FUD, but essentially re-creating a site each time you do an incremental update (e.g. install a new product) seems really risky to me. A bad product install can sometimes mess up that product - if you started getting strange exceptions (and GS exceptions aren't always that easy to debug, I've found) from somewhere deep within an unrelated product, that could be pretty scary too. I don't share those concerns. I'm more concerned about artifacts left behind by imperfect uninstall routines. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
