[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
yuppie wrote: Since CMF 1.5.1 CMFSetup/GenericSetup supports extension profiles. They are quite useful, but their current format has some fundamental issues: I posted something in a similar vein to this on plone-developers last night, about using GenericSetup profiles for product installation and un-installation. The basic premise is that people install Plone (which has a base profile) and the install and un-install products, trying them out. They do this on sites they intend to keep (even if they're just test instance) and may not want to re-create the site from scratch (including content!) each time they decide they don't like some simple product. See http://www.nabble.com/Installing-products-using-GenericSetup-tf1995512.html 1.) Extension profiles are hard to create and maintain +1 - I've just been learning to do this and intend to document it, but there are some conceptual barriers if nothing else. The thing that gets to me is that when you want to apply an extension profile (i.e. get the extra functionality it installs) you click run all import steps after applying the profile. It seems very counter-intuitive (and potentially dangerous) to re-run e.g. all workflow installation for all products when I only wanted the new workflows provided by my new product that registered that new extension profile. You need a lot of knowledge about the import handlers. Only if you know how their non-purging mode is implemented and which update directives are available you can write good extension profiles. There are no tools that help you to create extension profiles. Bearing in mind I don't understand purging and non-purging modes, take that as a yes :) 2.) Sites using extension profiles are hard to create and maintain You have to know which profiles depend on other profiles. This was another problem I hit. It seemed to me profiles were being applied alphabetically by the Plone ZMI setup form, but who knows ;-) It's hard to control the order in which profiles are applied. The Comparison tab becomes useless if the site uses several extensions that are not part of the 'initial_configuration' snapshot. (Sometimes I create a new site to get a clean snapshot of the applied profiles and delete that site again. A quite expensive procedure.) A snapshot contains no information which original profiles it 'contains'. The tool has no control on the applied profiles. There is no way to uninstall profiles. This is the other big one - we can't reasonably tell people to make extension profiles for their add-on products if users won't be able to remove them again cleanly. The irony is that the information contained in the GS XML files should make it way easier to figure out how to do clean un-installs compared to the Extensions.Install.uninstall() method idiom. And now forget everything you know about extension profiles. Imagine they were like this: Extension profiles describe how to transform base profiles. They know which profiles they can transform (AKA dependencies). They are applied to an other profile, *not* to the site. That sounds sensible. Only complete profiles (base profiles or extended base profiles) are applied to the site. The handlers only need to understand complete configuration files. That sounds easier, but also a bit scary. If I want to just install some product after a site has been populated with content, applying a complete profile implies that it may stomp on changes I've made manually afterwards (people click around the ZMI after reading some how-to all the time). And what about being able to re-install (i.e. reset to original state as defined by the extension profile) and un-install (remove/reset the state that existed before the profile) a single product only? The setup tool knows how to merge profiles without modifying the site. You can compare your snapshot with any valid combination of profiles you like. There might even be a function that searches for the profile combination that has the smallest diff to your snapshot and finds this way the best dependencies list. And now imagine we have a tool that can *create* extension profiles based on the current customizations. This would allow to provide many new features: - customization snapshots: Snapshots would just contain the customizations and a list of dependencies, not a complete base profile. - install/uninstall: Basically any setup change is applied by recreating the site from scratch. At least for some expensive to create items like indexes we have to make sure we don't delete/recreate them if they were not modified. Uninstall would be no different procedure, we just don't reapply a specific profile. This is the bit that worries me. Recreating the site from scratch sounds like it'd be a nightmare. What if some bug causes portal_setup to effectively wipe people's sites or people's settings? What about the cases where the XML vocabulary known to
[Zope-CMF] CMF Collector: Open Issues
The following supporters have open issues assigned to them in this collector (http://www.zope.org/Collectors/CMF). Assigned and Open jens - CachingPolicyManager: Support OFS.Cache.CacheManager, [Accepted] http://www.zope.org/Collectors/CMF/408 mhammond - Windows DevelopmentMode penalty in CMFCore.DirectoryView, [Accepted] http://www.zope.org/Collectors/CMF/366 shh - CMFUid: cmf_uid attribute acquired when cataloging, [Accepted] http://www.zope.org/Collectors/CMF/446 Pending / Deferred Issues - Wrong cache association for FSObject, [Pending] http://www.zope.org/Collectors/CMF/255 - CMFSetup: Windows exports contain CR/LF, LF and even CR newlines, [Pending] http://www.zope.org/Collectors/CMF/266 - FSPropertiesObject.py cannot handle multiline input for lines, text attributes, [Deferred] http://www.zope.org/Collectors/CMF/271 - Can't invalidate skin items in a RAMCacheManager, [Pending] http://www.zope.org/Collectors/CMF/343 - CMFSetup: Workflow Tool export fails with workflows which have scripts, [Pending] http://www.zope.org/Collectors/CMF/373 - CMFCore.Skinnable.SkinnableObjectManager can merge skin data, [Pending] http://www.zope.org/Collectors/CMF/375 - Proxy Roles does't work for a Script using portal_catalog.searchResults, [Pending] http://www.zope.org/Collectors/CMF/380 - workflow notify success should be after reindex, [Pending] http://www.zope.org/Collectors/CMF/389 - 'except ...: pass' in CMFCore/TypesTool.py:_queryFactoryMethod() nullifies VerboseSecurity, [Pending] http://www.zope.org/Collectors/CMF/410 - Possible bug when using a BTreeFolder Member folder, [Pending] http://www.zope.org/Collectors/CMF/441 Pending / Deferred Features - Favorite.py: queries and anchors in remote_url, [Pending] http://www.zope.org/Collectors/CMF/26 - DefaultDublinCore should have Creator property, [Pending] http://www.zope.org/Collectors/CMF/61 - Document.py: universal newlines, [Pending] http://www.zope.org/Collectors/CMF/174 - Add condition for transition's action like other action, [Pending] http://www.zope.org/Collectors/CMF/207 - Major action enhancement, [Pending] http://www.zope.org/Collectors/CMF/232 - portal_type is undefined in initialization code, [Pending] http://www.zope.org/Collectors/CMF/248 - CMFTopic Does Not Cache, [Deferred] http://www.zope.org/Collectors/CMF/295 - Wishlist: a flag that tags the selected action., [Pending] http://www.zope.org/Collectors/CMF/301 - CMFDefault should make use of allowCreate(), [Pending] http://www.zope.org/Collectors/CMF/340 - Nested Skins, [Deferred] http://www.zope.org/Collectors/CMF/377 - CatalogVariableProvider code + tests, [Pending] http://www.zope.org/Collectors/CMF/378 - manage_doCustomize() : minor additions, [Pending] http://www.zope.org/Collectors/CMF/382 - CMF needs View-based TypeInformation, [Pending] http://www.zope.org/Collectors/CMF/437 - Marker attributes should be deprecated, [Pending] http://www.zope.org/Collectors/CMF/440 ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
yuppie-2 wrote: GenericSetup uses a completely different approach than CMFQuickInstaller. It is focused on states, not on changes. Indeed. From having used it recently, it just seems to be an easier way of working, so I'm trying to find out how we can meet the use cases that CMFQI meets now based on this technology - if at all. The early versions of GenericSetup (FKA CMFSetup) didn't even have extension profiles and importVarious handlers. I invented those hacks to give GenericSetup more momentum, but especially importVarious was never meant as a permanent solution. The GenericSetup UI is counter-intuitive because it was built for complete profiles/snapshots. importVarious is dangerous because it is a hack. That was my feeling too. I can see that it will be a necessary hack, though, until we've covered everyhing anyone ever wanted to do with some import/export handler. I know that so far GenericSetup can't replace CMFQuickInstaller. But the fact people are missing CMFQuickInstaller functionality doesn't mean it has to be implemented following the old patterns. /quiote No, definitely. What I was trying to get back to was the original use cases for CMFQI - people download some product, want to try it out, install it, decide they don't like it, uninstall it, or re-install it for an upgrade, or leave it in place. Incidentally, one of the biggest problem peoplems people have with Plone is that they configure a site a certain way (including third party products) and then want to deploy that to a production server, leaving content intact. GS is obviously the answer to that, so I'm keen on trying to promote third party products that integrate with it as well as possible. Install/uninstall code for CMFQuickInstaller is hard to write, usually only add-on programmers do that. Just replacing python code by XML-files will not make it much easier. But 90% of what we do in an installer is boilerplate - add a workflow, register a new content type, install a skin - all stuff that's done easier with GS than with python APIs. And now regarding your concerns: - I don't think we should use the same machinery for configuration data and content. (I know the distinction is fuzzy, but the big mass is pure content.) AFAICS it is not very hard to specify areas that contain content and should not be touched if profiles are reapplied. +1 - The procedure I have in mind depends on the ability to create customization snapshots. As a first step the setup tool would create this snapshot. In the next step it would combine all dependencies of that snapshot minus uninstalled extensions plus new extensions. The result is a profile that contains the latest data of all selected profiles plus the customizations from the snapshot. This profile would be used to reset the site. Bearing in mind that my understanding is not completely solid, the option I keep thinking we need is to say run all import steps, but only for this one extension profile. I have a workflows.xml and a types.xml and whatever else in my extension profile for my product. I'd like to uninstall or re-install or install workflows and types *for my product*, but I'm not interested in evaluating every workflows.xml and types.xml in every active profile everywhere (e.g. those that came with Plone or CMF or some other product), lest they'd changed or I'd changed them and didn't want my changes stomped on. Even though the actual import step is defined in another product (e.g. CMFCore) I assume it'd be possible to simply not include steps for other profiles? - Speed is not that important. It doesn't hurt if it takes a few seconds until a profile is installed or uninstalled. There are a few very expensive tasks like creating indexes. The handler can make sure indexes are not removed if the new profile needs the same index. Probably true. On plone.org we've had cases where things result in reindexes that take 45 minutes or more to reindex, causing Squid timeouts and other nastiness. We just to be a bit careful, I guess. - I don't know if we really need a way to reset specific products. AFAICT the more common use case is to reset specific objects like tools. I'd prefer a tab on the object that allows to load preconfigured settings instead of using the setup tool for that. This doesn't resonate with my experience from Plone, if I'm understanding you correctly. I agree that some way of resetting individual tools may be useful, but that's still very low level. If Joe Average downloads CMFCrazyPollingProduct, he may install it, and need to un-install it cleanly. He may also need to re-set it if he messed up its configuration or maybe he's the developer and wants to test the effects of a clean re-install. Or, if the product is updated, he may need to re-run its installation. - While I hope we can get rid of importVarious it would be less dangerous to use if only complete profiles are
Re: [Zope-CMF] [dev] RFC: rethinking GenericSetup extension profiles
yuppie-2 wrote: GenericSetup uses a completely different approach than CMFQuickInstaller. It is focused on states, not on changes. Indeed. From having used it recently, it just seems to be an easier way of working, so I'm trying to find out how we can meet the use cases that CMFQI meets now based on this technology - if at all. The early versions of GenericSetup (FKA CMFSetup) didn't even have extension profiles and importVarious handlers. I invented those hacks to give GenericSetup more momentum, but especially importVarious was never meant as a permanent solution. The GenericSetup UI is counter-intuitive because it was built for complete profiles/snapshots. importVarious is dangerous because it is a hack. That was my feeling too. I can see that it will be a necessary hack, though, until we've covered everyhing anyone ever wanted to do with some import/export handler. I know that so far GenericSetup can't replace CMFQuickInstaller. But the fact people are missing CMFQuickInstaller functionality doesn't mean it has to be implemented following the old patterns. /quiote No, definitely. What I was trying to get back to was the original use cases for CMFQI - people download some product, want to try it out, install it, decide they don't like it, uninstall it, or re-install it for an upgrade, or leave it in place. Incidentally, one of the biggest problem peoplems people have with Plone is that they configure a site a certain way (including third party products) and then want to deploy that to a production server, leaving content intact. GS is obviously the answer to that, so I'm keen on trying to promote third party products that integrate with it as well as possible. Install/uninstall code for CMFQuickInstaller is hard to write, usually only add-on programmers do that. Just replacing python code by XML-files will not make it much easier. But 90% of what we do in an installer is boilerplate - add a workflow, register a new content type, install a skin - all stuff that's done easier with GS than with python APIs. And now regarding your concerns: - I don't think we should use the same machinery for configuration data and content. (I know the distinction is fuzzy, but the big mass is pure content.) AFAICS it is not very hard to specify areas that contain content and should not be touched if profiles are reapplied. +1 - The procedure I have in mind depends on the ability to create customization snapshots. As a first step the setup tool would create this snapshot. In the next step it would combine all dependencies of that snapshot minus uninstalled extensions plus new extensions. The result is a profile that contains the latest data of all selected profiles plus the customizations from the snapshot. This profile would be used to reset the site. Bearing in mind that my understanding is not completely solid, the option I keep thinking we need is to say run all import steps, but only for this one extension profile. I have a workflows.xml and a types.xml and whatever else in my extension profile for my product. I'd like to uninstall or re-install or install workflows and types *for my product*, but I'm not interested in evaluating every workflows.xml and types.xml in every active profile everywhere (e.g. those that came with Plone or CMF or some other product), lest they'd changed or I'd changed them and didn't want my changes stomped on. Even though the actual import step is defined in another product (e.g. CMFCore) I assume it'd be possible to simply not include steps for other profiles? - Speed is not that important. It doesn't hurt if it takes a few seconds until a profile is installed or uninstalled. There are a few very expensive tasks like creating indexes. The handler can make sure indexes are not removed if the new profile needs the same index. Probably true. On plone.org we've had cases where things result in reindexes that take 45 minutes or more to reindex, causing Squid timeouts and other nastiness. We just to be a bit careful, I guess. - I don't know if we really need a way to reset specific products. AFAICT the more common use case is to reset specific objects like tools. I'd prefer a tab on the object that allows to load preconfigured settings instead of using the setup tool for that. This doesn't resonate with my experience from Plone, if I'm understanding you correctly. I agree that some way of resetting individual tools may be useful, but that's still very low level. If Joe Average downloads CMFCrazyPollingProduct, he may install it, and need to un-install it cleanly. He may also need to re-set it if he messed up its configuration or maybe he's the developer and wants to test the effects of a clean re-install. Or, if the product is updated, he may need to re-run its installation. - While I hope we can get rid of importVarious it would be less dangerous to use if only complete profiles are
[Zope-CMF] Re: [dev] RFC: rethinking GenericSetup extension profiles
Hi Martin! Martin Aspeli wrote: - The procedure I have in mind depends on the ability to create customization snapshots. As a first step the setup tool would create this snapshot. In the next step it would combine all dependencies of that snapshot minus uninstalled extensions plus new extensions. The result is a profile that contains the latest data of all selected profiles plus the customizations from the snapshot. This profile would be used to reset the site. Bearing in mind that my understanding is not completely solid, the option I keep thinking we need is to say run all import steps, but only for this one extension profile. I have a workflows.xml and a types.xml and whatever else in my extension profile for my product. I'd like to uninstall or re-install or install workflows and types *for my product*, but I'm not interested in evaluating every workflows.xml and types.xml in every active profile everywhere (e.g. those that came with Plone or CMF or some other product), lest they'd changed or I'd changed them and didn't want my changes stomped on. Even though the actual import step is defined in another product (e.g. CMFCore) I assume it'd be possible to simply not include steps for other profiles? The concept of import steps is confusing because it was invented before extension profile support was added. Originally an import step represented a piece of the profile. The handler property just provided the information which handler to use for this piece of the profile. That did no longer work with extension profiles. Now an import step just registers an handler. Only the last selected profile is active. Normal import steps depend on configuration files and do nothing if the active profile does not contain related files. importVarious steps have to simulate that behavior by testing if they are already applied. If the import handlers follow those rules (CMF handlers do) only the profile *for your product* is applied. Running all steps just means that all handlers try to find related files in the active profile and apply them *if* they exist. So your example should work right now. My proposal would mean that everything is reset, but without loosing customizations. So while the procedure would be much more expensive the result should be similar. - I don't know if we really need a way to reset specific products. AFAICT the more common use case is to reset specific objects like tools. I'd prefer a tab on the object that allows to load preconfigured settings instead of using the setup tool for that. This doesn't resonate with my experience from Plone, if I'm understanding you correctly. I agree that some way of resetting individual tools may be useful, but that's still very low level. If Joe Average downloads CMFCrazyPollingProduct, he may install it, and need to un-install it cleanly. This is the normal uninstall procedure I described in my last mail. He may also need to re-set it if he messed up its configuration or maybe he's the developer and wants to test the effects of a clean re-install. Well. You can't always tell which customizations belong to which product. But after uninstalling a product customizations of related objects will fail if they no longer exist. So we if we uninstall and reinstall a product the effect should be similar to a reset. Or, if the product is updated, he may need to re-run its installation. Do you mean you want to switch to the latest version of a specific profile but not of all profiles? - While I hope we can get rid of importVarious it would be less dangerous to use if only complete profiles are applied to the site. I suppose, except if I'm trying to just install product X, running some (potentially destructive) importVarious from some other unrelated product still seems risky. It's still a hack. But I think we should focus on making writing handlers easier instead on making it easier to work around writing handlers. The changes I propose make it easier to write handlers. Writing handlers for PropertyManagers is already quite easy and generic support for Zope 3 schemas could be added. Perhaps its FUD, but essentially re-creating a site each time you do an incremental update (e.g. install a new product) seems really risky to me. A bad product install can sometimes mess up that product - if you started getting strange exceptions (and GS exceptions aren't always that easy to debug, I've found) from somewhere deep within an unrelated product, that could be pretty scary too. I don't share those concerns. I'm more concerned about artifacts left behind by imperfect uninstall routines. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] [dev] RFC: rethinking GenericSetup extension profiles
The concept of import steps is confusing because it was invented before extension profile support was added. Originally an import step represented a piece of the profile. The handler property just provided the information which handler to use for this piece of the profile. That did no longer work with extension profiles. Now an import step just registers an handler. Only the last selected profile is active. Normal import steps depend on configuration files and do nothing if the active profile does not contain related files. importVarious steps have to simulate that behavior by testing if they are already applied. If the import handlers follow those rules (CMF handlers do) only the profile *for your product* is applied. Running all steps just means that all handlers try to find related files in the active profile and apply them *if* they exist. So your example should work right now. My proposal would mean that everything is reset, but without loosing customizations. So while the procedure would be much more expensive the result should be similar. Aha - so it seems was confused :) So, to be clear, run all import steps will only apply to the last selected active profile, even if that's an extension-profile? I guess we'd need to switch active profiles around from time to time, e.g. if we were selecting a profile for un-install or re-install. The point of not losting customisations is really important. However, I'd also like to have a way to force the profile application to override customisations, when you explicitly want a clean install. Or, if the product is updated, he may need to re-run its installation. Do you mean you want to switch to the latest version of a specific profile but not of all profiles? I guess. Even if that isn't too sensible, it may be necessary for example to control the order in which updates happen. It's still a hack. But I think we should focus on making writing handlers easier instead on making it easier to work around writing handlers. The changes I propose make it easier to write handlers. Writing handlers for PropertyManagers is already quite easy and generic support for Zope 3 schemas could be added. +1, so long as we acknowledge that sometimes it may be necessary to write some custom python code ala importVarious, if only due to time constraints. Writing a generic parser must always be harder than writing a line of code to configure something. :) Perhaps its FUD, but essentially re-creating a site each time you do an incremental update (e.g. install a new product) seems really risky to me. A bad product install can sometimes mess up that product - if you started getting strange exceptions (and GS exceptions aren't always that easy to debug, I've found) from somewhere deep within an unrelated product, that could be pretty scary too. I don't share those concerns. I'm more concerned about artifacts left behind by imperfect uninstall routines. I have those concerns too ;) Thanks for the update. So long as we keep the trying out a new product use case in mind and focus on making it as easy to make import handlers, I think we're very much on the same page. By the way - two other things I brought up in the post to plone-dev was: - We need some way of exposing to the user the *version* of the profile being applied, i.e. the version of the product. You have Foo version 1.0 installed, but version 2.0 on the filesystem. Do you want to upgrade? - We need some way of expressing dependencies between profiles, so that installing Foo also installs Bar. It should be possible to do optional dependencies, too, so that Foo installs Bar if Bar is available, but doesn't fail if it's not. Martin -- View this message in context: http://www.nabble.com/-dev--RFC%3A-rethinking-GenericSetup-extension-profiles-tf1996759.html#a5489196 Sent from the Zope - CMF list2 forum at Nabble.com. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
