[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Martin Aspeli wrote: Hi guys, After a lot of is-this-a-bug type discussions with Rob and Wichert, I've come to feel pretty strongly about the following: When you load an extension profile for the first time in GS, it looks to see if it has any import steps (in import_steps.xml) that are not already known. If so, it adds them to the import step registry and then runs all steps in the registry. Any time a profile (the same, or another one) is run subsequently, that import step will be run too. In Rob's words, the import steps and the profile that they came from (import_steps.xml-wise) get decoupled. We normally don't mind too much, because steps are run in the context of a particular profile, and if that context has no matching XML file, nothing is loaded. At least so we assume - there's of course nothing to stop the setuphandler from having side effects even if the XML file is not found. We *do* see a problem with importVarious-type hacks, though. Here, people resort to checking for a flag (normally an empty text file) to determine if the setuphandler should be run. I believe this is fundamentally Bad(tm) for three reasons: - People don't expect it to work that way. - Authors of setuphandlers have to be a lot more careful about having side effects on entirely alien packages. - It's impossible to predict which steps actually get run for an extension profile. Say, for example, that I wrote extension profile Y and I wanted to use an import handler used in extension profile X. If X had been run (or in fact, looked at under the Import tab, the registry filling happens as soon as you look at a profile there, even without running it) at least once, then it would get attempted for Y as well. If the appropriate XML or flag file was found it'd be run. But there is no way for Y to know that X was run (or looked at). In fact, it may be that users of Y do not want X to be loaded at all. The only predictable way for Y's author to run said import step is to explicitly list a setuphandler from X's package in Y's own import_steps.xml. I cannot see a good use case for the current behaviour, and it causes a lot of confusion (either why was this step run again? or why wasn't this step run?). Having to have flag files for pure-python setuphandlers is also pretty hacky (even if such handlers themselves are a bit hacky, but sometimes unavoidable). One example where the current behavior makes sense is when you write an add-on product which wants to install itself into a standard Plone site and change the settings of the Archetypes tool. The import handler for the Archetypes tool is only specified in the extension profile for Archetypes and not in the Plone base profile. It is however always installed into a Plone site via an import various step. I think this kind of decoupling the different profiles into the packages they came from is a sensible idea. In order for that to work reliable we need to have some profile dependency support, though. When we would have that I would extend your idea in the following way: The only sane behaviour, IMHO, is this: - When importing a base profile, only steps from the base profile are run. - When importing an extension profile, only steps from the *current* base profile and any additional steps explicitly defined in the *selected* extension profile (via an import_steps.xml file) are run. In both cases all the steps from any (recursively) defined dependencies are run as well. So the Plone Base profile would depend on the Archetypes extension profile (as well as the other half a dozen we install manually right now) and you could always assume the Archetypes tool import step to be present. For an add-on X it can decide to depend on Y and thus re-use all of their import steps as well, without having to specify the handler itself. No steps are ever implicitly pulled in from other extension profiles, and switching base profiles resets the base set of steps that are run for any extension profile. Another solution (better than the current behaviour, less good imho than the solution above) would be to load all steps from all profiles at Zope startup, when the profile registry is populated, so that all steps are always run if a package registering a new step is installed. This still means having lots of flag/file-checking and could be quite inefficient. This is really bad. That way if there is some experimental profile registered somewhere it would get loaded automatically. So as soon as I have one broken import step somewhere I would have no chance to exclude it. The registration of an import step should still be an explicit task. What do you think? Is there a sane use case for the current behaviour? Apart from the decoupled but guaranteed-to-be-there extension profiles I cannot think of any. Hanno ___ Zope-CMF maillist - Zope-CMF@lists.zope.org
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Hanno Schlichting wrote: One example where the current behavior makes sense is when you write an add-on product which wants to install itself into a standard Plone site and change the settings of the Archetypes tool. The import handler for the Archetypes tool is only specified in the extension profile for Archetypes and not in the Plone base profile. It is however always installed into a Plone site via an import various step. Mmm, well that's another problem: Installing one profile during installation of another also does not work reliably (I believe Rob was working on a generic dependency solution for this). I think this kind of decoupling the different profiles into the packages they came from is a sensible idea. In order for that to work reliable we need to have some profile dependency support, though. Right. You could extend my argument to include and those extension profiles explicitly marked as dependencies of the current base profile or the extension profile being installed. When we would have that I would extend your idea in the following way: The only sane behaviour, IMHO, is this: - When importing a base profile, only steps from the base profile are run. - When importing an extension profile, only steps from the *current* base profile and any additional steps explicitly defined in the *selected* extension profile (via an import_steps.xml file) are run. In both cases all the steps from any (recursively) defined dependencies are run as well. Yes. So the Plone Base profile would depend on the Archetypes extension profile (as well as the other half a dozen we install manually right now) and you could always assume the Archetypes tool import step to be present. For an add-on X it can decide to depend on Y and thus re-use all of their import steps as well, without having to specify the handler itself. No steps are ever implicitly pulled in from other extension profiles, and switching base profiles resets the base set of steps that are run for any extension profile. Another solution (better than the current behaviour, less good imho than the solution above) would be to load all steps from all profiles at Zope startup, when the profile registry is populated, so that all steps are always run if a package registering a new step is installed. This still means having lots of flag/file-checking and could be quite inefficient. This is really bad. That way if there is some experimental profile registered somewhere it would get loaded automatically. So as soon as I have one broken import step somewhere I would have no chance to exclude it. The registration of an import step should still be an explicit task. Yeah, good point. :) What do you think? Is there a sane use case for the current behaviour? Apart from the decoupled but guaranteed-to-be-there extension profiles I cannot think of any. Explicit being better than implicit and all that. :) Martin -- Acquisition is a jealous mistress ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
yuppie-3 wrote: Hi! The proposed solution doesn't work: The ImportStepRegistry is not only used for imports - it is also used for creating new 'import_steps.xml' files for exports. Exported profiles are always base profiles, they have to specify *all* import handlers used for creating the site. Right, that's fine. I think the current export functionality is right. However, the current *import* functionality for extension profiles is arbitrary and unpredictable. What if we stored in the import step registry: - The setuphandler to call - The base or extension profile it came from For an export operation, you just take the union of all the steps (de-duplicated, of course). For an import operation, you run: - All steps that came from the current base profile - All steps that came from explicit, transitive dependencies of the current base profile (provided we get support for declaring profile dependencies) - All steps explicitly defined in the extension profile being run - All steps explicitly defined in transitive dependencies of the extension profile being run (provided we get support for declaring profile dependencies) Transitive here means that if A depends on B and B depends on C, we run steps from A, B and C. I can see two ways to specify available import steps: 1.) 'import_steps.xml' is used as an index of available import steps. That's the way it was used before extension profile support was added to CMFSetup. It requires to maintain a lot of redundant information and a way to identify and remove duplicate steps for exports. Yes... this sounds like what I am talking about above? 2.) The files in the profile determine which import steps are available. This is the current behavior, but further changes would make this more clear: Getting rid of 'import_steps.xml', using a global handler registry instead. And adding new code that checks if a step is available without actually running it. I'm not sure how this gets any clearer... e.g. how does the author of an extension profile for an add-on product ensure that the steps he needs are in the registry? To me (and for a long time when I was using this blissfully unaware) extension profiles really are separate entities that will generally be applied in an uncontrolled order (unless we get explicit dependency support, which would be really great). Flag files (be they empty boolean switches or XML files with processing instructions) feel like a dirty way of dealing with this unpredictability. Martin -- View this message in context: http://www.nabble.com/-Zope-CMF--GenericSetup-extension-profile-import-step-mis-feature-tf4190394.html#a11922157 Sent from the Zope - CMF list1 mailing list archive at Nabble.com. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Previously Martin Aspeli wrote: yuppie-3 wrote: Hi! The proposed solution doesn't work: The ImportStepRegistry is not only used for imports - it is also used for creating new 'import_steps.xml' files for exports. Exported profiles are always base profiles, they have to specify *all* import handlers used for creating the site. Right, that's fine. I think the current export functionality is right. I think it's wrong :) The export functionality suffers from the same problen as the import functionality: the export steps defined in extension profiles are only used if they have been selected once. In other words: it is just as unpredictable. I like Yvo's idea of make import and export registration global instead of per-profile. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Hi! Martin Aspeli wrote: yuppie-3 wrote: The proposed solution doesn't work: The ImportStepRegistry is not only used for imports - it is also used for creating new 'import_steps.xml' files for exports. Exported profiles are always base profiles, they have to specify *all* import handlers used for creating the site. Right, that's fine. I think the current export functionality is right. However, the current *import* functionality for extension profiles is arbitrary and unpredictable. What if we stored in the import step registry: - The setuphandler to call - The base or extension profile it came from For an export operation, you just take the union of all the steps (de-duplicated, of course). For an import operation, you run: - All steps that came from the current base profile - All steps that came from explicit, transitive dependencies of the current base profile (provided we get support for declaring profile dependencies) ??? Base profiles have per definition no dependencies. No? - All steps explicitly defined in the extension profile being run - All steps explicitly defined in transitive dependencies of the extension profile being run (provided we get support for declaring profile dependencies) Transitive here means that if A depends on B and B depends on C, we run steps from A, B and C. I can see two ways to specify available import steps: 1.) 'import_steps.xml' is used as an index of available import steps. That's the way it was used before extension profile support was added to CMFSetup. It requires to maintain a lot of redundant information and a way to identify and remove duplicate steps for exports. Yes... this sounds like what I am talking about above? Not really. You are mixing both ways: Some import steps come from the registry (or are looked up in other profiles?), some from 'import_steps.xml'. 2.) The files in the profile determine which import steps are available. This is the current behavior, but further changes would make this more clear: Getting rid of 'import_steps.xml', using a global handler registry instead. And adding new code that checks if a step is available without actually running it. I'm not sure how this gets any clearer... e.g. how does the author of an extension profile for an add-on product ensure that the steps he needs are in the registry? The author of the add-on product is not responsible for this - just like he is not responsible for registering all adapters or utilities used by his product. The author of the setup handler should add a ZCML file that registers the setup handler. To me (and for a long time when I was using this blissfully unaware) extension profiles really are separate entities that will generally be applied in an uncontrolled order (unless we get explicit dependency support, which would be really great). Flag files (be they empty boolean switches or XML files with processing instructions) feel like a dirty way of dealing with this unpredictability. Why is it more explicit to have a 'flag' import step in 'import_steps.xml' or in 'import_steps.xml' of a dependency than a flag file in the current profile? Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Previously yuppie wrote: Base profiles have per definition no dependencies. No? It would be useful if they had. Or rather it would be useful if there was an include-option for them so you don't have to duplicate the configuration for products that you want to install directly from a base profile or optionally as an extension, depending on the specific deployment. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin Aspeli wrote: Hi guys, After a lot of is-this-a-bug type discussions with Rob and Wichert, I've come to feel pretty strongly about the following: When you load an extension profile for the first time in GS, it looks to see if it has any import steps (in import_steps.xml) that are not already known. If so, it adds them to the import step registry and then runs all steps in the registry. Any time a profile (the same, or another one) is run subsequently, that import step will be run too. In Rob's words, the import steps and the profile that they came from (import_steps.xml-wise) get decoupled. We normally don't mind too much, because steps are run in the context of a particular profile, and if that context has no matching XML file, nothing is loaded. At least so we assume - there's of course nothing to stop the setuphandler from having side effects even if the XML file is not found. We *do* see a problem with importVarious-type hacks, though. Here, people resort to checking for a flag (normally an empty text file) to determine if the setuphandler should be run. 'importVarious' is a brutal hack: better to focus efforts on making it disappear. The entire point of the tool is to externalize configuration as declarative data in the profile; accomodating imperative configuration is not something I care to support. The 'upgrade' stuff Rob ported in from CPS is the only sane case I know of for imperative action. I believe this is fundamentally Bad(tm) for three reasons: - People don't expect it to work that way. - Authors of setuphandlers have to be a lot more careful about having side effects on entirely alien packages. Authors of setup handlers shouls be *extremely* careful, and should not be running unless they have information in the profile to be applied. - It's impossible to predict which steps actually get run for an extension profile. Yes it is: *all* steps will get run, and will consult the profile to see whether they have configuration to apply. Say, for example, that I wrote extension profile Y and I wanted to use an import handler used in extension profile X. If X had been run (or in fact, looked at under the Import tab, the registry filling happens as soon as you look at a profile there, even without running it) at least once, then it would get attempted for Y as well. If the appropriate XML or flag file was found it'd be run. That is by design. Once an import step is grafted into the baseline, it is part of the site's configuration for ever. Resetting the baseline profile (a drastic step) is the only way to wipe it out. But there is no way for Y to know that X was run (or looked at). In fact, it may be that users of Y do not want X to be loaded at all. You'd better make this un-abstract: the presence of the enabling files in the profile being imported *is* the signal that those steps should be active. The only predictable way for Y's author to run said import step is to explicitly list a setuphandler from X's package in Y's own import_steps.xml. I cannot see a good use case for the current behaviour, and it causes a lot of confusion (either why was this step run again? or why wasn't this step run?). Having to have flag files for pure-python setuphandlers is also pretty hacky (even if such handlers themselves are a bit hacky, but sometimes unavoidable). I doubt they are unavoidable: rather, I think nobody has wanted to think about reworking them. Not the same thing atall. The only sane behaviour, IMHO, is this: - When importing a base profile, only steps from the base profile are run. Do you mean when setting a new baseline? Otherwise, -1: snapshots, for instance, are baseline profiles, and represent accumulated steps. - When importing an extension profile, only steps from the *current* base profile and any additional steps explicitly defined in the *selected* extension profile (via an import_steps.xml file) are run. No steps are ever implicitly pulled in from other extension profiles, and switching base profiles resets the base set of steps that are run for any extension profile. There is nothing implicit about this: those steps are part of the configured state of the site. If the profile-being-imported has configuration for them, they should do their stuff; otherwise, they should pass. Another solution (better than the current behaviour, less good imho than the solution above) would be to load all steps from all profiles at Zope startup, when the profile registry is populated, so that all steps are always run if a package registering a new step is installed. This still means having lots of flag/file-checking and could be quite inefficient. What do you think? Is there a sane use case for the current behaviour? As above: quit doing imperative configuration, and the problems go away. Tres. - --
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Wichert Akkerman wrote: Previously Martin Aspeli wrote: yuppie-3 wrote: Hi! The proposed solution doesn't work: The ImportStepRegistry is not only used for imports - it is also used for creating new 'import_steps.xml' files for exports. Exported profiles are always base profiles, they have to specify *all* import handlers used for creating the site. Right, that's fine. I think the current export functionality is right. I think it's wrong :) The export functionality suffers from the same problen as the import functionality: the export steps defined in extension profiles are only used if they have been selected once. In other words: it is just as unpredictable. What is unpredictable? If you import an extension profile, its steps (both export and import) get grafted into the effective baseline profile. I like Yvo's idea of make import and export registration global instead of per-profile. They are global to the site, if I understand you. I would *not* want them global across the whole instance, because they aren't relevant for all users of GenericSetup. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGrzts+gerLs4ltQ4RAuIaAKC7/hSWycvx+MWaKeL7E1t3a8SOqgCfabPC FFhIkaAaqH3oytpLpDjCZAQ= =9yFm -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Tres Seaver wrote: I think it's wrong :) The export functionality suffers from the same problen as the import functionality: the export steps defined in extension profiles are only used if they have been selected once. In other words: it is just as unpredictable. What is unpredictable? If you import an extension profile, its steps (both export and import) get grafted into the effective baseline profile. Yes, that's true and right. What's unpredictable is that it gets grafted in (at least to import steps, I haven't tested export steps) even if you only *look* at the profile in the Import tab. As soon as you select a profile to look at what steps it offers, those steps are forever part of the registry, whether you run them or not. Martin -- View this message in context: http://www.nabble.com/-Zope-CMF--GenericSetup-extension-profile-import-step-mis-feature-tf4190394.html#a11924305 Sent from the Zope - CMF list1 mailing list archive at Nabble.com. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
yuppie-3 wrote: For an import operation, you run: - All steps that came from the current base profile - All steps that came from explicit, transitive dependencies of the current base profile (provided we get support for declaring profile dependencies) ??? Base profiles have per definition no dependencies. No? Probably, though in Plone, for example, it'd be very useful to be able to say when installing Plone, please also install extension profiles from Archetypes and CMFEditions, where those are dependencies. That's mostly from a code-management point of view, though. Anyway, that functionality doesn't exist yet, so let's not quibble. Not really. You are mixing both ways: Some import steps come from the registry (or are looked up in other profiles?), some from 'import_steps.xml'. I think looked up in other profiles is more predictable for imports, where other profiles means the current base profile and any explicit dependencies, plus the one being installed right now. 2.) The files in the profile determine which import steps are available. This is the current behavior, but further changes would make this more clear: Getting rid of 'import_steps.xml', using a global handler registry instead. And adding new code that checks if a step is available without actually running it. I'm not sure how this gets any clearer... e.g. how does the author of an extension profile for an add-on product ensure that the steps he needs are in the registry? The author of the add-on product is not responsible for this - just like he is not responsible for registering all adapters or utilities used by his product. The author of the setup handler should add a ZCML file that registers the setup handler. Okay, that's different, now we have an instance-global registry. That may make sense, though Tres pointed out you'd need to not run unexpected export steps on export, for example. To me (and for a long time when I was using this blissfully unaware) extension profiles really are separate entities that will generally be applied in an uncontrolled order (unless we get explicit dependency support, which would be really great). Flag files (be they empty boolean switches or XML files with processing instructions) feel like a dirty way of dealing with this unpredictability. Why is it more explicit to have a 'flag' import step in 'import_steps.xml' or in 'import_steps.xml' of a dependency than a flag file in the current profile? The flag file is magical. It's not used for anything. But it's the reverse case that's the problem: As an author of product Y with an extension profile, if I want to make sure I run steps that were in setuphandlers in product X, I don't have a sane way of making sure they get run. They *may* get run if the user looked at and/or ran the import steps for X, but I have no way of controlling that. Therefore, I have to be explicit and include those steps. Therefore, having import steps from extension profiles be additive is an unpredictable feature. Therefore, why require magic flag files? Obviously, that goes away with an instance-global setuphandler registry. Then we still have flag files, but at least it's predictable. Martin -- View this message in context: http://www.nabble.com/-Zope-CMF--GenericSetup-extension-profile-import-step-mis-feature-tf4190394.html#a11924378 Sent from the Zope - CMF list1 mailing list archive at Nabble.com. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Tres Seaver wrote: 'importVarious' is a brutal hack: better to focus efforts on making it disappear. The entire point of the tool is to externalize configuration as declarative data in the profile; accomodating imperative configuration is not something I care to support. I think that's a bit puritanical. Sometimes it's just not possible to have declarative configuration for anything, either for time or complexity reasons. There will be setup tasks that can only feasibly be achieved in an imperative way, and need to run during the installation of a product. - It's impossible to predict which steps actually get run for an extension profile. Yes it is: *all* steps will get run, and will consult the profile to see whether they have configuration to apply. All steps of what? The way it works now, it's all steps of any base or extension profile that has ever been examined by the user or imported, which is not terribly predictable. Say, for example, that I wrote extension profile Y and I wanted to use an import handler used in extension profile X. If X had been run (or in fact, looked at under the Import tab, the registry filling happens as soon as you look at a profile there, even without running it) at least once, then it would get attempted for Y as well. If the appropriate XML or flag file was found it'd be run. That is by design. Once an import step is grafted into the baseline, it is part of the site's configuration for ever. Resetting the baseline profile (a drastic step) is the only way to wipe it out. That makes a lot of sense for export or creation of a new base profile. However, due to the lack of predictability of what has actually been grafted in to date, third party product authors writing an extension profile will never be able to safely depend on steps from anything else than a supported base profile (or explicit dependencies if and when we get those). But there is no way for Y to know that X was run (or looked at). In fact, it may be that users of Y do not want X to be loaded at all. You'd better make this un-abstract: the presence of the enabling files in the profile being imported *is* the signal that those steps should be active. It's not about files, it's about the registry. Here's an example from Plone: - Product CacheFu has an extension profile that declares a new import handler for the Cache Settings tool/control panel. Let's say it creates a handler for the cache.xml file. - Product MyContentType wants to configure caching, with a cache.xml file. Now, if CacheFu has been looked at once or imported, cache.xml is run. CacheFu may or may not be in a sane state at the point (e.g. if you just looked at the profile, you may get an error because the setuphandler is expecting CacheFu to be installed when it isn't). Equally, if CacheFu isn't installed and was never looked at, the cache.xml file is never run, even though the product author probably intended it to be. Of course, there may be a good reason to have cache.xml not be processed if CacheFu isn't installed, but it's still unpredictable. The only way I can make it predictable would be if I explicitly declared a dependency on this profile and/or I explicitly declared a dependency on the product that provided it. In other cases, the setuphandler may be more generic and thus not really depend on e.g. a tool being installed. In that case, it may be very desirable to have the XML file always be processed, but it won't happen unless the product with the setuphandler was installed or looked at once. The only sane behaviour, IMHO, is this: - When importing a base profile, only steps from the base profile are run. Do you mean when setting a new baseline? Otherwise, -1: snapshots, for instance, are baseline profiles, and represent accumulated steps. Yeah, you're right. - When importing an extension profile, only steps from the *current* base profile and any additional steps explicitly defined in the *selected* extension profile (via an import_steps.xml file) are run. No steps are ever implicitly pulled in from other extension profiles, and switching base profiles resets the base set of steps that are run for any extension profile. There is nothing implicit about this: those steps are part of the configured state of the site. If the profile-being-imported has configuration for them, they should do their stuff; otherwise, they should pass. As above: quit doing imperative configuration, and the problems go away. I wish I could, but I've yet to come across a non-trivial product where I didnt need at least a tiny bit of imperative configuration. Most often, it's because things I depend on don't yet have (sufficiently expressive) XML configuration for the option I need to set. I don't really want to invent my own XML syntax for third party products all the time. Martin -- View this message in context:
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Hi! Tres Seaver wrote: Wichert Akkerman wrote: I like Yvo's idea of make import and export registration global instead of per-profile. They are global to the site, if I understand you. I would *not* want them global across the whole instance, because they aren't relevant for all users of GenericSetup. Why is that a problem? Imports are defined by the set of configuration files. As long as there are no conflicting import handlers that depend on files with the same name, I can't see any import problems. Exports are defined by the set of objects in the site. As long as there are no conflicting export handlers that export the object's configuration twice, I can't see any export problems. If there are conflicts, it should be sufficient to resolve them on instance level by fixing the registrations. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin Aspeli wrote: Tres Seaver wrote: 'importVarious' is a brutal hack: better to focus efforts on making it disappear. The entire point of the tool is to externalize configuration as declarative data in the profile; accomodating imperative configuration is not something I care to support. I think that's a bit puritanical. Sometimes it's just not possible to have declarative configuration for anything, either for time or complexity reasons. There will be setup tasks that can only feasibly be achieved in an imperative way, and need to run during the installation of a product. My point is that imperative configuration creates a logically-insoluble rats-nest: Goedel's Law comes into play, because imperative stuff induces ordering dependencies, which are irrelevant for purely declarative stuff. Plone's 'importVarious' was supposed to be a temporary hack when Rob added it almost two years ago; it should *still* die. In the meantime, it is the responsibility of the perpetrator to obey the Hippocratic maxim, first do no harm. - It's impossible to predict which steps actually get run for an extension profile. Yes it is: *all* steps will get run, and will consult the profile to see whether they have configuration to apply. All steps of what? The way it works now, it's all steps of any base or extension profile that has ever been examined by the user or imported, which is not terribly predictable. The examined part is a wart: I missed earlier that you were advocating its destruction. It isn't logically any cleaner for the author of an extension profile, however: she still needs to do the Right Thing when an unexpected step is run (e.g., one imported from another extension). Say, for example, that I wrote extension profile Y and I wanted to use an import handler used in extension profile X. If X had been run (or in fact, looked at under the Import tab, the registry filling happens as soon as you look at a profile there, even without running it) at least once, then it would get attempted for Y as well. If the appropriate XML or flag file was found it'd be run. That is by design. Once an import step is grafted into the baseline, it is part of the site's configuration for ever. Resetting the baseline profile (a drastic step) is the only way to wipe it out. That makes a lot of sense for export or creation of a new base profile. However, due to the lack of predictability of what has actually been grafted in to date, third party product authors writing an extension profile will never be able to safely depend on steps from anything else than a supported base profile (or explicit dependencies if and when we get those). Note that the extension author's convenience was not highest in priority in the design, which tries to make the integrators job easier. Extension authors have to be extremely cautious, and may need to clone steps (in the absence of dependencies). But there is no way for Y to know that X was run (or looked at). In fact, it may be that users of Y do not want X to be loaded at all. You'd better make this un-abstract: the presence of the enabling files in the profile being imported *is* the signal that those steps should be active. It's not about files, it's about the registry. Here's an example from Plone: - Product CacheFu has an extension profile that declares a new import handler for the Cache Settings tool/control panel. Let's say it creates a handler for the cache.xml file. - Product MyContentType wants to configure caching, with a cache.xml file. Now, if CacheFu has been looked at once or imported, cache.xml is run. CacheFu may or may not be in a sane state at the point (e.g. if you just looked at the profile, you may get an error because the setuphandler is expecting CacheFu to be installed when it isn't). That is a bug in CacheFu's handler: first do no harm still applies. Equally, if CacheFu isn't installed and was never looked at, the cache.xml file is never run, even though the product author probably intended it to be. You can't have it both ways: either the handler runs after any inspectiion, or only after importing the relevant extension profile (which is what I would prefer). If MyContentType's author *must* get her cache.xml installed, then she needs to declare the handler (using Products.CacheFu.exportimport.handle_whatever) in her own import_steps.xml. Of course, there may be a good reason to have cache.xml not be processed if CacheFu isn't installed, but it's still unpredictable. The only way I can make it predictable would be if I explicitly declared a dependency on this profile and/or I explicitly declared a dependency on the product that provided it. In other cases, the setuphandler may be more generic and thus not really depend on e.g. a tool being installed. In that case, it may be very desirable to have the XML file always
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Tres Seaver wrote: Martin Aspeli wrote: Tres Seaver wrote: 'importVarious' is a brutal hack: better to focus efforts on making it disappear. The entire point of the tool is to externalize configuration as declarative data in the profile; accomodating imperative configuration is not something I care to support. I think that's a bit puritanical. Sometimes it's just not possible to have declarative configuration for anything, either for time or complexity reasons. There will be setup tasks that can only feasibly be achieved in an imperative way, and need to run during the installation of a product. My point is that imperative configuration creates a logically-insoluble rats-nest: Goedel's Law comes into play, because imperative stuff induces ordering dependencies, which are irrelevant for purely declarative stuff. Plone's 'importVarious' was supposed to be a temporary hack when Rob added it almost two years ago; it should *still* die. In the meantime, it is the responsibility of the perpetrator to obey the Hippocratic maxim, first do no harm. Just as a side note, Plone itself (not counting the ones from its dependencies) currently has five different interdependent import various handlers. One is enabling the portal as a Zope3 site, two are quickinstalling products based on extension profiles (which are again dependent on each other), one is doing some configuration that really should be in xml files and one is installing content. The content creation as well as some site settings are dependent on the current browser language. All this works reasonable well right now and while I don't like all of it, I think realistically we need to have some sensible support for imperative import handlers for some time in the future. Hanno ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Tres Seaver wrote: [..] Note that the extension author's convenience was not highest in priority in the design, which tries to make the integrators job easier. Extension authors have to be extremely cautious, and may need to clone steps (in the absence of dependencies). [sorry if this may come out as OT but somehow I'm getting lost here] Even for the integrator it might be advantageous if add-ons could declare dependencies. And it doesn't stop with Plone. Some things only start there ;-) Let me illustrate this by way of example: I provide an add-on for convenient handling of bibliographic data in Plone (CMFBibliographAT; some of you may know it even). It uses a specific field and widget (for handling the author lists) that I provide via ATExtensions (because I think its generic enough to be useful in its own right). Now, when moving CMFBib to GS-based install I went for the clone way (aka included ATExtensions' profile within CMFBib's profile - after all I'm the author of both and I take it as my responsibility to take care of the redundancy as things evolve further) so you could say that's no big deal. But, there are further add-ons by other people based on CMFBib, e.g., ATBiblioStyles, ATBiblioList, ATBiblioTopic, ... which all depend on CMFBib. Previously it was no problem for any of these products to put in their install something like (pseudocode of course; qi is Plone's QuickInstaller): for p in dependencies: if not qi.isInstalled(p): qi.install(p) And now? What are those add-on authors (e.g. David Convent for ATBiblioStyles or Mike Gabriel for ATBiblioTopic) supposed to do? Include the ATExtensions and CMFBib profiles in their products as well? Leave them out and explain in their README (which those who should never read) that they have to load the other profiles first? Is that making the integrators life easier? I guess I just don't see what's wrong with supporting dependencies. And to stress it as explicitly as I can: I sometimes get the impression here that some don't realize what stacks get build up when it comes to real-world applications/deployments. It's a matter of fact, that many, many applications are not build on plain CMF (or Plone for that matter) but often include a significant number of add-ons. E.g., a 'simple' site for a university institute that wants to have decent publication handling based on what I mentioned above may be using: CMF, Plone, Archetypes, ATExtensions, CMFBibliographyAT, ATBiblioStyles, ATBiblioList, ATBiblioTopic and maybe the AmazonTool and the PubMedClient Now don't tell me its overkill to resort to 7(!) add-ons just to handle bibliographic data on a website. All these packages provide different yet useful features and it is on purpose that they are kept apart (in fact I strongly believe I should break down CMFBib into several packages as it became quite overgrown by now) but without any decent support for handling dependencies the *integrators* job won't become easier. rant-mode off Raphael ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Raphael Ritz wrote: ---SNIP--- And now? What are those add-on authors (e.g. David Convent for ATBiblioStyles or Mike Gabriel for ATBiblioTopic) supposed to do? Include the ATExtensions and CMFBib profiles in their products as well? Leave them out and explain in their README (which those who should never read) that they have to load the other profiles first? Is that making the integrators life easier? I guess I just don't see what's wrong with supporting dependencies. i don't think anybody is saying there's something wrong w/ supporting dependencies. in fact, i'm pretty sure we have consensus that profile dependency support is desirable. the problem is that it's not yet in the code. i put support for declaring profile dependencies into the metadata.xml file (where you can also specify the profile version), but i ran out of time to work on GS and so nothing is being done with this information. -r ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin Aspeli wrote: Hi guys, After a lot of is-this-a-bug type discussions with Rob and Wichert, I've come to feel pretty strongly about the following: When you load an extension profile for the first time in GS, it looks to see if it has any import steps (in import_steps.xml) that are not already known. If so, it adds them to the import step registry and then runs all steps in the registry. Any time a profile (the same, or another one) is run subsequently, that import step will be run too. In Rob's words, the import steps and the profile that they came from (import_steps.xml-wise) get decoupled. We normally don't mind too much, because steps are run in the context of a particular profile, and if that context has no matching XML file, nothing is loaded. At least so we assume - there's of course nothing to stop the setuphandler from having side effects even if the XML file is not found. We *do* see a problem with importVarious-type hacks, though. Here, people resort to checking for a flag (normally an empty text file) to determine if the setuphandler should be run. 'importVarious' is a brutal hack: better to focus efforts on making it disappear. The entire point of the tool is to externalize configuration as declarative data in the profile; accomodating imperative configuration is not something I care to support. The 'upgrade' stuff Rob ported in from CPS is the only sane case I know of for imperative action. while i agree with this in principal, i think the truth of the matter is that plenty of relatively inexperienced developers will be coding GenericSetup profiles. these developers are the ones most likely to write imperative import steps (b/c they won't understand how to write a new XML parser, and it will seem like a lot of extra work compared to just twiddling a setting somewhere). they are also the _least_ likely to understand the subtleties of GS's behaviour. the problem is exacerbated by the symptoms; it's not that there will necesarily be problems running the profile in question. it's when you try to import OTHER profiles that there will be problems, because these naively coded steps will be performing actions when they shouldn't be. this has already shown itself to be the most misunderstood aspect of GenericSetup, i've had to explain the behaviour over and over again. the documentation can be improved, sure, but we all know that people will be bitten by this no matter how clearly this issue is spelled out. i'm pretty much in support of the proposal as it's evolved so far. here's my understanding of it at this point: - the import step registry will record the profile from which a given import step is included. if multiple profiles include the same step in their import_steps.xml, they will both be recorded as related to that import step. - when a profile is imported, only the import steps from the base profile and all transitively dependent profiles will be run. (clearly this depends on actually finishing the dependency support.) - when a profile is exported (or a snapshot is generated), all of the registered steps will be used. this way, a naively coded import handler can only cause a problem when the profile it is associated with (or any other profile that explicitly depends on the problematic profile) is run. writing imperative handlers should still be discouraged, but the damage that they can cause is contained. -r ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Tres Seaver, on 2007-07-31: That is by design. Once an import step is grafted into the baseline, it is part of the site's configuration for ever. Resetting the baseline profile (a drastic step) is the only way to wipe it out. Which is unfortunate as it means that any Product that defines its own import step (be it via import_steps.xml or e.g. in the case of CacheFu cache.xml) needs to stay in the Products directory forever. If you remove it (even after uninstalling it and restarting Zope and removing it from Control_Panel/Products) the step is still in the registry. So it will give an error when you explicitly run it or when you run all available steps, like this: 2007-07-31 19:43:34 ERROR Zope.SiteErrorLog http://localhost:8080/kantoor/portal_setup/manage_importSelectedSteps Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Products.GenericSetup.tool, line 414, in manage_importSelectedSteps Module Products.GenericSetup.tool, line 235, in runImportStep Module Products.GenericSetup.tool, line 727, in _doRunImportStep - __traceback_info__: kantoorsetup_various Module Products.GenericSetup.registry, line 146, in getStep Module Products.GenericSetup.utils, line 103, in _resolveDottedName - __traceback_info__: Products.kantoorsetup.setuphandlers.importVarious AttributeError: 'module' object has no attribute 'kantoorsetup' I went in with a zopectl debug session a few weeks ago to remove such a step manually from the registry. That worked, but having a method that you can call in an uninstall method that does this for you, would be better. -- Maurits van Rees | http://maurits.vanrees.org/ [NL] Work | http://zestsoftware.nl/ Do not worry about your difficulties in computers, I can assure you mine are still greater. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Martin Aspeli, on 2007-07-31: Of course, there may be a good reason to have cache.xml not be processed if CacheFu isn't installed, but it's still unpredictable. The only way I can make it predictable would be if I explicitly declared a dependency on this profile and/or I explicitly declared a dependency on the product that provided it. Is there anything wrong with declaring CacheFu a dependency when you actually depend on it? And on the other hand: if you have a cache.xml file in e.g. Poi and someone else uses that product but does not want to use CacheFu, then without declaring a dependency the cache.xml just does not get processed, which is exactly what the user wants then. -- Maurits van Rees | http://maurits.vanrees.org/ [NL] Work | http://zestsoftware.nl/ Do not worry about your difficulties in computers, I can assure you mine are still greater. ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Hi! Rob Miller wrote: i'm pretty much in support of the proposal as it's evolved so far. I'm not. here's my understanding of it at this point: - the import step registry will record the profile from which a given import step is included. if multiple profiles include the same step in their import_steps.xml, they will both be recorded as related to that import step. What means same step? Step with same handler, same ID, same version, same title or same description? I guess you mean same handler, but what about the other metadata? Which profile has the 'right' metadata? - when a profile is imported, only the import steps from the base profile and all transitively dependent profiles will be run. (clearly this depends on actually finishing the dependency support.) - when a profile is exported (or a snapshot is generated), all of the registered steps will be used. this way, a naively coded import handler can only cause a problem when the profile it is associated with (or any other profile that explicitly depends on the problematic profile) is run. writing imperative handlers should still be discouraged, but the damage that they can cause is contained. I'd prefer it if naively coded import handlers always cause problems - this way the problems are easily identified and fixed. A global handler registry is *much* simpler. It allows to remove a lot of code instead of making things even more complex than they already are. Cheers, Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: GenericSetup extension profile import step mis-feature
Maurits van Rees wrote: 2007-07-31 19:43:34 ERROR Zope.SiteErrorLog http://localhost:8080/kantoor/portal_setup/manage_importSelectedSteps Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Products.GenericSetup.tool, line 414, in manage_importSelectedSteps Module Products.GenericSetup.tool, line 235, in runImportStep Module Products.GenericSetup.tool, line 727, in _doRunImportStep - __traceback_info__: kantoorsetup_various Module Products.GenericSetup.registry, line 146, in getStep Module Products.GenericSetup.utils, line 103, in _resolveDottedName - __traceback_info__: Products.kantoorsetup.setuphandlers.importVarious AttributeError: 'module' object has no attribute 'kantoorsetup' I went in with a zopectl debug session a few weeks ago to remove such a step manually from the registry. That worked, but having a method that you can call in an uninstall method that does this for you, would be better. One more reason for a non-persistent global registry. Yuppie ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: GenericSetup extension profile import step mis-feature
On 8/1/07, Martin Aspeli [EMAIL PROTECTED] wrote: yuppie-3 wrote: For an import operation, you run: - All steps that came from the current base profile - All steps that came from explicit, transitive dependencies of the current base profile (provided we get support for declaring profile dependencies) ??? Base profiles have per definition no dependencies. No? Probably, though in Plone, for example, it'd be very useful to be able to say when installing Plone, please also install extension profiles from Archetypes and CMFEditions, where those are dependencies. That's mostly from a code-management point of view, though. Anyway, that functionality doesn't exist yet, so let's not quibble. While I like the idea of the freedom from constantly checking xml diffs of updated products' base profiles against my current base profile, I think that installing extension profiles of product dependencies is a bit heavy handed unless extension profiles are known to be independent of each other in what they modify in a site. It seems more relevant to me to be explicitly calling import steps from extension profiles of other products in the order that you want them to be called (or not called). As it is, doing diffs and merges between extension profiles and a base profile (including trying to figure out whether I really do want to run another products importVarious scariness) to attempt to get a new base profile that is supposed to reflect the desired base profile of my site is frustrating and feels like the wrong workflow anyway. What I would like to be able to do is to start with a vanilla base profile, then apply particular import steps of relevant extension profiles, then export that state as my new base profile. Updating a product would simply mean running the steps you want again, checking the consistency of your site, then if happy, exporting that state as your new base profile. Problem is, for plone I can't even get an exported profile of a vanilla plone that was setup with the default base profile to work as a new base profile - mostly because various imperative configuration steps have altered the site in ways that get reflected partially in the export (but not fully). ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
