Re: [Zope-CMF] GenericSetup proposal: upgrade profiles
Hi Rob,
Thanks for the feedback!
Rob Miller, on 2009-06-24:
> i'm not actively working on GenericSetup these days, so i don't know that my
> opinion counts for much, but i'm -1 on this. to me it seems to introduce an
> entirely new way of handling upgrades that is likely to confuse people.
> GenericSetup is more than complex enough already, IMO.
>
> Maurits van Rees wrote:
>> Use case
>>
>>
>> For our use case we assume that you have made an add-on product. It
>> has an EXTENSION profile. You have applied the profile in a website.
>> Now you want to add a catalog index. Your options are:
>>
>> A. Add this in the current extension profile. This means that any
>>time you apply this profile this index will be removed and
>>recreated so you lose data; you need to reindex the index (manually
>>or in python code - not GenericSetup) and that takes time.
>
> in my mind this is the core problem, that there is no way to
> non-destructively apply an index to a catalog. this is a weakness
> in GenericSetup, but developing an entirely new alternative
> mechanism for handling upgrades seems to me the wrong solution.
In general, I think we could use an option like 'override' or
'only_once', next to the purge and remove options, like this:
Site Administrator
utf-8
This would then mean:
- at first the properties do not exist yet
- apply this profile -> properties do exist, with the values given
- change email_from_name to 'Acme Inc' and email_charset to 'ascii'
- reapply this profile
- email_from_name is untouched at 'Acme Inc' and email_charset has
been overridden to 'utf-8'
Problem may be that if you do this in one import handler (say for the
properties) users may be caught off guard when this option does
nothing in another (like for the catalog) and loose data/time that
way.
>> B. Add an upgrade step. Works fine, but this needs python code along
>>these lines:
>>
>> indexes = catalog.indexes()
>> if 'new' not in indexes:
>> catalog.addIndex('new', 'some_meta_type')
>> catalog.manage_reindexIndex(ids=['new'])
>>
>>But why add python code when there is already a perfectly fine
>>import handler that can read a catalog.xml? We are back to the
>>dark days before there was GenericSetup then. (Well, okay, writing
>>python code is not *that* bad. ;-))
>>
>> C. Add it to the current extension profile. Do not apply this profile
>>ever again: only once at install time.
>
> this confuses me. in my mind, extension profiles should be idempotent. IOW,
> it should _always_ be safe to re-apply an extension profile to a site,
> assuming that extension profile does indeed represent the final state that
> you'd like the site to have.
In my mind, an extension profile is just a starting point, at least
partly. Like above, you set a property email_from_name so the code
that expects this property to exist actually works. Then your client
comes along and changes this property to something he likes. You do
not want to reset this to the default when reapplying the profile.
For another part, an extension profile can serve as a set of known
good settings and be reapplied to cleanup the mess that the client
made when clicking through the ZMI.
>> Instead add an upgrade step
>>that applies the catalog step from this profile. But this has
>>basically the same problem as option A: if you later add a second
>>catalog index you would need to create a new upgrade step to apply
>>the catalog step and then your first index would be removed and
>>recreated empty again.
>
> again, this seems to me a weakness of the catalog index importing, and it
> should be fixed there.
>
> it is option (C) which i would recommend, so much so that there is even a
> dedicated ZCML tag specific for this purpose: genericsetup:upgradeDepends.
> here's an example from the GenericSetup tests:
>
>... ... title="Bar Upgrade dependency profile import steps"
>... description="Re-imports steps from the profile"
>... import_profile="profile-Products.CMFDefault:default"
>... import_steps="baz bat"
>... run_deps="True"
>... purge="True"
>... />
>
> upgradeDepends lets you specify any arbitrary import steps from any arbitrary
> profile, so that you don't have to repeat the boilerplate code of invoking
> those import steps by hand whenever you need to write such an upgrade step.
This is definitely an improvement, and I am certainly going to use
that when GenericSetup 1.5 becomes available. But it does not help
against a catalog index importer that recreates an empty index; or
against a properties importer that resets properties to their factory
default.
>> D. Define a new EXTENSION profile that just has a catalog.xml. Either
>>go to the import tab of portal_setup to run all steps of that
>>profil
Re: [Zope-CMF] GenericSetup proposal: upgrade profiles
i'm not actively working on GenericSetup these days, so i don't know that my
opinion counts for much, but i'm -1 on this. to me it seems to introduce an
entirely new way of handling upgrades that is likely to confuse people.
GenericSetup is more than complex enough already, IMO.
Maurits van Rees wrote:
> Use case
>
>
> For our use case we assume that you have made an add-on product. It
> has an EXTENSION profile. You have applied the profile in a website.
> Now you want to add a catalog index. Your options are:
>
> A. Add this in the current extension profile. This means that any
>time you apply this profile this index will be removed and
>recreated so you lose data; you need to reindex the index (manually
>or in python code - not GenericSetup) and that takes time.
in my mind this is the core problem, that there is no way to non-destructively
apply an index to a catalog. this is a weakness in GenericSetup, but
developing an entirely new alternative mechanism for handling upgrades seems
to me the wrong solution.
> B. Add an upgrade step. Works fine, but this needs python code along
>these lines:
>
> indexes = catalog.indexes()
> if 'new' not in indexes:
> catalog.addIndex('new', 'some_meta_type')
> catalog.manage_reindexIndex(ids=['new'])
>
>But why add python code when there is already a perfectly fine
>import handler that can read a catalog.xml? We are back to the
>dark days before there was GenericSetup then. (Well, okay, writing
>python code is not *that* bad. ;-))
>
> C. Add it to the current extension profile. Do not apply this profile
>ever again: only once at install time.
this confuses me. in my mind, extension profiles should be idempotent. IOW,
it should _always_ be safe to re-apply an extension profile to a site,
assuming that extension profile does indeed represent the final state that
you'd like the site to have.
> Instead add an upgrade step
>that applies the catalog step from this profile. But this has
>basically the same problem as option A: if you later add a second
>catalog index you would need to create a new upgrade step to apply
>the catalog step and then your first index would be removed and
>recreated empty again.
again, this seems to me a weakness of the catalog index importing, and it
should be fixed there.
it is option (C) which i would recommend, so much so that there is even a
dedicated ZCML tag specific for this purpose: genericsetup:upgradeDepends.
here's an example from the GenericSetup tests:
...
upgradeDepends lets you specify any arbitrary import steps from any arbitrary
profile, so that you don't have to repeat the boilerplate code of invoking
those import steps by hand whenever you need to write such an upgrade step.
> D. Define a new EXTENSION profile that just has a catalog.xml. Either
>go to the import tab of portal_setup to run all steps of that
>profile, or create an upgrade step that does this. But this means
>that in several spots the list of available profiles gets longer,
>which threatens to make it hard to use.
i think the use of profiles for upgrades was an understandable stop-gap when
GS did not yet provide any other mechanism for handling ugprades. doing so
now is, IMO, a mistake, even if you designate a specific type of profile for
this purpose.
>The migration machinery of CMFPlone is moving to the
>plone.app.upgrade package (not released yet), which already defines
>about 24 upgrade steps with 11 extension profiles and that will
>only increase, potentially one for each point release of Plone.
>See http://svn.plone.org/svn/plone/plone.app.upgrade/trunk
>
>(And the CMFQuickInstaller can get confused when a package has more
>than one profile; but that is not a problem for this mailing list,
>and I expect that in Plone 4 either the CMFQuickInstaller is gone
>or it ends up as just a small wrapper around GenericSetup.)
>
> Similar arguments could be made about adding a property to a property
> sheet: you only want to do that once and not override any changes made
> to that property by a site admin when you reapply the profile.
>
> So I think another option would help:
>
> E. Define an UPGRADE profile that just has a catalog.xml. Going to
>the import tab of portal_setup should not list this profile, but
>only EXTENSION and BASE profiles, so that list is not so cluttered
>anymore. You now just create an upgrade step that applies the
>profile.
so if your product requires this new index, so much so that you made an
upgrade profile for it, it's reasonable to assume that you'd want any new
installations of this product to also have this new index. does this mean
that when you apply an extension profile, all upgrade profiles are
automatically applied? or do you have to add the index to the extension
profile as well?
maybe this horse
Re: [Zope-CMF] GenericSetup proposal: upgrade profiles
Hi Maurits,
I have a product where I use an extension profile and 2 other
extension profiles used with upgradeStep.
For my use case, when GenericSetup 1.5 will be released, I will
rewrite my migration code to use the new upgradeDepends directive so I
don't need the 2 other extension profiles.
You gave good examples. If you add a new index in catalog.xml of the
*main* extension profile, even if you use the new upgradeDepends
directive to reimport the catalog step, it will recreate all indexes
specified in catalog.xml.
When I reinstall a product generated in ArchGenXML which have a
catalog.xml file, all indexes are reinitialized and I go to
portal_catalog to rebuild the catalog manually.
So using an UPGRADE profile with a catalog.xml containing only the new
index and if it can rebuild the index when the step is imported, it
will be good.
And I had the use case for creating a new property in a property sheet
in portal_properties.
I simply gave up doing it with XML because it flushed the value every
time I reinstalled.
I've done a step in python to not reinitialize the property when I
reinstall the product.
Vincent Fretin
On Wed, Jun 24, 2009 at 2:42 AM, Maurits van
Rees wrote:
> Hi,
>
> Products.GenericSetup/interfaces.py says:
> BASE, EXTENSION = 1, 2
>
> I propose adding another one:
> UPGRADE = 3
> and possibly a fourth one:
> UNINSTALL = 4
>
>
> Intro
> -
>
> This refs a small discussion on the Plone developers list at the end
> of May, which can be interesting (but not necessary) as background:
> http://plone.org/support/forums/core#nabble-tc2966873|a2973467
>
> The meaning should be reasonably clear from the names:
>
> - BASE: completely new install; purge existing settings; usually
> applied once during the lifetime of the website: at creation time
>
> - EXTENSION: extend a BASE profile; do not purge existing settings;
> might get applied multiple times during the lifetime of the website.
>
> - UPGRADE: upgrade for a BASE or EXTENSION profile; no purge; usually
> applied once. Actually, it does not need to be tied to another
> profile in any way: it is just a bunch of import steps that need to
> be applied once.
>
> - UNINSTALL: undo what was done while applying another profile (BASE,
> EXTENSION or less likely an UPGRADE); no purge; usually applied once:
> when uninstalling the product that the profile belongs to.
>
> - Hm, let's throw TEST (value 5) in as an idea as well: a non-purging
> profile that is meant to be applied during the automated tests but
> not during normal use. Can anyone come up with a use case?
>
> - And I suddenly wonder if it is handy to define SNAPSHOT (value 6)
> here too. Might this make some of the special handling for
> snapshots easier or unnecessary? If someone wants to plea for that:
> go ahead. :)
>
> I do not have plans for UNINSTALL, TEST or SNAPSHOT, but I think
> UPGRADE would be handy, so let's concentrate on that one.
>
>
> Proposal
>
>
> - Add UPGRADE to the GS interfaces.py, for use in the 'provides' line
> of a profile registration in zcml:
>
> provides="Products.GenericSetup.interfaces.UPGRADE"
>
> - Show only BASE and EXTENSION profiles and not UPGRADE (or other)
> profiles in some parts of the UI, so that those parts are not
> cluttered by too many profiles.
>
> Third party products can then register one true EXTENSION profile that
> is meant to be applied when installing the product, and several
> UPGRADE profiles meant to be applied once, without confusing for
> example the CMFQuickInstaller in Plone.
>
>
> Use case
>
>
> For our use case we assume that you have made an add-on product. It
> has an EXTENSION profile. You have applied the profile in a website.
> Now you want to add a catalog index. Your options are:
>
> A. Add this in the current extension profile. This means that any
> time you apply this profile this index will be removed and
> recreated so you lose data; you need to reindex the index (manually
> or in python code - not GenericSetup) and that takes time.
>
> B. Add an upgrade step. Works fine, but this needs python code along
> these lines:
>
> indexes = catalog.indexes()
> if 'new' not in indexes:
> catalog.addIndex('new', 'some_meta_type')
> catalog.manage_reindexIndex(ids=['new'])
>
> But why add python code when there is already a perfectly fine
> import handler that can read a catalog.xml? We are back to the
> dark days before there was GenericSetup then. (Well, okay, writing
> python code is not *that* bad. ;-))
>
> C. Add it to the current extension profile. Do not apply this profile
> ever again: only once at install time. Instead add an upgrade step
> that applies the catalog step from this profile. But this has
> basically the same problem as option A: if you later add a second
> catalog index you would need to create a new upgrade step to apply
> the catalog step and then your first index would be removed and
> recrea
[Zope-CMF] GenericSetup proposal: upgrade profiles
Hi,
Products.GenericSetup/interfaces.py says:
BASE, EXTENSION = 1, 2
I propose adding another one:
UPGRADE = 3
and possibly a fourth one:
UNINSTALL = 4
Intro
-
This refs a small discussion on the Plone developers list at the end
of May, which can be interesting (but not necessary) as background:
http://plone.org/support/forums/core#nabble-tc2966873|a2973467
The meaning should be reasonably clear from the names:
- BASE: completely new install; purge existing settings; usually
applied once during the lifetime of the website: at creation time
- EXTENSION: extend a BASE profile; do not purge existing settings;
might get applied multiple times during the lifetime of the website.
- UPGRADE: upgrade for a BASE or EXTENSION profile; no purge; usually
applied once. Actually, it does not need to be tied to another
profile in any way: it is just a bunch of import steps that need to
be applied once.
- UNINSTALL: undo what was done while applying another profile (BASE,
EXTENSION or less likely an UPGRADE); no purge; usually applied once:
when uninstalling the product that the profile belongs to.
- Hm, let's throw TEST (value 5) in as an idea as well: a non-purging
profile that is meant to be applied during the automated tests but
not during normal use. Can anyone come up with a use case?
- And I suddenly wonder if it is handy to define SNAPSHOT (value 6)
here too. Might this make some of the special handling for
snapshots easier or unnecessary? If someone wants to plea for that:
go ahead. :)
I do not have plans for UNINSTALL, TEST or SNAPSHOT, but I think
UPGRADE would be handy, so let's concentrate on that one.
Proposal
- Add UPGRADE to the GS interfaces.py, for use in the 'provides' line
of a profile registration in zcml:
provides="Products.GenericSetup.interfaces.UPGRADE"
- Show only BASE and EXTENSION profiles and not UPGRADE (or other)
profiles in some parts of the UI, so that those parts are not
cluttered by too many profiles.
Third party products can then register one true EXTENSION profile that
is meant to be applied when installing the product, and several
UPGRADE profiles meant to be applied once, without confusing for
example the CMFQuickInstaller in Plone.
Use case
For our use case we assume that you have made an add-on product. It
has an EXTENSION profile. You have applied the profile in a website.
Now you want to add a catalog index. Your options are:
A. Add this in the current extension profile. This means that any
time you apply this profile this index will be removed and
recreated so you lose data; you need to reindex the index (manually
or in python code - not GenericSetup) and that takes time.
B. Add an upgrade step. Works fine, but this needs python code along
these lines:
indexes = catalog.indexes()
if 'new' not in indexes:
catalog.addIndex('new', 'some_meta_type')
catalog.manage_reindexIndex(ids=['new'])
But why add python code when there is already a perfectly fine
import handler that can read a catalog.xml? We are back to the
dark days before there was GenericSetup then. (Well, okay, writing
python code is not *that* bad. ;-))
C. Add it to the current extension profile. Do not apply this profile
ever again: only once at install time. Instead add an upgrade step
that applies the catalog step from this profile. But this has
basically the same problem as option A: if you later add a second
catalog index you would need to create a new upgrade step to apply
the catalog step and then your first index would be removed and
recreated empty again.
D. Define a new EXTENSION profile that just has a catalog.xml. Either
go to the import tab of portal_setup to run all steps of that
profile, or create an upgrade step that does this. But this means
that in several spots the list of available profiles gets longer,
which threatens to make it hard to use.
The migration machinery of CMFPlone is moving to the
plone.app.upgrade package (not released yet), which already defines
about 24 upgrade steps with 11 extension profiles and that will
only increase, potentially one for each point release of Plone.
See http://svn.plone.org/svn/plone/plone.app.upgrade/trunk
(And the CMFQuickInstaller can get confused when a package has more
than one profile; but that is not a problem for this mailing list,
and I expect that in Plone 4 either the CMFQuickInstaller is gone
or it ends up as just a small wrapper around GenericSetup.)
Similar arguments could be made about adding a property to a property
sheet: you only want to do that once and not override any changes made
to that property by a site admin when you reapply the profile.
So I think another option would help:
E. Define an UPGRADE profile that just has a catalog.xml. Going to
the import tab of portal_setup should not list this profile, but
only EXTENSION and
