Sorry, I forgot to mention : it's pushed in the patch branch :
https://github.com/apache/incubator-unomi/tree/patch
On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier <tdra...@jahia.com> wrote:
> Hi,
>
> I've committed the changes for the patch. The files need to be put in the
> same folder as the definitions ( META-INF/cxs/xxx ), and named with a
> suffix -patch.json . They will be deployed only once, if the definitions
> was already there. Each patch has an id, so if you need to deploy a new
> patch, just use a new id. The list of deployed patches is stored in ES.
> Patches can also be manually executed with the "unomi:deploy-definition"
> karaf command, or through a rest endpoint at /cxs/patches/apply . I've
> removed the behaviour with -SNAPSHOT - definitions are never redeployed,
> even in snapshot versions, so you'll need to explicitely provide a patch to
> override.
>
> The syntax is slightly different from what has been sent previously :
>
> {
> "itemId": "firstName-patch1",
> "patchedItemId": "firstName",
> "operation": "patch",
> "data": [
>
>
> {
> "op": "replace",
> "path": "/defaultValue",
> "value": "foo"
> }
> ]
> }
>
> For an override :
>
> {
> "itemId": "gender-patch1",
> "patchedItemId": "gender",
> "operation": "override",
> "data": {
> "metadata": {
> "id": "gender",
> "name": "Gender",
>
> "systemTags": [
> "properties",
> "profileProperties"
> ]
> },
> "type": "string",
> "defaultValue": "",
> "automaticMappingsFrom": [ ],
>
> "rank": "105.0"
> }
> }
>
> Or a remove :
>
> {
> "itemId": "firstName-patch3",
> "patchedItemId": "firstName",
> "operation": "remove"
> }
>
>
> Regards
>
>
>
>
> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier <tdra...@jahia.com> wrote:
>
>> Hi,
>>
>> I did a quick test with json patch and actually it looks quite easy to
>> integrate and seems to fill more use cases than just a simple "override".
>> However I think it can still be useful to have a complete override and also
>> the possisbility to completely remove the definition, so I was thinking of
>> some "migration" or "update" json file wrapping the patch, within the same
>> folder as the definitions :
>>
>> {
>> "itemId": "firstName",
>> "updateId": "firstName-patch1",
>> "operation": "patch",
>> "patch": [
>> {
>> "op": "replace", "path": "/defaultValue", "value": "foo"
>> }
>> ]
>> }
>>
>>
>> A removal :
>>
>> {
>> "itemId": "firstName",
>> "updateId": "firstName-patch1",
>> "operation": "remove"
>> }
>>
>> Or a complete override :
>>
>> {
>> "itemId": "firstName",
>> "updateId": "firstName-patch1",
>> "operation": "override",
>> "definition": {
>> "metadata": {
>> "id": "firstName",
>> "name": "First name",
>> "systemTags": [
>> "properties",
>> "profileProperties",
>> "basicProfileProperties",
>> "personalIdentifierProperties"
>> ]
>> },
>> "type": "string",
>> "defaultValue": "",
>> "automaticMappingsFrom": [ ],
>> "rank": "101.0"
>> }
>> }
>>
>>
>> This could actually be extended with other type of operation.
>>
>> For now I do store "updateId" in the metadata of the item, so that it's
>> not executed multiple times. But we have have an issue with the delete if
>> the item is recreated after.. Maybe I should store the list of updates in a
>> new structure ? That could also be accessible with karaf commands, to see
>> which patch have been executed and reexecute them if needed.
>>
>> Thomas
>>
>>
>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
>> francois.pa...@openobject.fr> wrote:
>>
>>> Hi,
>>>
>>> Very interesting flow :)
>>>
>>> What is the format to update the definitions, it's just a new JSON, is
>>> there some directives about the changes ?
>>>
>>> For example, in Karaf we have something like this to override
>>> configuration files :
>>>
>>> <edits>
>>> <edit>
>>> <file>config.properties</file>
>>> <operation>put</operation>
>>> <key>karaf.framework</key>
>>> <value>equinox</value>
>>> </edit>
>>> <edit>
>>> <file>config.properties</file>
>>> <operation>extend</operation>
>>> <key>org.osgi.framework.system.capabilities</key>
>>> <value>my-magic-capability</value>
>>> </edit>
>>> <edit>
>>> <file>config.properties</file>
>>> <operation>remove</operation>
>>> <key>org.apache.karaf.security.providers</key>
>>> </edit>
>>> </edits>
>>>
>>> May be a solution is a file with a patch description of the changes ?
>>>
>>> regards,
>>>
>>> François Papon
>>> fpa...@apache.org
>>>
>>> Le 18/09/2018 à 13:38, Serge Huber a écrit :
>>> > Not sure if you are talking about the patching system, but if yes you
>>> > would provide a patch file that removes all the contents of file.
>>> >
>>> > If we are just using versions it is indeed a bit more tricky. We would
>>> > also need to have a way to list the definitions to remove.
>>> >
>>> > Regards,
>>> > Serge...
>>> >
>>> > On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier <rgauth...@jahia.com>
>>> wrote:
>>> >> Hi Serge,
>>> >>
>>> >> In that context, how would we manage objects deletion from a plugin?
>>> >>
>>> >> Thanks!
>>> >>
>>> >> Romain
>>> >>
>>> >> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber <shu...@apache.org>
>>> wrote:
>>> >>
>>> >>> Sorry, I didn't get around to this discussion but I think that
>>> >>> versions would probably be cleaner. I was even looking at solutions
>>> >>> such as JSON patch (https://github.com/java-json-tools/json-patch)
>>> to
>>> >>> be able to perform modifications on parts of definitions instead of
>>> >>> having to deploy a whole definition again.
>>> >>>
>>> >>> We had discussions about adding lots of metadata, including last
>>> >>> modification dates, maybe even hashes for the definition files, to
>>> >>> know if a definition has been changed or not.
>>> >>>
>>> >>> I think that one of the advantages of the patch idea is that you
>>> could
>>> >>> try to patch something if it's compatible, and if it's not you would
>>> >>> not deploy it. But this idea would mostly be for use cases where you
>>> >>> are interested in modifying something that comes out of the box in
>>> >>> Unomi, not for upgrading a plugin.
>>> >>>
>>> >>> With versions at least we could also track if a definition is on the
>>> >>> "default" version or has been modified somehow.
>>> >>>
>>> >>> For the cfg idea it feels weird to use a config file that way, maybe
>>> >>> we should put the marker file elsewhere but to be honest I don't have
>>> >>> a better idea :)
>>> >>>
>>> >>> Would love some input from JB or other Karaf experts :)
>>> >>>
>>> >>> Regards,
>>> >>> Serge...
>>> >>>
>>> >>> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier <tdra...@jahia.com>
>>> wrote:
>>> >>>> Hi,
>>> >>>>
>>> >>>> I have to check, but the idea would be to be able to package this
>>> cfg
>>> >>> file
>>> >>>> beside the bundle. I'm not familiar with unomi:migrate command, but
>>> we
>>> >>> can
>>> >>>> imagine that this command would extract and deploy the cfg file
>>> from the
>>> >>>> bundle - which would make the packaging easier (cfg directly inside
>>> the
>>> >>>> plugin). The thing is that we need to be able to remove the file
>>> once the
>>> >>>> definition has been "overridden" so that it's only done once, so
>>> the cfg
>>> >>>> file should not be deployed automatically on each startup.
>>> >>>>
>>> >>>> Another idea I had was to store sort of "version" of the definition
>>> >>> inside
>>> >>>> the metadata, and check if the json file has a greater "version"
>>> defined.
>>> >>>> That makes things even more easier when developing plugin and looks
>>> a
>>> >>>> little more clean, but I don't know if it would cover correctly all
>>> use
>>> >>>> cases.
>>> >>>>
>>> >>>> We should also extends the DeployDefinition shell command to be
>>> able to
>>> >>>> redeploy all definitions of a bundle.
>>> >>>>
>>> >>>> Thomas
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier <
>>> rgauth...@jahia.com>
>>> >>> wrote:
>>> >>>>> Hello Thomas,
>>> >>>>>
>>> >>>>> Nice proposition.How would it work for developers that are building
>>> >>>>> plugins? Would we have to add our own cfg file or somehow update
>>> the
>>> >>> cfg
>>> >>>>> file that you want to add?
>>> >>>>>
>>> >>>>> Since there is already a "unomi:migrate" command would you reuse
>>> that
>>> >>>>> trigger to update / delete the definitions of the objects?
>>> >>>>>
>>> >>>>> Same question regarding the plugins, should we also implement
>>> "migrate"
>>> >>>>> command?
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>>
>>> >>>>> Romain
>>> >>>>>
>>> >>>>> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier <tdra...@jahia.com>
>>> >>> wrote:
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>> I'm having some issues with the deployment and redeployment of the
>>> >>> json
>>> >>>>>> files containing default definitions (segments, properties,
>>> etc..).
>>> >>> Many
>>> >>>>>> services have these loadPredefined* methods which load data into
>>> ES,
>>> >>>>> based
>>> >>>>>> on json definition files. These methods only load the content
>>> once,
>>> >>> if
>>> >>>>> the
>>> >>>>>> object does not exist yet in ES. So if a bundle comes with an
>>> update
>>> >>> of
>>> >>>>> one
>>> >>>>>> of these file, it will just be ignored.
>>> >>>>>>
>>> >>>>>> Actually there was one ticket about this :
>>> >>>>>> https://issues.apache.org/jira/browse/UNOMI-182 , where
>>> definitions
>>> >>> are
>>> >>>>>> always redeployed when a bundle is in "snapshot" version - but
>>> never
>>> >>> for
>>> >>>>> a
>>> >>>>>> release version. So that's nice when you are in development, but
>>> not
>>> >>>>>> possible when upgrading a production environment.
>>> >>>>>>
>>> >>>>>> There also a console command to manually deploy a definition -
>>> which
>>> >>> is
>>> >>>>>> also nice but cannot be automated in a migration.
>>> >>>>>>
>>> >>>>>> Of course there's a reason - these objects can be modified
>>> >>> afterwards by
>>> >>>>>> using the API, and so restarting should not overwrite the
>>> >>> modifications
>>> >>>>> by
>>> >>>>>> reimporting the file. Even if a module is updated and object is
>>> >>> modified,
>>> >>>>>> the user may not want to overwrite his changes.
>>> >>>>>>
>>> >>>>>> However in some cases the developer may want to force the update
>>> of
>>> >>> some
>>> >>>>>> specific objects ( it actually happened to us ). I'm looking for
>>> some
>>> >>>>>> additional solution to force redeployment of a definition.
>>> >>>>>>
>>> >>>>>> So far i was thinking of adding a cfg file, which would list the
>>> >>>>> properties
>>> >>>>>> to redeploy. This file would be removed once the properties are
>>> <https://maps.google.com/?q=ould+be+removed+once+the+properties+are&entry=gmail&source=g>
>>> >>> updated,
>>> >>>>> so
>>> >>>>>> that they are reloaded only once and not on every startup.
>>> >>>>>>
>>> >>>>>> As a bonus, I would also need to be able to remove definitions the
>>> >>> same
>>> >>>>>> way. There's no way to do that now except using the REST API , it
>>> >>> could
>>> >>>>> be
>>> >>>>>> good to also have a console command and to be able to remove
>>> >>> definitions
>>> >>>>>> with the same cfg file.
>>> >>>>>>
>>> >>>>>> What do you think .. ?
>>> >>>>>>
>>> >>>>>> Regards,
>>> >>>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Romain Gauthier
>>> >>>>> Product Manager at Jahia
>>> >>>>> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025>
>>> <+33%206%2020%2018%2035%2025>
>>> >>>>> 8 Rue du sentier | 75002 Paris | France
>>> >>>>> <
>>> https://www.jahia.com/about-us/news-events/events-webinars/jahiadays>
>>> >>>>>
>>> >>
>>> >> --
>>> >> Romain Gauthier
>>> >> Product Manager at Jahia
>>> >> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025>
>>> >> 8 Rue du sentier | 75002 Paris | France
>>> >> <https://www.jahia.com/about-us/news-events/events-webinars/jahiadays
>>> >
>>>
>>>
>>>
