Hi, I did some more changes, now patches are read in /META-INF/cxs/patches , and do not have the -patch suffix. They should include an "patchedItemType" property along with the "patchedItemId". That simplifies the code a lot and make things easier to deploy through rest / console. If nobody has anything against it I will merge the branch
Thomas On Wed, Sep 26, 2018 at 5:19 PM Jean-Baptiste Onofré <[email protected]> wrote: > Cool, gonna take a look. > > Regards > JB > > Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier <[email protected]> a > écrit: > >Sorry, I forgot to mention : it's pushed in the patch branch : > >https://github.com/apache/incubator-unomi/tree/patch > > > > > >On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier <[email protected]> > >wrote: > > > >> Hi, > >> > >> I've committed the changes for the patch. The files need to be put in > >the > >> same folder as the definitions ( META-INF/cxs/xxx ), and named with a > >> suffix -patch.json . They will be deployed only once, if the > >definitions > >> was already there. Each patch has an id, so if you need to deploy a > >new > >> patch, just use a new id. The list of deployed patches is stored in > >ES. > >> Patches can also be manually executed with the > >"unomi:deploy-definition" > >> karaf command, or through a rest endpoint at /cxs/patches/apply . > >I've > >> removed the behaviour with -SNAPSHOT - definitions are never > >redeployed, > >> even in snapshot versions, so you'll need to explicitely provide a > >patch to > >> override. > >> > >> The syntax is slightly different from what has been sent previously : > >> > >> { > >> "itemId": "firstName-patch1", > >> "patchedItemId": "firstName", > >> "operation": "patch", > >> "data": [ > >> > >> > >> { > >> "op": "replace", > >> "path": "/defaultValue", > >> "value": "foo" > >> } > >> ] > >> } > >> > >> For an override : > >> > >> { > >> "itemId": "gender-patch1", > >> "patchedItemId": "gender", > >> "operation": "override", > >> "data": { > >> "metadata": { > >> "id": "gender", > >> "name": "Gender", > >> > >> "systemTags": [ > >> "properties", > >> "profileProperties" > >> ] > >> }, > >> "type": "string", > >> "defaultValue": "", > >> "automaticMappingsFrom": [ ], > >> > >> "rank": "105.0" > >> } > >> } > >> > >> Or a remove : > >> > >> { > >> "itemId": "firstName-patch3", > >> "patchedItemId": "firstName", > >> "operation": "remove" > >> } > >> > >> > >> Regards > >> > >> > >> > >> > >> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier <[email protected]> > >wrote: > >> > >>> Hi, > >>> > >>> I did a quick test with json patch and actually it looks quite easy > >to > >>> integrate and seems to fill more use cases than just a simple > >"override". > >>> However I think it can still be useful to have a complete override > >and also > >>> the possisbility to completely remove the definition, so I was > >thinking of > >>> some "migration" or "update" json file wrapping the patch, within > >the same > >>> folder as the definitions : > >>> > >>> { > >>> "itemId": "firstName", > >>> "updateId": "firstName-patch1", > >>> "operation": "patch", > >>> "patch": [ > >>> { > >>> "op": "replace", "path": "/defaultValue", "value": "foo" > >>> } > >>> ] > >>> } > >>> > >>> > >>> A removal : > >>> > >>> { > >>> "itemId": "firstName", > >>> "updateId": "firstName-patch1", > >>> "operation": "remove" > >>> } > >>> > >>> Or a complete override : > >>> > >>> { > >>> "itemId": "firstName", > >>> "updateId": "firstName-patch1", > >>> "operation": "override", > >>> "definition": { > >>> "metadata": { > >>> "id": "firstName", > >>> "name": "First name", > >>> "systemTags": [ > >>> "properties", > >>> "profileProperties", > >>> "basicProfileProperties", > >>> "personalIdentifierProperties" > >>> ] > >>> }, > >>> "type": "string", > >>> "defaultValue": "", > >>> "automaticMappingsFrom": [ ], > >>> "rank": "101.0" > >>> } > >>> } > >>> > >>> > >>> This could actually be extended with other type of operation. > >>> > >>> For now I do store "updateId" in the metadata of the item, so that > >it's > >>> not executed multiple times. But we have have an issue with the > >delete if > >>> the item is recreated after.. Maybe I should store the list of > >updates in a > >>> new structure ? That could also be accessible with karaf commands, > >to see > >>> which patch have been executed and reexecute them if needed. > >>> > >>> Thomas > >>> > >>> > >>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon < > >>> [email protected]> wrote: > >>> > >>>> Hi, > >>>> > >>>> Very interesting flow :) > >>>> > >>>> What is the format to update the definitions, it's just a new JSON, > >is > >>>> there some directives about the changes ? > >>>> > >>>> For example, in Karaf we have something like this to override > >>>> configuration files : > >>>> > >>>> <edits> > >>>> <edit> > >>>> <file>config.properties</file> > >>>> <operation>put</operation> > >>>> <key>karaf.framework</key> > >>>> <value>equinox</value> > >>>> </edit> > >>>> <edit> > >>>> <file>config.properties</file> > >>>> <operation>extend</operation> > >>>> <key>org.osgi.framework.system.capabilities</key> > >>>> <value>my-magic-capability</value> > >>>> </edit> > >>>> <edit> > >>>> <file>config.properties</file> > >>>> <operation>remove</operation> > >>>> <key>org.apache.karaf.security.providers</key> > >>>> </edit> > >>>> </edits> > >>>> > >>>> May be a solution is a file with a patch description of the changes > >? > >>>> > >>>> regards, > >>>> > >>>> François Papon > >>>> [email protected] > >>>> > >>>> Le 18/09/2018 à 13:38, Serge Huber a écrit : > >>>> > Not sure if you are talking about the patching system, but if yes > >you > >>>> > would provide a patch file that removes all the contents of file. > >>>> > > >>>> > If we are just using versions it is indeed a bit more tricky. We > >would > >>>> > also need to have a way to list the definitions to remove. > >>>> > > >>>> > Regards, > >>>> > Serge... > >>>> > > >>>> > On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier > ><[email protected]> > >>>> wrote: > >>>> >> Hi Serge, > >>>> >> > >>>> >> In that context, how would we manage objects deletion from a > >plugin? > >>>> >> > >>>> >> Thanks! > >>>> >> > >>>> >> Romain > >>>> >> > >>>> >> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber <[email protected]> > >>>> wrote: > >>>> >> > >>>> >>> Sorry, I didn't get around to this discussion but I think that > >>>> >>> versions would probably be cleaner. I was even looking at > >solutions > >>>> >>> such as JSON patch > >(https://github.com/java-json-tools/json-patch) > >>>> to > >>>> >>> be able to perform modifications on parts of definitions > >instead of > >>>> >>> having to deploy a whole definition again. > >>>> >>> > >>>> >>> We had discussions about adding lots of metadata, including > >last > >>>> >>> modification dates, maybe even hashes for the definition files, > >to > >>>> >>> know if a definition has been changed or not. > >>>> >>> > >>>> >>> I think that one of the advantages of the patch idea is that > >you > >>>> could > >>>> >>> try to patch something if it's compatible, and if it's not you > >would > >>>> >>> not deploy it. But this idea would mostly be for use cases > >where you > >>>> >>> are interested in modifying something that comes out of the box > >in > >>>> >>> Unomi, not for upgrading a plugin. > >>>> >>> > >>>> >>> With versions at least we could also track if a definition is > >on the > >>>> >>> "default" version or has been modified somehow. > >>>> >>> > >>>> >>> For the cfg idea it feels weird to use a config file that way, > >maybe > >>>> >>> we should put the marker file elsewhere but to be honest I > >don't have > >>>> >>> a better idea :) > >>>> >>> > >>>> >>> Would love some input from JB or other Karaf experts :) > >>>> >>> > >>>> >>> Regards, > >>>> >>> Serge... > >>>> >>> > >>>> >>> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier > ><[email protected]> > >>>> wrote: > >>>> >>>> Hi, > >>>> >>>> > >>>> >>>> I have to check, but the idea would be to be able to package > >this > >>>> cfg > >>>> >>> file > >>>> >>>> beside the bundle. I'm not familiar with unomi:migrate > >command, but > >>>> we > >>>> >>> can > >>>> >>>> imagine that this command would extract and deploy the cfg > >file > >>>> from the > >>>> >>>> bundle - which would make the packaging easier (cfg directly > >inside > >>>> the > >>>> >>>> plugin). The thing is that we need to be able to remove the > >file > >>>> once the > >>>> >>>> definition has been "overridden" so that it's only done once, > >so > >>>> the cfg > >>>> >>>> file should not be deployed automatically on each startup. > >>>> >>>> > >>>> >>>> Another idea I had was to store sort of "version" of the > >definition > >>>> >>> inside > >>>> >>>> the metadata, and check if the json file has a greater > >"version" > >>>> defined. > >>>> >>>> That makes things even more easier when developing plugin and > >looks > >>>> a > >>>> >>>> little more clean, but I don't know if it would cover > >correctly all > >>>> use > >>>> >>>> cases. > >>>> >>>> > >>>> >>>> We should also extends the DeployDefinition shell command to > >be > >>>> able to > >>>> >>>> redeploy all definitions of a bundle. > >>>> >>>> > >>>> >>>> Thomas > >>>> >>>> > >>>> >>>> > >>>> >>>> On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier < > >>>> [email protected]> > >>>> >>> wrote: > >>>> >>>>> Hello Thomas, > >>>> >>>>> > >>>> >>>>> Nice proposition.How would it work for developers that are > >building > >>>> >>>>> plugins? Would we have to add our own cfg file or somehow > >update > >>>> the > >>>> >>> cfg > >>>> >>>>> file that you want to add? > >>>> >>>>> > >>>> >>>>> Since there is already a "unomi:migrate" command would you > >reuse > >>>> that > >>>> >>>>> trigger to update / delete the definitions of the objects? > >>>> >>>>> > >>>> >>>>> Same question regarding the plugins, should we also implement > >>>> "migrate" > >>>> >>>>> command? > >>>> >>>>> > >>>> >>>>> Thanks, > >>>> >>>>> > >>>> >>>>> Romain > >>>> >>>>> > >>>> >>>>> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier > ><[email protected]> > >>>> >>> wrote: > >>>> >>>>>> Hi, > >>>> >>>>>> > >>>> >>>>>> I'm having some issues with the deployment and redeployment > >of the > >>>> >>> json > >>>> >>>>>> files containing default definitions (segments, properties, > >>>> etc..). > >>>> >>> Many > >>>> >>>>>> services have these loadPredefined* methods which load data > >into > >>>> ES, > >>>> >>>>> based > >>>> >>>>>> on json definition files. These methods only load the > >content > >>>> once, > >>>> >>> if > >>>> >>>>> the > >>>> >>>>>> object does not exist yet in ES. So if a bundle comes with > >an > >>>> update > >>>> >>> of > >>>> >>>>> one > >>>> >>>>>> of these file, it will just be ignored. > >>>> >>>>>> > >>>> >>>>>> Actually there was one ticket about this : > >>>> >>>>>> https://issues.apache.org/jira/browse/UNOMI-182 , where > >>>> definitions > >>>> >>> are > >>>> >>>>>> always redeployed when a bundle is in "snapshot" version - > >but > >>>> never > >>>> >>> for > >>>> >>>>> a > >>>> >>>>>> release version. So that's nice when you are in development, > >but > >>>> not > >>>> >>>>>> possible when upgrading a production environment. > >>>> >>>>>> > >>>> >>>>>> There also a console command to manually deploy a definition > >- > >>>> which > >>>> >>> is > >>>> >>>>>> also nice but cannot be automated in a migration. > >>>> >>>>>> > >>>> >>>>>> Of course there's a reason - these objects can be modified > >>>> >>> afterwards by > >>>> >>>>>> using the API, and so restarting should not overwrite the > >>>> >>> modifications > >>>> >>>>> by > >>>> >>>>>> reimporting the file. Even if a module is updated and object > >is > >>>> >>> modified, > >>>> >>>>>> the user may not want to overwrite his changes. > >>>> >>>>>> > >>>> >>>>>> However in some cases the developer may want to force the > >update > >>>> of > >>>> >>> some > >>>> >>>>>> specific objects ( it actually happened to us ). I'm looking > >for > >>>> some > >>>> >>>>>> additional solution to force redeployment of a definition. > >>>> >>>>>> > >>>> >>>>>> So far i was thinking of adding a cfg file, which would list > >the > >>>> >>>>> properties > >>>> >>>>>> to redeploy. This file would be removed once the properties > >are > >>>> > >< > https://maps.google.com/?q=ould+be+removed+once+the+properties+are&entry=gmail&source=g > > > >>>> >>> updated, > >>>> >>>>> so > >>>> >>>>>> that they are reloaded only once and not on every startup. > >>>> >>>>>> > >>>> >>>>>> As a bonus, I would also need to be able to remove > >definitions the > >>>> >>> same > >>>> >>>>>> way. There's no way to do that now except using the REST API > >, it > >>>> >>> could > >>>> >>>>> be > >>>> >>>>>> good to also have a console command and to be able to remove > >>>> >>> definitions > >>>> >>>>>> with the same cfg file. > >>>> >>>>>> > >>>> >>>>>> What do you think .. ? > >>>> >>>>>> > >>>> >>>>>> Regards, > >>>> >>>>>> > >>>> >>>>> > >>>> >>>>> -- > >>>> >>>>> Romain Gauthier > >>>> >>>>> Product Manager at Jahia > >>>> >>>>> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025> > <+33%206%2020%2018%2035%2025> > >>>> <+33%206%2020%2018%2035%2025> > >>>> >>>>> 8 Rue du sentier | 75002 Paris | France > >>>> >>>>> < > >>>> > >https://www.jahia.com/about-us/news-events/events-webinars/jahiadays> > >>>> >>>>> > >>>> >> > >>>> >> -- > >>>> >> Romain Gauthier > >>>> >> Product Manager at Jahia > >>>> >> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025> > <+33%206%2020%2018%2035%2025> > >>>> >> 8 Rue du sentier | 75002 Paris | France > >>>> >> > ><https://www.jahia.com/about-us/news-events/events-webinars/jahiadays > >>>> > > >>>> > >>>> > >>>> >
