+1 to merge. Thanks Thomas, it's a great work.
Regards JB On 01/10/2018 17:07, Thomas Draier wrote: > Hi, > > I did some more changes, now patches are read in /META-INF/cxs/patches , > and do not have the -patch suffix. They should include an "patchedItemType" > property along with the "patchedItemId". That simplifies the code a lot and > make things easier to deploy through rest / console. > If nobody has anything against it I will merge the branch > > Thomas > > > On Wed, Sep 26, 2018 at 5:19 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Cool, gonna take a look. >> >> Regards >> JB >> >> Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier <[email protected]> a >> écrit: >>> Sorry, I forgot to mention : it's pushed in the patch branch : >>> https://github.com/apache/incubator-unomi/tree/patch >>> >>> >>> On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I've committed the changes for the patch. The files need to be put in >>> the >>>> same folder as the definitions ( META-INF/cxs/xxx ), and named with a >>>> suffix -patch.json . They will be deployed only once, if the >>> definitions >>>> was already there. Each patch has an id, so if you need to deploy a >>> new >>>> patch, just use a new id. The list of deployed patches is stored in >>> ES. >>>> Patches can also be manually executed with the >>> "unomi:deploy-definition" >>>> karaf command, or through a rest endpoint at /cxs/patches/apply . >>> I've >>>> removed the behaviour with -SNAPSHOT - definitions are never >>> redeployed, >>>> even in snapshot versions, so you'll need to explicitely provide a >>> patch to >>>> override. >>>> >>>> The syntax is slightly different from what has been sent previously : >>>> >>>> { >>>> "itemId": "firstName-patch1", >>>> "patchedItemId": "firstName", >>>> "operation": "patch", >>>> "data": [ >>>> >>>> >>>> { >>>> "op": "replace", >>>> "path": "/defaultValue", >>>> "value": "foo" >>>> } >>>> ] >>>> } >>>> >>>> For an override : >>>> >>>> { >>>> "itemId": "gender-patch1", >>>> "patchedItemId": "gender", >>>> "operation": "override", >>>> "data": { >>>> "metadata": { >>>> "id": "gender", >>>> "name": "Gender", >>>> >>>> "systemTags": [ >>>> "properties", >>>> "profileProperties" >>>> ] >>>> }, >>>> "type": "string", >>>> "defaultValue": "", >>>> "automaticMappingsFrom": [ ], >>>> >>>> "rank": "105.0" >>>> } >>>> } >>>> >>>> Or a remove : >>>> >>>> { >>>> "itemId": "firstName-patch3", >>>> "patchedItemId": "firstName", >>>> "operation": "remove" >>>> } >>>> >>>> >>>> Regards >>>> >>>> >>>> >>>> >>>> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier <[email protected]> >>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I did a quick test with json patch and actually it looks quite easy >>> to >>>>> integrate and seems to fill more use cases than just a simple >>> "override". >>>>> However I think it can still be useful to have a complete override >>> and also >>>>> the possisbility to completely remove the definition, so I was >>> thinking of >>>>> some "migration" or "update" json file wrapping the patch, within >>> the same >>>>> folder as the definitions : >>>>> >>>>> { >>>>> "itemId": "firstName", >>>>> "updateId": "firstName-patch1", >>>>> "operation": "patch", >>>>> "patch": [ >>>>> { >>>>> "op": "replace", "path": "/defaultValue", "value": "foo" >>>>> } >>>>> ] >>>>> } >>>>> >>>>> >>>>> A removal : >>>>> >>>>> { >>>>> "itemId": "firstName", >>>>> "updateId": "firstName-patch1", >>>>> "operation": "remove" >>>>> } >>>>> >>>>> Or a complete override : >>>>> >>>>> { >>>>> "itemId": "firstName", >>>>> "updateId": "firstName-patch1", >>>>> "operation": "override", >>>>> "definition": { >>>>> "metadata": { >>>>> "id": "firstName", >>>>> "name": "First name", >>>>> "systemTags": [ >>>>> "properties", >>>>> "profileProperties", >>>>> "basicProfileProperties", >>>>> "personalIdentifierProperties" >>>>> ] >>>>> }, >>>>> "type": "string", >>>>> "defaultValue": "", >>>>> "automaticMappingsFrom": [ ], >>>>> "rank": "101.0" >>>>> } >>>>> } >>>>> >>>>> >>>>> This could actually be extended with other type of operation. >>>>> >>>>> For now I do store "updateId" in the metadata of the item, so that >>> it's >>>>> not executed multiple times. But we have have an issue with the >>> delete if >>>>> the item is recreated after.. Maybe I should store the list of >>> updates in a >>>>> new structure ? That could also be accessible with karaf commands, >>> to see >>>>> which patch have been executed and reexecute them if needed. >>>>> >>>>> Thomas >>>>> >>>>> >>>>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Very interesting flow :) >>>>>> >>>>>> What is the format to update the definitions, it's just a new JSON, >>> is >>>>>> there some directives about the changes ? >>>>>> >>>>>> For example, in Karaf we have something like this to override >>>>>> configuration files : >>>>>> >>>>>> <edits> >>>>>> <edit> >>>>>> <file>config.properties</file> >>>>>> <operation>put</operation> >>>>>> <key>karaf.framework</key> >>>>>> <value>equinox</value> >>>>>> </edit> >>>>>> <edit> >>>>>> <file>config.properties</file> >>>>>> <operation>extend</operation> >>>>>> <key>org.osgi.framework.system.capabilities</key> >>>>>> <value>my-magic-capability</value> >>>>>> </edit> >>>>>> <edit> >>>>>> <file>config.properties</file> >>>>>> <operation>remove</operation> >>>>>> <key>org.apache.karaf.security.providers</key> >>>>>> </edit> >>>>>> </edits> >>>>>> >>>>>> May be a solution is a file with a patch description of the changes >>> ? >>>>>> >>>>>> regards, >>>>>> >>>>>> François Papon >>>>>> [email protected] >>>>>> >>>>>> Le 18/09/2018 à 13:38, Serge Huber a écrit : >>>>>>> Not sure if you are talking about the patching system, but if yes >>> you >>>>>>> would provide a patch file that removes all the contents of file. >>>>>>> >>>>>>> If we are just using versions it is indeed a bit more tricky. We >>> would >>>>>>> also need to have a way to list the definitions to remove. >>>>>>> >>>>>>> Regards, >>>>>>> Serge... >>>>>>> >>>>>>> On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier >>> <[email protected]> >>>>>> wrote: >>>>>>>> Hi Serge, >>>>>>>> >>>>>>>> In that context, how would we manage objects deletion from a >>> plugin? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Romain >>>>>>>> >>>>>>>> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber <[email protected]> >>>>>> wrote: >>>>>>>> >>>>>>>>> Sorry, I didn't get around to this discussion but I think that >>>>>>>>> versions would probably be cleaner. I was even looking at >>> solutions >>>>>>>>> such as JSON patch >>> (https://github.com/java-json-tools/json-patch) >>>>>> to >>>>>>>>> be able to perform modifications on parts of definitions >>> instead of >>>>>>>>> having to deploy a whole definition again. >>>>>>>>> >>>>>>>>> We had discussions about adding lots of metadata, including >>> last >>>>>>>>> modification dates, maybe even hashes for the definition files, >>> to >>>>>>>>> know if a definition has been changed or not. >>>>>>>>> >>>>>>>>> I think that one of the advantages of the patch idea is that >>> you >>>>>> could >>>>>>>>> try to patch something if it's compatible, and if it's not you >>> would >>>>>>>>> not deploy it. But this idea would mostly be for use cases >>> where you >>>>>>>>> are interested in modifying something that comes out of the box >>> in >>>>>>>>> Unomi, not for upgrading a plugin. >>>>>>>>> >>>>>>>>> With versions at least we could also track if a definition is >>> on the >>>>>>>>> "default" version or has been modified somehow. >>>>>>>>> >>>>>>>>> For the cfg idea it feels weird to use a config file that way, >>> maybe >>>>>>>>> we should put the marker file elsewhere but to be honest I >>> don't have >>>>>>>>> a better idea :) >>>>>>>>> >>>>>>>>> Would love some input from JB or other Karaf experts :) >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Serge... >>>>>>>>> >>>>>>>>> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier >>> <[email protected]> >>>>>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have to check, but the idea would be to be able to package >>> this >>>>>> cfg >>>>>>>>> file >>>>>>>>>> beside the bundle. I'm not familiar with unomi:migrate >>> command, but >>>>>> we >>>>>>>>> can >>>>>>>>>> imagine that this command would extract and deploy the cfg >>> file >>>>>> from the >>>>>>>>>> bundle - which would make the packaging easier (cfg directly >>> inside >>>>>> the >>>>>>>>>> plugin). The thing is that we need to be able to remove the >>> file >>>>>> once the >>>>>>>>>> definition has been "overridden" so that it's only done once, >>> so >>>>>> the cfg >>>>>>>>>> file should not be deployed automatically on each startup. >>>>>>>>>> >>>>>>>>>> Another idea I had was to store sort of "version" of the >>> definition >>>>>>>>> inside >>>>>>>>>> the metadata, and check if the json file has a greater >>> "version" >>>>>> defined. >>>>>>>>>> That makes things even more easier when developing plugin and >>> looks >>>>>> a >>>>>>>>>> little more clean, but I don't know if it would cover >>> correctly all >>>>>> use >>>>>>>>>> cases. >>>>>>>>>> >>>>>>>>>> We should also extends the DeployDefinition shell command to >>> be >>>>>> able to >>>>>>>>>> redeploy all definitions of a bundle. >>>>>>>>>> >>>>>>>>>> Thomas >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier < >>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>>> Hello Thomas, >>>>>>>>>>> >>>>>>>>>>> Nice proposition.How would it work for developers that are >>> building >>>>>>>>>>> plugins? Would we have to add our own cfg file or somehow >>> update >>>>>> the >>>>>>>>> cfg >>>>>>>>>>> file that you want to add? >>>>>>>>>>> >>>>>>>>>>> Since there is already a "unomi:migrate" command would you >>> reuse >>>>>> that >>>>>>>>>>> trigger to update / delete the definitions of the objects? >>>>>>>>>>> >>>>>>>>>>> Same question regarding the plugins, should we also implement >>>>>> "migrate" >>>>>>>>>>> command? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Romain >>>>>>>>>>> >>>>>>>>>>> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier >>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I'm having some issues with the deployment and redeployment >>> of the >>>>>>>>> json >>>>>>>>>>>> files containing default definitions (segments, properties, >>>>>> etc..). >>>>>>>>> Many >>>>>>>>>>>> services have these loadPredefined* methods which load data >>> into >>>>>> ES, >>>>>>>>>>> based >>>>>>>>>>>> on json definition files. These methods only load the >>> content >>>>>> once, >>>>>>>>> if >>>>>>>>>>> the >>>>>>>>>>>> object does not exist yet in ES. So if a bundle comes with >>> an >>>>>> update >>>>>>>>> of >>>>>>>>>>> one >>>>>>>>>>>> of these file, it will just be ignored. >>>>>>>>>>>> >>>>>>>>>>>> Actually there was one ticket about this : >>>>>>>>>>>> https://issues.apache.org/jira/browse/UNOMI-182 , where >>>>>> definitions >>>>>>>>> are >>>>>>>>>>>> always redeployed when a bundle is in "snapshot" version - >>> but >>>>>> never >>>>>>>>> for >>>>>>>>>>> a >>>>>>>>>>>> release version. So that's nice when you are in development, >>> but >>>>>> not >>>>>>>>>>>> possible when upgrading a production environment. >>>>>>>>>>>> >>>>>>>>>>>> There also a console command to manually deploy a definition >>> - >>>>>> which >>>>>>>>> is >>>>>>>>>>>> also nice but cannot be automated in a migration. >>>>>>>>>>>> >>>>>>>>>>>> Of course there's a reason - these objects can be modified >>>>>>>>> afterwards by >>>>>>>>>>>> using the API, and so restarting should not overwrite the >>>>>>>>> modifications >>>>>>>>>>> by >>>>>>>>>>>> reimporting the file. Even if a module is updated and object >>> is >>>>>>>>> modified, >>>>>>>>>>>> the user may not want to overwrite his changes. >>>>>>>>>>>> >>>>>>>>>>>> However in some cases the developer may want to force the >>> update >>>>>> of >>>>>>>>> some >>>>>>>>>>>> specific objects ( it actually happened to us ). I'm looking >>> for >>>>>> some >>>>>>>>>>>> additional solution to force redeployment of a definition. >>>>>>>>>>>> >>>>>>>>>>>> So far i was thinking of adding a cfg file, which would list >>> the >>>>>>>>>>> properties >>>>>>>>>>>> to redeploy. This file would be removed once the properties >>> are >>>>>> >>> < >> https://maps.google.com/?q=ould+be+removed+once+the+properties+are&entry=gmail&source=g >>> >>>>>>>>> updated, >>>>>>>>>>> so >>>>>>>>>>>> that they are reloaded only once and not on every startup. >>>>>>>>>>>> >>>>>>>>>>>> As a bonus, I would also need to be able to remove >>> definitions the >>>>>>>>> same >>>>>>>>>>>> way. There's no way to do that now except using the REST API >>> , it >>>>>>>>> could >>>>>>>>>>> be >>>>>>>>>>>> good to also have a console command and to be able to remove >>>>>>>>> definitions >>>>>>>>>>>> with the same cfg file. >>>>>>>>>>>> >>>>>>>>>>>> What do you think .. ? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Romain Gauthier >>>>>>>>>>> Product Manager at Jahia >>>>>>>>>>> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025> >> <+33%206%2020%2018%2035%2025> >>>>>> <+33%206%2020%2018%2035%2025> >>>>>>>>>>> 8 Rue du sentier | 75002 Paris | France >>>>>>>>>>> < >>>>>> >>> https://www.jahia.com/about-us/news-events/events-webinars/jahiadays> >>>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Romain Gauthier >>>>>>>> Product Manager at Jahia >>>>>>>> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025> >> <+33%206%2020%2018%2035%2025> >>>>>>>> 8 Rue du sentier | 75002 Paris | France >>>>>>>> >>> <https://www.jahia.com/about-us/news-events/events-webinars/jahiadays >>>>>>> >>>>>> >>>>>> >>>>>> >> > -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
