Hi,
I did a quick test with json patch and actually it looks quite easy to
integrate and seems to fill more use cases than just a simple "override".
However I think it can still be useful to have a complete override and also
the possisbility to completely remove the definition, so I was thinking of
some "migration" or "update" json file wrapping the patch, within the same
folder as the definitions :
{
"itemId": "firstName",
"updateId": "firstName-patch1",
"operation": "patch",
"patch": [
{
"op": "replace", "path": "/defaultValue", "value": "foo"
}
]
}
A removal :
{
"itemId": "firstName",
"updateId": "firstName-patch1",
"operation": "remove"
}
Or a complete override :
{
"itemId": "firstName",
"updateId": "firstName-patch1",
"operation": "override",
"definition": {
"metadata": {
"id": "firstName",
"name": "First name",
"systemTags": [
"properties",
"profileProperties",
"basicProfileProperties",
"personalIdentifierProperties"
]
},
"type": "string",
"defaultValue": "",
"automaticMappingsFrom": [ ],
"rank": "101.0"
}
}
This could actually be extended with other type of operation.
For now I do store "updateId" in the metadata of the item, so that it's not
executed multiple times. But we have have an issue with the delete if the
item is recreated after.. Maybe I should store the list of updates in a new
structure ? That could also be accessible with karaf commands, to see which
patch have been executed and reexecute them if needed.
Thomas
On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
[email protected]> wrote:
> Hi,
>
> Very interesting flow :)
>
> What is the format to update the definitions, it's just a new JSON, is
> there some directives about the changes ?
>
> For example, in Karaf we have something like this to override
> configuration files :
>
> <edits>
> <edit>
> <file>config.properties</file>
> <operation>put</operation>
> <key>karaf.framework</key>
> <value>equinox</value>
> </edit>
> <edit>
> <file>config.properties</file>
> <operation>extend</operation>
> <key>org.osgi.framework.system.capabilities</key>
> <value>my-magic-capability</value>
> </edit>
> <edit>
> <file>config.properties</file>
> <operation>remove</operation>
> <key>org.apache.karaf.security.providers</key>
> </edit>
> </edits>
>
> May be a solution is a file with a patch description of the changes ?
>
> regards,
>
> François Papon
> [email protected]
>
> Le 18/09/2018 à 13:38, Serge Huber a écrit :
> > Not sure if you are talking about the patching system, but if yes you
> > would provide a patch file that removes all the contents of file.
> >
> > If we are just using versions it is indeed a bit more tricky. We would
> > also need to have a way to list the definitions to remove.
> >
> > Regards,
> > Serge...
> >
> > On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier <[email protected]>
> wrote:
> >> Hi Serge,
> >>
> >> In that context, how would we manage objects deletion from a plugin?
> >>
> >> Thanks!
> >>
> >> Romain
> >>
> >> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber <[email protected]> wrote:
> >>
> >>> Sorry, I didn't get around to this discussion but I think that
> >>> versions would probably be cleaner. I was even looking at solutions
> >>> such as JSON patch (https://github.com/java-json-tools/json-patch) to
> >>> be able to perform modifications on parts of definitions instead of
> >>> having to deploy a whole definition again.
> >>>
> >>> We had discussions about adding lots of metadata, including last
> >>> modification dates, maybe even hashes for the definition files, to
> >>> know if a definition has been changed or not.
> >>>
> >>> I think that one of the advantages of the patch idea is that you could
> >>> try to patch something if it's compatible, and if it's not you would
> >>> not deploy it. But this idea would mostly be for use cases where you
> >>> are interested in modifying something that comes out of the box in
> >>> Unomi, not for upgrading a plugin.
> >>>
> >>> With versions at least we could also track if a definition is on the
> >>> "default" version or has been modified somehow.
> >>>
> >>> For the cfg idea it feels weird to use a config file that way, maybe
> >>> we should put the marker file elsewhere but to be honest I don't have
> >>> a better idea :)
> >>>
> >>> Would love some input from JB or other Karaf experts :)
> >>>
> >>> Regards,
> >>> Serge...
> >>>
> >>> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier <[email protected]>
> wrote:
> >>>> Hi,
> >>>>
> >>>> I have to check, but the idea would be to be able to package this cfg
> >>> file
> >>>> beside the bundle. I'm not familiar with unomi:migrate command, but we
> >>> can
> >>>> imagine that this command would extract and deploy the cfg file from
> the
> >>>> bundle - which would make the packaging easier (cfg directly inside
> the
> >>>> plugin). The thing is that we need to be able to remove the file once
> the
> >>>> definition has been "overridden" so that it's only done once, so the
> cfg
> >>>> file should not be deployed automatically on each startup.
> >>>>
> >>>> Another idea I had was to store sort of "version" of the definition
> >>> inside
> >>>> the metadata, and check if the json file has a greater "version"
> defined.
> >>>> That makes things even more easier when developing plugin and looks a
> >>>> little more clean, but I don't know if it would cover correctly all
> use
> >>>> cases.
> >>>>
> >>>> We should also extends the DeployDefinition shell command to be able
> to
> >>>> redeploy all definitions of a bundle.
> >>>>
> >>>> Thomas
> >>>>
> >>>>
> >>>> On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier <[email protected]>
> >>> wrote:
> >>>>> Hello Thomas,
> >>>>>
> >>>>> Nice proposition.How would it work for developers that are building
> >>>>> plugins? Would we have to add our own cfg file or somehow update the
> >>> cfg
> >>>>> file that you want to add?
> >>>>>
> >>>>> Since there is already a "unomi:migrate" command would you reuse that
> >>>>> trigger to update / delete the definitions of the objects?
> >>>>>
> >>>>> Same question regarding the plugins, should we also implement
> "migrate"
> >>>>> command?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Romain
> >>>>>
> >>>>> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier <[email protected]>
> >>> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'm having some issues with the deployment and redeployment of the
> >>> json
> >>>>>> files containing default definitions (segments, properties, etc..).
> >>> Many
> >>>>>> services have these loadPredefined* methods which load data into ES,
> >>>>> based
> >>>>>> on json definition files. These methods only load the content once,
> >>> if
> >>>>> the
> >>>>>> object does not exist yet in ES. So if a bundle comes with an update
> >>> of
> >>>>> one
> >>>>>> of these file, it will just be ignored.
> >>>>>>
> >>>>>> Actually there was one ticket about this :
> >>>>>> https://issues.apache.org/jira/browse/UNOMI-182 , where definitions
> >>> are
> >>>>>> always redeployed when a bundle is in "snapshot" version - but never
> >>> for
> >>>>> a
> >>>>>> release version. So that's nice when you are in development, but not
> >>>>>> possible when upgrading a production environment.
> >>>>>>
> >>>>>> There also a console command to manually deploy a definition - which
> >>> is
> >>>>>> also nice but cannot be automated in a migration.
> >>>>>>
> >>>>>> Of course there's a reason - these objects can be modified
> >>> afterwards by
> >>>>>> using the API, and so restarting should not overwrite the
> >>> modifications
> >>>>> by
> >>>>>> reimporting the file. Even if a module is updated and object is
> >>> modified,
> >>>>>> the user may not want to overwrite his changes.
> >>>>>>
> >>>>>> However in some cases the developer may want to force the update of
> >>> some
> >>>>>> specific objects ( it actually happened to us ). I'm looking for
> some
> >>>>>> additional solution to force redeployment of a definition.
> >>>>>>
> >>>>>> So far i was thinking of adding a cfg file, which would list the
> >>>>> properties
> >>>>>> to redeploy. This file would be removed once the properties are
> <https://maps.google.com/?q=ould+be+removed+once+the+properties+are&entry=gmail&source=g>
> >>> updated,
> >>>>> so
> >>>>>> that they are reloaded only once and not on every startup.
> >>>>>>
> >>>>>> As a bonus, I would also need to be able to remove definitions the
> >>> same
> >>>>>> way. There's no way to do that now except using the REST API , it
> >>> could
> >>>>> be
> >>>>>> good to also have a console command and to be able to remove
> >>> definitions
> >>>>>> with the same cfg file.
> >>>>>>
> >>>>>> What do you think .. ?
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> Romain Gauthier
> >>>>> Product Manager at Jahia
> >>>>> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025>
> <+33%206%2020%2018%2035%2025>
> >>>>> 8 Rue du sentier | 75002 Paris | France
> >>>>> <
> https://www.jahia.com/about-us/news-events/events-webinars/jahiadays>
> >>>>>
> >>
> >> --
> >> Romain Gauthier
> >> Product Manager at Jahia
> >> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025>
> >> 8 Rue du sentier | 75002 Paris | France
> >> <https://www.jahia.com/about-us/news-events/events-webinars/jahiadays>
>
>
>
