Re: Override/migration of predefined definition files

[Thomas Draier]

Ok, merged

thomas

On Mon, Oct 1, 2018 at 6:53 PM Jean-Baptiste Onofré  wrote:

> +1 to merge.
>
> Thanks Thomas, it's a great work.
>
> Regards
> JB
>
> On 01/10/2018 17:07, Thomas Draier wrote:
> > Hi,
> >
> > I did some more changes, now patches are read in /META-INF/cxs/patches ,
> > and do not have the -patch suffix. They should include an
> "patchedItemType"
> > property along with the "patchedItemId". That simplifies the code a lot
> and
> > make things easier to deploy through rest / console.
> > If nobody has anything against it I will merge the branch
> >
> > Thomas
> >
> >
> > On Wed, Sep 26, 2018 at 5:19 PM Jean-Baptiste Onofré 
> > wrote:
> >
> >> Cool, gonna take a look.
> >>
> >> Regards
> >> JB
> >>
> >> Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier  a
> >> écrit:
> >>> Sorry, I forgot to mention : it's pushed in the patch branch :
> >>> https://github.com/apache/incubator-unomi/tree/patch
> >>>
> >>>
> >>> On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier 
> >>> wrote:
> >>>
>  Hi,
> 
>  I've committed the changes for the patch. The files need to be put in
> >>> the
>  same folder as the definitions ( META-INF/cxs/xxx ), and named with a
>  suffix -patch.json . They will be deployed only once, if the
> >>> definitions
>  was already there. Each patch has an id, so if you need to deploy a
> >>> new
>  patch, just use a new id. The list of deployed patches is stored in
> >>> ES.
>  Patches can also be manually executed with the
> >>> "unomi:deploy-definition"
>  karaf command, or through a rest endpoint at /cxs/patches/apply .
> >>> I've
>  removed the behaviour with -SNAPSHOT - definitions are never
> >>> redeployed,
>  even in snapshot versions, so you'll need to explicitely provide a
> >>> patch to
>  override.
> 
>  The syntax is slightly different from what has been sent previously :
> 
>  {
>    "itemId": "firstName-patch1",
>    "patchedItemId": "firstName",
>    "operation": "patch",
>    "data": [
> 
> 
>  {
>    "op": "replace",
>    "path": "/defaultValue",
>    "value": "foo"
>  }
>    ]
>  }
> 
>  For an override :
> 
>  {
>    "itemId": "gender-patch1",
>    "patchedItemId": "gender",
>    "operation": "override",
>    "data": {
>  "metadata": {
>    "id": "gender",
>    "name": "Gender",
> 
>    "systemTags": [
>  "properties",
>  "profileProperties"
>    ]
>  },
>  "type": "string",
>  "defaultValue": "",
>  "automaticMappingsFrom": [ ],
> 
>  "rank": "105.0"
>    }
>  }
> 
>  Or a remove :
> 
>  {
>    "itemId": "firstName-patch3",
>    "patchedItemId": "firstName",
>    "operation": "remove"
>  }
> 
> 
>  Regards
> 
> 
> 
> 
>  On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier 
> >>> wrote:
> 
> > Hi,
> >
> > I did a quick test with json patch and actually it looks quite easy
> >>> to
> > integrate and seems to fill more use cases than just a simple
> >>> "override".
> > However I think it can still be useful to have a complete override
> >>> and also
> > the possisbility to completely remove the definition, so I was
> >>> thinking of
> > some "migration" or "update" json file wrapping the patch, within
> >>> the same
> > folder as the definitions :
> >
> > {
> >   "itemId": "firstName",
> >   "updateId": "firstName-patch1",
> >   "operation": "patch",
> >   "patch": [
> > {
> >   "op": "replace", "path": "/defaultValue", "value": "foo"
> > }
> >   ]
> > }
> >
> >
> > A removal :
> >
> > {
> >   "itemId": "firstName",
> >   "updateId": "firstName-patch1",
> >   "operation": "remove"
> > }
> >
> > Or a complete override :
> >
> > {
> >   "itemId": "firstName",
> >   "updateId": "firstName-patch1",
> >   "operation": "override",
> >   "definition": {
> > "metadata": {
> >   "id": "firstName",
> >   "name": "First name",
> >   "systemTags": [
> > "properties",
> > "profileProperties",
> > "basicProfileProperties",
> > "personalIdentifierProperties"
> >   ]
> > },
> > "type": "string",
> > "defaultValue": "",
> > "automaticMappingsFrom": [ ],
> > "rank": "101.0"
> >   }
> > }
> >
> >
> > This could actually be extended with other type of operation.
> >
> > For now I do store "updateId" in the metadata of the item, so that
> >>> it's
> > not executed multiple times. But we have have an issue with the
> >>> delete if
> > the item is recreated after.. Maybe I should store the list of
> >>> updates in a
> > new 

Re: Override/migration of predefined definition files

[Jean-Baptiste Onofré]

+1 to merge.

Thanks Thomas, it's a great work.

Regards
JB

On 01/10/2018 17:07, Thomas Draier wrote:
> Hi,
> 
> I did some more changes, now patches are read in /META-INF/cxs/patches ,
> and do not have the -patch suffix. They should include an "patchedItemType"
> property along with the "patchedItemId". That simplifies the code a lot and
> make things easier to deploy through rest / console.
> If nobody has anything against it I will merge the branch
> 
> Thomas
> 
> 
> On Wed, Sep 26, 2018 at 5:19 PM Jean-Baptiste Onofré 
> wrote:
> 
>> Cool, gonna take a look.
>>
>> Regards
>> JB
>>
>> Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier  a
>> écrit:
>>> Sorry, I forgot to mention : it's pushed in the patch branch :
>>> https://github.com/apache/incubator-unomi/tree/patch
>>>
>>>
>>> On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier 
>>> wrote:
>>>
 Hi,

 I've committed the changes for the patch. The files need to be put in
>>> the
 same folder as the definitions ( META-INF/cxs/xxx ), and named with a
 suffix -patch.json . They will be deployed only once, if the
>>> definitions
 was already there. Each patch has an id, so if you need to deploy a
>>> new
 patch, just use a new id. The list of deployed patches is stored in
>>> ES.
 Patches can also be manually executed with the
>>> "unomi:deploy-definition"
 karaf command, or through a rest endpoint at /cxs/patches/apply .
>>> I've
 removed the behaviour with -SNAPSHOT - definitions are never
>>> redeployed,
 even in snapshot versions, so you'll need to explicitely provide a
>>> patch to
 override.

 The syntax is slightly different from what has been sent previously :

 {
   "itemId": "firstName-patch1",
   "patchedItemId": "firstName",
   "operation": "patch",
   "data": [


 {
   "op": "replace",
   "path": "/defaultValue",
   "value": "foo"
 }
   ]
 }

 For an override :

 {
   "itemId": "gender-patch1",
   "patchedItemId": "gender",
   "operation": "override",
   "data": {
 "metadata": {
   "id": "gender",
   "name": "Gender",

   "systemTags": [
 "properties",
 "profileProperties"
   ]
 },
 "type": "string",
 "defaultValue": "",
 "automaticMappingsFrom": [ ],

 "rank": "105.0"
   }
 }

 Or a remove :

 {
   "itemId": "firstName-patch3",
   "patchedItemId": "firstName",
   "operation": "remove"
 }


 Regards




 On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier 
>>> wrote:

> Hi,
>
> I did a quick test with json patch and actually it looks quite easy
>>> to
> integrate and seems to fill more use cases than just a simple
>>> "override".
> However I think it can still be useful to have a complete override
>>> and also
> the possisbility to completely remove the definition, so I was
>>> thinking of
> some "migration" or "update" json file wrapping the patch, within
>>> the same
> folder as the definitions :
>
> {
>   "itemId": "firstName",
>   "updateId": "firstName-patch1",
>   "operation": "patch",
>   "patch": [
> {
>   "op": "replace", "path": "/defaultValue", "value": "foo"
> }
>   ]
> }
>
>
> A removal :
>
> {
>   "itemId": "firstName",
>   "updateId": "firstName-patch1",
>   "operation": "remove"
> }
>
> Or a complete override :
>
> {
>   "itemId": "firstName",
>   "updateId": "firstName-patch1",
>   "operation": "override",
>   "definition": {
> "metadata": {
>   "id": "firstName",
>   "name": "First name",
>   "systemTags": [
> "properties",
> "profileProperties",
> "basicProfileProperties",
> "personalIdentifierProperties"
>   ]
> },
> "type": "string",
> "defaultValue": "",
> "automaticMappingsFrom": [ ],
> "rank": "101.0"
>   }
> }
>
>
> This could actually be extended with other type of operation.
>
> For now I do store "updateId" in the metadata of the item, so that
>>> it's
> not executed multiple times. But we have have an issue with the
>>> delete if
> the item is recreated after.. Maybe I should store the list of
>>> updates in a
> new structure ? That could also be accessible with karaf commands,
>>> to see
> which patch have been executed and reexecute them if needed.
>
> Thomas
>
>
> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
> francois.pa...@openobject.fr> wrote:
>
>> Hi,
>>
>> Very interesting flow :)
>>
>> What is the format to update the definitions, it's just a new JSON,
>>> is
>> there some directives 

Re: Override/migration of predefined definition files

[Serge Huber]

I agree you should merge the branch.

Happy to see the patches idea was proven to be something useful!

cheers,
  Serge…
On Mon, Oct 1, 2018 at 5:08 PM Thomas Draier  wrote:
>
> Hi,
>
> I did some more changes, now patches are read in /META-INF/cxs/patches ,
> and do not have the -patch suffix. They should include an "patchedItemType"
> property along with the "patchedItemId". That simplifies the code a lot and
> make things easier to deploy through rest / console.
> If nobody has anything against it I will merge the branch
>
> Thomas
>
>
> On Wed, Sep 26, 2018 at 5:19 PM Jean-Baptiste Onofré 
> wrote:
>
> > Cool, gonna take a look.
> >
> > Regards
> > JB
> >
> > Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier  a
> > écrit:
> > >Sorry, I forgot to mention : it's pushed in the patch branch :
> > >https://github.com/apache/incubator-unomi/tree/patch
> > >
> > >
> > >On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier 
> > >wrote:
> > >
> > >> Hi,
> > >>
> > >> I've committed the changes for the patch. The files need to be put in
> > >the
> > >> same folder as the definitions ( META-INF/cxs/xxx ), and named with a
> > >> suffix -patch.json . They will be deployed only once, if the
> > >definitions
> > >> was already there. Each patch has an id, so if you need to deploy a
> > >new
> > >> patch, just use a new id. The list of deployed patches is stored in
> > >ES.
> > >> Patches can also be manually executed with the
> > >"unomi:deploy-definition"
> > >> karaf command, or through a rest endpoint at /cxs/patches/apply .
> > >I've
> > >> removed the behaviour with -SNAPSHOT - definitions are never
> > >redeployed,
> > >> even in snapshot versions, so you'll need to explicitely provide a
> > >patch to
> > >> override.
> > >>
> > >> The syntax is slightly different from what has been sent previously :
> > >>
> > >> {
> > >>   "itemId": "firstName-patch1",
> > >>   "patchedItemId": "firstName",
> > >>   "operation": "patch",
> > >>   "data": [
> > >>
> > >>
> > >> {
> > >>   "op": "replace",
> > >>   "path": "/defaultValue",
> > >>   "value": "foo"
> > >> }
> > >>   ]
> > >> }
> > >>
> > >> For an override :
> > >>
> > >> {
> > >>   "itemId": "gender-patch1",
> > >>   "patchedItemId": "gender",
> > >>   "operation": "override",
> > >>   "data": {
> > >> "metadata": {
> > >>   "id": "gender",
> > >>   "name": "Gender",
> > >>
> > >>   "systemTags": [
> > >> "properties",
> > >> "profileProperties"
> > >>   ]
> > >> },
> > >> "type": "string",
> > >> "defaultValue": "",
> > >> "automaticMappingsFrom": [ ],
> > >>
> > >> "rank": "105.0"
> > >>   }
> > >> }
> > >>
> > >> Or a remove :
> > >>
> > >> {
> > >>   "itemId": "firstName-patch3",
> > >>   "patchedItemId": "firstName",
> > >>   "operation": "remove"
> > >> }
> > >>
> > >>
> > >> Regards
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier 
> > >wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I did a quick test with json patch and actually it looks quite easy
> > >to
> > >>> integrate and seems to fill more use cases than just a simple
> > >"override".
> > >>> However I think it can still be useful to have a complete override
> > >and also
> > >>> the possisbility to completely remove the definition, so I was
> > >thinking of
> > >>> some "migration" or "update" json file wrapping the patch, within
> > >the same
> > >>> folder as the definitions :
> > >>>
> > >>> {
> > >>>   "itemId": "firstName",
> > >>>   "updateId": "firstName-patch1",
> > >>>   "operation": "patch",
> > >>>   "patch": [
> > >>> {
> > >>>   "op": "replace", "path": "/defaultValue", "value": "foo"
> > >>> }
> > >>>   ]
> > >>> }
> > >>>
> > >>>
> > >>> A removal :
> > >>>
> > >>> {
> > >>>   "itemId": "firstName",
> > >>>   "updateId": "firstName-patch1",
> > >>>   "operation": "remove"
> > >>> }
> > >>>
> > >>> Or a complete override :
> > >>>
> > >>> {
> > >>>   "itemId": "firstName",
> > >>>   "updateId": "firstName-patch1",
> > >>>   "operation": "override",
> > >>>   "definition": {
> > >>> "metadata": {
> > >>>   "id": "firstName",
> > >>>   "name": "First name",
> > >>>   "systemTags": [
> > >>> "properties",
> > >>> "profileProperties",
> > >>> "basicProfileProperties",
> > >>> "personalIdentifierProperties"
> > >>>   ]
> > >>> },
> > >>> "type": "string",
> > >>> "defaultValue": "",
> > >>> "automaticMappingsFrom": [ ],
> > >>> "rank": "101.0"
> > >>>   }
> > >>> }
> > >>>
> > >>>
> > >>> This could actually be extended with other type of operation.
> > >>>
> > >>> For now I do store "updateId" in the metadata of the item, so that
> > >it's
> > >>> not executed multiple times. But we have have an issue with the
> > >delete if
> > >>> the item is recreated after.. Maybe I should store the list of
> > >updates in a
> > >>> new structure ? That could also be accessible with karaf commands,
> > >to see
> > >>> 

Re: Override/migration of predefined definition files

[Thomas Draier]

Hi,

I did some more changes, now patches are read in /META-INF/cxs/patches ,
and do not have the -patch suffix. They should include an "patchedItemType"
property along with the "patchedItemId". That simplifies the code a lot and
make things easier to deploy through rest / console.
If nobody has anything against it I will merge the branch

Thomas


On Wed, Sep 26, 2018 at 5:19 PM Jean-Baptiste Onofré 
wrote:

> Cool, gonna take a look.
>
> Regards
> JB
>
> Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier  a
> écrit:
> >Sorry, I forgot to mention : it's pushed in the patch branch :
> >https://github.com/apache/incubator-unomi/tree/patch
> >
> >
> >On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier 
> >wrote:
> >
> >> Hi,
> >>
> >> I've committed the changes for the patch. The files need to be put in
> >the
> >> same folder as the definitions ( META-INF/cxs/xxx ), and named with a
> >> suffix -patch.json . They will be deployed only once, if the
> >definitions
> >> was already there. Each patch has an id, so if you need to deploy a
> >new
> >> patch, just use a new id. The list of deployed patches is stored in
> >ES.
> >> Patches can also be manually executed with the
> >"unomi:deploy-definition"
> >> karaf command, or through a rest endpoint at /cxs/patches/apply .
> >I've
> >> removed the behaviour with -SNAPSHOT - definitions are never
> >redeployed,
> >> even in snapshot versions, so you'll need to explicitely provide a
> >patch to
> >> override.
> >>
> >> The syntax is slightly different from what has been sent previously :
> >>
> >> {
> >>   "itemId": "firstName-patch1",
> >>   "patchedItemId": "firstName",
> >>   "operation": "patch",
> >>   "data": [
> >>
> >>
> >> {
> >>   "op": "replace",
> >>   "path": "/defaultValue",
> >>   "value": "foo"
> >> }
> >>   ]
> >> }
> >>
> >> For an override :
> >>
> >> {
> >>   "itemId": "gender-patch1",
> >>   "patchedItemId": "gender",
> >>   "operation": "override",
> >>   "data": {
> >> "metadata": {
> >>   "id": "gender",
> >>   "name": "Gender",
> >>
> >>   "systemTags": [
> >> "properties",
> >> "profileProperties"
> >>   ]
> >> },
> >> "type": "string",
> >> "defaultValue": "",
> >> "automaticMappingsFrom": [ ],
> >>
> >> "rank": "105.0"
> >>   }
> >> }
> >>
> >> Or a remove :
> >>
> >> {
> >>   "itemId": "firstName-patch3",
> >>   "patchedItemId": "firstName",
> >>   "operation": "remove"
> >> }
> >>
> >>
> >> Regards
> >>
> >>
> >>
> >>
> >> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier 
> >wrote:
> >>
> >>> Hi,
> >>>
> >>> I did a quick test with json patch and actually it looks quite easy
> >to
> >>> integrate and seems to fill more use cases than just a simple
> >"override".
> >>> However I think it can still be useful to have a complete override
> >and also
> >>> the possisbility to completely remove the definition, so I was
> >thinking of
> >>> some "migration" or "update" json file wrapping the patch, within
> >the same
> >>> folder as the definitions :
> >>>
> >>> {
> >>>   "itemId": "firstName",
> >>>   "updateId": "firstName-patch1",
> >>>   "operation": "patch",
> >>>   "patch": [
> >>> {
> >>>   "op": "replace", "path": "/defaultValue", "value": "foo"
> >>> }
> >>>   ]
> >>> }
> >>>
> >>>
> >>> A removal :
> >>>
> >>> {
> >>>   "itemId": "firstName",
> >>>   "updateId": "firstName-patch1",
> >>>   "operation": "remove"
> >>> }
> >>>
> >>> Or a complete override :
> >>>
> >>> {
> >>>   "itemId": "firstName",
> >>>   "updateId": "firstName-patch1",
> >>>   "operation": "override",
> >>>   "definition": {
> >>> "metadata": {
> >>>   "id": "firstName",
> >>>   "name": "First name",
> >>>   "systemTags": [
> >>> "properties",
> >>> "profileProperties",
> >>> "basicProfileProperties",
> >>> "personalIdentifierProperties"
> >>>   ]
> >>> },
> >>> "type": "string",
> >>> "defaultValue": "",
> >>> "automaticMappingsFrom": [ ],
> >>> "rank": "101.0"
> >>>   }
> >>> }
> >>>
> >>>
> >>> This could actually be extended with other type of operation.
> >>>
> >>> For now I do store "updateId" in the metadata of the item, so that
> >it's
> >>> not executed multiple times. But we have have an issue with the
> >delete if
> >>> the item is recreated after.. Maybe I should store the list of
> >updates in a
> >>> new structure ? That could also be accessible with karaf commands,
> >to see
> >>> which patch have been executed and reexecute them if needed.
> >>>
> >>> Thomas
> >>>
> >>>
> >>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
> >>> francois.pa...@openobject.fr> wrote:
> >>>
>  Hi,
> 
>  Very interesting flow :)
> 
>  What is the format to update the definitions, it's just a new JSON,
> >is
>  there some directives about the changes ?
> 
>  For example, in Karaf we have something like this to override
>  configuration files :
> 
> 
>   

Re: Override/migration of predefined definition files

[Jean-Baptiste Onofré]

Cool, gonna take a look.

Regards
JB

Le 26 sept. 2018 à 11:06, à 11:06, Thomas Draier  a écrit:
>Sorry, I forgot to mention : it's pushed in the patch branch :
>https://github.com/apache/incubator-unomi/tree/patch
>
>
>On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier 
>wrote:
>
>> Hi,
>>
>> I've committed the changes for the patch. The files need to be put in
>the
>> same folder as the definitions ( META-INF/cxs/xxx ), and named with a
>> suffix -patch.json . They will be deployed only once, if the
>definitions
>> was already there. Each patch has an id, so if you need to deploy a
>new
>> patch, just use a new id. The list of deployed patches is stored in
>ES.
>> Patches can also be manually executed with the
>"unomi:deploy-definition"
>> karaf command, or through a rest endpoint at /cxs/patches/apply .
>I've
>> removed the behaviour with -SNAPSHOT - definitions are never
>redeployed,
>> even in snapshot versions, so you'll need to explicitely provide a
>patch to
>> override.
>>
>> The syntax is slightly different from what has been sent previously :
>>
>> {
>>   "itemId": "firstName-patch1",
>>   "patchedItemId": "firstName",
>>   "operation": "patch",
>>   "data": [
>>
>>
>> {
>>   "op": "replace",
>>   "path": "/defaultValue",
>>   "value": "foo"
>> }
>>   ]
>> }
>>
>> For an override :
>>
>> {
>>   "itemId": "gender-patch1",
>>   "patchedItemId": "gender",
>>   "operation": "override",
>>   "data": {
>> "metadata": {
>>   "id": "gender",
>>   "name": "Gender",
>>
>>   "systemTags": [
>> "properties",
>> "profileProperties"
>>   ]
>> },
>> "type": "string",
>> "defaultValue": "",
>> "automaticMappingsFrom": [ ],
>>
>> "rank": "105.0"
>>   }
>> }
>>
>> Or a remove :
>>
>> {
>>   "itemId": "firstName-patch3",
>>   "patchedItemId": "firstName",
>>   "operation": "remove"
>> }
>>
>>
>> Regards
>>
>>
>>
>>
>> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier 
>wrote:
>>
>>> Hi,
>>>
>>> I did a quick test with json patch and actually it looks quite easy
>to
>>> integrate and seems to fill more use cases than just a simple
>"override".
>>> However I think it can still be useful to have a complete override
>and also
>>> the possisbility to completely remove the definition, so I was
>thinking of
>>> some "migration" or "update" json file wrapping the patch, within
>the same
>>> folder as the definitions :
>>>
>>> {
>>>   "itemId": "firstName",
>>>   "updateId": "firstName-patch1",
>>>   "operation": "patch",
>>>   "patch": [
>>> {
>>>   "op": "replace", "path": "/defaultValue", "value": "foo"
>>> }
>>>   ]
>>> }
>>>
>>>
>>> A removal :
>>>
>>> {
>>>   "itemId": "firstName",
>>>   "updateId": "firstName-patch1",
>>>   "operation": "remove"
>>> }
>>>
>>> Or a complete override :
>>>
>>> {
>>>   "itemId": "firstName",
>>>   "updateId": "firstName-patch1",
>>>   "operation": "override",
>>>   "definition": {
>>> "metadata": {
>>>   "id": "firstName",
>>>   "name": "First name",
>>>   "systemTags": [
>>> "properties",
>>> "profileProperties",
>>> "basicProfileProperties",
>>> "personalIdentifierProperties"
>>>   ]
>>> },
>>> "type": "string",
>>> "defaultValue": "",
>>> "automaticMappingsFrom": [ ],
>>> "rank": "101.0"
>>>   }
>>> }
>>>
>>>
>>> This could actually be extended with other type of operation.
>>>
>>> For now I do store "updateId" in the metadata of the item, so that
>it's
>>> not executed multiple times. But we have have an issue with the
>delete if
>>> the item is recreated after.. Maybe I should store the list of
>updates in a
>>> new structure ? That could also be accessible with karaf commands,
>to see
>>> which patch have been executed and reexecute them if needed.
>>>
>>> Thomas
>>>
>>>
>>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
>>> francois.pa...@openobject.fr> wrote:
>>>
 Hi,

 Very interesting flow :)

 What is the format to update the definitions, it's just a new JSON,
>is
 there some directives about the changes ?

 For example, in Karaf we have something like this to override
 configuration files :


  
config.properties
put
karaf.framework
equinox
  
  
config.properties
extend
org.osgi.framework.system.capabilities
my-magic-capability
  
  
config.properties
remove
org.apache.karaf.security.providers
  


 May be a solution is a file with a patch description of the changes
>?

 regards,

 François Papon
 fpa...@apache.org

 Le 18/09/2018 à 13:38, Serge Huber a écrit :
 > Not sure if you are talking about the patching system, but if yes
>you
 > would provide a patch file that 

Re: Override/migration of predefined definition files

[Francois Papon]

It look's very nice :)

regards,

François Papon
fpa...@apache.org

Le 26/09/2018 à 19:06, Thomas Draier a écrit :
> Sorry, I forgot to mention : it's pushed in the patch branch :
> https://github.com/apache/incubator-unomi/tree/patch
>
>
> On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier  wrote:
>
>> Hi,
>>
>> I've committed the changes for the patch. The files need to be put in the
>> same folder as the definitions ( META-INF/cxs/xxx ), and named with a
>> suffix -patch.json . They will be deployed only once, if the definitions
>> was already there. Each patch has an id, so if you need to deploy a new
>> patch, just use a new id. The list of deployed patches is stored in ES.
>> Patches can also be manually executed with the "unomi:deploy-definition"
>> karaf command, or through a rest endpoint at /cxs/patches/apply . I've
>> removed the behaviour with -SNAPSHOT - definitions are never redeployed,
>> even in snapshot versions, so you'll need to explicitely provide a patch to
>> override.
>>
>> The syntax is slightly different from what has been sent previously :
>>
>> {
>>   "itemId": "firstName-patch1",
>>   "patchedItemId": "firstName",
>>   "operation": "patch",
>>   "data": [
>>
>>
>> {
>>   "op": "replace",
>>   "path": "/defaultValue",
>>   "value": "foo"
>> }
>>   ]
>> }
>>
>> For an override :
>>
>> {
>>   "itemId": "gender-patch1",
>>   "patchedItemId": "gender",
>>   "operation": "override",
>>   "data": {
>> "metadata": {
>>   "id": "gender",
>>   "name": "Gender",
>>
>>   "systemTags": [
>> "properties",
>> "profileProperties"
>>   ]
>> },
>> "type": "string",
>> "defaultValue": "",
>> "automaticMappingsFrom": [ ],
>>
>> "rank": "105.0"
>>   }
>> }
>>
>> Or a remove :
>>
>> {
>>   "itemId": "firstName-patch3",
>>   "patchedItemId": "firstName",
>>   "operation": "remove"
>> }
>>
>>
>> Regards
>>
>>
>>
>>
>> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier  wrote:
>>
>>> Hi,
>>>
>>> I did a quick test with json patch and actually it looks quite easy to
>>> integrate and seems to fill more use cases than just a simple "override".
>>> However I think it can still be useful to have a complete override and also
>>> the possisbility to completely remove the definition, so I was thinking of
>>> some "migration" or "update" json file wrapping the patch, within the same
>>> folder as the definitions :
>>>
>>> {
>>>   "itemId": "firstName",
>>>   "updateId": "firstName-patch1",
>>>   "operation": "patch",
>>>   "patch": [
>>> {
>>>   "op": "replace", "path": "/defaultValue", "value": "foo"
>>> }
>>>   ]
>>> }
>>>
>>>
>>> A removal :
>>>
>>> {
>>>   "itemId": "firstName",
>>>   "updateId": "firstName-patch1",
>>>   "operation": "remove"
>>> }
>>>
>>> Or a complete override :
>>>
>>> {
>>>   "itemId": "firstName",
>>>   "updateId": "firstName-patch1",
>>>   "operation": "override",
>>>   "definition": {
>>> "metadata": {
>>>   "id": "firstName",
>>>   "name": "First name",
>>>   "systemTags": [
>>> "properties",
>>> "profileProperties",
>>> "basicProfileProperties",
>>> "personalIdentifierProperties"
>>>   ]
>>> },
>>> "type": "string",
>>> "defaultValue": "",
>>> "automaticMappingsFrom": [ ],
>>> "rank": "101.0"
>>>   }
>>> }
>>>
>>>
>>> This could actually be extended with other type of operation.
>>>
>>> For now I do store "updateId" in the metadata of the item, so that it's
>>> not executed multiple times. But we have have an issue with the delete if
>>> the item is recreated after.. Maybe I should store the list of updates in a
>>> new structure ? That could also be accessible with karaf commands, to see
>>> which patch have been executed and reexecute them if needed.
>>>
>>> Thomas
>>>
>>>
>>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
>>> francois.pa...@openobject.fr> wrote:
>>>
 Hi,

 Very interesting flow :)

 What is the format to update the definitions, it's just a new JSON, is
 there some directives about the changes ?

 For example, in Karaf we have something like this to override
 configuration files :


  
config.properties
put
karaf.framework
equinox
  
  
config.properties
extend
org.osgi.framework.system.capabilities
my-magic-capability
  
  
config.properties
remove
org.apache.karaf.security.providers
  


 May be a solution is a file with a patch description of the changes ?

 regards,

 François Papon
 fpa...@apache.org

 Le 18/09/2018 à 13:38, Serge Huber a écrit :
> Not sure if you are talking about the patching system, but if yes you
> would provide a patch file that 

Re: Override/migration of predefined definition files

[Thomas Draier]

Sorry, I forgot to mention : it's pushed in the patch branch :
https://github.com/apache/incubator-unomi/tree/patch


On Wed, Sep 26, 2018 at 3:14 PM Thomas Draier  wrote:

> Hi,
>
> I've committed the changes for the patch. The files need to be put in the
> same folder as the definitions ( META-INF/cxs/xxx ), and named with a
> suffix -patch.json . They will be deployed only once, if the definitions
> was already there. Each patch has an id, so if you need to deploy a new
> patch, just use a new id. The list of deployed patches is stored in ES.
> Patches can also be manually executed with the "unomi:deploy-definition"
> karaf command, or through a rest endpoint at /cxs/patches/apply . I've
> removed the behaviour with -SNAPSHOT - definitions are never redeployed,
> even in snapshot versions, so you'll need to explicitely provide a patch to
> override.
>
> The syntax is slightly different from what has been sent previously :
>
> {
>   "itemId": "firstName-patch1",
>   "patchedItemId": "firstName",
>   "operation": "patch",
>   "data": [
>
>
> {
>   "op": "replace",
>   "path": "/defaultValue",
>   "value": "foo"
> }
>   ]
> }
>
> For an override :
>
> {
>   "itemId": "gender-patch1",
>   "patchedItemId": "gender",
>   "operation": "override",
>   "data": {
> "metadata": {
>   "id": "gender",
>   "name": "Gender",
>
>   "systemTags": [
> "properties",
> "profileProperties"
>   ]
> },
> "type": "string",
> "defaultValue": "",
> "automaticMappingsFrom": [ ],
>
> "rank": "105.0"
>   }
> }
>
> Or a remove :
>
> {
>   "itemId": "firstName-patch3",
>   "patchedItemId": "firstName",
>   "operation": "remove"
> }
>
>
> Regards
>
>
>
>
> On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier  wrote:
>
>> Hi,
>>
>> I did a quick test with json patch and actually it looks quite easy to
>> integrate and seems to fill more use cases than just a simple "override".
>> However I think it can still be useful to have a complete override and also
>> the possisbility to completely remove the definition, so I was thinking of
>> some "migration" or "update" json file wrapping the patch, within the same
>> folder as the definitions :
>>
>> {
>>   "itemId": "firstName",
>>   "updateId": "firstName-patch1",
>>   "operation": "patch",
>>   "patch": [
>> {
>>   "op": "replace", "path": "/defaultValue", "value": "foo"
>> }
>>   ]
>> }
>>
>>
>> A removal :
>>
>> {
>>   "itemId": "firstName",
>>   "updateId": "firstName-patch1",
>>   "operation": "remove"
>> }
>>
>> Or a complete override :
>>
>> {
>>   "itemId": "firstName",
>>   "updateId": "firstName-patch1",
>>   "operation": "override",
>>   "definition": {
>> "metadata": {
>>   "id": "firstName",
>>   "name": "First name",
>>   "systemTags": [
>> "properties",
>> "profileProperties",
>> "basicProfileProperties",
>> "personalIdentifierProperties"
>>   ]
>> },
>> "type": "string",
>> "defaultValue": "",
>> "automaticMappingsFrom": [ ],
>> "rank": "101.0"
>>   }
>> }
>>
>>
>> This could actually be extended with other type of operation.
>>
>> For now I do store "updateId" in the metadata of the item, so that it's
>> not executed multiple times. But we have have an issue with the delete if
>> the item is recreated after.. Maybe I should store the list of updates in a
>> new structure ? That could also be accessible with karaf commands, to see
>> which patch have been executed and reexecute them if needed.
>>
>> Thomas
>>
>>
>> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
>> francois.pa...@openobject.fr> wrote:
>>
>>> Hi,
>>>
>>> Very interesting flow :)
>>>
>>> What is the format to update the definitions, it's just a new JSON, is
>>> there some directives about the changes ?
>>>
>>> For example, in Karaf we have something like this to override
>>> configuration files :
>>>
>>>
>>>  
>>>config.properties
>>>put
>>>karaf.framework
>>>equinox
>>>  
>>>  
>>>config.properties
>>>extend
>>>org.osgi.framework.system.capabilities
>>>my-magic-capability
>>>  
>>>  
>>>config.properties
>>>remove
>>>org.apache.karaf.security.providers
>>>  
>>>
>>>
>>> May be a solution is a file with a patch description of the changes ?
>>>
>>> regards,
>>>
>>> François Papon
>>> fpa...@apache.org
>>>
>>> Le 18/09/2018 à 13:38, Serge Huber a écrit :
>>> > Not sure if you are talking about the patching system, but if yes you
>>> > would provide a patch file that removes all the contents of file.
>>> >
>>> > If we are just using versions it is indeed a bit more tricky. We would
>>> > also need to have a way to list the definitions to remove.
>>> >
>>> > Regards,
>>> >   Serge...
>>> >
>>> > On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier 
>>> wrote:
>>> >> Hi 

Re: Override/migration of predefined definition files

[Thomas Draier]

Hi,

I've committed the changes for the patch. The files need to be put in the
same folder as the definitions ( META-INF/cxs/xxx ), and named with a
suffix -patch.json . They will be deployed only once, if the definitions
was already there. Each patch has an id, so if you need to deploy a new
patch, just use a new id. The list of deployed patches is stored in ES.
Patches can also be manually executed with the "unomi:deploy-definition"
karaf command, or through a rest endpoint at /cxs/patches/apply . I've
removed the behaviour with -SNAPSHOT - definitions are never redeployed,
even in snapshot versions, so you'll need to explicitely provide a patch to
override.

The syntax is slightly different from what has been sent previously :

{
  "itemId": "firstName-patch1",
  "patchedItemId": "firstName",
  "operation": "patch",
  "data": [
{
  "op": "replace",
  "path": "/defaultValue",
  "value": "foo"
}
  ]
}

For an override :

{
  "itemId": "gender-patch1",
  "patchedItemId": "gender",
  "operation": "override",
  "data": {
"metadata": {
  "id": "gender",
  "name": "Gender",
  "systemTags": [
"properties",
"profileProperties"
  ]
},
"type": "string",
"defaultValue": "",
"automaticMappingsFrom": [ ],
"rank": "105.0"
  }
}

Or a remove :

{
  "itemId": "firstName-patch3",
  "patchedItemId": "firstName",
  "operation": "remove"
}


Regards




On Mon, Sep 24, 2018 at 11:00 AM Thomas Draier  wrote:

> Hi,
>
> I did a quick test with json patch and actually it looks quite easy to
> integrate and seems to fill more use cases than just a simple "override".
> However I think it can still be useful to have a complete override and also
> the possisbility to completely remove the definition, so I was thinking of
> some "migration" or "update" json file wrapping the patch, within the same
> folder as the definitions :
>
> {
>   "itemId": "firstName",
>   "updateId": "firstName-patch1",
>   "operation": "patch",
>   "patch": [
> {
>   "op": "replace", "path": "/defaultValue", "value": "foo"
> }
>   ]
> }
>
>
> A removal :
>
> {
>   "itemId": "firstName",
>   "updateId": "firstName-patch1",
>   "operation": "remove"
> }
>
> Or a complete override :
>
> {
>   "itemId": "firstName",
>   "updateId": "firstName-patch1",
>   "operation": "override",
>   "definition": {
> "metadata": {
>   "id": "firstName",
>   "name": "First name",
>   "systemTags": [
> "properties",
> "profileProperties",
> "basicProfileProperties",
> "personalIdentifierProperties"
>   ]
> },
> "type": "string",
> "defaultValue": "",
> "automaticMappingsFrom": [ ],
> "rank": "101.0"
>   }
> }
>
>
> This could actually be extended with other type of operation.
>
> For now I do store "updateId" in the metadata of the item, so that it's
> not executed multiple times. But we have have an issue with the delete if
> the item is recreated after.. Maybe I should store the list of updates in a
> new structure ? That could also be accessible with karaf commands, to see
> which patch have been executed and reexecute them if needed.
>
> Thomas
>
>
> On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
> francois.pa...@openobject.fr> wrote:
>
>> Hi,
>>
>> Very interesting flow :)
>>
>> What is the format to update the definitions, it's just a new JSON, is
>> there some directives about the changes ?
>>
>> For example, in Karaf we have something like this to override
>> configuration files :
>>
>>
>>  
>>config.properties
>>put
>>karaf.framework
>>equinox
>>  
>>  
>>config.properties
>>extend
>>org.osgi.framework.system.capabilities
>>my-magic-capability
>>  
>>  
>>config.properties
>>remove
>>org.apache.karaf.security.providers
>>  
>>
>>
>> May be a solution is a file with a patch description of the changes ?
>>
>> regards,
>>
>> François Papon
>> fpa...@apache.org
>>
>> Le 18/09/2018 à 13:38, Serge Huber a écrit :
>> > Not sure if you are talking about the patching system, but if yes you
>> > would provide a patch file that removes all the contents of file.
>> >
>> > If we are just using versions it is indeed a bit more tricky. We would
>> > also need to have a way to list the definitions to remove.
>> >
>> > Regards,
>> >   Serge...
>> >
>> > On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier 
>> wrote:
>> >> Hi Serge,
>> >>
>> >> In that context, how would we manage objects deletion from a plugin?
>> >>
>> >> Thanks!
>> >>
>> >> Romain
>> >>
>> >> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber 
>> wrote:
>> >>
>> >>> Sorry, I didn't get around to this discussion but I think that
>> >>> versions would probably be cleaner. I was even looking at solutions
>> >>> such as JSON patch (https://github.com/java-json-tools/json-patch) 

Re: Override/migration of predefined definition files

[Thomas Draier]

Hi,

I did a quick test with json patch and actually it looks quite easy to
integrate and seems to fill more use cases than just a simple "override".
However I think it can still be useful to have a complete override and also
the possisbility to completely remove the definition, so I was thinking of
some "migration" or "update" json file wrapping the patch, within the same
folder as the definitions :

{
  "itemId": "firstName",
  "updateId": "firstName-patch1",
  "operation": "patch",
  "patch": [
{
  "op": "replace", "path": "/defaultValue", "value": "foo"
}
  ]
}


A removal :

{
  "itemId": "firstName",
  "updateId": "firstName-patch1",
  "operation": "remove"
}

Or a complete override :

{
  "itemId": "firstName",
  "updateId": "firstName-patch1",
  "operation": "override",
  "definition": {
"metadata": {
  "id": "firstName",
  "name": "First name",
  "systemTags": [
"properties",
"profileProperties",
"basicProfileProperties",
"personalIdentifierProperties"
  ]
},
"type": "string",
"defaultValue": "",
"automaticMappingsFrom": [ ],
"rank": "101.0"
  }
}


This could actually be extended with other type of operation.

For now I do store "updateId" in the metadata of the item, so that it's not
executed multiple times. But we have have an issue with the delete if the
item is recreated after.. Maybe I should store the list of updates in a new
structure ? That could also be accessible with karaf commands, to see which
patch have been executed and reexecute them if needed.

Thomas


On Tue, Sep 18, 2018 at 11:46 AM Francois Papon <
francois.pa...@openobject.fr> wrote:

> Hi,
>
> Very interesting flow :)
>
> What is the format to update the definitions, it's just a new JSON, is
> there some directives about the changes ?
>
> For example, in Karaf we have something like this to override
> configuration files :
>
>
>  
>config.properties
>put
>karaf.framework
>equinox
>  
>  
>config.properties
>extend
>org.osgi.framework.system.capabilities
>my-magic-capability
>  
>  
>config.properties
>remove
>org.apache.karaf.security.providers
>  
>
>
> May be a solution is a file with a patch description of the changes ?
>
> regards,
>
> François Papon
> fpa...@apache.org
>
> Le 18/09/2018 à 13:38, Serge Huber a écrit :
> > Not sure if you are talking about the patching system, but if yes you
> > would provide a patch file that removes all the contents of file.
> >
> > If we are just using versions it is indeed a bit more tricky. We would
> > also need to have a way to list the definitions to remove.
> >
> > Regards,
> >   Serge...
> >
> > On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier 
> wrote:
> >> Hi Serge,
> >>
> >> In that context, how would we manage objects deletion from a plugin?
> >>
> >> Thanks!
> >>
> >> Romain
> >>
> >> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber  wrote:
> >>
> >>> Sorry, I didn't get around to this discussion but I think that
> >>> versions would probably be cleaner. I was even looking at solutions
> >>> such as JSON patch (https://github.com/java-json-tools/json-patch) to
> >>> be able to perform modifications on parts of definitions instead of
> >>> having to deploy a whole definition again.
> >>>
> >>> We had discussions about adding lots of metadata, including last
> >>> modification dates, maybe even hashes for the definition files, to
> >>> know if a definition has been changed or not.
> >>>
> >>> I think that one of the advantages of the patch idea is that you could
> >>> try to patch something if it's compatible, and if it's not you would
> >>> not deploy it. But this idea would mostly be for use cases where you
> >>> are interested in modifying something that comes out of the box in
> >>> Unomi, not for upgrading a plugin.
> >>>
> >>> With versions at least we could also track if a definition is on the
> >>> "default" version or has been modified somehow.
> >>>
> >>> For the cfg idea it feels weird to use a config file that way, maybe
> >>> we should put the marker file elsewhere but to be honest I don't have
> >>> a better idea :)
> >>>
> >>> Would love some input from JB or other Karaf experts :)
> >>>
> >>> Regards,
> >>>   Serge...
> >>>
> >>> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier 
> wrote:
>  Hi,
> 
>  I have to check, but the idea would be to be able to package this cfg
> >>> file
>  beside the bundle. I'm not familiar with unomi:migrate command, but we
> >>> can
>  imagine that this command would extract and deploy the cfg file from
> the
>  bundle - which would make the packaging easier (cfg directly inside
> the
>  plugin). The thing is that we need to be able to remove the file once
> the
>  definition has been "overridden" so that it's only done once, so 

Re: Override/migration of predefined definition files

[Francois Papon]

Hi,

Very interesting flow :)

What is the format to update the definitions, it's just a new JSON, is
there some directives about the changes ?

For example, in Karaf we have something like this to override
configuration files :

   
 
   config.properties
   put
   karaf.framework
   equinox
 
 
   config.properties
   extend
   org.osgi.framework.system.capabilities
   my-magic-capability
 
 
   config.properties
   remove
   org.apache.karaf.security.providers
 
   

May be a solution is a file with a patch description of the changes ?

regards,

François Papon
fpa...@apache.org

Le 18/09/2018 à 13:38, Serge Huber a écrit :
> Not sure if you are talking about the patching system, but if yes you
> would provide a patch file that removes all the contents of file.
>
> If we are just using versions it is indeed a bit more tricky. We would
> also need to have a way to list the definitions to remove.
>
> Regards,
>   Serge...
>
> On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier  wrote:
>> Hi Serge,
>>
>> In that context, how would we manage objects deletion from a plugin?
>>
>> Thanks!
>>
>> Romain
>>
>> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber  wrote:
>>
>>> Sorry, I didn't get around to this discussion but I think that
>>> versions would probably be cleaner. I was even looking at solutions
>>> such as JSON patch (https://github.com/java-json-tools/json-patch) to
>>> be able to perform modifications on parts of definitions instead of
>>> having to deploy a whole definition again.
>>>
>>> We had discussions about adding lots of metadata, including last
>>> modification dates, maybe even hashes for the definition files, to
>>> know if a definition has been changed or not.
>>>
>>> I think that one of the advantages of the patch idea is that you could
>>> try to patch something if it's compatible, and if it's not you would
>>> not deploy it. But this idea would mostly be for use cases where you
>>> are interested in modifying something that comes out of the box in
>>> Unomi, not for upgrading a plugin.
>>>
>>> With versions at least we could also track if a definition is on the
>>> "default" version or has been modified somehow.
>>>
>>> For the cfg idea it feels weird to use a config file that way, maybe
>>> we should put the marker file elsewhere but to be honest I don't have
>>> a better idea :)
>>>
>>> Would love some input from JB or other Karaf experts :)
>>>
>>> Regards,
>>>   Serge...
>>>
>>> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier  wrote:
 Hi,

 I have to check, but the idea would be to be able to package this cfg
>>> file
 beside the bundle. I'm not familiar with unomi:migrate command, but we
>>> can
 imagine that this command would extract and deploy the cfg file from the
 bundle - which would make the packaging easier (cfg directly inside the
 plugin). The thing is that we need to be able to remove the file once the
 definition has been "overridden" so that it's only done once, so the cfg
 file should not be deployed automatically on each startup.

 Another idea I had was to store sort of "version" of the definition
>>> inside
 the metadata, and check if the json file has a greater "version" defined.
 That makes things even more easier when developing plugin and looks a
 little more clean, but I don't know if it would cover correctly all use
 cases.

 We should also extends the DeployDefinition shell command to be able to
 redeploy all definitions of a bundle.

 Thomas


 On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier 
>>> wrote:
> Hello Thomas,
>
> Nice proposition.How would it work for developers that are building
> plugins? Would we have to add our own cfg file or somehow update the
>>> cfg
> file that you want to add?
>
> Since there is already a "unomi:migrate" command would you reuse that
> trigger to update / delete the definitions of the objects?
>
> Same question regarding the plugins, should we also implement "migrate"
> command?
>
> Thanks,
>
> Romain
>
> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier 
>>> wrote:
>> Hi,
>>
>> I'm having some issues with the deployment and redeployment of the
>>> json
>> files containing default definitions (segments, properties, etc..).
>>> Many
>> services have these loadPredefined* methods which load data into ES,
> based
>> on json definition files. These methods only load the content once,
>>> if
> the
>> object does not exist yet in ES. So if a bundle comes with an update
>>> of
> one
>> of these file, it will just be ignored.
>>
>> Actually there was one ticket about this :
>> https://issues.apache.org/jira/browse/UNOMI-182 , where definitions
>>> are
>> always redeployed when a 

Re: Override/migration of predefined definition files

[Serge Huber]

Not sure if you are talking about the patching system, but if yes you
would provide a patch file that removes all the contents of file.

If we are just using versions it is indeed a bit more tricky. We would
also need to have a way to list the definitions to remove.

Regards,
  Serge...

On Tue, Sep 18, 2018 at 11:23 AM Romain Gauthier  wrote:
>
> Hi Serge,
>
> In that context, how would we manage objects deletion from a plugin?
>
> Thanks!
>
> Romain
>
> On Tue, Sep 18, 2018 at 10:45 AM Serge Huber  wrote:
>
> > Sorry, I didn't get around to this discussion but I think that
> > versions would probably be cleaner. I was even looking at solutions
> > such as JSON patch (https://github.com/java-json-tools/json-patch) to
> > be able to perform modifications on parts of definitions instead of
> > having to deploy a whole definition again.
> >
> > We had discussions about adding lots of metadata, including last
> > modification dates, maybe even hashes for the definition files, to
> > know if a definition has been changed or not.
> >
> > I think that one of the advantages of the patch idea is that you could
> > try to patch something if it's compatible, and if it's not you would
> > not deploy it. But this idea would mostly be for use cases where you
> > are interested in modifying something that comes out of the box in
> > Unomi, not for upgrading a plugin.
> >
> > With versions at least we could also track if a definition is on the
> > "default" version or has been modified somehow.
> >
> > For the cfg idea it feels weird to use a config file that way, maybe
> > we should put the marker file elsewhere but to be honest I don't have
> > a better idea :)
> >
> > Would love some input from JB or other Karaf experts :)
> >
> > Regards,
> >   Serge...
> >
> > On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier  wrote:
> > >
> > > Hi,
> > >
> > > I have to check, but the idea would be to be able to package this cfg
> > file
> > > beside the bundle. I'm not familiar with unomi:migrate command, but we
> > can
> > > imagine that this command would extract and deploy the cfg file from the
> > > bundle - which would make the packaging easier (cfg directly inside the
> > > plugin). The thing is that we need to be able to remove the file once the
> > > definition has been "overridden" so that it's only done once, so the cfg
> > > file should not be deployed automatically on each startup.
> > >
> > > Another idea I had was to store sort of "version" of the definition
> > inside
> > > the metadata, and check if the json file has a greater "version" defined.
> > > That makes things even more easier when developing plugin and looks a
> > > little more clean, but I don't know if it would cover correctly all use
> > > cases.
> > >
> > > We should also extends the DeployDefinition shell command to be able to
> > > redeploy all definitions of a bundle.
> > >
> > > Thomas
> > >
> > >
> > > On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier 
> > wrote:
> > >
> > > > Hello Thomas,
> > > >
> > > > Nice proposition.How would it work for developers that are building
> > > > plugins? Would we have to add our own cfg file or somehow update the
> > cfg
> > > > file that you want to add?
> > > >
> > > > Since there is already a "unomi:migrate" command would you reuse that
> > > > trigger to update / delete the definitions of the objects?
> > > >
> > > > Same question regarding the plugins, should we also implement "migrate"
> > > > command?
> > > >
> > > > Thanks,
> > > >
> > > > Romain
> > > >
> > > > On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier 
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm having some issues with the deployment and redeployment of the
> > json
> > > > > files containing default definitions (segments, properties, etc..).
> > Many
> > > > > services have these loadPredefined* methods which load data into ES,
> > > > based
> > > > > on json definition files. These methods only load the content once,
> > if
> > > > the
> > > > > object does not exist yet in ES. So if a bundle comes with an update
> > of
> > > > one
> > > > > of these file, it will just be ignored.
> > > > >
> > > > > Actually there was one ticket about this :
> > > > > https://issues.apache.org/jira/browse/UNOMI-182 , where definitions
> > are
> > > > > always redeployed when a bundle is in "snapshot" version - but never
> > for
> > > > a
> > > > > release version. So that's nice when you are in development, but not
> > > > > possible when upgrading a production environment.
> > > > >
> > > > > There also a console command to manually deploy a definition - which
> > is
> > > > > also nice but cannot be automated in a migration.
> > > > >
> > > > > Of course there's a reason - these objects can be modified
> > afterwards by
> > > > > using the API, and so restarting should not overwrite the
> > modifications
> > > > by
> > > > > reimporting the file. Even if a module is updated and object is
> > modified,
> > > > > the user may not want to overwrite his 

Re: Override/migration of predefined definition files

[Romain Gauthier]

Hi Serge,

In that context, how would we manage objects deletion from a plugin?

Thanks!

Romain

On Tue, Sep 18, 2018 at 10:45 AM Serge Huber  wrote:

> Sorry, I didn't get around to this discussion but I think that
> versions would probably be cleaner. I was even looking at solutions
> such as JSON patch (https://github.com/java-json-tools/json-patch) to
> be able to perform modifications on parts of definitions instead of
> having to deploy a whole definition again.
>
> We had discussions about adding lots of metadata, including last
> modification dates, maybe even hashes for the definition files, to
> know if a definition has been changed or not.
>
> I think that one of the advantages of the patch idea is that you could
> try to patch something if it's compatible, and if it's not you would
> not deploy it. But this idea would mostly be for use cases where you
> are interested in modifying something that comes out of the box in
> Unomi, not for upgrading a plugin.
>
> With versions at least we could also track if a definition is on the
> "default" version or has been modified somehow.
>
> For the cfg idea it feels weird to use a config file that way, maybe
> we should put the marker file elsewhere but to be honest I don't have
> a better idea :)
>
> Would love some input from JB or other Karaf experts :)
>
> Regards,
>   Serge...
>
> On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier  wrote:
> >
> > Hi,
> >
> > I have to check, but the idea would be to be able to package this cfg
> file
> > beside the bundle. I'm not familiar with unomi:migrate command, but we
> can
> > imagine that this command would extract and deploy the cfg file from the
> > bundle - which would make the packaging easier (cfg directly inside the
> > plugin). The thing is that we need to be able to remove the file once the
> > definition has been "overridden" so that it's only done once, so the cfg
> > file should not be deployed automatically on each startup.
> >
> > Another idea I had was to store sort of "version" of the definition
> inside
> > the metadata, and check if the json file has a greater "version" defined.
> > That makes things even more easier when developing plugin and looks a
> > little more clean, but I don't know if it would cover correctly all use
> > cases.
> >
> > We should also extends the DeployDefinition shell command to be able to
> > redeploy all definitions of a bundle.
> >
> > Thomas
> >
> >
> > On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier 
> wrote:
> >
> > > Hello Thomas,
> > >
> > > Nice proposition.How would it work for developers that are building
> > > plugins? Would we have to add our own cfg file or somehow update the
> cfg
> > > file that you want to add?
> > >
> > > Since there is already a "unomi:migrate" command would you reuse that
> > > trigger to update / delete the definitions of the objects?
> > >
> > > Same question regarding the plugins, should we also implement "migrate"
> > > command?
> > >
> > > Thanks,
> > >
> > > Romain
> > >
> > > On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm having some issues with the deployment and redeployment of the
> json
> > > > files containing default definitions (segments, properties, etc..).
> Many
> > > > services have these loadPredefined* methods which load data into ES,
> > > based
> > > > on json definition files. These methods only load the content once,
> if
> > > the
> > > > object does not exist yet in ES. So if a bundle comes with an update
> of
> > > one
> > > > of these file, it will just be ignored.
> > > >
> > > > Actually there was one ticket about this :
> > > > https://issues.apache.org/jira/browse/UNOMI-182 , where definitions
> are
> > > > always redeployed when a bundle is in "snapshot" version - but never
> for
> > > a
> > > > release version. So that's nice when you are in development, but not
> > > > possible when upgrading a production environment.
> > > >
> > > > There also a console command to manually deploy a definition - which
> is
> > > > also nice but cannot be automated in a migration.
> > > >
> > > > Of course there's a reason - these objects can be modified
> afterwards by
> > > > using the API, and so restarting should not overwrite the
> modifications
> > > by
> > > > reimporting the file. Even if a module is updated and object is
> modified,
> > > > the user may not want to overwrite his changes.
> > > >
> > > > However in some cases the developer may want to force the update of
> some
> > > > specific objects ( it actually happened to us ). I'm looking for some
> > > > additional solution to force redeployment of a definition.
> > > >
> > > > So far i was thinking of adding a cfg file, which would list the
> > > properties
> > > > to redeploy. This file would be removed once the properties are
> updated,
> > > so
> > > > that they are reloaded only once and not on every startup.
> > > >
> > > > As a bonus, I would also need to be able to remove definitions the
> same
> 

Re: Override/migration of predefined definition files

[Serge Huber]

Sorry, I didn't get around to this discussion but I think that
versions would probably be cleaner. I was even looking at solutions
such as JSON patch (https://github.com/java-json-tools/json-patch) to
be able to perform modifications on parts of definitions instead of
having to deploy a whole definition again.

We had discussions about adding lots of metadata, including last
modification dates, maybe even hashes for the definition files, to
know if a definition has been changed or not.

I think that one of the advantages of the patch idea is that you could
try to patch something if it's compatible, and if it's not you would
not deploy it. But this idea would mostly be for use cases where you
are interested in modifying something that comes out of the box in
Unomi, not for upgrading a plugin.

With versions at least we could also track if a definition is on the
"default" version or has been modified somehow.

For the cfg idea it feels weird to use a config file that way, maybe
we should put the marker file elsewhere but to be honest I don't have
a better idea :)

Would love some input from JB or other Karaf experts :)

Regards,
  Serge...

On Tue, Sep 18, 2018 at 9:51 AM Thomas Draier  wrote:
>
> Hi,
>
> I have to check, but the idea would be to be able to package this cfg file
> beside the bundle. I'm not familiar with unomi:migrate command, but we can
> imagine that this command would extract and deploy the cfg file from the
> bundle - which would make the packaging easier (cfg directly inside the
> plugin). The thing is that we need to be able to remove the file once the
> definition has been "overridden" so that it's only done once, so the cfg
> file should not be deployed automatically on each startup.
>
> Another idea I had was to store sort of "version" of the definition inside
> the metadata, and check if the json file has a greater "version" defined.
> That makes things even more easier when developing plugin and looks a
> little more clean, but I don't know if it would cover correctly all use
> cases.
>
> We should also extends the DeployDefinition shell command to be able to
> redeploy all definitions of a bundle.
>
> Thomas
>
>
> On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier  wrote:
>
> > Hello Thomas,
> >
> > Nice proposition.How would it work for developers that are building
> > plugins? Would we have to add our own cfg file or somehow update the cfg
> > file that you want to add?
> >
> > Since there is already a "unomi:migrate" command would you reuse that
> > trigger to update / delete the definitions of the objects?
> >
> > Same question regarding the plugins, should we also implement "migrate"
> > command?
> >
> > Thanks,
> >
> > Romain
> >
> > On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier  wrote:
> >
> > > Hi,
> > >
> > > I'm having some issues with the deployment and redeployment of the json
> > > files containing default definitions (segments, properties, etc..). Many
> > > services have these loadPredefined* methods which load data into ES,
> > based
> > > on json definition files. These methods only load the content once, if
> > the
> > > object does not exist yet in ES. So if a bundle comes with an update of
> > one
> > > of these file, it will just be ignored.
> > >
> > > Actually there was one ticket about this :
> > > https://issues.apache.org/jira/browse/UNOMI-182 , where definitions are
> > > always redeployed when a bundle is in "snapshot" version - but never for
> > a
> > > release version. So that's nice when you are in development, but not
> > > possible when upgrading a production environment.
> > >
> > > There also a console command to manually deploy a definition - which is
> > > also nice but cannot be automated in a migration.
> > >
> > > Of course there's a reason - these objects can be modified afterwards by
> > > using the API, and so restarting should not overwrite the modifications
> > by
> > > reimporting the file. Even if a module is updated and object is modified,
> > > the user may not want to overwrite his changes.
> > >
> > > However in some cases the developer may want to force the update of some
> > > specific objects ( it actually happened to us ). I'm looking for some
> > > additional solution to force redeployment of a definition.
> > >
> > > So far i was thinking of adding a cfg file, which would list the
> > properties
> > > to redeploy. This file would be removed once the properties are updated,
> > so
> > > that they are reloaded only once and not on every startup.
> > >
> > > As a bonus, I would also need to be able to remove definitions the same
> > > way. There's no way to do that now except using the REST API , it could
> > be
> > > good to also have a console command and to be able to remove definitions
> > > with the same cfg file.
> > >
> > > What do you think .. ?
> > >
> > > Regards,
> > >
> >
> >
> > --
> > Romain Gauthier
> > Product Manager at Jahia
> > + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025>
> > 8 Rue du sentier | 75002 Paris 

Re: Override/migration of predefined definition files

[Thomas Draier]

Hi,

I have to check, but the idea would be to be able to package this cfg file
beside the bundle. I'm not familiar with unomi:migrate command, but we can
imagine that this command would extract and deploy the cfg file from the
bundle - which would make the packaging easier (cfg directly inside the
plugin). The thing is that we need to be able to remove the file once the
definition has been "overridden" so that it's only done once, so the cfg
file should not be deployed automatically on each startup.

Another idea I had was to store sort of "version" of the definition inside
the metadata, and check if the json file has a greater "version" defined.
That makes things even more easier when developing plugin and looks a
little more clean, but I don't know if it would cover correctly all use
cases.

We should also extends the DeployDefinition shell command to be able to
redeploy all definitions of a bundle.

Thomas


On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier  wrote:

> Hello Thomas,
>
> Nice proposition.How would it work for developers that are building
> plugins? Would we have to add our own cfg file or somehow update the cfg
> file that you want to add?
>
> Since there is already a "unomi:migrate" command would you reuse that
> trigger to update / delete the definitions of the objects?
>
> Same question regarding the plugins, should we also implement "migrate"
> command?
>
> Thanks,
>
> Romain
>
> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier  wrote:
>
> > Hi,
> >
> > I'm having some issues with the deployment and redeployment of the json
> > files containing default definitions (segments, properties, etc..). Many
> > services have these loadPredefined* methods which load data into ES,
> based
> > on json definition files. These methods only load the content once, if
> the
> > object does not exist yet in ES. So if a bundle comes with an update of
> one
> > of these file, it will just be ignored.
> >
> > Actually there was one ticket about this :
> > https://issues.apache.org/jira/browse/UNOMI-182 , where definitions are
> > always redeployed when a bundle is in "snapshot" version - but never for
> a
> > release version. So that's nice when you are in development, but not
> > possible when upgrading a production environment.
> >
> > There also a console command to manually deploy a definition - which is
> > also nice but cannot be automated in a migration.
> >
> > Of course there's a reason - these objects can be modified afterwards by
> > using the API, and so restarting should not overwrite the modifications
> by
> > reimporting the file. Even if a module is updated and object is modified,
> > the user may not want to overwrite his changes.
> >
> > However in some cases the developer may want to force the update of some
> > specific objects ( it actually happened to us ). I'm looking for some
> > additional solution to force redeployment of a definition.
> >
> > So far i was thinking of adding a cfg file, which would list the
> properties
> > to redeploy. This file would be removed once the properties are updated,
> so
> > that they are reloaded only once and not on every startup.
> >
> > As a bonus, I would also need to be able to remove definitions the same
> > way. There's no way to do that now except using the REST API , it could
> be
> > good to also have a console command and to be able to remove definitions
> > with the same cfg file.
> >
> > What do you think .. ?
> >
> > Regards,
> >
>
>
> --
> Romain Gauthier
> Product Manager at Jahia
> + 33 6 20 18 35 25 <+33%206%2020%2018%2035%2025>
> 8 Rue du sentier | 75002 Paris | France
> 
>

Re: Override/migration of predefined definition files

[Damien GAILLARD]

I like the idea.

Another idea will be to add to new Unomi commands, we have
unomi:deploy-definition which allow you to redeploy one definition at a
time, we could easily add a command to redeploy all the definition of a
given bundle and another one to redeploy all the definitions in all bundles.

Cheers

On Tue, Sep 18, 2018 at 1:33 AM Romain Gauthier  wrote:

> Hello Thomas,
>
> Nice proposition.How would it work for developers that are building
> plugins? Would we have to add our own cfg file or somehow update the cfg
> file that you want to add?
>
> Since there is already a "unomi:migrate" command would you reuse that
> trigger to update / delete the definitions of the objects?
>
> Same question regarding the plugins, should we also implement "migrate"
> command?
>
> Thanks,
>
> Romain
>
> On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier  wrote:
>
> > Hi,
> >
> > I'm having some issues with the deployment and redeployment of the json
> > files containing default definitions (segments, properties, etc..). Many
> > services have these loadPredefined* methods which load data into ES,
> based
> > on json definition files. These methods only load the content once, if
> the
> > object does not exist yet in ES. So if a bundle comes with an update of
> one
> > of these file, it will just be ignored.
> >
> > Actually there was one ticket about this :
> > https://issues.apache.org/jira/browse/UNOMI-182 , where definitions are
> > always redeployed when a bundle is in "snapshot" version - but never for
> a
> > release version. So that's nice when you are in development, but not
> > possible when upgrading a production environment.
> >
> > There also a console command to manually deploy a definition - which is
> > also nice but cannot be automated in a migration.
> >
> > Of course there's a reason - these objects can be modified afterwards by
> > using the API, and so restarting should not overwrite the modifications
> by
> > reimporting the file. Even if a module is updated and object is modified,
> > the user may not want to overwrite his changes.
> >
> > However in some cases the developer may want to force the update of some
> > specific objects ( it actually happened to us ). I'm looking for some
> > additional solution to force redeployment of a definition.
> >
> > So far i was thinking of adding a cfg file, which would list the
> properties
> > to redeploy. This file would be removed once the properties are updated,
> so
> > that they are reloaded only once and not on every startup.
> >
> > As a bonus, I would also need to be able to remove definitions the same
> > way. There's no way to do that now except using the REST API , it could
> be
> > good to also have a console command and to be able to remove definitions
> > with the same cfg file.
> >
> > What do you think .. ?
> >
> > Regards,
> >
>
>
> --
> Romain Gauthier
> Product Manager at Jahia
> + 33 6 20 18 35 25
> 8 Rue du sentier | 75002 Paris | France
> 
>


-- 
Damien GAILLARD
Software Developer


8 rue du sentier | 75002 | France


jahia.com 
SKYPE | TWITTER  | VCARD


> JAHIA  empowers organizations to create
sophisticated websites that drive their business growth

Re: Override/migration of predefined definition files

[Romain Gauthier]

Hello Thomas,

Nice proposition.How would it work for developers that are building
plugins? Would we have to add our own cfg file or somehow update the cfg
file that you want to add?

Since there is already a "unomi:migrate" command would you reuse that
trigger to update / delete the definitions of the objects?

Same question regarding the plugins, should we also implement "migrate"
command?

Thanks,

Romain

On Thu, Sep 13, 2018 at 6:22 PM Thomas Draier  wrote:

> Hi,
>
> I'm having some issues with the deployment and redeployment of the json
> files containing default definitions (segments, properties, etc..). Many
> services have these loadPredefined* methods which load data into ES, based
> on json definition files. These methods only load the content once, if the
> object does not exist yet in ES. So if a bundle comes with an update of one
> of these file, it will just be ignored.
>
> Actually there was one ticket about this :
> https://issues.apache.org/jira/browse/UNOMI-182 , where definitions are
> always redeployed when a bundle is in "snapshot" version - but never for a
> release version. So that's nice when you are in development, but not
> possible when upgrading a production environment.
>
> There also a console command to manually deploy a definition - which is
> also nice but cannot be automated in a migration.
>
> Of course there's a reason - these objects can be modified afterwards by
> using the API, and so restarting should not overwrite the modifications by
> reimporting the file. Even if a module is updated and object is modified,
> the user may not want to overwrite his changes.
>
> However in some cases the developer may want to force the update of some
> specific objects ( it actually happened to us ). I'm looking for some
> additional solution to force redeployment of a definition.
>
> So far i was thinking of adding a cfg file, which would list the properties
> to redeploy. This file would be removed once the properties are updated, so
> that they are reloaded only once and not on every startup.
>
> As a bonus, I would also need to be able to remove definitions the same
> way. There's no way to do that now except using the REST API , it could be
> good to also have a console command and to be able to remove definitions
> with the same cfg file.
>
> What do you think .. ?
>
> Regards,
>


-- 
Romain Gauthier
Product Manager at Jahia
+ 33 6 20 18 35 25
8 Rue du sentier | 75002 Paris | France