Re: [Context Aware Configs] Merging?

[Bertrand Delacretaz]

On Thu, Aug 18, 2016 at 11:52 AM, Stefan Seifert  wrote:
> ...perhaps we find a way to support both ways with the pluggable architecture 
> i've in mind and
> that in good parts is already in use in wcm.io - activate the property-level 
> merging osgi service or
> bundle if you want you want to use it, or leave it out if not

Having a service to control merging things sounds like the safest thing to do.

We might provide a basic implementation along with an extension point
for fancier logic.

And the logging that helps troubleshooting who merged what.

-Bertrand

RE: [Context Aware Configs] Merging?

[Stefan Seifert]

>So far, no one needed it, as the implementation which is used today does
>not support it.

well, depends which implementation is referenced - in the wcm.io Config 
property-level merging for parameter inheritance is supported.

i agree with the arguments from justin and georg, and see more usecases where 
property-level merging is helpful than where it is harmful. keep in mind that 
content-aware configuration is often outside the scope of the devops-controlled 
automated configuration.

in our usecases (worldwide-distributed marketing websites) we often have a big 
number of "business parameters" that control features on the websites which are 
switched on and off be editors using a graphical configuration editor within 
their CMS GUI. the devops team only maintains a sensible set of global/default 
confgurations which can be overridden by each of the hundreds of websites. 
without property merging a new parameter in a global/default configuration 
would mean to adjust hundreds of inherited configuration objects as well.

in wcm.io config we also added support to "lock" a single configuration 
property on a certain context level, e.g. this allows to define a global or 
region-wide parameter and disallows the editors in the website to override it. 
useful e.g. to switch off a feature centrally which is currently malfunctioning 
until the application is fixed, and then the lock is removed again preserving 
the previous configuration state in each site.

if we support object merging we already open "pandora's box" - if it is feared 
that property-level inheritance can do harm, the same can apply to object-level 
inheritance as well.

we also have a third special case in inheritance: object/resource collection 
inheritance (object/resource collections are comparable to osgi factory 
configurations). we need to decide what happens when globally there are objects 
A and B in the config collection, and local there is C. is the result A+B+C or 
only C? some of the existing configuration solutions already support special 
properties to decide what is happening, because there are usecases for both 
results. the resource merger also has concepts for this. this usecase is 
currently missing in ians specification draft 
https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869

in my pov it is not always correct the have the analogy of osgi configuration 
vs. context-aware configuration when deciding if thinks like property merging 
are useful or not - osgi configuration is per definition always global, never 
tenant-specific, never depending on context e.g. content tree, never 
hierarchical. this makes the usecases for inheritance and property merging more 
complex for context-aware configuration than for osgi configurations.

perhaps we find a way to support both ways with the pluggable architecture i've 
in mind and that in good parts is already in use in wcm.io - activate the 
property-level merging osgi service or bundle if you want you want to use it, 
or leave it out if not.

stefan

Re: [Context Aware Configs] Merging?

[Georg Henzler]

>> * Puppet (not too sure about the others) is not exactly great at dealing
>> with xml files
>> * FileVault [1] is only available in Java, if you want to create proper
>> packages with it you have to call out to java from other languages
>
> Packages are zip files with a defined structure and some required files.
> If you have a template, a few lines of bash (cp -r, sed, zip) will create
> a package containing OSGi configs. This works well in Kubernetes within
> Docker images. (btw, without a package and its version number, the
> OSGIPackageInstaller doesnt install replacement config files)

Simple property filtering works fine yes, but to implement a
replacement for server side config merging you quickly come to a
situation where you  would have to read the structure of xml files and
add config xml files and filter rules for it dynamically (think of
creating a config /conf/tenant/myprof/path/to/my/page/myconfig.xml iff
the prop 
puppetproproot.myprof.path.to.my.page.myconfig.theOneConfigPropToBeOverWrittenHere
exists), doing so using FileVault/XmlApis is more robust/easier than
using manual, text-based assembly.

-Georg

Re: [Context Aware Configs] Merging?

[Justin Edelson]

On Mon, Aug 1, 2016 at 7:24 AM Robert Munteanu  wrote:

> On Mon, 2016-08-01 at 11:34 +0100, Ian Boston wrote:
> > Hi,
> >
> > On 1 August 2016 at 11:02, Carsten Ziegeler 
> > wrote:
> >
> > >
> > > Well, I think you're misinterpreting my comments - I never said
> > > that it
> > > will require a special tooling - I said it would be insane to edit
> > > a
> > > complex setup by editing nodes in the repository.
> > >
> >
> > I was, apologies and thank you for the clarification.
> >
> > No special tooling *and* a simple set of rules that are natural to
> > communicate and document.
> >
> > I don't have a strong opinion on merging or inheritance, provided its
> > simple to use and the result if obvious with no strange side effects
> > to
> > catch a deployer out.
>
> I think it's possible to have merging on a less fine-grained level, and
> thus more controllable, by using multiple nodes for the same
> configuration instance.
>
> For example:
>
> Global: {
> 'services' : {
> 'serviceA' {
> 'endpoint' : 'https://api.com/foo'
> 'params': {
>'timeout': 500
> }
> }
> }
> }
>
> Site: {
> 'services' : {
> 'serviceA' {
> 'endpoint' : '
> https://stage.api.com/foo'
> }
> }
> }
>
> the override only affects the 'endpoint' parameter. If at somepoint we
> decide to add a 'retryCount' parameter to 'params' this will also be
> propagated to inherited configurations.
>

I feel like this example only makes the argument for property-level merging
all the more obvious. If we support only object-level merging, then you
will inevitably end up with a state like this:

Global: {
'services' : {
'serviceA' {
'endpoint' : 'https://api.com/foo',
'params': {
   'timeout': 500
},
'params2' : {
   'connectTimeout' : 5000
}
}
}
}

Site: {
'services' : {
'serviceA' {
'endpoint' : 'https://stage.api.com/foo',
'params': {
   'timeout': 200
}
}
}
}

where the fact that the 'params' object has been overridden forces the
global config editor to define a 'params2' object to store some new
properties which will be inherited and then when those are overridden, a
'params3' and so on.

Justin


>
> Of course, this comes with the caveats that:
>
> - we need to decide upfront what about how configurations are split into
> nodes
> - we won't always be able to split configurations into nodes so that all
> properties are overrideable



> Robert
>
> >
> >
> > >
> > >
> > > Clearly with devops automation you have some tooling already in
> > > place
> > > which is using text files and does something to start AEM, so you
> > > can
> > > add the stuff there - or - the configuration are content packages
> > > after
> > > all, so if the automation supports content packages in some form,
> > > you're
> > > done.
> > >
> >
> > Agreed.
> > Best Regards
> > Ian
> >
> >
> > >
> > >
> > > Carsten
> > >
> > > >
> > > > Hi,
> > > >
> > > > I think it must be possible to configure a Sling/AEM instance
> > > > without
> > > using
> > > >
> > > > specific Sling/AEM tooling to create and manage the source
> > > > configuration.
> > > > The reason I say this is almost all serious deployments use some
> > > > form of
> > > > DevOps automation ranging from Puppet/Chef/AWS etc through to
> > > > Docker/Mesos/Kubernetes etc. Almost without exception those
> > > > systems work
> > > > with configuration stored in text files (property, json, yaml
> > > > etc), often
> > > > stored in Git/SVN/Hg, edited by a human in a text editor. There
> > > > are
> > > systems
> > > >
> > > > that only allow config via a GUI, but most of those only used,
> > > > under
> > > > sufferance, when there is no other option, (or competing
> > > > product). The
> > > > "customer" here is not a Sling/AEM expert, but a DevOps engineer
> > > > who has
> > > > never seen either before.
> > > >
> > > > If the only way to configure Sling/AEM is via special Sling/AEM
> > > > tooling
> > > > then we will have made the job of the TechOps/DevOps team even
> > > > harder.
> > > >
> > > > I can think of lots of ways that merging at the property level
> > > > could be
> > > > made to work, but if we are not able to communicate clearly how
> > > > it works,
> > > > and all the potential side effects, to those whose job it is to
> > > > make
> > > > Sling/AEM work in production then we will have failed. If we have
> > > > to
> > > resort
> > > >
> > > > to validation tools, compilers, special editors thats an
> > > > indication the
> > > > system is so complex we are not able to document it in plain
> > > > english.
> > > For
> > > >
> > > > that reason, I beg, please keep whatever solution is chosen as
> > > > simple as
> > > > possible.
> > > >
> > > > I did try to document "object" 

Re: [Context Aware Configs] Merging?

[Justin Edelson]

Hi,

On Sun, Jul 31, 2016 at 7:07 AM Carsten Ziegeler 
wrote:

> > Allow me to disagree. This lack of merging is actually one of the key
> pain
> > points I have with OSGi ConfigAdmin and I would really love to see it
> > accommodated here. The problem with not merging is that it results in a
> lot
> > of error-prone duplication and you can easily get into a state where new
> > default values (in the global configuration) are not propagated as
> expected
> > because once override any configuration property you are effectively
> > hard-coding the *current* global configuration in your override. Future
> > changes (or even additions) to that global configuration does not have
> the
> > expected result.
> >
> > Consider this case:
> >
> > Global: {
> > 'services' : {
> > 'serviceA' {
> > 'endpoint' : 'https://api.com/foo'
> > }
> > }
> > }
> >
> >
> > Site: {
> > 'services' : {
> > 'serviceA' {
> > 'endpoint' : 'https://stage.api.com/foo'
> > }
> > }
> > }
> >
> > Works OK. But at some point down the line, the developer of the component
> > consuming this configuration finds that a timeout value needs to be
> > specified. They do the needful and modify the global configuration to be:
> >
> > {
> > 'services' : {
> > 'serviceA' {
> > 'endpoint' : 'https://api.com/foo',
> > 'timeout' : 500
> > }
> > }
> > }
> >
> > I'd suggest that the reasonable expectation is that the timeout would
> apply
> > to all contexts. But without merging, this won't happen.
> >
> > Merging is hard to add after-the-fact because it is almost impossible to
> > make backwards-compatible. Since we are starting with a blank slate here,
> > I'd strongly suggest that this new capability support merging from the
> > start.
> >
> > I would also disagree that this isn't a problem already. The adoption of
> > service users is a recent specific case where the failure of JcrInstaller
> > to support merging was an impediment. We had to add the "amended"
> > configuration PID specifically to work around that problem because it was
> > pretty clearly nonscalable to have a centralized list of service user
> > configurations. I could go through my archives of 6 years of AEM projects
> > and easily find hundreds of overridden sling:OsgiConfig nodes which had
> > properties totally unrelated to the desired configuration change. But
> since
> > the configuration is all there, there's no real way to know (save for
> > documentation) what *specific* configuration change was desired vs. what
> > properties had to be copied just to get a component to retain its
> original
> > behavior.
> >
>
> Well, I could argue the opposite: if you have merging (or inheritance) then
> you change the attribute of a parent and this influences all childs and you
> have no idea what happens there and worst case you end up editing all child
> configurations by overriding that property.
>

Do you have a concrete example of when this wouldn't be desired? I'm
struggling to think of a case. But even if this was wrong, we can always
have a flag at either the parent or child level which disables the
inheritance. Still, I wouldn't suggest implementing this without a specific
use case which requires it as this would be a relatively simple and
backwards compatible addition.


>
> I personally think this comes all down to tooling - no one (at least I
> hope so) is editing all these configurations by hand in the repository.
> That's - regardless if you have merging or not - error prone.
> So a provisioning tool can do all what is requested here: it knows the
> parent configuration, it knows the inheritance/merging rules and it
> can apply the changes in the way the editor of the configuration intents
> to. And that means we *could* move all this functionality into the
> tooling but keep the implementation at run time crystal clear.
>

I'm not really sure what kind of "tooling" you are describing and based on
the subsequent emails in this thread I'm even more confused. If it isn't
"special" tooling, then how does this tooling know the inheritance/merging
rules?

Regards,
Justin


>
> Regards
> Carsten
>
>
>
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziege...@apache.org
>
>

Re: [Context Aware Configs] Merging?

[Ian Boston]

Hi,

On 2 August 2016 at 10:30, Georg Henzler  wrote:

> On 2016-08-01 13:29, Konrad Windszus wrote:
>
>> Can't we just use the resource merger approach for merging configs
>> (
>> http://sling.staging.apache.org/documentation/bundles/resource-merger.html
>> <
>> http://sling.staging.apache.org/documentation/bundles/resource-merger.html
>> >)?
>> There probably needs to be a dedicated resource picker for
>> configurations, because neither the search path approach nor the
>> resource super type approach is applicable for configurations.
>>
>
> Sounds like a good match, I could imagine that way we could implement
> merged configs fairly easily!
>
> Clearly with devops automation you have some tooling already in place
>>> which is using text files and does something to start AEM, so you can
>>> add the stuff there - or - the configuration are content packages after
>>> all, so if the automation supports content packages in some form, you're
>>> done.
>>>
>>>
>> Agreed.
>>
>
> DevOps automation is nice, but for generating CRX packages with xml config
> files it's not a great match:
> * Puppet (not too sure about the others) is not exactly great at dealing
> with xml files
> * FileVault [1] is only available in Java, if you want to create proper
> packages with it you have to call out to java from other languages
>

Packages are zip files with a defined structure and some required files.
 If you have a template, a few lines of bash (cp -r, sed, zip) will create
a package containing OSGi configs. This works well in Kubernetes within
Docker images. (btw, without a package and its version number, the
OSGIPackageInstaller doesnt install replacement config files)



> * Not having a standard approach makes every project implement their
> homegrown solution (and waste budget where it could really be standard).
> I've have seen a few awkward solutions for generating OSGi configs
> * Alternatively  (and this is what I've seen happening the most for OSGi
> configs), people just accept the fact that they have to duplicate
> configuration sets and then run into all the frustration Justin described
> earlier



>
> So I'm still in big favour of having a runtime solution.
>

+1.

Will OSGi configs be context aware or do they only make sense in a Global
context (overridden by /libs /apps )?

I can see how a Factory might generate multiple configs, but OSGi
components are shared and survive outside most Request or Job contexts.

Only where a component is actually created and dies on a thread with
context will context aware OSGi configurations make sense... or am I
missing something ?

Best Regards
Ian



>
> Best Regards
> Georg
>
> [1]
> http://jackrabbit.apache.org/filevault/
>

Re: [Context Aware Configs] Merging?

[Georg Henzler]

On 2016-08-01 13:29, Konrad Windszus wrote:

Can't we just use the resource merger approach for merging configs
(http://sling.staging.apache.org/documentation/bundles/resource-merger.html
)?
There probably needs to be a dedicated resource picker for
configurations, because neither the search path approach nor the
resource super type approach is applicable for configurations.


Sounds like a good match, I could imagine that way we could implement 
merged configs fairly easily!



Clearly with devops automation you have some tooling already in place
which is using text files and does something to start AEM, so you can
add the stuff there - or - the configuration are content packages 
after
all, so if the automation supports content packages in some form, 
you're

done.



Agreed.


DevOps automation is nice, but for generating CRX packages with xml 
config files it's not a great match:
* Puppet (not too sure about the others) is not exactly great at dealing 
with xml files
* FileVault [1] is only available in Java, if you want to create proper 
packages with it you have to call out to java from other languages
* Not having a standard approach makes every project implement their 
homegrown solution (and waste budget where it could really be standard). 
I've have seen a few awkward solutions for generating OSGi configs
* Alternatively  (and this is what I've seen happening the most for OSGi 
configs), people just accept the fact that they have to duplicate 
configuration sets and then run into all the frustration Justin 
described earlier


So I'm still in big favour of having a runtime solution.

Best Regards
Georg

[1]
http://jackrabbit.apache.org/filevault/

Re: [Context Aware Configs] Merging?

[Konrad Windszus]

Can't we just use the resource merger approach for merging configs 
(http://sling.staging.apache.org/documentation/bundles/resource-merger.html 
)?
There probably needs to be a dedicated resource picker for configurations, 
because neither the search path approach nor the resource super type approach 
is applicable for configurations.
That way the merging aspect is nicely separated.
Konrad


> On 01 Aug 2016, at 13:24, Robert Munteanu  wrote:
> 
> On Mon, 2016-08-01 at 11:34 +0100, Ian Boston wrote:
>> Hi,
>> 
>> On 1 August 2016 at 11:02, Carsten Ziegeler 
>> wrote:
>> 
>>> 
>>> Well, I think you're misinterpreting my comments - I never said
>>> that it
>>> will require a special tooling - I said it would be insane to edit
>>> a
>>> complex setup by editing nodes in the repository.
>>> 
>> 
>> I was, apologies and thank you for the clarification.
>> 
>> No special tooling *and* a simple set of rules that are natural to
>> communicate and document.
>> 
>> I don't have a strong opinion on merging or inheritance, provided its
>> simple to use and the result if obvious with no strange side effects
>> to
>> catch a deployer out.
> 
> I think it's possible to have merging on a less fine-grained level, and
> thus more controllable, by using multiple nodes for the same
> configuration instance.
> 
> For example:
> 
> Global: {
> 'services' : {
> 'serviceA' {
> 'endpoint' : 'https://api.com/foo' 
> 'params': {
>'timeout': 500
> }
> }
> }
> }
> 
> Site: {
> 'services' : {
> 'serviceA' {
> 'endpoint' : '
> https://stage.api.com/foo' 
> }
> }
> }
> 
> the override only affects the 'endpoint' parameter. If at somepoint we decide 
> to add a 'retryCount' parameter to 'params' this will also be propagated to 
> inherited configurations.
> 
> Of course, this comes with the caveats that:
> 
> - we need to decide upfront what about how configurations are split into nodes
> - we won't always be able to split configurations into nodes so that all 
> properties are overrideable
> 
> Robert
> 
>> 
>> 
>>> 
>>> 
>>> Clearly with devops automation you have some tooling already in
>>> place
>>> which is using text files and does something to start AEM, so you
>>> can
>>> add the stuff there - or - the configuration are content packages
>>> after
>>> all, so if the automation supports content packages in some form,
>>> you're
>>> done.
>>> 
>> 
>> Agreed.
>> Best Regards
>> Ian
>> 
>> 
>>> 
>>> 
>>> Carsten
>>> 
 
 Hi,
 
 I think it must be possible to configure a Sling/AEM instance
 without
>>> using
 
 specific Sling/AEM tooling to create and manage the source
 configuration.
 The reason I say this is almost all serious deployments use some
 form of
 DevOps automation ranging from Puppet/Chef/AWS etc through to
 Docker/Mesos/Kubernetes etc. Almost without exception those
 systems work
 with configuration stored in text files (property, json, yaml
 etc), often
 stored in Git/SVN/Hg, edited by a human in a text editor. There
 are
>>> systems
 
 that only allow config via a GUI, but most of those only used,
 under
 sufferance, when there is no other option, (or competing
 product). The
 "customer" here is not a Sling/AEM expert, but a DevOps engineer
 who has
 never seen either before.
 
 If the only way to configure Sling/AEM is via special Sling/AEM
 tooling
 then we will have made the job of the TechOps/DevOps team even
 harder.
 
 I can think of lots of ways that merging at the property level
 could be
 made to work, but if we are not able to communicate clearly how
 it works,
 and all the potential side effects, to those whose job it is to
 make
 Sling/AEM work in production then we will have failed. If we have
 to
>>> resort
 
 to validation tools, compilers, special editors thats an
 indication the
 system is so complex we are not able to document it in plain
 english.
>>> For
 
 that reason, I beg, please keep whatever solution is chosen as
 simple as
 possible.
 
 I did try to document "object" level inheritance (ie aggregating
 properties), with all the side effects, but when I realised it
 was not
 going to be simple, I gave up and went back "object" level
 merging, which
 is brutal and simple by comparison.
 
 Best Regards
 Ian
 
 
 
 On 1 August 2016 at 08:29, Carsten Ziegeler  
>>> wrote:
 
 
> 
>> 
>> Hi everyone,
>> 
>> I personally see a tooling based approach complicated due to
>> the
> ownership
>> 
>> scenarios.
> Let's 

Re: [Context Aware Configs] Merging?

[Robert Munteanu]

On Mon, 2016-08-01 at 11:34 +0100, Ian Boston wrote:
> Hi,
> 
> On 1 August 2016 at 11:02, Carsten Ziegeler 
> wrote:
> 
> > 
> > Well, I think you're misinterpreting my comments - I never said
> > that it
> > will require a special tooling - I said it would be insane to edit
> > a
> > complex setup by editing nodes in the repository.
> > 
> 
> I was, apologies and thank you for the clarification.
> 
> No special tooling *and* a simple set of rules that are natural to
> communicate and document.
> 
> I don't have a strong opinion on merging or inheritance, provided its
> simple to use and the result if obvious with no strange side effects
> to
> catch a deployer out.

I think it's possible to have merging on a less fine-grained level, and
thus more controllable, by using multiple nodes for the same
configuration instance.

For example:

Global: {
'services' : {
'serviceA' {
'endpoint' : 'https://api.com/foo'
            'params': {
               'timeout': 500
            }
}
}
}

Site: {
'services' : {
'serviceA' {
'endpoint' : '
https://stage.api.com/foo'
}
}
}

the override only affects the 'endpoint' parameter. If at somepoint we decide 
to add a 'retryCount' parameter to 'params' this will also be propagated to 
inherited configurations.

Of course, this comes with the caveats that:

- we need to decide upfront what about how configurations are split into nodes
- we won't always be able to split configurations into nodes so that all 
properties are overrideable

Robert

> 
> 
> > 
> > 
> > Clearly with devops automation you have some tooling already in
> > place
> > which is using text files and does something to start AEM, so you
> > can
> > add the stuff there - or - the configuration are content packages
> > after
> > all, so if the automation supports content packages in some form,
> > you're
> > done.
> > 
> 
> Agreed.
> Best Regards
> Ian
> 
> 
> > 
> > 
> > Carsten
> > 
> > > 
> > > Hi,
> > > 
> > > I think it must be possible to configure a Sling/AEM instance
> > > without
> > using
> > > 
> > > specific Sling/AEM tooling to create and manage the source
> > > configuration.
> > > The reason I say this is almost all serious deployments use some
> > > form of
> > > DevOps automation ranging from Puppet/Chef/AWS etc through to
> > > Docker/Mesos/Kubernetes etc. Almost without exception those
> > > systems work
> > > with configuration stored in text files (property, json, yaml
> > > etc), often
> > > stored in Git/SVN/Hg, edited by a human in a text editor. There
> > > are
> > systems
> > > 
> > > that only allow config via a GUI, but most of those only used,
> > > under
> > > sufferance, when there is no other option, (or competing
> > > product). The
> > > "customer" here is not a Sling/AEM expert, but a DevOps engineer
> > > who has
> > > never seen either before.
> > > 
> > > If the only way to configure Sling/AEM is via special Sling/AEM
> > > tooling
> > > then we will have made the job of the TechOps/DevOps team even
> > > harder.
> > > 
> > > I can think of lots of ways that merging at the property level
> > > could be
> > > made to work, but if we are not able to communicate clearly how
> > > it works,
> > > and all the potential side effects, to those whose job it is to
> > > make
> > > Sling/AEM work in production then we will have failed. If we have
> > > to
> > resort
> > > 
> > > to validation tools, compilers, special editors thats an
> > > indication the
> > > system is so complex we are not able to document it in plain
> > > english.
> > For
> > > 
> > > that reason, I beg, please keep whatever solution is chosen as
> > > simple as
> > > possible.
> > > 
> > > I did try to document "object" level inheritance (ie aggregating
> > > properties), with all the side effects, but when I realised it
> > > was not
> > > going to be simple, I gave up and went back "object" level
> > > merging, which
> > > is brutal and simple by comparison.
> > > 
> > > Best Regards
> > > Ian
> > > 
> > > 
> > > 
> > > On 1 August 2016 at 08:29, Carsten Ziegeler  > > >
> > wrote:
> > > 
> > > 
> > > > 
> > > > > 
> > > > > Hi everyone,
> > > > > 
> > > > > I personally see a tooling based approach complicated due to
> > > > > the
> > > > ownership
> > > > > 
> > > > > scenarios.
> > > > Let's assume two scenarios, configuration A and configuration B
> > > > inherits
> > > > from A.
> > > > 
> > > > Now, if we support merging, someone is allowed to change A,
> > > > adds a new
> > > > property and this automatically influences B - and in this case
> > > > the
> > > > person who made changes to A does not even need to have access
> > > > rights to
> > > > change anything for B. (Which I think is scary).
> > > > 
> > > > If we don't support inheritance, there is still someone who is
> > > > allowed
> > > > to make a change to A - at first this has no influence on B. If
> > > > that
> > > > 

Re: [Context Aware Configs] Merging?

[Ian Boston]

Hi,

On 1 August 2016 at 11:02, Carsten Ziegeler  wrote:

> Well, I think you're misinterpreting my comments - I never said that it
> will require a special tooling - I said it would be insane to edit a
> complex setup by editing nodes in the repository.
>

I was, apologies and thank you for the clarification.

No special tooling *and* a simple set of rules that are natural to
communicate and document.

I don't have a strong opinion on merging or inheritance, provided its
simple to use and the result if obvious with no strange side effects to
catch a deployer out.


>
> Clearly with devops automation you have some tooling already in place
> which is using text files and does something to start AEM, so you can
> add the stuff there - or - the configuration are content packages after
> all, so if the automation supports content packages in some form, you're
> done.
>

Agreed.
Best Regards
Ian


>
> Carsten
>
> > Hi,
> >
> > I think it must be possible to configure a Sling/AEM instance without
> using
> > specific Sling/AEM tooling to create and manage the source configuration.
> > The reason I say this is almost all serious deployments use some form of
> > DevOps automation ranging from Puppet/Chef/AWS etc through to
> > Docker/Mesos/Kubernetes etc. Almost without exception those systems work
> > with configuration stored in text files (property, json, yaml etc), often
> > stored in Git/SVN/Hg, edited by a human in a text editor. There are
> systems
> > that only allow config via a GUI, but most of those only used, under
> > sufferance, when there is no other option, (or competing product). The
> > "customer" here is not a Sling/AEM expert, but a DevOps engineer who has
> > never seen either before.
> >
> > If the only way to configure Sling/AEM is via special Sling/AEM tooling
> > then we will have made the job of the TechOps/DevOps team even harder.
> >
> > I can think of lots of ways that merging at the property level could be
> > made to work, but if we are not able to communicate clearly how it works,
> > and all the potential side effects, to those whose job it is to make
> > Sling/AEM work in production then we will have failed. If we have to
> resort
> > to validation tools, compilers, special editors thats an indication the
> > system is so complex we are not able to document it in plain english.
> For
> > that reason, I beg, please keep whatever solution is chosen as simple as
> > possible.
> >
> > I did try to document "object" level inheritance (ie aggregating
> > properties), with all the side effects, but when I realised it was not
> > going to be simple, I gave up and went back "object" level merging, which
> > is brutal and simple by comparison.
> >
> > Best Regards
> > Ian
> >
> >
> >
> > On 1 August 2016 at 08:29, Carsten Ziegeler 
> wrote:
> >
> >>> Hi everyone,
> >>>
> >>> I personally see a tooling based approach complicated due to the
> >> ownership
> >>> scenarios.
> >> Let's assume two scenarios, configuration A and configuration B inherits
> >> from A.
> >>
> >> Now, if we support merging, someone is allowed to change A, adds a new
> >> property and this automatically influences B - and in this case the
> >> person who made changes to A does not even need to have access rights to
> >> change anything for B. (Which I think is scary).
> >>
> >> If we don't support inheritance, there is still someone who is allowed
> >> to make a change to A - at first this has no influence on B. If that
> >> same person is allowed to also change B and wants to "inherit" the new
> >> property to B, the tooling will allow so.
> >>
> >> With that respect, not having automatic inheritance sounds like the
> >> better option when it comes to access rights.
> >>
> >> Carsten
> >>
> >>> We are talking about context aware configuration that typically
> >>> would be applied by some runtime user that might have limited write
> >> access
> >>> (just for a specific context). The assumption that who writes to the
> >>> inheriting configuration node also has write access to the subsequent
> >>> levels is something we cannot and should not make here. I know multiple
> >>> customer scenarios where a central site (including global
> configuration)
> >>> was maintained by a central marketing organization while the markets
> >> owned
> >>> their own sites - so they declare inheritance (pull) instead of having
> >> the
> >>> global one pushed.
> >>>
> >>> The one thing in between that I could imagine might work sufficiently
> is
> >>> referencing (by path to property) instead of inheritance. This still
> >>> provides the lookup of "inheritance" at runtime where changes are
> >> reflected
> >>> in the lookup immediately but at the same time gets rid of magic
> >>> inheritance rules.
> >>>
> >>> Just a thought about an alternative in between.
> >>>
> >>> Cheers
> >>> Dominik
> >>>
> >>> On Mon, Aug 1, 2016 at 8:36 AM, Carsten Ziegeler  >
> >>> wrote:
> >>>
> 

Re: [Context Aware Configs] Merging?

[Carsten Ziegeler]

Well, I think you're misinterpreting my comments - I never said that it
will require a special tooling - I said it would be insane to edit a
complex setup by editing nodes in the repository.

Clearly with devops automation you have some tooling already in place
which is using text files and does something to start AEM, so you can
add the stuff there - or - the configuration are content packages after
all, so if the automation supports content packages in some form, you're
done.

Carsten

> Hi,
> 
> I think it must be possible to configure a Sling/AEM instance without using
> specific Sling/AEM tooling to create and manage the source configuration.
> The reason I say this is almost all serious deployments use some form of
> DevOps automation ranging from Puppet/Chef/AWS etc through to
> Docker/Mesos/Kubernetes etc. Almost without exception those systems work
> with configuration stored in text files (property, json, yaml etc), often
> stored in Git/SVN/Hg, edited by a human in a text editor. There are systems
> that only allow config via a GUI, but most of those only used, under
> sufferance, when there is no other option, (or competing product). The
> "customer" here is not a Sling/AEM expert, but a DevOps engineer who has
> never seen either before.
> 
> If the only way to configure Sling/AEM is via special Sling/AEM tooling
> then we will have made the job of the TechOps/DevOps team even harder.
> 
> I can think of lots of ways that merging at the property level could be
> made to work, but if we are not able to communicate clearly how it works,
> and all the potential side effects, to those whose job it is to make
> Sling/AEM work in production then we will have failed. If we have to resort
> to validation tools, compilers, special editors thats an indication the
> system is so complex we are not able to document it in plain english.  For
> that reason, I beg, please keep whatever solution is chosen as simple as
> possible.
> 
> I did try to document "object" level inheritance (ie aggregating
> properties), with all the side effects, but when I realised it was not
> going to be simple, I gave up and went back "object" level merging, which
> is brutal and simple by comparison.
> 
> Best Regards
> Ian
> 
> 
> 
> On 1 August 2016 at 08:29, Carsten Ziegeler  wrote:
> 
>>> Hi everyone,
>>>
>>> I personally see a tooling based approach complicated due to the
>> ownership
>>> scenarios.
>> Let's assume two scenarios, configuration A and configuration B inherits
>> from A.
>>
>> Now, if we support merging, someone is allowed to change A, adds a new
>> property and this automatically influences B - and in this case the
>> person who made changes to A does not even need to have access rights to
>> change anything for B. (Which I think is scary).
>>
>> If we don't support inheritance, there is still someone who is allowed
>> to make a change to A - at first this has no influence on B. If that
>> same person is allowed to also change B and wants to "inherit" the new
>> property to B, the tooling will allow so.
>>
>> With that respect, not having automatic inheritance sounds like the
>> better option when it comes to access rights.
>>
>> Carsten
>>
>>> We are talking about context aware configuration that typically
>>> would be applied by some runtime user that might have limited write
>> access
>>> (just for a specific context). The assumption that who writes to the
>>> inheriting configuration node also has write access to the subsequent
>>> levels is something we cannot and should not make here. I know multiple
>>> customer scenarios where a central site (including global configuration)
>>> was maintained by a central marketing organization while the markets
>> owned
>>> their own sites - so they declare inheritance (pull) instead of having
>> the
>>> global one pushed.
>>>
>>> The one thing in between that I could imagine might work sufficiently is
>>> referencing (by path to property) instead of inheritance. This still
>>> provides the lookup of "inheritance" at runtime where changes are
>> reflected
>>> in the lookup immediately but at the same time gets rid of magic
>>> inheritance rules.
>>>
>>> Just a thought about an alternative in between.
>>>
>>> Cheers
>>> Dominik
>>>
>>> On Mon, Aug 1, 2016 at 8:36 AM, Carsten Ziegeler 
>>> wrote:
>>>
>> ... if you have merging (or inheritance) then you change the
>> attribute of a parent and this influences all childs and you
>> have no idea what happens there...
>
> We can create a felix console plugin that shows for a given
> context path and config name the value of the property and the
> path from where it was taken - that way both developers and ops
> can quickly check what's going wrong if results are unexpected.
>
>> I personally think this comes all down to tooling ...
>
> So this could mean e.g. an additional maven plugin, which handles
> merging of configurations 

Re: [Context Aware Configs] Merging?

[Ian Boston]

Hi,

I think it must be possible to configure a Sling/AEM instance without using
specific Sling/AEM tooling to create and manage the source configuration.
The reason I say this is almost all serious deployments use some form of
DevOps automation ranging from Puppet/Chef/AWS etc through to
Docker/Mesos/Kubernetes etc. Almost without exception those systems work
with configuration stored in text files (property, json, yaml etc), often
stored in Git/SVN/Hg, edited by a human in a text editor. There are systems
that only allow config via a GUI, but most of those only used, under
sufferance, when there is no other option, (or competing product). The
"customer" here is not a Sling/AEM expert, but a DevOps engineer who has
never seen either before.

If the only way to configure Sling/AEM is via special Sling/AEM tooling
then we will have made the job of the TechOps/DevOps team even harder.

I can think of lots of ways that merging at the property level could be
made to work, but if we are not able to communicate clearly how it works,
and all the potential side effects, to those whose job it is to make
Sling/AEM work in production then we will have failed. If we have to resort
to validation tools, compilers, special editors thats an indication the
system is so complex we are not able to document it in plain english.  For
that reason, I beg, please keep whatever solution is chosen as simple as
possible.

I did try to document "object" level inheritance (ie aggregating
properties), with all the side effects, but when I realised it was not
going to be simple, I gave up and went back "object" level merging, which
is brutal and simple by comparison.

Best Regards
Ian



On 1 August 2016 at 08:29, Carsten Ziegeler  wrote:

> > Hi everyone,
> >
> > I personally see a tooling based approach complicated due to the
> ownership
> > scenarios.
> Let's assume two scenarios, configuration A and configuration B inherits
> from A.
>
> Now, if we support merging, someone is allowed to change A, adds a new
> property and this automatically influences B - and in this case the
> person who made changes to A does not even need to have access rights to
> change anything for B. (Which I think is scary).
>
> If we don't support inheritance, there is still someone who is allowed
> to make a change to A - at first this has no influence on B. If that
> same person is allowed to also change B and wants to "inherit" the new
> property to B, the tooling will allow so.
>
> With that respect, not having automatic inheritance sounds like the
> better option when it comes to access rights.
>
> Carsten
>
> > We are talking about context aware configuration that typically
> > would be applied by some runtime user that might have limited write
> access
> > (just for a specific context). The assumption that who writes to the
> > inheriting configuration node also has write access to the subsequent
> > levels is something we cannot and should not make here. I know multiple
> > customer scenarios where a central site (including global configuration)
> > was maintained by a central marketing organization while the markets
> owned
> > their own sites - so they declare inheritance (pull) instead of having
> the
> > global one pushed.
> >
> > The one thing in between that I could imagine might work sufficiently is
> > referencing (by path to property) instead of inheritance. This still
> > provides the lookup of "inheritance" at runtime where changes are
> reflected
> > in the lookup immediately but at the same time gets rid of magic
> > inheritance rules.
> >
> > Just a thought about an alternative in between.
> >
> > Cheers
> > Dominik
> >
> > On Mon, Aug 1, 2016 at 8:36 AM, Carsten Ziegeler 
> > wrote:
> >
>  ... if you have merging (or inheritance) then you change the
>  attribute of a parent and this influences all childs and you
>  have no idea what happens there...
> >>>
> >>> We can create a felix console plugin that shows for a given
> >>> context path and config name the value of the property and the
> >>> path from where it was taken - that way both developers and ops
> >>> can quickly check what's going wrong if results are unexpected.
> >>>
>  I personally think this comes all down to tooling ...
> >>>
> >>> So this could mean e.g. an additional maven plugin, which handles
> >>> merging of configurations and then the runtime can work without
> >>> merging. The problem I see with that is that
> >>> * you end up having two similar configuration model types (one in
> >>>   the source code that supports merging and one effective one for
> >>>   the runtime) - this makes the mechanism harder to understand
> >>>   for everyone
> >>> * the tooling has to be created and IMHO it'll not be easier/less
> >>>   code than creating a smarter runtime (even if we take into
> >>>   account that we have to create the felix console plugin to make
> >>>   it traceable)
> >>>
> >> First of all, you need 

Re: [Context Aware Configs] Merging?

[Carsten Ziegeler]

> Hi everyone,
> 
> I personally see a tooling based approach complicated due to the ownership
> scenarios. 
Let's assume two scenarios, configuration A and configuration B inherits
from A.

Now, if we support merging, someone is allowed to change A, adds a new
property and this automatically influences B - and in this case the
person who made changes to A does not even need to have access rights to
change anything for B. (Which I think is scary).

If we don't support inheritance, there is still someone who is allowed
to make a change to A - at first this has no influence on B. If that
same person is allowed to also change B and wants to "inherit" the new
property to B, the tooling will allow so.

With that respect, not having automatic inheritance sounds like the
better option when it comes to access rights.

Carsten

> We are talking about context aware configuration that typically
> would be applied by some runtime user that might have limited write access
> (just for a specific context). The assumption that who writes to the
> inheriting configuration node also has write access to the subsequent
> levels is something we cannot and should not make here. I know multiple
> customer scenarios where a central site (including global configuration)
> was maintained by a central marketing organization while the markets owned
> their own sites - so they declare inheritance (pull) instead of having the
> global one pushed.
> 
> The one thing in between that I could imagine might work sufficiently is
> referencing (by path to property) instead of inheritance. This still
> provides the lookup of "inheritance" at runtime where changes are reflected
> in the lookup immediately but at the same time gets rid of magic
> inheritance rules.
> 
> Just a thought about an alternative in between.
> 
> Cheers
> Dominik
> 
> On Mon, Aug 1, 2016 at 8:36 AM, Carsten Ziegeler 
> wrote:
> 
 ... if you have merging (or inheritance) then you change the
 attribute of a parent and this influences all childs and you
 have no idea what happens there...
>>>
>>> We can create a felix console plugin that shows for a given
>>> context path and config name the value of the property and the
>>> path from where it was taken - that way both developers and ops
>>> can quickly check what's going wrong if results are unexpected.
>>>
 I personally think this comes all down to tooling ...
>>>
>>> So this could mean e.g. an additional maven plugin, which handles
>>> merging of configurations and then the runtime can work without
>>> merging. The problem I see with that is that
>>> * you end up having two similar configuration model types (one in
>>>   the source code that supports merging and one effective one for
>>>   the runtime) - this makes the mechanism harder to understand
>>>   for everyone
>>> * the tooling has to be created and IMHO it'll not be easier/less
>>>   code than creating a smarter runtime (even if we take into
>>>   account that we have to create the felix console plugin to make
>>>   it traceable)
>>>
>> First of all, you need tooling anyway. It would be foolish to make
>> these configurations without any tooling by hand. Tooling can easily
>> be customized to anyones needs without tampering the implementation.
>> The implementation at runtime can be simply and effective.
>>
>> We had similar discussions while starting the work for the OSGi
>> Configurator which allows you to put OSGi configurations into a bundle
>> which then get applied at runtime. My initial proposal had all the
>> things requested here as well (merging, inheritance etc.). It turned
>> out that this messed up the specification and made the implementation
>> extraordinary complicated. While moving this into the tooling which
>> creates the bundles in the first place solves all the problems.
>>
>> Just think of all the different cases you have to solve once you
>> start support merging. And not to mention the runtime implications as
>> you have to do the merging dynamically at runtime as well, or you
>> start introducing all kinds of caching again. All of this is not
>> required if you move this to the tooling level.
>>
>>  Regards
>>
>> Carsten
>>
>> --
>> Carsten Ziegeler
>> Adobe Research Switzerland
>> cziege...@apache.org
>>
>>
> 


 

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: [Context Aware Configs] Merging?

[Dominik Süß]

Hi everyone,

I personally see a tooling based approach complicated due to the ownership
scenarios. We are talking about context aware configuration that typically
would be applied by some runtime user that might have limited write access
(just for a specific context). The assumption that who writes to the
inheriting configuration node also has write access to the subsequent
levels is something we cannot and should not make here. I know multiple
customer scenarios where a central site (including global configuration)
was maintained by a central marketing organization while the markets owned
their own sites - so they declare inheritance (pull) instead of having the
global one pushed.

The one thing in between that I could imagine might work sufficiently is
referencing (by path to property) instead of inheritance. This still
provides the lookup of "inheritance" at runtime where changes are reflected
in the lookup immediately but at the same time gets rid of magic
inheritance rules.

Just a thought about an alternative in between.

Cheers
Dominik

On Mon, Aug 1, 2016 at 8:36 AM, Carsten Ziegeler 
wrote:

> >> ... if you have merging (or inheritance) then you change the
> >> attribute of a parent and this influences all childs and you
> >> have no idea what happens there...
> >
> > We can create a felix console plugin that shows for a given
> > context path and config name the value of the property and the
> > path from where it was taken - that way both developers and ops
> > can quickly check what's going wrong if results are unexpected.
> >
> >> I personally think this comes all down to tooling ...
> >
> > So this could mean e.g. an additional maven plugin, which handles
> > merging of configurations and then the runtime can work without
> > merging. The problem I see with that is that
> > * you end up having two similar configuration model types (one in
> >   the source code that supports merging and one effective one for
> >   the runtime) - this makes the mechanism harder to understand
> >   for everyone
> > * the tooling has to be created and IMHO it'll not be easier/less
> >   code than creating a smarter runtime (even if we take into
> >   account that we have to create the felix console plugin to make
> >   it traceable)
> >
> First of all, you need tooling anyway. It would be foolish to make
> these configurations without any tooling by hand. Tooling can easily
> be customized to anyones needs without tampering the implementation.
> The implementation at runtime can be simply and effective.
>
> We had similar discussions while starting the work for the OSGi
> Configurator which allows you to put OSGi configurations into a bundle
> which then get applied at runtime. My initial proposal had all the
> things requested here as well (merging, inheritance etc.). It turned
> out that this messed up the specification and made the implementation
> extraordinary complicated. While moving this into the tooling which
> creates the bundles in the first place solves all the problems.
>
> Just think of all the different cases you have to solve once you
> start support merging. And not to mention the runtime implications as
> you have to do the merging dynamically at runtime as well, or you
> start introducing all kinds of caching again. All of this is not
> required if you move this to the tooling level.
>
>  Regards
>
> Carsten
>
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziege...@apache.org
>
>

Re: [Context Aware Configs] Merging?

[Carsten Ziegeler]

>> ... if you have merging (or inheritance) then you change the
>> attribute of a parent and this influences all childs and you
>> have no idea what happens there...
> 
> We can create a felix console plugin that shows for a given
> context path and config name the value of the property and the
> path from where it was taken - that way both developers and ops
> can quickly check what's going wrong if results are unexpected.
> 
>> I personally think this comes all down to tooling ...
> 
> So this could mean e.g. an additional maven plugin, which handles
> merging of configurations and then the runtime can work without
> merging. The problem I see with that is that
> * you end up having two similar configuration model types (one in
>   the source code that supports merging and one effective one for
>   the runtime) - this makes the mechanism harder to understand
>   for everyone
> * the tooling has to be created and IMHO it'll not be easier/less
>   code than creating a smarter runtime (even if we take into
>   account that we have to create the felix console plugin to make
>   it traceable)
> 
First of all, you need tooling anyway. It would be foolish to make
these configurations without any tooling by hand. Tooling can easily
be customized to anyones needs without tampering the implementation.
The implementation at runtime can be simply and effective.

We had similar discussions while starting the work for the OSGi
Configurator which allows you to put OSGi configurations into a bundle
which then get applied at runtime. My initial proposal had all the
things requested here as well (merging, inheritance etc.). It turned
out that this messed up the specification and made the implementation
extraordinary complicated. While moving this into the tooling which
creates the bundles in the first place solves all the problems.

Just think of all the different cases you have to solve once you
start support merging. And not to mention the runtime implications as
you have to do the merging dynamically at runtime as well, or you
start introducing all kinds of caching again. All of this is not
required if you move this to the tooling level.

 Regards

Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: [Context Aware Configs] Merging?

[Georg Henzler]

... if you have merging (or inheritance) then you change the
attribute of a parent and this influences all childs and you
have no idea what happens there...


We can create a felix console plugin that shows for a given
context path and config name the value of the property and the
path from where it was taken - that way both developers and ops
can quickly check what's going wrong if results are unexpected.


I personally think this comes all down to tooling ...


So this could mean e.g. an additional maven plugin, which handles
merging of configurations and then the runtime can work without
merging. The problem I see with that is that
* you end up having two similar configuration model types (one in
  the source code that supports merging and one effective one for
  the runtime) - this makes the mechanism harder to understand
  for everyone
* the tooling has to be created and IMHO it'll not be easier/less
  code than creating a smarter runtime (even if we take into
  account that we have to create the felix console plugin to make
  it traceable)

Regards
Georg

Re: [Context Aware Configs] Merging?

[Carsten Ziegeler]

> Allow me to disagree. This lack of merging is actually one of the key pain
> points I have with OSGi ConfigAdmin and I would really love to see it
> accommodated here. The problem with not merging is that it results in a lot
> of error-prone duplication and you can easily get into a state where new
> default values (in the global configuration) are not propagated as expected
> because once override any configuration property you are effectively
> hard-coding the *current* global configuration in your override. Future
> changes (or even additions) to that global configuration does not have the
> expected result.
> 
> Consider this case:
> 
> Global: {
> 'services' : {
> 'serviceA' {
> 'endpoint' : 'https://api.com/foo'
> }
> }
> }
> 
> 
> Site: {
> 'services' : {
> 'serviceA' {
> 'endpoint' : 'https://stage.api.com/foo'
> }
> }
> }
> 
> Works OK. But at some point down the line, the developer of the component
> consuming this configuration finds that a timeout value needs to be
> specified. They do the needful and modify the global configuration to be:
> 
> {
> 'services' : {
> 'serviceA' {
> 'endpoint' : 'https://api.com/foo',
> 'timeout' : 500
> }
> }
> }
> 
> I'd suggest that the reasonable expectation is that the timeout would apply
> to all contexts. But without merging, this won't happen.
> 
> Merging is hard to add after-the-fact because it is almost impossible to
> make backwards-compatible. Since we are starting with a blank slate here,
> I'd strongly suggest that this new capability support merging from the
> start.
> 
> I would also disagree that this isn't a problem already. The adoption of
> service users is a recent specific case where the failure of JcrInstaller
> to support merging was an impediment. We had to add the "amended"
> configuration PID specifically to work around that problem because it was
> pretty clearly nonscalable to have a centralized list of service user
> configurations. I could go through my archives of 6 years of AEM projects
> and easily find hundreds of overridden sling:OsgiConfig nodes which had
> properties totally unrelated to the desired configuration change. But since
> the configuration is all there, there's no real way to know (save for
> documentation) what *specific* configuration change was desired vs. what
> properties had to be copied just to get a component to retain its original
> behavior.
> 

Well, I could argue the opposite: if you have merging (or inheritance) then
you change the attribute of a parent and this influences all childs and you
have no idea what happens there and worst case you end up editing all child
configurations by overriding that property.

I personally think this comes all down to tooling - no one (at least I
hope so) is editing all these configurations by hand in the repository.
That's - regardless if you have merging or not - error prone.
So a provisioning tool can do all what is requested here: it knows the
parent configuration, it knows the inheritance/merging rules and it
can apply the changes in the way the editor of the configuration intents
to. And that means we *could* move all this functionality into the
tooling but keep the implementation at run time crystal clear.

Regards
Carsten

 

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: [Context Aware Configs] Merging?

[Roy Teeuwen]

+1 for me too, one of the real life examples that I have in most of my projects 
is that you have a site /content/mysite and you have some languages like 
/content/mysite/en and /content/mysite/fr

It would be nice to put all your global mysite configs on /content/mysite, but 
still have the possibility to override and make new ones on language level. 
Or is this not even applicable with the context aware configs?

Greets
Roy
> On 30 Jul 2016, at 10:10, Georg Henzler  wrote:
> 
> A big +1 from my side to support merging (all the way down to property level) 
> from the very first release. IMHO it's not that hard to implement and it 
> saves a lot of frustration downstream.
> 
> On 2016-07-29 17:10, Justin Edelson wrote:
>> Allow me to disagree. This lack of merging is actually one of the key pain
>> points I have with OSGi ConfigAdmin and I would really love to see it
>> accommodated here. The problem with not merging is that it results in a lot
>> of error-prone duplication and you can easily get into a state where new
>> default values (in the global configuration) are not propagated as expected
>> because once override any configuration property you are effectively
>> hard-coding the *current* global configuration in your override. Future
>> changes (or even additions) to that global configuration does not have the
>> expected result.
>> Consider this case:
>> Global: {
>>'services' : {
>>'serviceA' {
>>'endpoint' : 'https://api.com/foo'
>>}
>>}
>> }
>> Site: {
>>'services' : {
>>'serviceA' {
>>'endpoint' : 'https://stage.api.com/foo'
>>}
>>}
>> }
>> Works OK. But at some point down the line, the developer of the component
>> consuming this configuration finds that a timeout value needs to be
>> specified. They do the needful and modify the global configuration to be:
>> {
>>'services' : {
>>'serviceA' {
>>'endpoint' : 'https://api.com/foo',
>>'timeout' : 500
>>}
>>}
>> }
>> I'd suggest that the reasonable expectation is that the timeout would apply
>> to all contexts. But without merging, this won't happen.
>> Merging is hard to add after-the-fact because it is almost impossible to
>> make backwards-compatible. Since we are starting with a blank slate here,
>> I'd strongly suggest that this new capability support merging from the
>> start.
>> I would also disagree that this isn't a problem already. The adoption of
>> service users is a recent specific case where the failure of JcrInstaller
>> to support merging was an impediment. We had to add the "amended"
>> configuration PID specifically to work around that problem because it was
>> pretty clearly nonscalable to have a centralized list of service user
>> configurations. I could go through my archives of 6 years of AEM projects
>> and easily find hundreds of overridden sling:OsgiConfig nodes which had
>> properties totally unrelated to the desired configuration change. But since
>> the configuration is all there, there's no real way to know (save for
>> documentation) what *specific* configuration change was desired vs. what
>> properties had to be copied just to get a component to retain its original
>> behavior.
>> Caveat - In a former life, I worked extensively with the dependency
>> injection framework ATG Nucleus for a large, multi-property, multi-site
>> deployment which supported configuration merging and I've spent the last
>> many years pining away over this one aspect of Nucleus.
>> Justin
>> On Fri, Jul 29, 2016 at 10:39 AM Carsten Ziegeler 
>> wrote:
>>> > Hi,
>>> > Properties should not be merged, as that will produce undesirable side
>>> > effects, however some things need to be merged.
>>> >
>>> > Having attempted to write documentation I found it easiest to describe in
>>> > terms of json documents.
>>> > Using the examples in
>>> > https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869
>>> >
>>> > given 2 configs from different locations
>>> >
>>> > Location A
>>> >
>>> > {
>>> > "socialmedia" : {
>>> > "youtube" : {
>>> > "enabled" : false,
>>> > "url" : "https://youtube.com;
>>> > },
>>> > "facebook" : {
>>> > "enabled" : false,
>>> > "apiusage" : false,
>>> > "url" : "https://youtube.com;
>>> > },
>>> >
>>> > Location B
>>> >
>>> >  {
>>> > "socialmedia" : {
>>> > "facebook" : {
>>> > "enabled" : true,
>>> > "url" : "https://facebook.com;
>>> > }
>>> > }
>>> > }
>>> >
>>> >
>>> > The result of A then B should be
>>> >
>>> > {
>>> > "socialmedia" : {
>>> > "youtube" : {
>>> > "enabled" : false,
>>> > "url" : "https://youtube.com;
>>> > },
>>> > "facebook" : {
>>> >

Re: [Context Aware Configs] Merging?

[Georg Henzler]

A big +1 from my side to support merging (all the way down to property 
level) from the very first release. IMHO it's not that hard to implement 
and it saves a lot of frustration downstream.


On 2016-07-29 17:10, Justin Edelson wrote:
Allow me to disagree. This lack of merging is actually one of the key 
pain

points I have with OSGi ConfigAdmin and I would really love to see it
accommodated here. The problem with not merging is that it results in a 
lot
of error-prone duplication and you can easily get into a state where 
new
default values (in the global configuration) are not propagated as 
expected

because once override any configuration property you are effectively
hard-coding the *current* global configuration in your override. Future
changes (or even additions) to that global configuration does not have 
the

expected result.

Consider this case:

Global: {
'services' : {
'serviceA' {
'endpoint' : 'https://api.com/foo'
}
}
}


Site: {
'services' : {
'serviceA' {
'endpoint' : 'https://stage.api.com/foo'
}
}
}

Works OK. But at some point down the line, the developer of the 
component

consuming this configuration finds that a timeout value needs to be
specified. They do the needful and modify the global configuration to 
be:


{
'services' : {
'serviceA' {
'endpoint' : 'https://api.com/foo',
'timeout' : 500
}
}
}

I'd suggest that the reasonable expectation is that the timeout would 
apply

to all contexts. But without merging, this won't happen.

Merging is hard to add after-the-fact because it is almost impossible 
to
make backwards-compatible. Since we are starting with a blank slate 
here,

I'd strongly suggest that this new capability support merging from the
start.

I would also disagree that this isn't a problem already. The adoption 
of
service users is a recent specific case where the failure of 
JcrInstaller

to support merging was an impediment. We had to add the "amended"
configuration PID specifically to work around that problem because it 
was

pretty clearly nonscalable to have a centralized list of service user
configurations. I could go through my archives of 6 years of AEM 
projects

and easily find hundreds of overridden sling:OsgiConfig nodes which had
properties totally unrelated to the desired configuration change. But 
since

the configuration is all there, there's no real way to know (save for
documentation) what *specific* configuration change was desired vs. 
what
properties had to be copied just to get a component to retain its 
original

behavior.

Caveat - In a former life, I worked extensively with the dependency
injection framework ATG Nucleus for a large, multi-property, multi-site
deployment which supported configuration merging and I've spent the 
last

many years pining away over this one aspect of Nucleus.

Justin

On Fri, Jul 29, 2016 at 10:39 AM Carsten Ziegeler 


wrote:


> Hi,
> Properties should not be merged, as that will produce undesirable side
> effects, however some things need to be merged.
>
> Having attempted to write documentation I found it easiest to describe in
> terms of json documents.
> Using the examples in
> https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869
>
> given 2 configs from different locations
>
> Location A
>
> {
> "socialmedia" : {
> "youtube" : {
> "enabled" : false,
> "url" : "https://youtube.com;
> },
> "facebook" : {
> "enabled" : false,
> "apiusage" : false,
> "url" : "https://youtube.com;
> },
>
> Location B
>
>  {
> "socialmedia" : {
> "facebook" : {
> "enabled" : true,
> "url" : "https://facebook.com;
> }
> }
> }
>
>
> The result of A then B should be
>
> {
> "socialmedia" : {
> "youtube" : {
> "enabled" : false,
> "url" : "https://youtube.com;
> },
> "facebook" : {
> "enabled" : true,
> "url" : "https://facebook.com;
> }
> }
> }
>
>
> ie Objects are merged but properties are not. socialmedia.facebook from A
> does not add @apiusage to the result, as socialmedia.facebook form B
> overwrites socialmedia.facebook from A.
>
> That will allow a deployer to completely replace a configuration "object"
> without having to get their head around more complex inheritance rules.
> This does assume that configurations are collected together into
"objects".
> I think its is important to keep it simple.
> If this isnt simple enough then it needs some more thought.
>

Yes, I think we agree on this, and that's how my implementation
currently works). So for your above example if we have "A then B" and
you get the collection for "socialmedia" you 

Re: [Context Aware Configs] Merging?

[Ian Boston]

Hi,
Properties should not be merged, as that will produce undesirable side
effects, however some things need to be merged.

Having attempted to write documentation I found it easiest to describe in
terms of json documents.
Using the examples in
https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869

given 2 configs from different locations

Location A

{
"socialmedia" : {
"youtube" : {
"enabled" : false,
"url" : "https://youtube.com;
},
"facebook" : {
"enabled" : false,
"apiusage" : false,
"url" : "https://youtube.com;
},

Location B

 {
"socialmedia" : {
"facebook" : {
"enabled" : true,
"url" : "https://facebook.com;
}
}
}


The result of A then B should be

{
"socialmedia" : {
"youtube" : {
"enabled" : false,
"url" : "https://youtube.com;
},
"facebook" : {
"enabled" : true,
"url" : "https://facebook.com;
}
}
}


ie Objects are merged but properties are not. socialmedia.facebook from A
does not add @apiusage to the result, as socialmedia.facebook form B
overwrites socialmedia.facebook from A.

That will allow a deployer to completely replace a configuration "object"
without having to get their head around more complex inheritance rules.
This does assume that configurations are collected together into "objects".
I think its is important to keep it simple.
If this isnt simple enough then it needs some more thought.


Best Regards
Ian

On 29 July 2016 at 15:11, Carsten Ziegeler  wrote:

> Hi,
>
> looking at various documentations on the different approaches and the
> implementations, there seem to be totally different opinions about if
> merging of resources (and therefore resources) should be done.
>
> For this topic, let's not discuss names or how the resolving really
> should work, let's focus on one topic: merging.
>
> Let's assume you have a global config at:
> /config/global/foo - with properties @a, @b
> and a content aware one at:
> /config/mysite/foo - with properties @b, @c
>
> A content resource now sets /config/mysite as its context. If you get
> the "foo" configuration, do we merge the properties and the result will
> have @a, @b, and @c? Or is it just @b and @c?
>
> I personally think we should *not* support any merging because as soon
> as you support merging you will need additional ways of not merging
> properties etc etc. I think a configuration should be the full set of
> properties. The confmgr implementation works exactly this way as well.
>
> Regards
> Carsten
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziege...@apache.org
>

Re: [Context Aware Configs] Merging?

[Justin Edelson]

Allow me to disagree. This lack of merging is actually one of the key pain
points I have with OSGi ConfigAdmin and I would really love to see it
accommodated here. The problem with not merging is that it results in a lot
of error-prone duplication and you can easily get into a state where new
default values (in the global configuration) are not propagated as expected
because once override any configuration property you are effectively
hard-coding the *current* global configuration in your override. Future
changes (or even additions) to that global configuration does not have the
expected result.

Consider this case:

Global: {
'services' : {
'serviceA' {
'endpoint' : 'https://api.com/foo'
}
}
}


Site: {
'services' : {
'serviceA' {
'endpoint' : 'https://stage.api.com/foo'
}
}
}

Works OK. But at some point down the line, the developer of the component
consuming this configuration finds that a timeout value needs to be
specified. They do the needful and modify the global configuration to be:

{
'services' : {
'serviceA' {
'endpoint' : 'https://api.com/foo',
'timeout' : 500
}
}
}

I'd suggest that the reasonable expectation is that the timeout would apply
to all contexts. But without merging, this won't happen.

Merging is hard to add after-the-fact because it is almost impossible to
make backwards-compatible. Since we are starting with a blank slate here,
I'd strongly suggest that this new capability support merging from the
start.

I would also disagree that this isn't a problem already. The adoption of
service users is a recent specific case where the failure of JcrInstaller
to support merging was an impediment. We had to add the "amended"
configuration PID specifically to work around that problem because it was
pretty clearly nonscalable to have a centralized list of service user
configurations. I could go through my archives of 6 years of AEM projects
and easily find hundreds of overridden sling:OsgiConfig nodes which had
properties totally unrelated to the desired configuration change. But since
the configuration is all there, there's no real way to know (save for
documentation) what *specific* configuration change was desired vs. what
properties had to be copied just to get a component to retain its original
behavior.

Caveat - In a former life, I worked extensively with the dependency
injection framework ATG Nucleus for a large, multi-property, multi-site
deployment which supported configuration merging and I've spent the last
many years pining away over this one aspect of Nucleus.

Justin

On Fri, Jul 29, 2016 at 10:39 AM Carsten Ziegeler 
wrote:

> > Hi,
> > Properties should not be merged, as that will produce undesirable side
> > effects, however some things need to be merged.
> >
> > Having attempted to write documentation I found it easiest to describe in
> > terms of json documents.
> > Using the examples in
> > https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869
> >
> > given 2 configs from different locations
> >
> > Location A
> >
> > {
> > "socialmedia" : {
> > "youtube" : {
> > "enabled" : false,
> > "url" : "https://youtube.com;
> > },
> > "facebook" : {
> > "enabled" : false,
> > "apiusage" : false,
> > "url" : "https://youtube.com;
> > },
> >
> > Location B
> >
> >  {
> > "socialmedia" : {
> > "facebook" : {
> > "enabled" : true,
> > "url" : "https://facebook.com;
> > }
> > }
> > }
> >
> >
> > The result of A then B should be
> >
> > {
> > "socialmedia" : {
> > "youtube" : {
> > "enabled" : false,
> > "url" : "https://youtube.com;
> > },
> > "facebook" : {
> > "enabled" : true,
> > "url" : "https://facebook.com;
> > }
> > }
> > }
> >
> >
> > ie Objects are merged but properties are not. socialmedia.facebook from A
> > does not add @apiusage to the result, as socialmedia.facebook form B
> > overwrites socialmedia.facebook from A.
> >
> > That will allow a deployer to completely replace a configuration "object"
> > without having to get their head around more complex inheritance rules.
> > This does assume that configurations are collected together into
> "objects".
> > I think its is important to keep it simple.
> > If this isnt simple enough then it needs some more thought.
> >
>
> Yes, I think we agree on this, and that's how my implementation
> currently works). So for your above example if we have "A then B" and
> you get the collection for "socialmedia" you get "youtube" and
> "facebook" with the properties you list above.
>
> This is merging on a resource collection level and yes, we must 

Re: [Context Aware Configs] Merging?

[Chetan Mehrotra]

Not having any specific opinion here but just for an example. Maven
allows merging of config [1] but that requires parent config to
declare/allow the child to merge there config. So merging "intention"
is explicitly mentioned.

So if we plan to support this sometime later (with a big if!) then
probably we can have such meta config settings to express this merge
intention.

Chetan Mehrotra
[1] 
http://blog.sonatype.com/2011/01/maven-how-to-merging-plugin-configuration-in-complex-projects/


On Fri, Jul 29, 2016 at 8:09 PM, Carsten Ziegeler  wrote:
>> Hi,
>> Properties should not be merged, as that will produce undesirable side
>> effects, however some things need to be merged.
>>
>> Having attempted to write documentation I found it easiest to describe in
>> terms of json documents.
>> Using the examples in
>> https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869
>>
>> given 2 configs from different locations
>>
>> Location A
>>
>> {
>> "socialmedia" : {
>> "youtube" : {
>> "enabled" : false,
>> "url" : "https://youtube.com;
>> },
>> "facebook" : {
>> "enabled" : false,
>> "apiusage" : false,
>> "url" : "https://youtube.com;
>> },
>>
>> Location B
>>
>>  {
>> "socialmedia" : {
>> "facebook" : {
>> "enabled" : true,
>> "url" : "https://facebook.com;
>> }
>> }
>> }
>>
>>
>> The result of A then B should be
>>
>> {
>> "socialmedia" : {
>> "youtube" : {
>> "enabled" : false,
>> "url" : "https://youtube.com;
>> },
>> "facebook" : {
>> "enabled" : true,
>> "url" : "https://facebook.com;
>> }
>> }
>> }
>>
>>
>> ie Objects are merged but properties are not. socialmedia.facebook from A
>> does not add @apiusage to the result, as socialmedia.facebook form B
>> overwrites socialmedia.facebook from A.
>>
>> That will allow a deployer to completely replace a configuration "object"
>> without having to get their head around more complex inheritance rules.
>> This does assume that configurations are collected together into "objects".
>> I think its is important to keep it simple.
>> If this isnt simple enough then it needs some more thought.
>>
>
> Yes, I think we agree on this, and that's how my implementation
> currently works). So for your above example if we have "A then B" and
> you get the collection for "socialmedia" you get "youtube" and
> "facebook" with the properties you list above.
>
> This is merging on a resource collection level and yes, we must do that.
> My post here was more about merging properties, sorry that I didn't make
> this clear :)
>
> But it's great that we have a common understand now
>
> Thanks
> Carsten
>
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziege...@apache.org
>

Re: [Context Aware Configs] Merging?

[Carsten Ziegeler]

> Hi,
> Properties should not be merged, as that will produce undesirable side
> effects, however some things need to be merged.
> 
> Having attempted to write documentation I found it easiest to describe in
> terms of json documents.
> Using the examples in
> https://gist.github.com/ieb/460504e79c861cb980f4f0154210a869
> 
> given 2 configs from different locations
> 
> Location A
> 
> {
> "socialmedia" : {
> "youtube" : {
> "enabled" : false,
> "url" : "https://youtube.com;
> },
> "facebook" : {
> "enabled" : false,
> "apiusage" : false,
> "url" : "https://youtube.com;
> },
> 
> Location B
> 
>  {
> "socialmedia" : {
> "facebook" : {
> "enabled" : true,
> "url" : "https://facebook.com;
> }
> }
> }
> 
> 
> The result of A then B should be
> 
> {
> "socialmedia" : {
> "youtube" : {
> "enabled" : false,
> "url" : "https://youtube.com;
> },
> "facebook" : {
> "enabled" : true,
> "url" : "https://facebook.com;
> }
> }
> }
> 
> 
> ie Objects are merged but properties are not. socialmedia.facebook from A
> does not add @apiusage to the result, as socialmedia.facebook form B
> overwrites socialmedia.facebook from A.
> 
> That will allow a deployer to completely replace a configuration "object"
> without having to get their head around more complex inheritance rules.
> This does assume that configurations are collected together into "objects".
> I think its is important to keep it simple.
> If this isnt simple enough then it needs some more thought.
> 

Yes, I think we agree on this, and that's how my implementation
currently works). So for your above example if we have "A then B" and
you get the collection for "socialmedia" you get "youtube" and
"facebook" with the properties you list above.

This is merging on a resource collection level and yes, we must do that.
My post here was more about merging properties, sorry that I didn't make
this clear :)

But it's great that we have a common understand now

Thanks
Carsten

--
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: [Context Aware Configs] Merging?

[Bertrand Delacretaz]

Hi,

On Fri, Jul 29, 2016 at 4:11 PM, Carsten Ziegeler  wrote:
> ...I personally think we should *not* support any merging because as soon
> as you support merging you will need additional ways of not merging
> properties etc etc

Combining configs sounds a lot like inheriting Java classes...maybe we
need attributes like final, abstract,
not-inherited-unless-explicitly-specified, etc.

IMO as soon as configs can be inherited/combined  the most important
thing is a tracing tool to be able to find out exactly why a given
value is what it is.

-Bertrand