On Fri, Dec 25, 2015 at 7:51 PM, David Jencks <[email protected]> wrote:
> Benson,
>
> What you did is pretty similar in many ways to what I did although I don’t
> have the json/yaml support.
I'm hardly attached to 4 hours of hacking. If you're willing to take
Jackson deps, I'd be game to collaborate and add json/yaml(/xml) to
what you have.
>
> I’m not sure the [] characters are appropriate for key tokens. Anyway my
> encoding uses key1.<number>.key2.<number>….. instead of
> key1[<number>].key2[<number>]…..
I think I could switch to that. On the other hand, can you elaborate
on what's wrong with []? I picked them because it was a less tricky
parse (no "oh, a _number_, that must mean it's a sequence") but I
don't feel very strongly about it.
>
> Since I’m interested in using DS spec features as much as possible, the
> object model is annotations or interfaces rather than concrete classes. So
> if your DS component has
>
> @Activate
> void activate(MyNestedConfigAnnotation config) {…}
>
> DS will generate implementations of the config annotation backed by the CA
> properties map.
I'd be interested to see a fully-worked-out example of this. I often
end up using a fair amount of Jackson @nnotation to tune the
representation. It seems possible that they could coexist; Jackson has
support deserializing to concrete classes as informed by annotations.
>
> I just realized that I haven’t yet made it easy to turn this on by committing
> the meta-annotation to generate the felix specific flag to allow nested
> annotations….. maybe this weekend.
>
> While both you and I picked complex keys, Peter Kriens seems to be strongly
> in favor of complex values encoded in json. I believe one of his reasons is
> that such configurations are less likely to completely break existing
> metatype-aware config viewers.
I waffled back and forth between 'CA is fundamentally a key-value
store, and I need fancy values' and 'I don't want to have a key-value
store at all'. I landed where I did because I wanted to be able to
design a configuration file however it made sense to me -- even with
an sequence rather than a map at the top. But I'm not surprised to
learn that my 'elders' in this design space feel that more
compatibility is more important than this ultimate flexibility, and I
could live with that. As to the contents of the config when in memory,
I don't see anything too horrible about strings of json; it would make
my deflattening scheme a bit slower, but who cares? How big could one
of these sensibly get?
>
> At work I have a proprietary system that relates metatype to these nested
> configurations, but so far there is no hint of spec support. Do you have a
> plan for using metatype?
The first three hits on google for metatype weren't informative enough
to get me off the ground here, so I deferred thinking about it. When
reading via Jackson, I know the type of each value, so I could, I
suppose, generate metatype data on the fly; I'd need to find my way
to/through the documentation about how to do that instead of having
XML sitting in OSGI-INF. Assuming that there is such a way.
>
> While I’m sure OSGI would like your employer to join and help pay $$, I think
> you personally can join as a non-voting person for free or a trivial amount
> of $$. I’d encourage you to consider this to help out with the spec
> development.
I'll have a look. To be honest, I've posted enough stupid stuff here
as I've been learning my way around that I don't think of myself much
of a candidate member of a spec team; I can certainly represent my use
case.
>
> thanks
> david jencks
>
>> On Dec 25, 2015, at 3:00 PM, Benson Margulies <[email protected]> wrote:
>>
>> David,
>>
>> Warning, this is going to be a long message.
>>
>> Background:
>>
>> I created an OSGi DS component which is initialized from a complex
>> configuration. I defined this configuration in terms of a Java object
>> model. I implemented it by having a single configuration admin
>> property that pointed to a file of YAML, and then I use Jackson to
>> read that file into the Java objects of the model. I've typed a subset
>> of it below.
>>
>> It all is quite readable and maintainable as a YAML file. That's the
>> driver here: I want a component with relatively complex configuration,
>> and I want that configuration to be maintainable and readable in a
>> natural way. I don't want the devops people to have to deal with the
>> consequences of force-fitting the complex configuration into a simple
>> key-value model, particularly because there are sequences of objects
>> in the natural model. This all works.
>>
>> How this (new) code came to be:
>>
>> It occurred to us that it would be desirable to be able to manipulate
>> this, to some extent, through ConfigurationAdmin -- to be able to drop
>> in a new config and have the service reconfigure, to see and edit some
>> details in the existing tools.
>>
>> After some discussion with other people who had done similar things on
>> the Karaf list, I decided to adopt 'flatten tree-shaped configurations
>> into a single CA configuration', because I was able to think my way
>> around it. The implementation uses complex keys to avoid complex
>> values; I didn't know that Felix had existing support; I invented my
>> own (rather obvious) mapping between the Jackson generic tree
>> structure and complex keys. (I suspected that something awful would
>> happen if I put complex values into the Dictionary and fed that to
>> ConfigurationAdmin.) Certainly the web console and the Karaf shell
>> would be unable to help edit such things.
>>
>> To illustrate:
>>
>> Consider a configuration file in YAML like:
>>
>> factories:
>> - id: factory1
>> componentId: tokenization
>> - id: factory2
>> componentId: ner
>> pipelines:
>> - endpoint: ner
>> stages:
>> - factoryId: factory1
>> languages: ['*']
>> - factoryId: factory2
>> languages: ['*']
>>
>> This corresponds to an overall Java class (Configuration), plus
>> Factory and Stage.
>>
>> class Configuration {
>> List<Factory> factories;
>> List<Pipeline> pipelines;
>> }; // etc, etc.
>>
>> The service component in that git repo reads a file like this as a
>> generic tree structure that can represent Json, Yaml (or even XML),
>> and then 'flattens' it using complex keys:
>>
>> factories[0].id = factory2
>> factories[0].componentId = tokenization
>> ...
>> pipelines[0].endpoint = ner
>> pipelines[0].stages[0].factoryId = factory1
>> ...
>>
>> Given all of this, my device reads the file, makes the Dictionary with
>> the complex 'flattened' keys, and pushed it into ConfigurationAdmin.
>> The service that does the work now receives the Dictionary with those
>> keys and their values. However, it does not have to work with them as
>> such. It can call a method I've supplied to 'deflatten' -- to convert
>> the key-value dictionary back into the tree structure of the Jackson
>> 'JsonNode' classes. But that's not all: once it has that tree, it can
>> ask Jackson to convert to the original nice object model. And now
>> we've got cake and and eating it too; the code of my service works
>> with the nice Java object model, the config in source control is
>> readable YAML, and you can at least look around and make simple
>> changes in the standard tools.
>>
>> The code I wrote today is not all that complex: one class maps back
>> and forth between JsonNode trees and Dictionary objects, and another
>> uses the NIO2 watcher to implement the 'watch the file system and poke
>> ConfigurationAdmin'.
>>
>> to answer your first question: I don't know if I have access to R7
>> docs. Neither I nor my employer has any particular relationship to the
>> OSGi consortium.
>>
>> --benson
>>
>>
>>
>>
>> On Fri, Dec 25, 2015 at 5:15 PM, David Jencks <[email protected]>
>> wrote:
>>> Hi Benson,
>>>
>>> Can you get to the OSGI RFCs under development for R7? There’s the
>>> Configurer one abstracted from the enRoute configurer that David mentioned
>>> and an object conversion service abstracting the config-by-annotations from
>>> DS 1.3/R6.
>>>
>>> I don’t quite understand what you mean by “Object model” and “flattening”.
>>> Does your project deal with converting nested configurations into e.g. DS
>>> references somehow? Or does it flatten tree-shaped configurations into a
>>> single CA configuration? I’m in favor of both of these ideas making it
>>> into the specs in some form; felix DS already has some support for
>>> tree-shaped configurations using complex keys rather than complex values.
>>>
>>> I’d appreciate some more description of what your code does….
>>>
>>> thanks
>>> david jencks
>>>> On Dec 25, 2015, at 1:56 PM, David Daniel <[email protected]>
>>>> wrote:
>>>>
>>>> Thank you for the explanation
>>>>
>>>> On Friday, December 25, 2015, Benson Margulies <[email protected]>
>>>> wrote:
>>>>
>>>>> On Fri, Dec 25, 2015 at 4:45 PM, David Daniel
>>>>> <[email protected] <javascript:;>> wrote:
>>>>>> What are the differences other than yaml with enroutes configuration
>>>>>> provider.
>>>>>>
>>>>> https://github.com/osgi/osgi.enroute.bundles/tree/master/osgi.enroute.configurer.simple.provider
>>>>>
>>>>> It does not look to me as if that does a full object model; it looks
>>>>> as if it just replaces property file syntax for a single-level map
>>>>> with json syntax. I may not be doing it justice in a fast read. Also,
>>>>> mine allows the receiving component to convert the Dictionary (back)
>>>>> to a Jackson JsonNode, and thence to a real class model. So, you can
>>>>> define a configuration in terms of full data model, but still push it
>>>>> through config admin and allow the use of the webconsole or karaf
>>>>> shell to examine or make minor changes on the fly. I am not educated
>>>>> in enroute, so I would not be surprised to learn that I've reinvented
>>>>> a wheel.
>>>>>
>>>>>
>>>>>>
>>>>>> On Fri, Dec 25, 2015 at 4:39 PM, Benson Margulies <[email protected]
>>>>> <javascript:;>>
>>>>>> wrote:
>>>>>>
>>>>>>> After some hints from folks on the Karaf list, I implemented a mutant
>>>>>>> cousin of fileinstall. It only does config admin, and it's role in
>>>>>>> life is to read yaml or json files, 'flatten' them, and push the
>>>>>>> results into config admin. My plan is for the things that use it to
>>>>>>> reconstruct the original object model by reversing the flattening, not
>>>>>>> to actually use keys like "foo.bar[3].baz' -- it includes an API that
>>>>>>> maps a CA Dictionary back to a JsonTree.
>>>>>>>
>>>>>>> For expediency, I made it depend on NIO2 (not to mention Jackson) and
>>>>> DS.
>>>>>>>
>>>>>>> I doubt that folks will see this as generally applicable, but I
>>>>>>> carefully set it up with AL so that I could contribute it if there is
>>>>>>> a surprising (to me) groundswell of interest. I know that some
>>>>>>> karafers read this list, so I'm not going to bother to crosspost.
>>>>>>>
>>>>>>> I haven't written any automated tests yet.
>>>>>>>
>>>>>>> https://github.com/benson-basis/yaml-configuration-admin
>>>>>>>
>>>>>
>>>
>
