First, thanks to David Daniel for verifying that you should have access to the rfcs and Ray for encouraging you to help with the specs :-)
I’ve managed to write a simple test example in the ds integration tests, see AnnoConfigTest.testNestedAnnoConfig. I also managed to finally commit the extension meta-annotation, into a new scr-ext-anno project. This lets you generate the configureWithInterfaces=“true” extension attribute using bnd: add @DSExt.ConfigureWithInterfaces to your class. I don’t remember if this works with bnd 3.0.0, but certainly with 3.1.0. More inline > On Dec 25, 2015, at 5:11 PM, Benson Margulies <[email protected]> wrote: > > On Fri, Dec 25, 2015 at 7:51 PM, David Jencks <[email protected]> > wrote: >> Benson, >> >> What you did is pretty similar in many ways to what I did although I don’t >> have the json/yaml support. > > I'm hardly attached to 4 hours of hacking. If you're willing to take > Jackson deps, I'd be game to collaborate and add json/yaml(/xml) to > what you have. I’m not sure where Jackson is going to fit in here. The DS spec insists on annotations and can be relaxed easily to accept interfaces. It’s easy (see my implementation) to take a complex-key encoding of a tree-shaped configuration and turn it into nested annotations or interfaces. We pretty much have to start with a configuration map, so I don’t see what’s missing in DS. Even if we decide to (also) support complex json values, I think a simple json parser would be more appropriate than a whole framework. The other end, converting nested or tree-shaped json into the flat configuration isn’t part of DS, but might be related to the Configurer RFC. However there too I don’t think going through an object model is going to help anyone, just flattening the json to either complex keys or complex values seems like plenty to me. Are you using Jackson somehow to generate the json you start with? Other than this, which I can’t quite imagine yet :-) I’m having trouble seeing where Jackson might fit. > >> >> I’m not sure the [] characters are appropriate for key tokens. Anyway my >> encoding uses key1.<number>.key2.<number>….. instead of >> key1[<number>].key2[<number>]….. > > I think I could switch to that. On the other hand, can you elaborate > on what's wrong with []? I picked them because it was a less tricky > parse (no "oh, a _number_, that must mean it's a sequence") but I > don't feel very strongly about it. Thinking back, I think I picked it because I got the idea for this from James Strachan’s spring configurer for ActiveMQ that I blueprint-ized as xbean-blueprint in the Geronimo project. At work I have a fairly large investment in this encoding…. besides it uses one fewer character per level :-D > >> >> Since I’m interested in using DS spec features as much as possible, the >> object model is annotations or interfaces rather than concrete classes. So >> if your DS component has >> >> @Activate >> void activate(MyNestedConfigAnnotation config) {…} >> >> DS will generate implementations of the config annotation backed by the CA >> properties map. > > I'd be interested to see a fully-worked-out example of this. I often > end up using a fair amount of Jackson @nnotation to tune the > representation. It seems possible that they could coexist; Jackson has > support deserializing to concrete classes as informed by annotations. As mentioned at the top, there’s now an integration test showing how it works. I thought I’d written at least some test for this (he admits with great embarrassment :-) > >> >> I just realized that I haven’t yet made it easy to turn this on by >> committing the meta-annotation to generate the felix specific flag to allow >> nested annotations….. maybe this weekend. >> >> While both you and I picked complex keys, Peter Kriens seems to be strongly >> in favor of complex values encoded in json. I believe one of his reasons is >> that such configurations are less likely to completely break existing >> metatype-aware config viewers. > > I waffled back and forth between 'CA is fundamentally a key-value > store, and I need fancy values' and 'I don't want to have a key-value > store at all'. I landed where I did because I wanted to be able to > design a configuration file however it made sense to me -- even with > an sequence rather than a map at the top. But I'm not surprised to > learn that my 'elders' in this design space feel that more > compatibility is more important than this ultimate flexibility, and I > could live with that. As to the contents of the config when in memory, > I don't see anything too horrible about strings of json; it would make > my deflattening scheme a bit slower, but who cares? How big could one > of these sensibly get? I think this needs more thought. I still think complex keys are way better than complex values. > >> >> At work I have a proprietary system that relates metatype to these nested >> configurations, but so far there is no hint of spec support. Do you have a >> plan for using metatype? > > The first three hits on google for metatype weren't informative enough > to get me off the ground here, so I deferred thinking about it. When > reading via Jackson, I know the type of each value, so I could, I > suppose, generate metatype data on the fly; I'd need to find my way > to/through the documentation about how to do that instead of having > XML sitting in OSGI-INF. Assuming that there is such a way. Read the spec :-) The OSGI metatype spec is sort of supposed to be a way of helping “UI”s to describe to the user what should go into a configuration. Any json you provide IMO should already be influenced by whatever metatype specifies. However, metatype currently assumes flat configurations. But now you generally generate the metatype by annotating your DS configuration annotation. @ObjectClassDefinition @interface MyConfig { String foo() default “foo”; int bar() default 0; } But we want to support something like @ObjectClassDefinition @interface MyConfig { Nested[] nested(); int bar() default 0; } @ObjectClassDefinition @interface Nested { String foo() default “foo”; } This looks good as annotations, but how are you going to represent nested() in the generated xml metatype descriptor? The proprietary solution I use is to treat it as a String[] but provide a metatype extension saying it’s a reference to something with a specified pid, which identifies the Nested metatype. So far there isn’t any agreement about if or how such a solution should get into a spec. I think this is related to my earlier question, “how are you generating the json”. > >> >> While I’m sure OSGI would like your employer to join and help pay $$, I >> think you personally can join as a non-voting person for free or a trivial >> amount of $$. I’d encourage you to consider this to help out with the spec >> development. > > I'll have a look. To be honest, I've posted enough stupid stuff here > as I've been learning my way around that I don't think of myself much > of a candidate member of a spec team; I can certainly represent my use > case. As Ray says, mostly you need persistence and interest :-) thanks david jencks > > >> >> thanks >> david jencks >> >>> On Dec 25, 2015, at 3:00 PM, Benson Margulies <[email protected]> wrote: >>> >>> David, >>> >>> Warning, this is going to be a long message. >>> >>> Background: >>> >>> I created an OSGi DS component which is initialized from a complex >>> configuration. I defined this configuration in terms of a Java object >>> model. I implemented it by having a single configuration admin >>> property that pointed to a file of YAML, and then I use Jackson to >>> read that file into the Java objects of the model. I've typed a subset >>> of it below. >>> >>> It all is quite readable and maintainable as a YAML file. That's the >>> driver here: I want a component with relatively complex configuration, >>> and I want that configuration to be maintainable and readable in a >>> natural way. I don't want the devops people to have to deal with the >>> consequences of force-fitting the complex configuration into a simple >>> key-value model, particularly because there are sequences of objects >>> in the natural model. This all works. >>> >>> How this (new) code came to be: >>> >>> It occurred to us that it would be desirable to be able to manipulate >>> this, to some extent, through ConfigurationAdmin -- to be able to drop >>> in a new config and have the service reconfigure, to see and edit some >>> details in the existing tools. >>> >>> After some discussion with other people who had done similar things on >>> the Karaf list, I decided to adopt 'flatten tree-shaped configurations >>> into a single CA configuration', because I was able to think my way >>> around it. The implementation uses complex keys to avoid complex >>> values; I didn't know that Felix had existing support; I invented my >>> own (rather obvious) mapping between the Jackson generic tree >>> structure and complex keys. (I suspected that something awful would >>> happen if I put complex values into the Dictionary and fed that to >>> ConfigurationAdmin.) Certainly the web console and the Karaf shell >>> would be unable to help edit such things. >>> >>> To illustrate: >>> >>> Consider a configuration file in YAML like: >>> >>> factories: >>> - id: factory1 >>> componentId: tokenization >>> - id: factory2 >>> componentId: ner >>> pipelines: >>> - endpoint: ner >>> stages: >>> - factoryId: factory1 >>> languages: ['*'] >>> - factoryId: factory2 >>> languages: ['*'] >>> >>> This corresponds to an overall Java class (Configuration), plus >>> Factory and Stage. >>> >>> class Configuration { >>> List<Factory> factories; >>> List<Pipeline> pipelines; >>> }; // etc, etc. >>> >>> The service component in that git repo reads a file like this as a >>> generic tree structure that can represent Json, Yaml (or even XML), >>> and then 'flattens' it using complex keys: >>> >>> factories[0].id = factory2 >>> factories[0].componentId = tokenization >>> ... >>> pipelines[0].endpoint = ner >>> pipelines[0].stages[0].factoryId = factory1 >>> ... >>> >>> Given all of this, my device reads the file, makes the Dictionary with >>> the complex 'flattened' keys, and pushed it into ConfigurationAdmin. >>> The service that does the work now receives the Dictionary with those >>> keys and their values. However, it does not have to work with them as >>> such. It can call a method I've supplied to 'deflatten' -- to convert >>> the key-value dictionary back into the tree structure of the Jackson >>> 'JsonNode' classes. But that's not all: once it has that tree, it can >>> ask Jackson to convert to the original nice object model. And now >>> we've got cake and and eating it too; the code of my service works >>> with the nice Java object model, the config in source control is >>> readable YAML, and you can at least look around and make simple >>> changes in the standard tools. >>> >>> The code I wrote today is not all that complex: one class maps back >>> and forth between JsonNode trees and Dictionary objects, and another >>> uses the NIO2 watcher to implement the 'watch the file system and poke >>> ConfigurationAdmin'. >>> >>> to answer your first question: I don't know if I have access to R7 >>> docs. Neither I nor my employer has any particular relationship to the >>> OSGi consortium. >>> >>> --benson >>> >>> >>> >>> >>> On Fri, Dec 25, 2015 at 5:15 PM, David Jencks <[email protected]> >>> wrote: >>>> Hi Benson, >>>> >>>> Can you get to the OSGI RFCs under development for R7? There’s the >>>> Configurer one abstracted from the enRoute configurer that David mentioned >>>> and an object conversion service abstracting the config-by-annotations >>>> from DS 1.3/R6. >>>> >>>> I don’t quite understand what you mean by “Object model” and “flattening”. >>>> Does your project deal with converting nested configurations into e.g. DS >>>> references somehow? Or does it flatten tree-shaped configurations into a >>>> single CA configuration? I’m in favor of both of these ideas making it >>>> into the specs in some form; felix DS already has some support for >>>> tree-shaped configurations using complex keys rather than complex values. >>>> >>>> I’d appreciate some more description of what your code does…. >>>> >>>> thanks >>>> david jencks >>>>> On Dec 25, 2015, at 1:56 PM, David Daniel <[email protected]> >>>>> wrote: >>>>> >>>>> Thank you for the explanation >>>>> >>>>> On Friday, December 25, 2015, Benson Margulies <[email protected]> >>>>> wrote: >>>>> >>>>>> On Fri, Dec 25, 2015 at 4:45 PM, David Daniel >>>>>> <[email protected] <javascript:;>> wrote: >>>>>>> What are the differences other than yaml with enroutes configuration >>>>>>> provider. >>>>>>> >>>>>> https://github.com/osgi/osgi.enroute.bundles/tree/master/osgi.enroute.configurer.simple.provider >>>>>> >>>>>> It does not look to me as if that does a full object model; it looks >>>>>> as if it just replaces property file syntax for a single-level map >>>>>> with json syntax. I may not be doing it justice in a fast read. Also, >>>>>> mine allows the receiving component to convert the Dictionary (back) >>>>>> to a Jackson JsonNode, and thence to a real class model. So, you can >>>>>> define a configuration in terms of full data model, but still push it >>>>>> through config admin and allow the use of the webconsole or karaf >>>>>> shell to examine or make minor changes on the fly. I am not educated >>>>>> in enroute, so I would not be surprised to learn that I've reinvented >>>>>> a wheel. >>>>>> >>>>>> >>>>>>> >>>>>>> On Fri, Dec 25, 2015 at 4:39 PM, Benson Margulies <[email protected] >>>>>> <javascript:;>> >>>>>>> wrote: >>>>>>> >>>>>>>> After some hints from folks on the Karaf list, I implemented a mutant >>>>>>>> cousin of fileinstall. It only does config admin, and it's role in >>>>>>>> life is to read yaml or json files, 'flatten' them, and push the >>>>>>>> results into config admin. My plan is for the things that use it to >>>>>>>> reconstruct the original object model by reversing the flattening, not >>>>>>>> to actually use keys like "foo.bar[3].baz' -- it includes an API that >>>>>>>> maps a CA Dictionary back to a JsonTree. >>>>>>>> >>>>>>>> For expediency, I made it depend on NIO2 (not to mention Jackson) and >>>>>> DS. >>>>>>>> >>>>>>>> I doubt that folks will see this as generally applicable, but I >>>>>>>> carefully set it up with AL so that I could contribute it if there is >>>>>>>> a surprising (to me) groundswell of interest. I know that some >>>>>>>> karafers read this list, so I'm not going to bother to crosspost. >>>>>>>> >>>>>>>> I haven't written any automated tests yet. >>>>>>>> >>>>>>>> https://github.com/benson-basis/yaml-configuration-admin >>>>>>>> >>>>>> >>>> >>
