Thanks for the comments -- more inline. Meanwhile I've fixed more stuff so that...
kar packaging: - you can supply a source src/main/feature/feature.xml that we add (or modify) the pom-generated feature to. This should let you list repositories, details, description (although that could come from the pom), and configuration and config files. - you can specify resolver in the pom plugin config karaf-server packaging: - the features installed from kars get added to the etc/org.apache.karaf.features.cfg file featuresRepositories key. I'm not sure how good an idea this is but the runtime state isn't available at this point. kar installer: - local repo paths are decoded into a mvn url when installing features. This actually needs a fallback to file url in case the file can't be decoded properly my geronimo sample: - I construct a server containing the framework kar and my sample geronimo-transaction kar. The build fails at this point due to missing legal files but the server starts up and you can start the tx feature. On Jan 24, 2011, at 3:59 PM, Guillaume Nodet wrote: > I haven't been able to dive enough in the code today, but want to > provide some comments anyway: > > On Sat, Jan 22, 2011 at 09:04, David Jencks <[email protected]> wrote: >> I now have all the basic functionality I wanted working at least to some >> extent: >> >> packagings: >> - feature packaging to just generate a feature.xml >> - kar packaging to in addition generate a .kar that can include resources >> and bundles (along with the feature.xml generated as above) >> - karaf-assembly packaging to take one or more .kars, unpack the resources, >> unpack the bundles (and feature files) to the local-repo), and package up >> the server as tar.gz and zip >> >> mojos: >> ArchiveKarMojo packs up a kar file using project resources and generated >> feature xml >> ArchiveServerMojo packs up a server >> GenerateFeaturesXmlMojo2 generates a features.xml containing a single >> feature with bundles from the (transitive) maven dependencies >> InstallKarsMojo unpack cars listed as maven dependencies to create a server >> image >> >> putting it to work: >> features/assembly/framework has the base karaf server resources and builds a >> kar containing these and the jars and bundles needed to start a minimal >> server >> assemblies/apache-karaf assembles and packages a server containing the >> framework kar. If you unpack the tar.gz, it actually starts. >> >> The code is mostly copied from geronimo and is pretty messy, with giant >> chunks commented out. What's left is pretty simple however. >> >> Comments on guillaumes comments inline below: more comments very welcome. >> >> >> On Jan 20, 2011, at 12:20 AM, Guillaume Nodet wrote: >> >>> On Thu, Jan 20, 2011 at 02:17, David Jencks <[email protected]> wrote: >>>> I've set up >>>> >>>> https://[email protected]/djencks/karaf.git >>>> >>>> So far I've... >>>> - set up a jaxb tree to read/write feature.xml. Presumably this would be >>>> better inside the feature bundle itself but that also presumably requires >>>> some discussion. >>> >>> We were envisioning using jaxb when we switch to JDK >= 1.6, i.e. for >>> 3.x branch. >> >> At the moment I have the jaxb classes in the feature plugin. I wonder if >> it's a reasonable restriction even for 2.2 that if you want to use this new >> way of generating features and kar files, you need to build on java 6. It's >> been a while but can't you also add the jaxb goo as maven dependencies >> perhaps in a profile to allow building on java 5? Eventually I assume they >> should go into the feature service itself. > > Can't we add dependencies on jaxb-api and jaxb-impl to the plugin so > that it would work on java 5 too ? We probably want a java 5 profile for this since they are probably the wrong versions and irrelevant for java 6. If this doesn't work we can just use the dependencies always. > >> >>> >>>> - added a GenerateFeaturesXmlMojo2 that generates feature.xml based on >>>> maven dependencies similar to how geronimo deals with them. It >>>> (supposedle) stops when one of the dependencies is a maven project that >>>> generates a feature. (not tested yet) >>>> - added an ArchiveKarMojo that takes the feature.xml and generates a kar >>>> file with the specified bundles inside in a maven repo-like structure. A >>>> couple obvious problems with this are that the feature.xml and legal files >>>> from remote-resources-plugin are not yet included. >>>> - added feature and kar maven packaging types >>> >>> Can't those be generated from the same project ? Or do you see >>> drawbacks to that? >> >> There are two related packagings and two mojos. The feature packaging only >> uses the feature mojo and stops there. The kar packaging uses the feature >> mojo to generate the feature xml and continues with the ArchiveKarMojo to >> package it and other goo into a kar. >> >>> >>>> >>>> This is all very experimental and there are still a lot of capabilities of >>>> features and the existing feature plugin that I don't understand. >>> >>> Yeah, I think we need to find some way to add them back step by step. >>> The two most important things that come to my mind are: >>> >>> 1/ the use of configurations: those are feed to ConfigAdmin before >>> starting the bundles >> >> Unless you want to do something like convert maven properties to config >> admin properties (I don't recommend this, it's too hard to get rid of the >> ones you don't want) I can think of two plausible approaches: >> >> 1. allow starting with a base features.xml in src/main/feature/feature.xml >> and add the bundles to it. This is easy. >> 2. configure this stuff in the maven feature-maven-plugin configuration. >> This is kind of yucky because maven uses a pretty non standard xml system so >> you have to write another model and translate to the jaxb one. > > In a kar, properties could be added into the kar itself and for > features, I wonder if an easy way would be to point the plugin to > properties files on the file system so that they would be added as > <configuration> elements in the generated features (or embedded in the > kar when using that packaging). I'm not sure exactly what you mean and how it relates to the "modify an existing feature.xml" and "unpack resources" in feature.xml generation and kar packaging. Could you sketch out a simple example showing the source files, a bit of expected generated configuration, and where files end up in the kar and unpacked into the server? > >> >> I suspect (1) will be most plausible. >>> >>> 2/ the use of OBR and flagging some bundles as dependencies >>> If you're familiar with the way Aries EBAs work, it's the same concept. >>> If obr is installed and the feature has resolver="obr" or >>> resolver="(obr)" (the >>> parenthesis mean that the obr resolver is optional), an OBR >>> repository will be >>> generated using the bundles in the feature, and OBR will be >>> asked to resolve >>> bundles that are not flagged with dependency="true". This >>> avoids duplication >>> of the bundles in the framework instead of reusing those already >>> installed. >> >> the resolver="OBR" should be easy to specify in the maven plugin >> configuration. Perhaps we can map the maven dependency scope to the feature >> bundle dependency attribute? >> <scope>compile</scope> ----> dependency="true" >> <scope>runtime</scope> ----> dependency="false" >> >> (or vice-versa, not sure I understand which way makes more sense) > > If we want to be the closest possible to the meaning, I think i would > put direct and non optional dependencies as dependency="false" and > transitive or optional dependencies as dependency="true". > The idea is that bundles flagged as "dependencies" may not be > installed by OBR if not really needed as their goal is just to fulfill > some requirements for the non "dependencies" bundles (i.e. those that > will always be installed as part of the features). > But I'm not sure this way is flexible enough when it comes to tweaking > the dependencies tree. Again I'm a little lost.... a more concrete example would be great. > >> >> Another current limitation is that the GenerateFeaturesXmlMojo2 really only >> looks at maven dependencies. What it does is traverse the entire >> (appropriately scoped) dependency tree. It keeps track of transitive >> project dependencies and also dependencies that are dependencies of other >> .kar or features projects. Then it removes all the dependencies found in >> other feature projects from the set of transitive project dependencies. >> This avoids duplication but means (if you want this pruning to work) that >> all the features.xml need to be generated using this plugin from maven >> dependencies. I don't know if this will be a noticeable limitation since >> there is the option of asking obr to resolve the dependency rather than >> directly installing it. > > Leveraging OBR here would definitely make sense and maybe it can be > done by computing the OBR repository for all transitive dependencies > and asking it to resolve the direct mandatory dependencies. OBR will > output a list of bundles that should be installed in order to solve > those direct mandatory dependencies, then we remove any bundle that > are mandatory bundles for dependant kars/features and flag them as > being dependencies in the feature. Confused again :-). I should warn everyone that I took out the code that tries to figure out package imports and exports because I couldn't understand how it is supposed to work. Here I'm having a related problem. I think of OBR as naturally having all bundles ever created in it so if you ask it to satisfy package constraints it will come up with an infinite list of acceptable bundles. However with maven dependencies you have specified which bundles you want to use for your dependencies. On the other hand the maven dependencies are hard to override with a different choice. So whenever someone says they want to use OBR to figure out which bundles to install I want to know how OBR has been configured to come up with a reasonable list of acceptable bundles. So far the only solution that makes much sense to me is to generate OBR repository.xmls for each feature or kar and construct and OBR that knows about the bundles specified in the features/kars you have actually installed. I'd really like to understand other peoples point of view on this, I've been discussing it for at least a year with everyone I can without much resolution. many thanks! david jencks > > >> >> thanks! >> david jencks >> >>> >>>> I have a sandbox project at geronimo I've been using to try these out. >>>> >>>> https://svn.apache.org/repos/asf/geronimo/sandbox/djencks/txmanager >>>> >>>> thanks >>>> david jencks >>>> >>>> >>>> On Jan 18, 2011, at 12:35 AM, Jean-Baptiste Onofré wrote: >>>> >>>>> David, >>>>> >>>>> No problem if you want to fork on github to work around these topics. >>>>> I will help you on that. >>>>> >>>>> Thanks >>>>> Regards >>>>> JB >>>>> >>>>> On 01/18/2011 09:15 AM, Guillaume Nodet wrote: >>>>>> On Tue, Jan 18, 2011 at 07:22, Jean-Baptiste Onofré<[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi David, >>>>>>> >>>>>>> my comments inline: >>>>>>> >>>>>>> 1. maven-generated features.xml and maven feature packaging type. >>>>>>>> Geronimo has some code that basically generates a features.xml that >>>>>>>> includes the (appropriately scoped) pom dependencies and transitive >>>>>>>> dependencies, stopping when a dependency is a feature. It even works >>>>>>>> with >>>>>>>> maven 3 :-) (AFAICT the current generate-features-xml mojo does not). >>>>>>>> I >>>>>>>> haven't been able to figure out what the current geronimo-feature-xml >>>>>>>> mojo >>>>>>>> tries to do but it appears to be something else. So, the output from >>>>>>>> such a >>>>>>>> maven project would be a features.xml file with a single feature >>>>>>>> including >>>>>>>> all the transitive dependencies. This is a totally non-code project. >>>>>>>> A >>>>>>>> feature maven packaging type would then deploy this as the sole >>>>>>>> generated >>>>>>>> artifact for this project. Then you can refer to this project as a >>>>>>>> dependency and it all fits into the maven infrastructure. >>>>>>>> >>>>>>> We have the features maven plugin: >>>>>>> >>>>>>> http://karaf.apache.org/manual/2.1.99-SNAPSHOT/developers-guide/features-maven-plugin.html >>>>>>> >>>>>>> Using the features plugin, you can: >>>>>>> - features:generate-features-xml to generate the features XML >>>>>>> - features:add-features-to-repo which reads a features descriptor and >>>>>>> resolves artifacts to the local repo >>>>>>> >>>>>>> FYI, we raised a JIRA to rename this maven plugin with a more generic >>>>>>> name: >>>>>>> karaf. >>>>>>> The purpose is to provide new goals such as karaf:branding, karaf:run, >>>>>>> karaf:distribution, etc. >>>>>>> Of course, the existing goals will still exist: >>>>>>> karaf:generate-features-xml, karaf:add-features-to-repo. >>>>>>> >>>>>>> >>>>>> I think the point here is that if we were to head toward one maven >>>>>> project >>>>>> == one feature, handling of maven dependencies would be much easier. One >>>>>> problem we had (and that's why camel and servicemix did not use our >>>>>> plugin), >>>>>> is that for non pure-OSGi projects such as Camel, the dependencies are >>>>>> mostly non OSGi-bundles, so having them globally is really difficult to >>>>>> deal >>>>>> with. However, if we were to have a single feature per project, it's way >>>>>> easier to use maven dependencies to actually reflect the content of the >>>>>> features. so definitely +1 for me, as I think the current goal has not >>>>>> proved very usable atm. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> 2. .kar files >>>>>>>> I just found out about .kar files today. IIUC this is a jar file that >>>>>>>> contains a feature.xml and the bundle dependencies listed in the >>>>>>>> feature.xml. I think an additional mojo and packaging type that >>>>>>>> generates >>>>>>>> the feature.xml as in (1) and then packages it and the bundles listed >>>>>>>> into >>>>>>>> an archive would be a good idea. >>>>>>>> >>>>>>> Agree, it's in the plan of the "new" generic karaf maven plugin: >>>>>>> - karaf:kar goal will take just a features descriptor (and eventually >>>>>>> some >>>>>>> resources) to create the kar by packaging the features description plus >>>>>>> the >>>>>>> bundles set >>>>>>> I raised a JIRA about that: KARAF-401. >>>>>>> >>>>>>> >>>>>> +1 >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> 3. resources in .kar files >>>>>>>> Geronimo .car files can include resources that get unpacked on >>>>>>>> installation into the server directory structure. While normally osgi >>>>>>>> bundles are supposed to look for their data in their own osgi-managed >>>>>>>> data >>>>>>>> area, sometimes you need something that the user can find :-) and it >>>>>>>> really >>>>>>>> belongs in a well known location not associated with a particular >>>>>>>> bundle. >>>>>>>> In particular, in geronimo we use this idea in server assembly. We >>>>>>>> have a >>>>>>>> .car file that has the basic directory structure of a server and the >>>>>>>> startup >>>>>>>> jars packed as resources. When assembling a server, we start with >>>>>>>> this .car >>>>>>>> file and unpack the server structure out of it. >>>>>>>> >>>>>>> Yeah, as I said before, the kar goal could contains an instructions >>>>>>> (with >>>>>>> includes/excludes tags) to embed resources into the generated kar >>>>>>> archive. >>>>>>> >>>>>>> >>>>>> Note that we recently added a<configfile> element to the<feature> >>>>>> element >>>>>> which is related to that. One example is in the http feature in >>>>>> >>>>>> http://svn.apache.org/repos/asf/karaf/trunk/features/assembly/standard/src/main/resources/features.xml >>>>>> Though, in the case of kar files, those files should definitely be >>>>>> embedded. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> 4. server assembly >>>>>>>> (cf. KARAF-56) Karaf really needs a really simple way to assemble a >>>>>>>> custom server using maven. Based on geronimo's experience, one way to >>>>>>>> do >>>>>>>> this would be using a maven plugin that installed features and .kar >>>>>>>> files, >>>>>>>> using the resource-in-kar idea from (3). You'd use a server-assembly >>>>>>>> packaging and list all the features and kar files you want as >>>>>>>> dependencies, >>>>>>>> and the plugin would install them and pack up the result. >>>>>>>> >>>>>>> Agree. I'm working about this. >>>>>>> The first step is to rename the features maven plugin into karaf maven >>>>>>> plugin to avoid to lost the users with several plugin name. >>>>>>> On the karaf maven plugin, the dist goal will create and package a >>>>>>> custom >>>>>>> karaf container. >>>>>>> >>>>>>> >>>>>> That was the plan. We just haven't got any time to get to it :-( >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Comments >>>>>>>> >>>>>>>> When assembling an osgi server, there's a battle between osgi/obr "I >>>>>>>> need >>>>>>>> these packages/services" and maven/file-system/require-bundle "I want >>>>>>>> this >>>>>>>> artifact". When a feature lists an artifact to install, its >>>>>>>> definitely on >>>>>>>> the "this artifact" side of the line. What do you do when you get to >>>>>>>> something you need but don't want to include in the feature? you can >>>>>>>> either >>>>>>>> specify the external requirements as Import-Package and some kind of >>>>>>>> require-service specifications or specify e.g. other features whose >>>>>>>> bundles >>>>>>>> provide the requirements. Geronimo has used something similar to the >>>>>>>> "specify other features" approach. Aries EBAs specify osgi package >>>>>>>> requirements. If you want to assemble a server with the minimum input >>>>>>>> information, relying on feature dependencies is very simple. However >>>>>>>> it >>>>>>>> makes it harder to use alternate sources of packages and services. >>>>>>>> One idea >>>>>>>> I had was to associate an OBR repository.xml with each feature that >>>>>>>> includes >>>>>>>> the OBR info for the bundles in the >>>>>>>> >>>>>>> feature. Then you can create a "virtual" OBR from a set of features by >>>>>>> including all the repository.xmls from the features. If a feature >>>>>>> specifies >>>>>>> its external requirements both as osgi requirements and as feature >>>>>>> dependencies, an installer/assembler could check if each feature >>>>>>> dependency >>>>>>> is needed to satisfy the osgi dependencies before installing it. >>>>>>> We added the obr resolver in a feature. >>>>>>> If we package an OBR repository.xml in a feature, I could be append to >>>>>>> the >>>>>>> Karaf instance OBR. >>>>>>> From my point of view, the OBR should be manage: >>>>>>> - by the Karaf instance itself >>>>>>> - by the provisioning tool >>>>>>> As we can consider the feature as a provisioning tool, it makes sense >>>>>>> that >>>>>>> it can provide an OBR repository.xml. >>>>>>> >>>>>> >>>>>> I don't think we need to package an OBR respository. Right now, it's >>>>>> generated based on the content of the features descriptor with all the >>>>>> bundles, and those marked as 'dependency' are not asked to be resolved, >>>>>> so >>>>>> OBR will only include those if they are actually needed. I'm still not >>>>>> convinced by requiring an OBR repository, as it's an extra thing users >>>>>> have >>>>>> to manage, and people may not want to do that. In addition OBR >>>>>> repositories >>>>>> consume quite a lot of memory when loaded, so I'd like to avoid requiring >>>>>> those. However, using OBR to avoid installing unnecessary bundles is >>>>>> quite >>>>>> neat. >>>>>> >>>>>> David, as I said on IRC, feel free to work on that on a github fork or >>>>>> in a >>>>>> branch in the sandbox, up to you. If you can get to have that working, >>>>>> that would be awesome !!! >>>>>> >>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> ---- >>>>>>>> >>>>>>>> I'd really appreciate comments on these ideas. I'm going to start >>>>>>>> seeing >>>>>>>> if I can munge the geronimo code to do some of this either in a karaf >>>>>>>> sandbox or in git. Having some code to try out may be a lot clearer >>>>>>>> than my >>>>>>>> sometimes convoluted explanations... >>>>>>>> >>>>>>>> thanks >>>>>>>> david jencks >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >>> >>> >>> >>> -- >>> Cheers, >>> Guillaume Nodet >>> ------------------------ >>> Blog: http://gnodet.blogspot.com/ >>> ------------------------ >>> Open Source SOA >>> http://fusesource.com >> >> > > > > -- > Cheers, > Guillaume Nodet > ------------------------ > Blog: http://gnodet.blogspot.com/ > ------------------------ > Open Source SOA > http://fusesource.com
