I was able to get the bnd-maven-plugin working with camel-core last night on the jdk8-lambda branch. I still need to finish the bnd.bnd configuration file to make the manifests match, but I’m getting close. Once Raul finishes his work and merges it back to master, I’ll get the PR going with the bnd-maven-plugin so everyone can see what that looks like.
Is there a list of which projects should not generate bundles? I’d like to try and build that into my PR as well. > On Mar 28, 2016, at 11:21 PM, Claus Ibsen <claus.ib...@gmail.com> wrote: > > On Tue, Mar 29, 2016 at 1:04 AM, Raul Kripalani <ra...@apache.org > <mailto:ra...@apache.org>> wrote: >> Thanks a lot Claus! >> >> Given that we now make no distinction between bundles and standard JARs >> through the packaging type, I was having the OSGi MANIFEST generation >> triggered for bundles where its irrelevant, e.g. camel-grape, >> camel-spring-boot, many examples, etc. So I added a profile which is >> activated by default and is disabled through a new POM property >> camel.osgi.skip, which I added to the POMs of such modules. >> > > Can we flip the switch so you have to enable it on the maven modules > that you want to be an osgi bundle. I am asking because people who are > not using OSGi should really not see camel.osgi.skip=true in the > examples / camel-spring-boot-starter etc. > > They should be clean and without any osgi stuff. > > Also I would rather make it explicit that this maven module is built > as an osgi bundle if it has camel.osgi=true. > > > > >> I was also able to upgrade the version of the maven-bundle-plugin to the >> latest 3.0.1. >> >> Hopefully I'll be able to finish my tests tomorrow and merge the branch >> into master for a full test in JKS. >> >> From then on... let the lambda fun begin (for features we don't intend to >> backport)! >> >> Cheers, >> >> *Raúl Kripalani* >> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and >> Messaging Engineer >> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani >> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk> >> >> On Sun, Mar 27, 2016 at 5:24 PM, Claus Ibsen <claus.ib...@gmail.com> wrote: >> >>> Hi >>> >>> I got the OSGi tests to work with karaf 4 and they are all passing now >>> for the features tests. There is a few that has been @Ignored because >>> of known issues. eg we need a new SMX bundle for them. >>> >>> You can run these tests reliable using a bash script >>> >>> cd tests/camel-itest-karaf >>> ./run-tests.sh >>> >>> You can also try running using mvn with >>> >>> mvn clean install >>> >>> But the latter tend to break on my computer after 20 or so tests. The >>> scripts runs reliable and it also cleanup dangling karaf JVMs. But >>> generally its been safer to run one mvn goal per test it seems. >>> >>> I also fixed the mistakes in karaf features and the tests so they are >>> working again. >>> >>> Raul you should be able to use that on your branch to give the osgi tests >>> a go. >>> >>> >>> >>> >>> >>> >>> On Sat, Mar 26, 2016 at 5:47 PM, Claus Ibsen <claus.ib...@gmail.com> >>> wrote: >>>> I managed to get the CI jobs to complete now with setting higher memory. >>>> >>>> The snapshots are now deployed and the CI jobs for osgi is triggered >>>> >>>> Triggering a new build of Camel.trunk.itest.karaf >>>> Triggering a new build of Camel.trunk.itest.osgi >>>> Finished: SUCCESS >>>> >>>> >>>> >>>> On Sat, Mar 26, 2016 at 12:36 PM, Claus Ibsen <claus.ib...@gmail.com> >>> wrote: >>>>> Oh I forgot the CI jobs are here >>>>> https://builds.apache.org/view/A-D/view/Camel/ >>>>> >>>>> I got the link from Mueller. Beforehand it was hard/impossible to find >>>>> Camel from >>>>> https://builds.apache.org/ >>>>> >>>>> So I was for a longer time not able to see all these jobs, and just >>>>> got the CI emails to look at. >>>>> >>>>> But the OSGi jobs rely on the SNAPSHOT of Camel has been built prior >>>>> to that, and that job is unreliable. I think we need to look at making >>>>> the -notest also publish the SNAPSHOT to the snapshot mvn repo, so the >>>>> CI jobs can pickup these new JARs. >>>>> >>>>> If we get that more reliable then Camel end users can also easier use >>>>> SNAPSHOT instead of having to build locally to be sure they get a >>>>> clean build of it, otherwise you can end up with mixed builds or >>>>> stale/old snapshots. >>>>> >>>>> >>>>> >>>>> On Sat, Mar 26, 2016 at 12:33 PM, Claus Ibsen <claus.ib...@gmail.com> >>> wrote: >>>>>> On Sat, Mar 26, 2016 at 12:12 PM, Raul Kripalani <ra...@apache.org> >>> wrote: >>>>>>> Agree with your points. >>>>>>> >>>>>>> 1. If you use lambdas on 2.18, you can't backport the code. >>>>>>> >>>>>>> 2. I'll spend some time this weekend installing the bundles on Karaf. >>> Karaf >>>>>>> 4 is OK, or are we sticking with older versions for 2.x? >>>>>>> >>>>>> >>>>>> Karaf 2.x is dropped. Use Karaf 4.x as the primary karaf version. >>>>>> We may want to add a -Pkaraf3 profile to run against older Karaf 3.x >>> containers. >>>>>> >>>>>> >>>>>>> 3. I'll also run our OSGi itests, but it may be worth having Jenkins >>> do all >>>>>>> this. Could you configure a job in JKS to build this branch? I don't >>> have >>>>>>> job admin rights I believe. >>>>>>> >>>>>> >>>>>> We already have some CI jobs to run 2 osgi based tests >>>>>> >>>>>> - tests/camel-itest-karaf >>>>>> - tests/camel-itest-osgi >>>>>> >>>>>> The former tests that the Camel component can be installed, it does >>>>>> this using pax-exam to bootup karaf and install the camel-xxx feature >>>>>> and find the component and then tear down the test. >>>>>> >>>>>> You can run these tests with the -Pkaraf4 profile. >>>>>> But mind I had trouble with these tests may stop after 20 or so tests >>>>>> and leave some karaf JVMs dangling. There is a kill-karaf.sh script to >>>>>> kill em. >>>>>> >>>>>> The camel-itest-osgi is likely in some state where some tests would >>>>>> fail even for older branches. But these are unit tests that run Camel >>>>>> apps in karaf and do some testing stuff like we do in regular >>>>>> component tests. This has not yet been migrated to karaf 4, and run on >>>>>> karaf 2.x. But you can give it a go and see how bad/good it is. It >>>>>> used to be able to run almost all tests and pass. And we had also had >>>>>> periods where everything was green. >>>>>> >>>>>> However people dont write so many osgi tests so this module do not >>>>>> have so much love. And there are plenty of components that are not >>>>>> being tested. >>>>>> >>>>>> >>>>>>> 4. It's necessary to configure the maven-jar-plugin to explicitly use >>> a >>>>>>> manifest file, otherwise the plugin generates its own. Originally I >>> tried >>>>>>> adding the config to the build plugins in the parent POM, but it >>> failed for >>>>>>> artifacts of type=maven-plugin. Hence I placed it in pluginManagement >>> and >>>>>>> then incorporated it in those intermediate POMs (components, examples, >>>>>>> etc.) whose children built normal JARs. However, I ended up tracing >>> the >>>>>>> issue to a conflict introduced by the config of the maven-jar-plugin >>> that >>>>>>> maven-plugin injects, and solved it with a couple >>>>>>> of combine.self="override". Hence, I think I might be able to push our >>>>>>> maven-jar-plugin config back to the parent POM and to the build >>> section. >>>>>>> >>>>>> >>>>>> Ah okay. I am okay with we can do "anything" we need to build all the >>>>>> Camel components and whatnot. >>>>>> >>>>>> Its more what the impact is for end users. If we can make out examples >>>>>> and maven archetypes simpler and easier that would be cool. But mind >>>>>> that not all examples are osgi examples. >>>>>> >>>>>> But it looks promising your work - thanks for starting this hard work. >>>>>> >>>>>> >>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> *Raúl Kripalani* >>>>>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data >>> and >>>>>>> Messaging Engineer >>>>>>> http://about.me/raulkripalani | >>> http://www.linkedin.com/in/raulkripalani >>>>>>> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk> >>>>>>> >>>>>>> On Sat, Mar 26, 2016 at 10:12 AM, Claus Ibsen <claus.ib...@gmail.com> >>> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> Good to hear that we may drop the <bundle> extension, nice work. >>>>>>>> >>>>>>>> For camel-core I have thought of build the manifest.mf by hand so we >>>>>>>> can better control it. It seems the bundle plugin has its issues with >>>>>>>> different versions to build it, as you say. >>>>>>>> >>>>>>>> Have you tried testing any of these JARs in OSGi? It would be good if >>>>>>>> you try that and install some of the examples in a karaf 4 container >>>>>>>> and see how it runs. >>>>>>>> >>>>>>>> For lambda code in the existing source code then we cannot do that >>> for >>>>>>>> code that need to be backported to 2.17 and 2.16 branches. We can >>> only >>>>>>>> use lambdas in new code that is not to be backported. >>>>>>>> >>>>>>>> For Camel 3.0 we can then covert the code to use lambads all over >>>>>>>> where applicable. There is likely some tools in IDEA or Eclipse etc >>>>>>>> that can assist with that. >>>>>>>> >>>>>>>> >>>>>>>> And Camel end users can use lamdas today with their own code, also >>>>>>>> with older Camel versions on java 8. >>>>>>>> >>>>>>>> It may be that those people are using a new bundle plugin, or more >>>>>>>> likely not using OSGi at all. >>>>>>>> >>>>>>>> >>>>>>>> I would like to see more OSGi testing on this branch before any >>> merge. >>>>>>>> OSGi is complex and pain to maintain and build modules. So we need to >>>>>>>> be sure we are on a path that we can do this. There is 250+ maven >>>>>>>> modules that are build as OSGi so it better have to work for us. >>>>>>>> >>>>>>>> But I really like that the bundle plugin is just generating a >>>>>>>> MANIFEST.MF file that the JAR plugin just slurps in. Can it not be >>>>>>>> configured somehow so the generated MANIFEST.MF is not replace so end >>>>>>>> users do not need to do anything with a JAR plugin? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Mar 26, 2016 at 1:09 AM, Raul Kripalani <ra...@apache.org> >>> wrote: >>>>>>>>> There is good news, bad news, and good news again ;-) >>>>>>>>> >>>>>>>>> *Good news:* Now that Camel 2.18 is officially on JDK8, we can use >>>>>>>> lambdas >>>>>>>>> in our code. Woohoo! >>>>>>>>> >>>>>>>>> *Bad news:* Our current Maven build breaks when doing so. >>>>>>>>> >>>>>>>>> This is what happens: >>>>>>>>> >>>>>>>>> 1. maven-bundle-plugin version 2.3.x cannot parse JDK8 lambdas. >>> [1] >>>>>>>>> >>>>>>>>> 2. Upgrading it to the latest version (3.0.1) makes the plugin >>> fail >>>>>>>>> miserably (at least on camel-core). >>>>>>>>> >>>>>>>>> 3. Using the latest 2.x version of the maven-bundle-plugin >>> (2.5.4) >>>>>>>> makes >>>>>>>>> the maven-shade-plugin fail with this error in turn: >>>>>>>>> >>>>>>>>> [ERROR] The project main artifact does not exist. This could have >>> the >>>>>>>>> following >>>>>>>>> [ERROR] reasons: >>>>>>>>> [ERROR] - You have invoked the goal directly from the command >>> line. This >>>>>>>> is >>>>>>>>> not >>>>>>>>> [ERROR] supported. Please add the goal to the default lifecycle >>> via an >>>>>>>>> [ERROR] <execution> element in your POM and use "mvn package" to >>> have >>>>>>>> it >>>>>>>>> run. >>>>>>>>> [ERROR] - You have bound the goal to a lifecycle phase before >>> "package". >>>>>>>>> Please >>>>>>>>> [ERROR] remove this binding from your POM such that the goal >>> will be >>>>>>>> run >>>>>>>>> in >>>>>>>>> [ERROR] the proper phase. >>>>>>>>> >>>>>>>>> 4. Upgrading the shade plugin doesn't help. >>>>>>>>> >>>>>>>>> 5. The situation is dirty :P >>>>>>>>> >>>>>>>>> The problem is the usage of the 'bundle' custom packaging type >>> enabled by >>>>>>>>> extensions='true' in the maven-bundle-plugin. >>>>>>>>> >>>>>>>>> *Good news again:* >>>>>>>>> >>>>>>>>> We can avoid the problem by reverting back to the standard 'jar' >>>>>>>> packaging >>>>>>>>> type, which even puts us in a less brittle situation wrt. to >>> future Maven >>>>>>>>> plugins we may want to add/upgrade. This implies: >>>>>>>>> >>>>>>>>> 1. Calling the maven-bundle-plugin:manifest goal (instead of the >>>>>>>> bundle >>>>>>>>> goal) to only generate the MANIFEST.MF. >>>>>>>>> >>>>>>>>> 2. Adding it to the JAR via an option in the the >>> maven-jar-plugin: >>>>>>>>> >>>>>>>>> <plugin> >>>>>>>>> <groupId>org.apache.maven.plugins</groupId> >>>>>>>>> <artifactId>maven-jar-plugin</artifactId> >>>>>>>>> <version>${maven-jar-plugin-version}</version> >>>>>>>>> <configuration> >>>>>>>>> <archive> >>>>>>>>> >>>>>>>>> >>>>>>>> >>> <manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile> >>>>>>>>> </archive> >>>>>>>>> </configuration> >>>>>>>>> </plugin> >>>>>>>>> >>>>>>>>> I've made the change. But since it's a big one, I didn't push to >>> master >>>>>>>> but >>>>>>>>> to a branch for us to review first: jdk8-lambdas. >>>>>>>>> >>>>>>>>> The build passes (yay!) but OSGi battle testing is a must to >>> ensure there >>>>>>>>> are no regressions. >>>>>>>>> >>>>>>>>> Could you have a quick look? If no feedback is received until >>> Monday EOD, >>>>>>>>> I'll merge to master. If feedback is positive, we can merge >>> earlier to >>>>>>>>> enable people to develop with lambdas finally. >>>>>>>>> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> Just in case you're not aware, the OSGi legend Neil Bartlett has >>> just >>>>>>>>> released a new bnd-maven-plugin [2] that claims to overcome such >>>>>>>>> dysfunctions, as well as others, of the Felix plugin. He >>> complained about >>>>>>>>> how the custom packaging type breaks integration with other plugins >>>>>>>>> downstream (what we're experiencing). >>>>>>>>> >>>>>>>>> I see no reason to migrate to his bnd-maven-plugin, but we should >>> keep it >>>>>>>>> in mind for the future. >>>>>>>>> >>>>>>>>> [1] https://issues.apache.org/jira/browse/FELIX-4005 >>>>>>>>> [2] >>> http://njbartlett.name/2015/03/27/announcing-bnd-maven-plugin.html >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> *Raúl Kripalani* >>>>>>>>> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big >>> Data and >>>>>>>>> Messaging Engineer >>>>>>>>> http://about.me/raulkripalani | >>> http://www.linkedin.com/in/raulkripalani >>>>>>>>> Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Claus Ibsen >>>>>>>> ----------------- >>>>>>>> http://davsclaus.com @davsclaus >>>>>>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claus Ibsen >>>>>> ----------------- >>>>>> http://davsclaus.com @davsclaus >>>>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> ----------------- >>>>> http://davsclaus.com @davsclaus >>>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> ----------------- >>>> http://davsclaus.com @davsclaus >>>> Camel in Action 2: https://www.manning.com/ibsen2 >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> http://davsclaus.com @davsclaus >>> Camel in Action 2: https://www.manning.com/ibsen2 >>> > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com <http://davsclaus.com/> @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > <https://www.manning.com/ibsen2>
