Tests in error: CamelDebeziumCommonTest.org.apache.camel.itest.springboot.CamelDebeziumCommonTest » Runtime org.apache.camel.itest.springboot.CamelHazelcastTest.componentTests(org.apache.camel.itest.springboot.CamelHazelcastTest) Run 1: CamelHazelcastTest>AbstractSpringBootTestSupport.startSpringBoot:44 » InvocationTarget Run 2: CamelHazelcastTest>AbstractSpringBootTestSupport.startSpringBoot:44 » InvocationTarget
CamelHttpCommonTest.org.apache.camel.itest.springboot.CamelHttpCommonTest » Runtime CamelJettyCommonTest.org.apache.camel.itest.springboot.CamelJettyCommonTest » Runtime Tests run: 285, Failures: 0, Errors: 4, Skipped: 6 There actually are such classes under tests/camel-itest-spring-boot/src/test/java, and there’s no indication they are automatically generated. I removed CamelDebeziumCommonTest, CamelHttpCommonTest, and CamelJettyCommonTest which eliminated those 3 errors. The CamelHazelcastTest problem has a stack trace of Caused by: java.io.FileNotFoundException: class path resource [org/apache/camel/component/hazelcast/atomicnumber/springboot/customizer/HazelcastInstanceCustomizer.class] cannot be opened because it does not exist at org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:187) ~[spring-core-5.3.10.jar!/:5.3.10] at org.springframework.core.type.classreading.SimpleMetadataReader.getClassReader(SimpleMetadataReader.java:55) ~[spring-core-5.3.10.jar!/:5.3.10] … and sure enough HazelcastInstanceCustomizer isn’t there any more… you removed it in commit a56ca8c09d5b6e50bb4891c623e09a470da0522e Author: Claus Ibsen <claus.ib...@gmail.com> Date: Sat Oct 9 09:54:37 2021 +0200 CAMEL-17056: camel-spring-boot-hazelcast-starter - Remove the old customer code as its standard in camel-core now but it’s still listed (with some others) in the spring.factories. Removing the HazelcastInstanceCustomizer lines from spring.factories has all the tests passing. I propose removing the excess projects in one commit in my upcoming PRs and then removing the HazelcastInstanceCustomizer lines in another commit, mentioning CAMEL-17056. David Jencks > On Oct 19, 2021, at 2:31 AM, Claus Ibsen <claus.ib...@gmail.com> wrote: > > And what integration test failures do you see when they are removed? > > eg maybe make sure that empty folders are removed etc, as the > integration test is some voodoo magic Nicola did that creates a spring > boot project per starter to see if it can install and startup spring > boot, and for some it also runs additional tests. > It's complex to maintain. > > On Tue, Oct 19, 2021 at 11:28 AM Claus Ibsen <claus.ib...@gmail.com> wrote: >> >> Hi >> >> Yeah these starters are not in use and should be removed >> >> camel-debezium-common-starter >> http-common-starter >> jetty-common-starter >> >> >> On Mon, Oct 18, 2021 at 8:34 PM David Jencks <david.a.jen...@gmail.com> >> wrote: >>> >>> I can easily remove the empty starter json files, but if e.g. >>> camel-debezium-common-starter isn’t a starter, what is it? Removing the 3 >>> starter projects from camel-spring-boot results in integration test >>> failures. Does some other process install these as dependencies as >>> appropriate? Are they actually unnecessary but something needs more >>> tweaking? >>> >>> David Jencks >>> >>>> On Oct 18, 2021, at 12:17 AM, Claus Ibsen <claus.ib...@gmail.com> wrote: >>>> >>>> Hi >>>> >>>> The -common JARs are not starters, and they should not be listed. >>>> >>>> On Mon, Oct 18, 2021 at 7:45 AM David Jencks <david.a.jen...@gmail.com >>>> <mailto:david.a.jen...@gmail.com>> wrote: >>>>> >>>>> Locally I got the ability to show unused starter json files working, and >>>>> fixed the obvious problems: >>>>> >>>>> avro-rpc was using avro >>>>> paho-mqtt5 was using paho >>>>> in 3.7.x, grape and joor were set up wrong. >>>>> >>>>> There are still some “starters” i’m not sure about: >>>>> >>>>> debezium-common >>>>> http-common >>>>> jetty-common >>>>> >>>>> In a previous commit I added empty json files for these on the theory >>>>> that if they were starters they should have instructions on how to use >>>>> them, but they don’t have corresponding main camel doc pages. Are these >>>>> starters automatically installed by the corresponding non-common starter >>>>> artifacts such as camel-debezium-mongodb-starter? If so, I can remove the >>>>> empty json files. If not, how should their usage be documented? >>>>> >>>>> I’m hoping to get my updated Antora extensions published in the next >>>>> couple of days, at which point we can upgrade Camel to use them. >>>>> >>>>> David Jencks >>>>> >>>>>> On Oct 13, 2021, at 8:14 PM, David Jencks <david.a.jen...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> They are part of the camel-core-starter, although it’s hard to find the >>>>>> 2 options per language among the 147 options for the starter. This >>>>>> suggests that, for starters that encompass more than one component, it >>>>>> would be useful to only show the options relevant to that component on >>>>>> the components’ page. >>>>>> >>>>>> In any case I believe everything is working properly and I’ve merged all >>>>>> the 9 PRs. >>>>>> >>>>>> The core and non-core languages are currently in different tables on the >>>>>> spring-boot list page: I should be able to put them back into one table >>>>>> after upgrading antora-indexer. >>>>>> >>>>>> David Jencks >>>>>> >>>>>>> On Oct 12, 2021, at 11:03 PM, David Jencks <david.a.jen...@gmail.com >>>>>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com >>>>>>> <mailto:david.a.jen...@gmail.com>>> wrote: >>>>>>> >>>>>>> There’s at least one problem…. (also asked on zulip) >>>>>>> >>>>>>> Are camel-core-languages actually part of camel-spring-boot? >>>>>>> >>>>>>> They are listed in the table in "list of starters" but with an >>>>>>> artifactid of "camel-base" which isn't a starter. The current site >>>>>>> doesn't have autoconfiguration info for them either. Since they are in >>>>>>> the table, I added info to them for my PRs, but it's at least partly >>>>>>> wrong, since it comes out as needing camel-core-languages-starter, >>>>>>> which doesn't exist. cf. >>>>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages> >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages>>, >>>>>>> >>>>>>> https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration> >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration>>. >>>>>>> Since they say they are part of camel-core, I've used core.json to >>>>>>> construct the autoconfig options table, but that doesn't look very >>>>>>> relevant to me. >>>>>>> >>>>>>> Is there a starter needed for them? If so, which one? >>>>>>> camel-core-starter seems like a possibility, is it correct? >>>>>>> >>>>>>> If no starter is needed, why are they in the spring-boot starters table? >>>>>>> >>>>>>> I hope to be able to merge everything tomorrow, it got too late tonight. >>>>>>> >>>>>>> David Jencks >>>>>>> >>>>>>>> On Oct 12, 2021, at 7:41 PM, David Jencks <david.a.jen...@gmail.com >>>>>>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com >>>>>>>> <mailto:david.a.jen...@gmail.com>>> wrote: >>>>>>>> >>>>>>>> The part of this work I’m ready for now is done, except for some >>>>>>>> commit-squashing and dealing with check style errors :-) >>>>>>>> >>>>>>>> Preview at https://pr-644--camel.netlify.app >>>>>>>> <https://pr-644--camel.netlify.app/> >>>>>>>> <https://pr-644--camel.netlify.app/ >>>>>>>> <https://pr-644--camel.netlify.app/>> >>>>>>>> >>>>>>>> I seem to only be able to write AsciiDoc by now…. :-) Here’s a list >>>>>>>> of the differences between table rows new/old: >>>>>>>> >>>>>>>> —— >>>>>>>> = Differences between spring-boot list-old and list >>>>>>>> >>>>>>>> == latest (main) >>>>>>>> >>>>>>>> === In list, not list-old: >>>>>>>> >>>>>>>> * disruptor-vm-component >>>>>>>> * splunk-hec-component >>>>>>>> >>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the >>>>>>>> single main bindy doc page) >>>>>>>> >>>>>>>> * csimple-language >>>>>>>> >>>>>>>> == 3.12.x >>>>>>>> === In list, not list-old: >>>>>>>> >>>>>>>> * disruptor-vm-component >>>>>>>> * splunk-hec-component >>>>>>>> >>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the >>>>>>>> single main bindy doc page) >>>>>>>> >>>>>>>> * csimple-language >>>>>>>> >>>>>>>> == 3.11.x >>>>>>>> === In list, not list-old: >>>>>>>> >>>>>>>> * splunk-hec-component >>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the >>>>>>>> single main bindy doc page) >>>>>>>> * csimple-language >>>>>>>> >>>>>>>> Moved to separate table: >>>>>>>> >>>>>>>> * camel-spring-cloud >>>>>>>> * camel-spring-cloud-consul >>>>>>>> * camel-spring-cloud-netflix >>>>>>>> * camel-spring-cloud-zookeeper >>>>>>>> >>>>>>>> == 3.7.x >>>>>>>> === In list, not list-old: >>>>>>>> * splunk-hec-component >>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the >>>>>>>> single main bindy doc page) >>>>>>>> * csimple-language >>>>>>>> >>>>>>>> >>>>>>>> Moved to separate table: >>>>>>>> >>>>>>>> * camel-spring-cloud >>>>>>>> * camel-spring-cloud-consul >>>>>>>> * camel-spring-cloud-netflix >>>>>>>> * camel-spring-cloud-zookeeper >>>>>>>> >>>>>>>> —— >>>>>>>> >>>>>>>> There are also some presentation differences in the tables, the new >>>>>>>> ones follow the tables in main camel components more closely, e.g. >>>>>>>> >>>>>>>> <td class="tableblock halign-left valign-top"><p >>>>>>>> class="tableblock">Stable</p></td> >>>>>>>>>> >>>>>>>> <td class="tableblock halign-left valign-top"><p >>>>>>>> class="tableblock">Stable-deprecated</p></td> >>>>>>>> >>>>>>>> as the deprecation marker. >>>>>>>> >>>>>>>> Also quite a few core languages now list the starter as >>>>>>>> >>>>>>>> <td class="tableblock halign-left valign-top"><p >>>>>>>> class="tableblock">camel-core-languages-starter</p></td> >>>>>>>> rather than >>>>>>>> <td class="tableblock halign-left valign-top"><p >>>>>>>> class="tableblock">camel-base</p></td> >>>>>>>> >>>>>>>> Is this correct? >>>>>>>> >>>>>>>> There are 8 PRs: >>>>>>>> (Camel) >>>>>>>> Main spring boot jsonpath (allow me to squash and merge) >>>>>>>> <https://github.com/apache/camel/pull/6261 >>>>>>>> <https://github.com/apache/camel/pull/6261>> >>>>>>>> Camel 3.12.x spring boot jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel/pull/6263 >>>>>>>> <https://github.com/apache/camel/pull/6263>> >>>>>>>> Camel 3.11.x spring boot jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel/pull/6264 >>>>>>>> <https://github.com/apache/camel/pull/6264>> >>>>>>>> Camel 3.7.x spring boot jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel/pull/6262 >>>>>>>> <https://github.com/apache/camel/pull/6262>> >>>>>>>> (Camel-spring-boot) >>>>>>>> Main jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/388 >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/388>> >>>>>>>> Camel spring boot 3.12.x jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/387 >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/387>> >>>>>>>> Camel spring boot 3.11.x jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/386 >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/386>> >>>>>>>> Camel spring boot 3.7.x jsonpath (please let me squash and merge) >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/385 >>>>>>>> <https://github.com/apache/camel-spring-boot/pull/385>> >>>>>>>> >>>>>>>> If everything looks good I plan to squash each PR to 2 or 3 commits >>>>>>>> (code changes and generated changes) and rebase/merge. I don’t think >>>>>>>> any playbook changes are needed. >>>>>>>> >>>>>>>> I’m completing a big upgrade of the antora-indexer and jsonpath >>>>>>>> extensions, and plan to finish that next. It will involve changing >>>>>>>> the syntax of most uses of these extensions. After that I expect to >>>>>>>> be able to implement automated checking for starters that aren’t >>>>>>>> linked to by main camel components. I think there are about 5, but >>>>>>>> have no way to find them at this point. >>>>>>>> >>>>>>>> >>>>>>>> David Jencks >>>>>>>> >>>>>>>>> On Oct 12, 2021, at 12:01 AM, Claus Ibsen <claus.ib...@gmail.com >>>>>>>>> <mailto:claus.ib...@gmail.com> <mailto:claus.ib...@gmail.com >>>>>>>>> <mailto:claus.ib...@gmail.com>>> wrote: >>>>>>>>> >>>>>>>>> On Mon, Oct 11, 2021 at 8:07 AM David Jencks >>>>>>>>> <david.a.jen...@gmail.com <mailto:david.a.jen...@gmail.com> >>>>>>>>> <mailto:david.a.jen...@gmail.com <mailto:david.a.jen...@gmail.com>>> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> I’ve been working on generating the spring-boot autoconfig info from >>>>>>>>>> the json files generated during the spring-boot build, and the >>>>>>>>>> spring boot “list.adoc” tables using the indexer extension (like the >>>>>>>>>> components tables are generated). >>>>>>>>>> >>>>>>>>>> My understanding of the spring-boot stuff is that: >>>>>>>>>> >>>>>>>>>> - any Camel component can be used in Spring >>>>>>>>>> >>>>>>>>>> - Some camel components/dataformats… can participate in spring-boot >>>>>>>>>> auto configure. These components are identified as having a >>>>>>>>>> corresponding camel-*-starter project in the spring-boot repo. >>>>>>>>>> >>>>>>>>>> - There are 3 kinds of these: >>>>>>>>>> >>>>>>>>>> 1. There’s (generated) java code and a generated json file >>>>>>>>>> >>>>>>>>>> 2. There’s generated java code but no generated json file (e.g. >>>>>>>>>> jasypt) >>>>>>>>>> >>>>>>>>>> 3. There’s no java code and no json file, just some properties files >>>>>>>>>> etc. (e.g. aws-xray) >>>>>>>>>> >>>>>>>>>> I’m really not familiar with spring-boot or the philosophy behind >>>>>>>>>> it, but my guess is that if there’s a spring-boot starter jar then >>>>>>>>>> spring will create a singleton instance of something in the jar, and >>>>>>>>>> if there’s a json file that will guide or describe configuring this >>>>>>>>>> instance with data from somewhere. If this is roughly correct, then >>>>>>>>>> the 2nd and 3rd kinds correspond to non-configurable instances. >>>>>>>>>> >>>>>>>>>> I’ve discovered that: >>>>>>>>>> >>>>>>>>>> - several components have starters that are not listed in the >>>>>>>>>> existing list >>>>>>>>>> >>>>>>>>> >>>>>>>>> Oh which ones for example is that? >>>>>>>>> >>>>>>>>> For Camel on Spring Boot then we only support the -starter JARs. So if >>>>>>>>> there is not a starter then its not supported. >>>>>>>>> And yes some of the starters are just shallow and empty/almost empty. >>>>>>>>> >>>>>>>>> But it's our way to curate what we support and make sense to use on >>>>>>>>> Spring Boot. >>>>>>>>> >>>>>>>>> For Camel on Karaf its the features.xml file that defines what we >>>>>>>>> support there. >>>>>>>>> And ditto for Quarkus with the extensions. >>>>>>>>> >>>>>>>>> >>>>>>>>>> - quite a few components main adoc documentation do not include the >>>>>>>>>> relevant autoconfig information (even if it’s “no options”) >>>>>>>>>> Most of these are linked to from the “list” page, so I’d imagine it >>>>>>>>>> would be pretty confusing to follow the link and find no information >>>>>>>>>> about spring-boot. >>>>>>>>>> >>>>>>>>>> Evidently, the current process is not maintainable. >>>>>>>>>> >>>>>>>>>> The (new, jsonpath based) autoconfig doc generation is based on >>>>>>>>>> putting the name of the appropriate json file in the main adoc file >>>>>>>>>> as a header attribute. >>>>>>>>>> >>>>>>>>>> I think we can produce something more resilient by: >>>>>>>>>> >>>>>>>>>> - for kinds (2) and (3), put a no-options appropriately named json >>>>>>>>>> file in the starter project under src/main/docs. This will not get >>>>>>>>>> added to the starter jar, unlike the properly generated json files. >>>>>>>>>> >>>>>>>>>> - put the corresponding header attribute in the appropriate main >>>>>>>>>> adoc file >>>>>>>>>> >>>>>>>>>> Now we should be able to do a basic check, that the number of unique >>>>>>>>>> json file names from header attributes is equal to the number of >>>>>>>>>> json files, which should be equal to the number of starters. This >>>>>>>>>> should be possible to do as part of the website build. >>>>>>>>>> >>>>>>>>>> This won’t check that for instance with a starter that serves many >>>>>>>>>> components, all the components will refer to the starter >>>>>>>>>> information, but it will detect most or all of the problems I’v >>>>>>>>>> found so far. >>>>>>>>>> >>>>>>>>>> The preview for https://github.com/apache/camel-website/pull/644 >>>>>>>>>> <https://github.com/apache/camel-website/pull/644> >>>>>>>>>> <https://github.com/apache/camel-website/pull/644 >>>>>>>>>> <https://github.com/apache/camel-website/pull/644>> shows how far >>>>>>>>>> I’ve gotten with this: the appropriate page to look at is >>>>>>>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html >>>>>>>>>> >>>>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html> >>>>>>>>>> >>>>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html >>>>>>>>>> >>>>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html>> >>>>>>>>>> This has, in addition to the tables of spring-boot >>>>>>>>>> autoconfigure-enabled components, counts of json files used and >>>>>>>>>> existing (there are 5 unused), and tables of non-spring-boot-enabled >>>>>>>>>> components/dataformats… >>>>>>>>>> All the dataformats and languages are set up properly. There are 2 >>>>>>>>>> components that aren’t spring-boot-enabled and a whole lot of >>>>>>>>>> “others" >>>>>>>>>> >>>>>>>>>> I think that actually failing the build when there are unused .json >>>>>>>>>> files and also finding out which json files are unused will have to >>>>>>>>>> wait for an upgrade to the antora-indexer… unfortunately this will >>>>>>>>>> require some syntax changes. >>>>>>>>>> >>>>>>>>>> I’d like to polish these changes, replace the list page with the new >>>>>>>>>> one (with the extra info removed), port this work to the other >>>>>>>>>> relevant branches, and then pursue releasing the upgraded >>>>>>>>>> antora-indexer extension and upgrading the syntax, and then work on >>>>>>>>>> making the checks work. >>>>>>>>>> >>>>>>>>>> Does this seem reasonable? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yeah sure continue your great work and effort on improving the >>>>>>>>> documentation for all the Camel projects on the website. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> —— >>>>>>>>>> >>>>>>>>>> Another feature of the preview is that I found out how to make the >>>>>>>>>> option names in all the jsonpath generated tables link to themselves >>>>>>>>>> and display a marker on hover, just like section titles, so you can >>>>>>>>>> now easily copy the link to an option and paste it somewhere. (the >>>>>>>>>> names have been links for a while, but there was no practical way to >>>>>>>>>> find out what the link was!) >>>>>>>>>> >>>>>>>>>> E.g. … >>>>>>>>>> https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo >>>>>>>>>> >>>>>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo><https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo >>>>>>>>>> >>>>>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo>> >>>>>>>>>> >>>>>>>>>> David Jencks >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Claus Ibsen >>>>>>>>> ----------------- >>>>>>>>> http://davsclaus.com <http://davsclaus.com/> <http://davsclaus.com/ >>>>>>>>> <http://davsclaus.com/>> @davsclaus >>>>>>>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>>>>>>> <https://www.manning.com/ibsen2> <https://www.manning.com/ibsen2 >>>>>>>>> <https://www.manning.com/ibsen2>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> ----------------- >>>> http://davsclaus.com <http://davsclaus.com/> @davsclaus >>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>> <https://www.manning.com/ibsen2> >> >> >> >> -- >> Claus Ibsen >> ----------------- >> http://davsclaus.com @davsclaus >> Camel in Action 2: https://www.manning.com/ibsen2 > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2