Tests in error:
CamelDebeziumCommonTest.org.apache.camel.itest.springboot.CamelDebeziumCommonTest
» Runtime
org.apache.camel.itest.springboot.CamelHazelcastTest.componentTests(org.apache.camel.itest.springboot.CamelHazelcastTest)
Run 1: CamelHazelcastTest>AbstractSpringBootTestSupport.startSpringBoot:44 »
InvocationTarget
Run 2: CamelHazelcastTest>AbstractSpringBootTestSupport.startSpringBoot:44 »
InvocationTarget
CamelHttpCommonTest.org.apache.camel.itest.springboot.CamelHttpCommonTest »
Runtime
CamelJettyCommonTest.org.apache.camel.itest.springboot.CamelJettyCommonTest »
Runtime
Tests run: 285, Failures: 0, Errors: 4, Skipped: 6
There actually are such classes under
tests/camel-itest-spring-boot/src/test/java, and there’s no indication they are
automatically generated.
I removed CamelDebeziumCommonTest, CamelHttpCommonTest, and
CamelJettyCommonTest which eliminated those 3 errors. The CamelHazelcastTest
problem has a stack trace of
Caused by: java.io.FileNotFoundException: class path resource
[org/apache/camel/component/hazelcast/atomicnumber/springboot/customizer/HazelcastInstanceCustomizer.class]
cannot be opened because it does not exist
at
org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:187)
~[spring-core-5.3.10.jar!/:5.3.10]
at
org.springframework.core.type.classreading.SimpleMetadataReader.getClassReader(SimpleMetadataReader.java:55)
~[spring-core-5.3.10.jar!/:5.3.10]
…
and sure enough HazelcastInstanceCustomizer isn’t there any more… you removed
it in
commit a56ca8c09d5b6e50bb4891c623e09a470da0522e
Author: Claus Ibsen <[email protected]>
Date: Sat Oct 9 09:54:37 2021 +0200
CAMEL-17056: camel-spring-boot-hazelcast-starter - Remove the old customer
code as its standard in camel-core now
but it’s still listed (with some others) in the spring.factories.
Removing the HazelcastInstanceCustomizer lines from spring.factories has all
the tests passing.
I propose removing the excess projects in one commit in my upcoming PRs and
then removing the HazelcastInstanceCustomizer lines in another commit,
mentioning CAMEL-17056.
David Jencks
> On Oct 19, 2021, at 2:31 AM, Claus Ibsen <[email protected]> wrote:
>
> And what integration test failures do you see when they are removed?
>
> eg maybe make sure that empty folders are removed etc, as the
> integration test is some voodoo magic Nicola did that creates a spring
> boot project per starter to see if it can install and startup spring
> boot, and for some it also runs additional tests.
> It's complex to maintain.
>
> On Tue, Oct 19, 2021 at 11:28 AM Claus Ibsen <[email protected]> wrote:
>>
>> Hi
>>
>> Yeah these starters are not in use and should be removed
>>
>> camel-debezium-common-starter
>> http-common-starter
>> jetty-common-starter
>>
>>
>> On Mon, Oct 18, 2021 at 8:34 PM David Jencks <[email protected]>
>> wrote:
>>>
>>> I can easily remove the empty starter json files, but if e.g.
>>> camel-debezium-common-starter isn’t a starter, what is it? Removing the 3
>>> starter projects from camel-spring-boot results in integration test
>>> failures. Does some other process install these as dependencies as
>>> appropriate? Are they actually unnecessary but something needs more
>>> tweaking?
>>>
>>> David Jencks
>>>
>>>> On Oct 18, 2021, at 12:17 AM, Claus Ibsen <[email protected]> wrote:
>>>>
>>>> Hi
>>>>
>>>> The -common JARs are not starters, and they should not be listed.
>>>>
>>>> On Mon, Oct 18, 2021 at 7:45 AM David Jencks <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>>
>>>>> Locally I got the ability to show unused starter json files working, and
>>>>> fixed the obvious problems:
>>>>>
>>>>> avro-rpc was using avro
>>>>> paho-mqtt5 was using paho
>>>>> in 3.7.x, grape and joor were set up wrong.
>>>>>
>>>>> There are still some “starters” i’m not sure about:
>>>>>
>>>>> debezium-common
>>>>> http-common
>>>>> jetty-common
>>>>>
>>>>> In a previous commit I added empty json files for these on the theory
>>>>> that if they were starters they should have instructions on how to use
>>>>> them, but they don’t have corresponding main camel doc pages. Are these
>>>>> starters automatically installed by the corresponding non-common starter
>>>>> artifacts such as camel-debezium-mongodb-starter? If so, I can remove the
>>>>> empty json files. If not, how should their usage be documented?
>>>>>
>>>>> I’m hoping to get my updated Antora extensions published in the next
>>>>> couple of days, at which point we can upgrade Camel to use them.
>>>>>
>>>>> David Jencks
>>>>>
>>>>>> On Oct 13, 2021, at 8:14 PM, David Jencks <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>> They are part of the camel-core-starter, although it’s hard to find the
>>>>>> 2 options per language among the 147 options for the starter. This
>>>>>> suggests that, for starters that encompass more than one component, it
>>>>>> would be useful to only show the options relevant to that component on
>>>>>> the components’ page.
>>>>>>
>>>>>> In any case I believe everything is working properly and I’ve merged all
>>>>>> the 9 PRs.
>>>>>>
>>>>>> The core and non-core languages are currently in different tables on the
>>>>>> spring-boot list page: I should be able to put them back into one table
>>>>>> after upgrading antora-indexer.
>>>>>>
>>>>>> David Jencks
>>>>>>
>>>>>>> On Oct 12, 2021, at 11:03 PM, David Jencks <[email protected]
>>>>>>> <mailto:[email protected]> <mailto:[email protected]
>>>>>>> <mailto:[email protected]>>> wrote:
>>>>>>>
>>>>>>> There’s at least one problem…. (also asked on zulip)
>>>>>>>
>>>>>>> Are camel-core-languages actually part of camel-spring-boot?
>>>>>>>
>>>>>>> They are listed in the table in "list of starters" but with an
>>>>>>> artifactid of "camel-base" which isn't a starter. The current site
>>>>>>> doesn't have autoconfiguration info for them either. Since they are in
>>>>>>> the table, I added info to them for my PRs, but it's at least partly
>>>>>>> wrong, since it comes out as needing camel-core-languages-starter,
>>>>>>> which doesn't exist. cf.
>>>>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages
>>>>>>>
>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages>
>>>>>>>
>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages
>>>>>>>
>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages>>,
>>>>>>>
>>>>>>> https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration
>>>>>>>
>>>>>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration>
>>>>>>>
>>>>>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration
>>>>>>>
>>>>>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration>>.
>>>>>>> Since they say they are part of camel-core, I've used core.json to
>>>>>>> construct the autoconfig options table, but that doesn't look very
>>>>>>> relevant to me.
>>>>>>>
>>>>>>> Is there a starter needed for them? If so, which one?
>>>>>>> camel-core-starter seems like a possibility, is it correct?
>>>>>>>
>>>>>>> If no starter is needed, why are they in the spring-boot starters table?
>>>>>>>
>>>>>>> I hope to be able to merge everything tomorrow, it got too late tonight.
>>>>>>>
>>>>>>> David Jencks
>>>>>>>
>>>>>>>> On Oct 12, 2021, at 7:41 PM, David Jencks <[email protected]
>>>>>>>> <mailto:[email protected]> <mailto:[email protected]
>>>>>>>> <mailto:[email protected]>>> wrote:
>>>>>>>>
>>>>>>>> The part of this work I’m ready for now is done, except for some
>>>>>>>> commit-squashing and dealing with check style errors :-)
>>>>>>>>
>>>>>>>> Preview at https://pr-644--camel.netlify.app
>>>>>>>> <https://pr-644--camel.netlify.app/>
>>>>>>>> <https://pr-644--camel.netlify.app/
>>>>>>>> <https://pr-644--camel.netlify.app/>>
>>>>>>>>
>>>>>>>> I seem to only be able to write AsciiDoc by now…. :-) Here’s a list
>>>>>>>> of the differences between table rows new/old:
>>>>>>>>
>>>>>>>> ——
>>>>>>>> = Differences between spring-boot list-old and list
>>>>>>>>
>>>>>>>> == latest (main)
>>>>>>>>
>>>>>>>> === In list, not list-old:
>>>>>>>>
>>>>>>>> * disruptor-vm-component
>>>>>>>> * splunk-hec-component
>>>>>>>>
>>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the
>>>>>>>> single main bindy doc page)
>>>>>>>>
>>>>>>>> * csimple-language
>>>>>>>>
>>>>>>>> == 3.12.x
>>>>>>>> === In list, not list-old:
>>>>>>>>
>>>>>>>> * disruptor-vm-component
>>>>>>>> * splunk-hec-component
>>>>>>>>
>>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the
>>>>>>>> single main bindy doc page)
>>>>>>>>
>>>>>>>> * csimple-language
>>>>>>>>
>>>>>>>> == 3.11.x
>>>>>>>> === In list, not list-old:
>>>>>>>>
>>>>>>>> * splunk-hec-component
>>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the
>>>>>>>> single main bindy doc page)
>>>>>>>> * csimple-language
>>>>>>>>
>>>>>>>> Moved to separate table:
>>>>>>>>
>>>>>>>> * camel-spring-cloud
>>>>>>>> * camel-spring-cloud-consul
>>>>>>>> * camel-spring-cloud-netflix
>>>>>>>> * camel-spring-cloud-zookeeper
>>>>>>>>
>>>>>>>> == 3.7.x
>>>>>>>> === In list, not list-old:
>>>>>>>> * splunk-hec-component
>>>>>>>> * (3 bindy dataformats collapsed to one link, corresponding to the
>>>>>>>> single main bindy doc page)
>>>>>>>> * csimple-language
>>>>>>>>
>>>>>>>>
>>>>>>>> Moved to separate table:
>>>>>>>>
>>>>>>>> * camel-spring-cloud
>>>>>>>> * camel-spring-cloud-consul
>>>>>>>> * camel-spring-cloud-netflix
>>>>>>>> * camel-spring-cloud-zookeeper
>>>>>>>>
>>>>>>>> ——
>>>>>>>>
>>>>>>>> There are also some presentation differences in the tables, the new
>>>>>>>> ones follow the tables in main camel components more closely, e.g.
>>>>>>>>
>>>>>>>> <td class="tableblock halign-left valign-top"><p
>>>>>>>> class="tableblock">Stable</p></td>
>>>>>>>>>>
>>>>>>>> <td class="tableblock halign-left valign-top"><p
>>>>>>>> class="tableblock">Stable-deprecated</p></td>
>>>>>>>>
>>>>>>>> as the deprecation marker.
>>>>>>>>
>>>>>>>> Also quite a few core languages now list the starter as
>>>>>>>>
>>>>>>>> <td class="tableblock halign-left valign-top"><p
>>>>>>>> class="tableblock">camel-core-languages-starter</p></td>
>>>>>>>> rather than
>>>>>>>> <td class="tableblock halign-left valign-top"><p
>>>>>>>> class="tableblock">camel-base</p></td>
>>>>>>>>
>>>>>>>> Is this correct?
>>>>>>>>
>>>>>>>> There are 8 PRs:
>>>>>>>> (Camel)
>>>>>>>> Main spring boot jsonpath (allow me to squash and merge)
>>>>>>>> <https://github.com/apache/camel/pull/6261
>>>>>>>> <https://github.com/apache/camel/pull/6261>>
>>>>>>>> Camel 3.12.x spring boot jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel/pull/6263
>>>>>>>> <https://github.com/apache/camel/pull/6263>>
>>>>>>>> Camel 3.11.x spring boot jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel/pull/6264
>>>>>>>> <https://github.com/apache/camel/pull/6264>>
>>>>>>>> Camel 3.7.x spring boot jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel/pull/6262
>>>>>>>> <https://github.com/apache/camel/pull/6262>>
>>>>>>>> (Camel-spring-boot)
>>>>>>>> Main jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/388
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/388>>
>>>>>>>> Camel spring boot 3.12.x jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/387
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/387>>
>>>>>>>> Camel spring boot 3.11.x jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/386
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/386>>
>>>>>>>> Camel spring boot 3.7.x jsonpath (please let me squash and merge)
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/385
>>>>>>>> <https://github.com/apache/camel-spring-boot/pull/385>>
>>>>>>>>
>>>>>>>> If everything looks good I plan to squash each PR to 2 or 3 commits
>>>>>>>> (code changes and generated changes) and rebase/merge. I don’t think
>>>>>>>> any playbook changes are needed.
>>>>>>>>
>>>>>>>> I’m completing a big upgrade of the antora-indexer and jsonpath
>>>>>>>> extensions, and plan to finish that next. It will involve changing
>>>>>>>> the syntax of most uses of these extensions. After that I expect to
>>>>>>>> be able to implement automated checking for starters that aren’t
>>>>>>>> linked to by main camel components. I think there are about 5, but
>>>>>>>> have no way to find them at this point.
>>>>>>>>
>>>>>>>>
>>>>>>>> David Jencks
>>>>>>>>
>>>>>>>>> On Oct 12, 2021, at 12:01 AM, Claus Ibsen <[email protected]
>>>>>>>>> <mailto:[email protected]> <mailto:[email protected]
>>>>>>>>> <mailto:[email protected]>>> wrote:
>>>>>>>>>
>>>>>>>>> On Mon, Oct 11, 2021 at 8:07 AM David Jencks
>>>>>>>>> <[email protected] <mailto:[email protected]>
>>>>>>>>> <mailto:[email protected] <mailto:[email protected]>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I’ve been working on generating the spring-boot autoconfig info from
>>>>>>>>>> the json files generated during the spring-boot build, and the
>>>>>>>>>> spring boot “list.adoc” tables using the indexer extension (like the
>>>>>>>>>> components tables are generated).
>>>>>>>>>>
>>>>>>>>>> My understanding of the spring-boot stuff is that:
>>>>>>>>>>
>>>>>>>>>> - any Camel component can be used in Spring
>>>>>>>>>>
>>>>>>>>>> - Some camel components/dataformats… can participate in spring-boot
>>>>>>>>>> auto configure. These components are identified as having a
>>>>>>>>>> corresponding camel-*-starter project in the spring-boot repo.
>>>>>>>>>>
>>>>>>>>>> - There are 3 kinds of these:
>>>>>>>>>>
>>>>>>>>>> 1. There’s (generated) java code and a generated json file
>>>>>>>>>>
>>>>>>>>>> 2. There’s generated java code but no generated json file (e.g.
>>>>>>>>>> jasypt)
>>>>>>>>>>
>>>>>>>>>> 3. There’s no java code and no json file, just some properties files
>>>>>>>>>> etc. (e.g. aws-xray)
>>>>>>>>>>
>>>>>>>>>> I’m really not familiar with spring-boot or the philosophy behind
>>>>>>>>>> it, but my guess is that if there’s a spring-boot starter jar then
>>>>>>>>>> spring will create a singleton instance of something in the jar, and
>>>>>>>>>> if there’s a json file that will guide or describe configuring this
>>>>>>>>>> instance with data from somewhere. If this is roughly correct, then
>>>>>>>>>> the 2nd and 3rd kinds correspond to non-configurable instances.
>>>>>>>>>>
>>>>>>>>>> I’ve discovered that:
>>>>>>>>>>
>>>>>>>>>> - several components have starters that are not listed in the
>>>>>>>>>> existing list
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Oh which ones for example is that?
>>>>>>>>>
>>>>>>>>> For Camel on Spring Boot then we only support the -starter JARs. So if
>>>>>>>>> there is not a starter then its not supported.
>>>>>>>>> And yes some of the starters are just shallow and empty/almost empty.
>>>>>>>>>
>>>>>>>>> But it's our way to curate what we support and make sense to use on
>>>>>>>>> Spring Boot.
>>>>>>>>>
>>>>>>>>> For Camel on Karaf its the features.xml file that defines what we
>>>>>>>>> support there.
>>>>>>>>> And ditto for Quarkus with the extensions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> - quite a few components main adoc documentation do not include the
>>>>>>>>>> relevant autoconfig information (even if it’s “no options”)
>>>>>>>>>> Most of these are linked to from the “list” page, so I’d imagine it
>>>>>>>>>> would be pretty confusing to follow the link and find no information
>>>>>>>>>> about spring-boot.
>>>>>>>>>>
>>>>>>>>>> Evidently, the current process is not maintainable.
>>>>>>>>>>
>>>>>>>>>> The (new, jsonpath based) autoconfig doc generation is based on
>>>>>>>>>> putting the name of the appropriate json file in the main adoc file
>>>>>>>>>> as a header attribute.
>>>>>>>>>>
>>>>>>>>>> I think we can produce something more resilient by:
>>>>>>>>>>
>>>>>>>>>> - for kinds (2) and (3), put a no-options appropriately named json
>>>>>>>>>> file in the starter project under src/main/docs. This will not get
>>>>>>>>>> added to the starter jar, unlike the properly generated json files.
>>>>>>>>>>
>>>>>>>>>> - put the corresponding header attribute in the appropriate main
>>>>>>>>>> adoc file
>>>>>>>>>>
>>>>>>>>>> Now we should be able to do a basic check, that the number of unique
>>>>>>>>>> json file names from header attributes is equal to the number of
>>>>>>>>>> json files, which should be equal to the number of starters. This
>>>>>>>>>> should be possible to do as part of the website build.
>>>>>>>>>>
>>>>>>>>>> This won’t check that for instance with a starter that serves many
>>>>>>>>>> components, all the components will refer to the starter
>>>>>>>>>> information, but it will detect most or all of the problems I’v
>>>>>>>>>> found so far.
>>>>>>>>>>
>>>>>>>>>> The preview for https://github.com/apache/camel-website/pull/644
>>>>>>>>>> <https://github.com/apache/camel-website/pull/644>
>>>>>>>>>> <https://github.com/apache/camel-website/pull/644
>>>>>>>>>> <https://github.com/apache/camel-website/pull/644>> shows how far
>>>>>>>>>> I’ve gotten with this: the appropriate page to look at is
>>>>>>>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html
>>>>>>>>>>
>>>>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html>
>>>>>>>>>>
>>>>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html
>>>>>>>>>>
>>>>>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html>>
>>>>>>>>>> This has, in addition to the tables of spring-boot
>>>>>>>>>> autoconfigure-enabled components, counts of json files used and
>>>>>>>>>> existing (there are 5 unused), and tables of non-spring-boot-enabled
>>>>>>>>>> components/dataformats…
>>>>>>>>>> All the dataformats and languages are set up properly. There are 2
>>>>>>>>>> components that aren’t spring-boot-enabled and a whole lot of
>>>>>>>>>> “others"
>>>>>>>>>>
>>>>>>>>>> I think that actually failing the build when there are unused .json
>>>>>>>>>> files and also finding out which json files are unused will have to
>>>>>>>>>> wait for an upgrade to the antora-indexer… unfortunately this will
>>>>>>>>>> require some syntax changes.
>>>>>>>>>>
>>>>>>>>>> I’d like to polish these changes, replace the list page with the new
>>>>>>>>>> one (with the extra info removed), port this work to the other
>>>>>>>>>> relevant branches, and then pursue releasing the upgraded
>>>>>>>>>> antora-indexer extension and upgrading the syntax, and then work on
>>>>>>>>>> making the checks work.
>>>>>>>>>>
>>>>>>>>>> Does this seem reasonable?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yeah sure continue your great work and effort on improving the
>>>>>>>>> documentation for all the Camel projects on the website.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> ——
>>>>>>>>>>
>>>>>>>>>> Another feature of the preview is that I found out how to make the
>>>>>>>>>> option names in all the jsonpath generated tables link to themselves
>>>>>>>>>> and display a marker on hover, just like section titles, so you can
>>>>>>>>>> now easily copy the link to an option and paste it somewhere. (the
>>>>>>>>>> names have been links for a while, but there was no practical way to
>>>>>>>>>> find out what the link was!)
>>>>>>>>>>
>>>>>>>>>> E.g. …
>>>>>>>>>> https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo
>>>>>>>>>>
>>>>>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo><https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo
>>>>>>>>>>
>>>>>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo>>
>>>>>>>>>>
>>>>>>>>>> David Jencks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> -----------------
>>>>>>>>> http://davsclaus.com <http://davsclaus.com/> <http://davsclaus.com/
>>>>>>>>> <http://davsclaus.com/>> @davsclaus
>>>>>>>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>>>>>>> <https://www.manning.com/ibsen2> <https://www.manning.com/ibsen2
>>>>>>>>> <https://www.manning.com/ibsen2>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> http://davsclaus.com <http://davsclaus.com/> @davsclaus
>>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>> <https://www.manning.com/ibsen2>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
