I can easily remove the empty starter json files, but if e.g. camel-debezium-common-starter isn’t a starter, what is it? Removing the 3 starter projects from camel-spring-boot results in integration test failures. Does some other process install these as dependencies as appropriate? Are they actually unnecessary but something needs more tweaking?
David Jencks > On Oct 18, 2021, at 12:17 AM, Claus Ibsen <claus.ib...@gmail.com> wrote: > > Hi > > The -common JARs are not starters, and they should not be listed. > > On Mon, Oct 18, 2021 at 7:45 AM David Jencks <david.a.jen...@gmail.com > <mailto:david.a.jen...@gmail.com>> wrote: >> >> Locally I got the ability to show unused starter json files working, and >> fixed the obvious problems: >> >> avro-rpc was using avro >> paho-mqtt5 was using paho >> in 3.7.x, grape and joor were set up wrong. >> >> There are still some “starters” i’m not sure about: >> >> debezium-common >> http-common >> jetty-common >> >> In a previous commit I added empty json files for these on the theory that >> if they were starters they should have instructions on how to use them, but >> they don’t have corresponding main camel doc pages. Are these starters >> automatically installed by the corresponding non-common starter artifacts >> such as camel-debezium-mongodb-starter? If so, I can remove the empty json >> files. If not, how should their usage be documented? >> >> I’m hoping to get my updated Antora extensions published in the next couple >> of days, at which point we can upgrade Camel to use them. >> >> David Jencks >> >>> On Oct 13, 2021, at 8:14 PM, David Jencks <david.a.jen...@gmail.com> wrote: >>> >>> They are part of the camel-core-starter, although it’s hard to find the 2 >>> options per language among the 147 options for the starter. This suggests >>> that, for starters that encompass more than one component, it would be >>> useful to only show the options relevant to that component on the >>> components’ page. >>> >>> In any case I believe everything is working properly and I’ve merged all >>> the 9 PRs. >>> >>> The core and non-core languages are currently in different tables on the >>> spring-boot list page: I should be able to put them back into one table >>> after upgrading antora-indexer. >>> >>> David Jencks >>> >>>> On Oct 12, 2021, at 11:03 PM, David Jencks <david.a.jen...@gmail.com >>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com >>>> <mailto:david.a.jen...@gmail.com>>> wrote: >>>> >>>> There’s at least one problem…. (also asked on zulip) >>>> >>>> Are camel-core-languages actually part of camel-spring-boot? >>>> >>>> They are listed in the table in "list of starters" but with an artifactid >>>> of "camel-base" which isn't a starter. The current site doesn't have >>>> autoconfiguration info for them either. Since they are in the table, I >>>> added info to them for my PRs, but it's at least partly wrong, since it >>>> comes out as needing camel-core-languages-starter, which doesn't exist. >>>> cf. >>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages >>>> >>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages> >>>> >>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages >>>> >>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages>>, >>>> >>>> https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration >>>> >>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration> >>>> >>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration >>>> >>>> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration>>. >>>> Since they say they are part of camel-core, I've used core.json to >>>> construct the autoconfig options table, but that doesn't look very >>>> relevant to me. >>>> >>>> Is there a starter needed for them? If so, which one? camel-core-starter >>>> seems like a possibility, is it correct? >>>> >>>> If no starter is needed, why are they in the spring-boot starters table? >>>> >>>> I hope to be able to merge everything tomorrow, it got too late tonight. >>>> >>>> David Jencks >>>> >>>>> On Oct 12, 2021, at 7:41 PM, David Jencks <david.a.jen...@gmail.com >>>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com >>>>> <mailto:david.a.jen...@gmail.com>>> wrote: >>>>> >>>>> The part of this work I’m ready for now is done, except for some >>>>> commit-squashing and dealing with check style errors :-) >>>>> >>>>> Preview at https://pr-644--camel.netlify.app >>>>> <https://pr-644--camel.netlify.app/> <https://pr-644--camel.netlify.app/ >>>>> <https://pr-644--camel.netlify.app/>> >>>>> >>>>> I seem to only be able to write AsciiDoc by now…. :-) Here’s a list of >>>>> the differences between table rows new/old: >>>>> >>>>> —— >>>>> = Differences between spring-boot list-old and list >>>>> >>>>> == latest (main) >>>>> >>>>> === In list, not list-old: >>>>> >>>>> * disruptor-vm-component >>>>> * splunk-hec-component >>>>> >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>>>> main bindy doc page) >>>>> >>>>> * csimple-language >>>>> >>>>> == 3.12.x >>>>> === In list, not list-old: >>>>> >>>>> * disruptor-vm-component >>>>> * splunk-hec-component >>>>> >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>>>> main bindy doc page) >>>>> >>>>> * csimple-language >>>>> >>>>> == 3.11.x >>>>> === In list, not list-old: >>>>> >>>>> * splunk-hec-component >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>>>> main bindy doc page) >>>>> * csimple-language >>>>> >>>>> Moved to separate table: >>>>> >>>>> * camel-spring-cloud >>>>> * camel-spring-cloud-consul >>>>> * camel-spring-cloud-netflix >>>>> * camel-spring-cloud-zookeeper >>>>> >>>>> == 3.7.x >>>>> === In list, not list-old: >>>>> * splunk-hec-component >>>>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>>>> main bindy doc page) >>>>> * csimple-language >>>>> >>>>> >>>>> Moved to separate table: >>>>> >>>>> * camel-spring-cloud >>>>> * camel-spring-cloud-consul >>>>> * camel-spring-cloud-netflix >>>>> * camel-spring-cloud-zookeeper >>>>> >>>>> —— >>>>> >>>>> There are also some presentation differences in the tables, the new ones >>>>> follow the tables in main camel components more closely, e.g. >>>>> >>>>> <td class="tableblock halign-left valign-top"><p >>>>> class="tableblock">Stable</p></td> >>>>>>> >>>>> <td class="tableblock halign-left valign-top"><p >>>>> class="tableblock">Stable-deprecated</p></td> >>>>> >>>>> as the deprecation marker. >>>>> >>>>> Also quite a few core languages now list the starter as >>>>> >>>>> <td class="tableblock halign-left valign-top"><p >>>>> class="tableblock">camel-core-languages-starter</p></td> >>>>> rather than >>>>> <td class="tableblock halign-left valign-top"><p >>>>> class="tableblock">camel-base</p></td> >>>>> >>>>> Is this correct? >>>>> >>>>> There are 8 PRs: >>>>> (Camel) >>>>> Main spring boot jsonpath (allow me to squash and merge) >>>>> <https://github.com/apache/camel/pull/6261 >>>>> <https://github.com/apache/camel/pull/6261>> >>>>> Camel 3.12.x spring boot jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel/pull/6263 >>>>> <https://github.com/apache/camel/pull/6263>> >>>>> Camel 3.11.x spring boot jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel/pull/6264 >>>>> <https://github.com/apache/camel/pull/6264>> >>>>> Camel 3.7.x spring boot jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel/pull/6262 >>>>> <https://github.com/apache/camel/pull/6262>> >>>>> (Camel-spring-boot) >>>>> Main jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel-spring-boot/pull/388 >>>>> <https://github.com/apache/camel-spring-boot/pull/388>> >>>>> Camel spring boot 3.12.x jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel-spring-boot/pull/387 >>>>> <https://github.com/apache/camel-spring-boot/pull/387>> >>>>> Camel spring boot 3.11.x jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel-spring-boot/pull/386 >>>>> <https://github.com/apache/camel-spring-boot/pull/386>> >>>>> Camel spring boot 3.7.x jsonpath (please let me squash and merge) >>>>> <https://github.com/apache/camel-spring-boot/pull/385 >>>>> <https://github.com/apache/camel-spring-boot/pull/385>> >>>>> >>>>> If everything looks good I plan to squash each PR to 2 or 3 commits (code >>>>> changes and generated changes) and rebase/merge. I don’t think any >>>>> playbook changes are needed. >>>>> >>>>> I’m completing a big upgrade of the antora-indexer and jsonpath >>>>> extensions, and plan to finish that next. It will involve changing the >>>>> syntax of most uses of these extensions. After that I expect to be able >>>>> to implement automated checking for starters that aren’t linked to by >>>>> main camel components. I think there are about 5, but have no way to >>>>> find them at this point. >>>>> >>>>> >>>>> David Jencks >>>>> >>>>>> On Oct 12, 2021, at 12:01 AM, Claus Ibsen <claus.ib...@gmail.com >>>>>> <mailto:claus.ib...@gmail.com> <mailto:claus.ib...@gmail.com >>>>>> <mailto:claus.ib...@gmail.com>>> wrote: >>>>>> >>>>>> On Mon, Oct 11, 2021 at 8:07 AM David Jencks <david.a.jen...@gmail.com >>>>>> <mailto:david.a.jen...@gmail.com> <mailto:david.a.jen...@gmail.com >>>>>> <mailto:david.a.jen...@gmail.com>>> wrote: >>>>>>> >>>>>>> I’ve been working on generating the spring-boot autoconfig info from >>>>>>> the json files generated during the spring-boot build, and the spring >>>>>>> boot “list.adoc” tables using the indexer extension (like the >>>>>>> components tables are generated). >>>>>>> >>>>>>> My understanding of the spring-boot stuff is that: >>>>>>> >>>>>>> - any Camel component can be used in Spring >>>>>>> >>>>>>> - Some camel components/dataformats… can participate in spring-boot >>>>>>> auto configure. These components are identified as having a >>>>>>> corresponding camel-*-starter project in the spring-boot repo. >>>>>>> >>>>>>> - There are 3 kinds of these: >>>>>>> >>>>>>> 1. There’s (generated) java code and a generated json file >>>>>>> >>>>>>> 2. There’s generated java code but no generated json file (e.g. jasypt) >>>>>>> >>>>>>> 3. There’s no java code and no json file, just some properties files >>>>>>> etc. (e.g. aws-xray) >>>>>>> >>>>>>> I’m really not familiar with spring-boot or the philosophy behind it, >>>>>>> but my guess is that if there’s a spring-boot starter jar then spring >>>>>>> will create a singleton instance of something in the jar, and if >>>>>>> there’s a json file that will guide or describe configuring this >>>>>>> instance with data from somewhere. If this is roughly correct, then >>>>>>> the 2nd and 3rd kinds correspond to non-configurable instances. >>>>>>> >>>>>>> I’ve discovered that: >>>>>>> >>>>>>> - several components have starters that are not listed in the existing >>>>>>> list >>>>>>> >>>>>> >>>>>> Oh which ones for example is that? >>>>>> >>>>>> For Camel on Spring Boot then we only support the -starter JARs. So if >>>>>> there is not a starter then its not supported. >>>>>> And yes some of the starters are just shallow and empty/almost empty. >>>>>> >>>>>> But it's our way to curate what we support and make sense to use on >>>>>> Spring Boot. >>>>>> >>>>>> For Camel on Karaf its the features.xml file that defines what we >>>>>> support there. >>>>>> And ditto for Quarkus with the extensions. >>>>>> >>>>>> >>>>>>> - quite a few components main adoc documentation do not include the >>>>>>> relevant autoconfig information (even if it’s “no options”) >>>>>>> Most of these are linked to from the “list” page, so I’d imagine it >>>>>>> would be pretty confusing to follow the link and find no information >>>>>>> about spring-boot. >>>>>>> >>>>>>> Evidently, the current process is not maintainable. >>>>>>> >>>>>>> The (new, jsonpath based) autoconfig doc generation is based on putting >>>>>>> the name of the appropriate json file in the main adoc file as a header >>>>>>> attribute. >>>>>>> >>>>>>> I think we can produce something more resilient by: >>>>>>> >>>>>>> - for kinds (2) and (3), put a no-options appropriately named json file >>>>>>> in the starter project under src/main/docs. This will not get added to >>>>>>> the starter jar, unlike the properly generated json files. >>>>>>> >>>>>>> - put the corresponding header attribute in the appropriate main adoc >>>>>>> file >>>>>>> >>>>>>> Now we should be able to do a basic check, that the number of unique >>>>>>> json file names from header attributes is equal to the number of json >>>>>>> files, which should be equal to the number of starters. This should be >>>>>>> possible to do as part of the website build. >>>>>>> >>>>>>> This won’t check that for instance with a starter that serves many >>>>>>> components, all the components will refer to the starter information, >>>>>>> but it will detect most or all of the problems I’v found so far. >>>>>>> >>>>>>> The preview for https://github.com/apache/camel-website/pull/644 >>>>>>> <https://github.com/apache/camel-website/pull/644> >>>>>>> <https://github.com/apache/camel-website/pull/644 >>>>>>> <https://github.com/apache/camel-website/pull/644>> shows how far I’ve >>>>>>> gotten with this: the appropriate page to look at is >>>>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html> >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html >>>>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html>> >>>>>>> This has, in addition to the tables of spring-boot >>>>>>> autoconfigure-enabled components, counts of json files used and >>>>>>> existing (there are 5 unused), and tables of non-spring-boot-enabled >>>>>>> components/dataformats… >>>>>>> All the dataformats and languages are set up properly. There are 2 >>>>>>> components that aren’t spring-boot-enabled and a whole lot of “others" >>>>>>> >>>>>>> I think that actually failing the build when there are unused .json >>>>>>> files and also finding out which json files are unused will have to >>>>>>> wait for an upgrade to the antora-indexer… unfortunately this will >>>>>>> require some syntax changes. >>>>>>> >>>>>>> I’d like to polish these changes, replace the list page with the new >>>>>>> one (with the extra info removed), port this work to the other relevant >>>>>>> branches, and then pursue releasing the upgraded antora-indexer >>>>>>> extension and upgrading the syntax, and then work on making the checks >>>>>>> work. >>>>>>> >>>>>>> Does this seem reasonable? >>>>>>> >>>>>> >>>>>> Yeah sure continue your great work and effort on improving the >>>>>> documentation for all the Camel projects on the website. >>>>>> >>>>>> >>>>>> >>>>>>> —— >>>>>>> >>>>>>> Another feature of the preview is that I found out how to make the >>>>>>> option names in all the jsonpath generated tables link to themselves >>>>>>> and display a marker on hover, just like section titles, so you can now >>>>>>> easily copy the link to an option and paste it somewhere. (the names >>>>>>> have been links for a while, but there was no practical way to find out >>>>>>> what the link was!) >>>>>>> >>>>>>> E.g. … >>>>>>> https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo><https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo >>>>>>> >>>>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo>> >>>>>>> >>>>>>> David Jencks >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claus Ibsen >>>>>> ----------------- >>>>>> http://davsclaus.com <http://davsclaus.com/> <http://davsclaus.com/ >>>>>> <http://davsclaus.com/>> @davsclaus >>>>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>>>> <https://www.manning.com/ibsen2> <https://www.manning.com/ibsen2 >>>>>> <https://www.manning.com/ibsen2>> >>>> >>> >> > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com <http://davsclaus.com/> @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > <https://www.manning.com/ibsen2>
