Locally I got the ability to show unused starter json files working, and fixed the obvious problems:
avro-rpc was using avro paho-mqtt5 was using paho in 3.7.x, grape and joor were set up wrong. There are still some “starters” i’m not sure about: debezium-common http-common jetty-common In a previous commit I added empty json files for these on the theory that if they were starters they should have instructions on how to use them, but they don’t have corresponding main camel doc pages. Are these starters automatically installed by the corresponding non-common starter artifacts such as camel-debezium-mongodb-starter? If so, I can remove the empty json files. If not, how should their usage be documented? I’m hoping to get my updated Antora extensions published in the next couple of days, at which point we can upgrade Camel to use them. David Jencks > On Oct 13, 2021, at 8:14 PM, David Jencks <david.a.jen...@gmail.com> wrote: > > They are part of the camel-core-starter, although it’s hard to find the 2 > options per language among the 147 options for the starter. This suggests > that, for starters that encompass more than one component, it would be useful > to only show the options relevant to that component on the components’ page. > > In any case I believe everything is working properly and I’ve merged all the > 9 PRs. > > The core and non-core languages are currently in different tables on the > spring-boot list page: I should be able to put them back into one table after > upgrading antora-indexer. > > David Jencks > >> On Oct 12, 2021, at 11:03 PM, David Jencks <david.a.jen...@gmail.com >> <mailto:david.a.jen...@gmail.com>> wrote: >> >> There’s at least one problem…. (also asked on zulip) >> >> Are camel-core-languages actually part of camel-spring-boot? >> >> They are listed in the table in "list of starters" but with an artifactid of >> "camel-base" which isn't a starter. The current site doesn't have >> autoconfiguration info for them either. Since they are in the table, I added >> info to them for my PRs, but it's at least partly wrong, since it comes out >> as needing camel-core-languages-starter, which doesn't exist. cf. >> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages >> >> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list.html#_camel_languages>, >> >> https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration >> >> <https://pr-644--camel.netlify.app/components/latest/languages/header-language.html#_spring_boot_auto_configuration>. >> Since they say they are part of camel-core, I've used core.json to construct >> the autoconfig options table, but that doesn't look very relevant to me. >> >> Is there a starter needed for them? If so, which one? camel-core-starter >> seems like a possibility, is it correct? >> >> If no starter is needed, why are they in the spring-boot starters table? >> >> I hope to be able to merge everything tomorrow, it got too late tonight. >> >> David Jencks >> >>> On Oct 12, 2021, at 7:41 PM, David Jencks <david.a.jen...@gmail.com >>> <mailto:david.a.jen...@gmail.com>> wrote: >>> >>> The part of this work I’m ready for now is done, except for some >>> commit-squashing and dealing with check style errors :-) >>> >>> Preview at https://pr-644--camel.netlify.app >>> <https://pr-644--camel.netlify.app/> >>> >>> I seem to only be able to write AsciiDoc by now…. :-) Here’s a list of the >>> differences between table rows new/old: >>> >>> —— >>> = Differences between spring-boot list-old and list >>> >>> == latest (main) >>> >>> === In list, not list-old: >>> >>> * disruptor-vm-component >>> * splunk-hec-component >>> >>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>> main bindy doc page) >>> >>> * csimple-language >>> >>> == 3.12.x >>> === In list, not list-old: >>> >>> * disruptor-vm-component >>> * splunk-hec-component >>> >>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>> main bindy doc page) >>> >>> * csimple-language >>> >>> == 3.11.x >>> === In list, not list-old: >>> >>> * splunk-hec-component >>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>> main bindy doc page) >>> * csimple-language >>> >>> Moved to separate table: >>> >>> * camel-spring-cloud >>> * camel-spring-cloud-consul >>> * camel-spring-cloud-netflix >>> * camel-spring-cloud-zookeeper >>> >>> == 3.7.x >>> === In list, not list-old: >>> * splunk-hec-component >>> * (3 bindy dataformats collapsed to one link, corresponding to the single >>> main bindy doc page) >>> * csimple-language >>> >>> >>> Moved to separate table: >>> >>> * camel-spring-cloud >>> * camel-spring-cloud-consul >>> * camel-spring-cloud-netflix >>> * camel-spring-cloud-zookeeper >>> >>> —— >>> >>> There are also some presentation differences in the tables, the new ones >>> follow the tables in main camel components more closely, e.g. >>> >>> <td class="tableblock halign-left valign-top"><p >>> class="tableblock">Stable</p></td> >>> >> >>> <td class="tableblock halign-left valign-top"><p >>> class="tableblock">Stable-deprecated</p></td> >>> >>> as the deprecation marker. >>> >>> Also quite a few core languages now list the starter as >>> >>> <td class="tableblock halign-left valign-top"><p >>> class="tableblock">camel-core-languages-starter</p></td> >>> rather than >>> <td class="tableblock halign-left valign-top"><p >>> class="tableblock">camel-base</p></td> >>> >>> Is this correct? >>> >>> There are 8 PRs: >>> (Camel) >>> Main spring boot jsonpath (allow me to squash and merge) >>> <https://github.com/apache/camel/pull/6261> >>> Camel 3.12.x spring boot jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel/pull/6263> >>> Camel 3.11.x spring boot jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel/pull/6264> >>> Camel 3.7.x spring boot jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel/pull/6262> >>> (Camel-spring-boot) >>> Main jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel-spring-boot/pull/388> >>> Camel spring boot 3.12.x jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel-spring-boot/pull/387> >>> Camel spring boot 3.11.x jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel-spring-boot/pull/386> >>> Camel spring boot 3.7.x jsonpath (please let me squash and merge) >>> <https://github.com/apache/camel-spring-boot/pull/385> >>> >>> If everything looks good I plan to squash each PR to 2 or 3 commits (code >>> changes and generated changes) and rebase/merge. I don’t think any >>> playbook changes are needed. >>> >>> I’m completing a big upgrade of the antora-indexer and jsonpath extensions, >>> and plan to finish that next. It will involve changing the syntax of most >>> uses of these extensions. After that I expect to be able to implement >>> automated checking for starters that aren’t linked to by main camel >>> components. I think there are about 5, but have no way to find them at >>> this point. >>> >>> >>> David Jencks >>> >>>> On Oct 12, 2021, at 12:01 AM, Claus Ibsen <claus.ib...@gmail.com >>>> <mailto:claus.ib...@gmail.com>> wrote: >>>> >>>> On Mon, Oct 11, 2021 at 8:07 AM David Jencks <david.a.jen...@gmail.com >>>> <mailto:david.a.jen...@gmail.com>> wrote: >>>>> >>>>> I’ve been working on generating the spring-boot autoconfig info from the >>>>> json files generated during the spring-boot build, and the spring boot >>>>> “list.adoc” tables using the indexer extension (like the components >>>>> tables are generated). >>>>> >>>>> My understanding of the spring-boot stuff is that: >>>>> >>>>> - any Camel component can be used in Spring >>>>> >>>>> - Some camel components/dataformats… can participate in spring-boot auto >>>>> configure. These components are identified as having a corresponding >>>>> camel-*-starter project in the spring-boot repo. >>>>> >>>>> - There are 3 kinds of these: >>>>> >>>>> 1. There’s (generated) java code and a generated json file >>>>> >>>>> 2. There’s generated java code but no generated json file (e.g. jasypt) >>>>> >>>>> 3. There’s no java code and no json file, just some properties files etc. >>>>> (e.g. aws-xray) >>>>> >>>>> I’m really not familiar with spring-boot or the philosophy behind it, but >>>>> my guess is that if there’s a spring-boot starter jar then spring will >>>>> create a singleton instance of something in the jar, and if there’s a >>>>> json file that will guide or describe configuring this instance with data >>>>> from somewhere. If this is roughly correct, then the 2nd and 3rd kinds >>>>> correspond to non-configurable instances. >>>>> >>>>> I’ve discovered that: >>>>> >>>>> - several components have starters that are not listed in the existing >>>>> list >>>>> >>>> >>>> Oh which ones for example is that? >>>> >>>> For Camel on Spring Boot then we only support the -starter JARs. So if >>>> there is not a starter then its not supported. >>>> And yes some of the starters are just shallow and empty/almost empty. >>>> >>>> But it's our way to curate what we support and make sense to use on Spring >>>> Boot. >>>> >>>> For Camel on Karaf its the features.xml file that defines what we support >>>> there. >>>> And ditto for Quarkus with the extensions. >>>> >>>> >>>>> - quite a few components main adoc documentation do not include the >>>>> relevant autoconfig information (even if it’s “no options”) >>>>> Most of these are linked to from the “list” page, so I’d imagine it would >>>>> be pretty confusing to follow the link and find no information about >>>>> spring-boot. >>>>> >>>>> Evidently, the current process is not maintainable. >>>>> >>>>> The (new, jsonpath based) autoconfig doc generation is based on putting >>>>> the name of the appropriate json file in the main adoc file as a header >>>>> attribute. >>>>> >>>>> I think we can produce something more resilient by: >>>>> >>>>> - for kinds (2) and (3), put a no-options appropriately named json file >>>>> in the starter project under src/main/docs. This will not get added to >>>>> the starter jar, unlike the properly generated json files. >>>>> >>>>> - put the corresponding header attribute in the appropriate main adoc file >>>>> >>>>> Now we should be able to do a basic check, that the number of unique json >>>>> file names from header attributes is equal to the number of json files, >>>>> which should be equal to the number of starters. This should be possible >>>>> to do as part of the website build. >>>>> >>>>> This won’t check that for instance with a starter that serves many >>>>> components, all the components will refer to the starter information, but >>>>> it will detect most or all of the problems I’v found so far. >>>>> >>>>> The preview for https://github.com/apache/camel-website/pull/644 >>>>> <https://github.com/apache/camel-website/pull/644> shows how far I’ve >>>>> gotten with this: the appropriate page to look at is >>>>> https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html >>>>> <https://pr-644--camel.netlify.app/camel-spring-boot/latest/list-2.html> >>>>> This has, in addition to the tables of spring-boot autoconfigure-enabled >>>>> components, counts of json files used and existing (there are 5 unused), >>>>> and tables of non-spring-boot-enabled components/dataformats… >>>>> All the dataformats and languages are set up properly. There are 2 >>>>> components that aren’t spring-boot-enabled and a whole lot of “others" >>>>> >>>>> I think that actually failing the build when there are unused .json files >>>>> and also finding out which json files are unused will have to wait for an >>>>> upgrade to the antora-indexer… unfortunately this will require some >>>>> syntax changes. >>>>> >>>>> I’d like to polish these changes, replace the list page with the new one >>>>> (with the extra info removed), port this work to the other relevant >>>>> branches, and then pursue releasing the upgraded antora-indexer extension >>>>> and upgrading the syntax, and then work on making the checks work. >>>>> >>>>> Does this seem reasonable? >>>>> >>>> >>>> Yeah sure continue your great work and effort on improving the >>>> documentation for all the Camel projects on the website. >>>> >>>> >>>> >>>>> —— >>>>> >>>>> Another feature of the preview is that I found out how to make the option >>>>> names in all the jsonpath generated tables link to themselves and display >>>>> a marker on hover, just like section titles, so you can now easily copy >>>>> the link to an option and paste it somewhere. (the names have been links >>>>> for a while, but there was no practical way to find out what the link >>>>> was!) >>>>> >>>>> E.g. … >>>>> https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo >>>>> >>>>> <https://pr-644--camel.netlify.app/components/latest/amqp-component.html#_endpoint_query_option_disableReplyTo> >>>>> >>>>> David Jencks >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> ----------------- >>>> http://davsclaus.com <http://davsclaus.com/> @davsclaus >>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>> <https://www.manning.com/ibsen2> >> >
