Hi, TL/DR Can I assume that only projects currently under connectors should have pages in the website? (These are the pages indexed in reference/index.html and the nav)
—— The longer version…. I’m investigating whether it’s practical to completely generate the camel-kafka-connector reference docs directly from the .json files under `connectors` during the Antora build, and I’m somewhat confused about what is supposed to show up in the website and whether it is kept up to date correctly. I haven’t found the parts of the build that copy json files to the catalog or adoc files to the docs directory. The connectors index and nav are generated from the project folders under `connectors`, suggesting that only projects that currently exist are supposed to be documented. However, several adoc files that don’t correspond to currently existing projects are in the docs/…./reference/connectors directory and can be found on the website, e.g. https://camel.apache.org/camel-kafka-connector/next/reference/connectors/camel-aws2-s3-kafka-sink-connector.html All the similarly named files in the source tree: % find . -name camel-aws2-s3* ./camel-kafka-connector-catalog/target/classes/connectors/camel-aws2-s3-source.json ./camel-kafka-connector-catalog/target/classes/connectors/camel-aws2-s3-sink.json ./camel-kafka-connector-catalog/src/generated/resources/connectors/camel-aws2-s3-source.json ./camel-kafka-connector-catalog/src/generated/resources/connectors/camel-aws2-s3-sink.json ./docs/modules/ROOT/pages/reference/connectors/camel-aws2-s3-kafka-source-connector.adoc ./docs/modules/ROOT/pages/reference/connectors/camel-aws2-s3-kafka-sink-connector.adoc so it appears likely that the corresponding json files are in the catalog. I think the most relevant commit is commit 9fd67af40ee37cabef79f15570fee45f32f64f47 Author: Andrea Tarocchi <andrea.taroc...@gmail.com> Date: Sat Sep 18 21:30:29 2021 +0200 Related to #423 : Regen after switching to kamelet generation. I haven’t been able to completely comprehend this commit but with —stat I see several lines like connectors/{camel-aws2-s3-kafka-connector => camel-aws-kinesis-source-kafka-connector}/src/main/assembly/package.xml | 0 I think this indicates that this commit removed the camel-aws2-s3-kafka-connector project (I’m not sure why the dates are not chronological): % git log --stat -- connectors/camel-aws2-s3* commit 9fd67af40ee37cabef79f15570fee45f32f64f47 Author: Andrea Tarocchi <andrea.taroc...@gmail.com> Date: Sat Sep 18 21:30:29 2021 +0200 Related to #423 : Regen after switching to kamelet generation. connectors/camel-aws2-s3-kafka-connector/pom.xml | 144 ------------------- connectors/camel-aws2-s3-kafka-connector/src/generated/descriptors/connector-sink.properties | 1 - connectors/camel-aws2-s3-kafka-connector/src/generated/descriptors/connector-source.properties | 1 - connectors/camel-aws2-s3-kafka-connector/src/generated/resources/camel-aws2-s3-sink.json | 529 --------------------------------------------------------------------- connectors/camel-aws2-s3-kafka-connector/src/generated/resources/camel-aws2-s3-source.json | 615 --------------------------------------------------------------------------------- connectors/camel-aws2-s3-kafka-connector/src/main/assembly/package.xml | 56 -------- connectors/camel-aws2-s3-kafka-connector/src/main/docs/camel-aws2-s3-kafka-sink-connector.adoc | 149 -------------------- connectors/camel-aws2-s3-kafka-connector/src/main/docs/camel-aws2-s3-kafka-source-connector.adoc | 165 ---------------------- connectors/camel-aws2-s3-kafka-connector/src/main/docs/examples/CamelAws2s3SinkConnector.properties | 33 ----- connectors/camel-aws2-s3-kafka-connector/src/main/docs/examples/CamelAws2s3SourceConnector.properties | 33 ----- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/CamelAws2s3SinkConnector.java | 35 ----- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/CamelAws2s3SinkConnectorConfig.java | 326 ------------------------------------------- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/CamelAws2s3SinkTask.java | 39 ------ connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/CamelAws2s3SourceConnector.java | 35 ----- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/CamelAws2s3SourceConnectorConfig.java | 392 --------------------------------------------------- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/CamelAws2s3SourceTask.java | 39 ------ connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/aggregation/NewlineAggregationStrategy.java | 44 ------ connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/converters/S3ObjectConverter.java | 45 ------ connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/models/StorageHeader.java | 28 ---- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/models/StorageRecord.java | 30 ---- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/serializers/S3ObjectSerializer.java | 57 -------- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/transformers/JSONToRecordTransforms.java | 76 ---------- connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/transformers/RecordToJSONTransforms.java | 86 ------------ connectors/camel-aws2-s3-kafka-connector/src/main/java/org/apache/camel/kafkaconnector/aws2s3/transformers/S3ObjectTransforms.java | 55 -------- connectors/camel-aws2-s3-kafka-connector/src/main/resources/META-INF/LICENSE.txt | 203 --------------------------- connectors/camel-aws2-s3-kafka-connector/src/main/resources/META-INF/NOTICE.txt | 11 -- 26 files changed, 3227 deletions(-) commit 2267ac23a54a2585d3cd9680d0a60c2dc6983f1f Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 8 03:44:59 2021 +0000 [create-pull-request] automated change connectors/camel-aws2-s3-kafka-connector/src/generated/resources/camel-aws2-s3-sink.json | 4 ++-- connectors/camel-aws2-s3-kafka-connector/src/generated/resources/camel-aws2-s3-source.json | 4 ++-- connectors/camel-aws2-s3-kafka-connector/src/main/docs/camel-aws2-s3-kafka-sink-connector.adoc | 4 ++-- connectors/camel-aws2-s3-kafka-connector/src/main/docs/camel-aws2-s3-kafka-source-connector.adoc | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) ... I’m wondering if the processes that copy the adoc files under docs and the json files into the catalog should have a cleanup step to remove no-longer-current files. As noted above, I haven’t found how this copying is done. If my experiment succeeds, the adoc files won’t exist any more, being generated by Antora from the json files under connectors, so only the catalog would potentially need this cleaning step. FWIW I found these files looking for adoc and json files with converters, transforms, or aggregation strategies…. I think these aws2-s3 ones are the only ones using these features. David Jencks
