This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch feature/CAMEL-24720-catalog-sample-components in repository https://gitbox.apache.org/repos/asf/camel.git
commit 645cc4c35f5ff43e5a54e17caeace84bc9143fa8 Author: Claus Ibsen <[email protected]> AuthorDate: Thu Sep 17 18:57:57 2026 +0200 CAMEL-24720: camel-jbang - camel_catalog_sample returns samples for components, data formats and languages from their documentation The tool served the EIPs and the file entries only; an agent asking for kafka, csv or jq got No sample. Now a component, data format or language name answers from the catalog's documentation of the Camel version in use: the component page and its sub-pages (the examples using the component's endpoint first, smtp and imap on the mail page, jackson on the jackson2 and jackson3 pages, bindyCsv on the bindy page), the data format page, the language page. The answer keeps its shape (name, placement, samples, count) and adds kind; placement says from:/to: for a component (consumer-only and producer-only say their side), the marshal step for a data format, the expression of a step for a language. A name in several kinds (avro, file) returns the kinds to choose from unless only one has samples; the optional kind argument picks one. An exact catalog name beats an EIP alias (chunk is a component before it is a split alias), and the timer intent gave way to the timer component. Suggestions for an unknown name include the catalog's components, data formats and languages that have a page. Co-Authored-By: Claude Fable 5.1 <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../modules/ROOT/pages/camel-jbang-mcp.adoc | 23 +- .../dsl/jbang/core/commands/ai/AuthoringTools.java | 17 +- .../dsl/jbang/core/commands/ai/CatalogSamples.java | 554 ++++++++++++++++++--- .../jbang/core/commands/ai/CatalogSamplesTest.java | 165 +++++- .../jbang/core/commands/mcp/AuthoringTools.java | 15 +- 5 files changed, 675 insertions(+), 99 deletions(-) diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc b/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc index 6b272ccadbd8..e063a4c71898 100644 --- a/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc +++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc @@ -297,15 +297,20 @@ project `directory` as an argument, the runtime tools take the integration `name `AggregationStrategy` lists the built-in implementations. | `camel_catalog_sample` -| Validated YAML DSL samples of an EIP or file entry (`onException`, `aggregate`, `split`, `rest`, `beans`) taken - from the documentation examples, with where it goes: a top-level entry next to the route, or a step inside - it. The name can be kebab-case, a part of another EIP (`doCatch`, `when`, `onFallback` show the whole - construct), an alias of the EIP (`fan-out`, `rate-limit`, from the catalog's EIP models) or what to do - (`read file`, `call service`, `retry`, `batch`). The EIP samples come from the - documentation of the catalog in use, so they follow its Camel version; the file entries come from the user - manual examples that the build validates (`generate-doc-samples` in - xref:camel-yaml-dsl-validator-maven-plugin.adoc[]). `limit` is 2 by default, at most 5. - Use it before writing an EIP the first time, or after a `not defined in the schema` validation error. +| Validated YAML DSL samples of an EIP or file entry (`onException`, `aggregate`, `split`, `rest`, `beans`), a + component (`kafka`, `file`), a data format (`csv`) or a language (`jq`) taken from the documentation + examples, with where it goes: a top-level entry next to the route, a step inside it, an endpoint uri in + `from:` or `to:` (the component's consumer or producer side), a `marshal`/`unmarshal` step, or the + expression of a step. The name can be kebab-case, a part of another EIP (`doCatch`, `when`, `onFallback` + show the whole construct), an alias of the EIP (`fan-out`, `rate-limit`, from the catalog's EIP models) or + what to do (`read file`, `call service`, `retry`, `batch`); `kind` (`eip`, `component`, `dataformat`, + `language`) is only needed for a name that is in several kinds (`avro`, `file`), otherwise the answer names + the kinds to choose from. The samples come from the documentation of the catalog in use, so they follow its + Camel version: the EIP page, the component page and its sub-pages (the examples using the component's + endpoint first), the data format or language page. The file entries come from the user manual examples that + the build validates (`generate-doc-samples` in xref:camel-yaml-dsl-validator-maven-plugin.adoc[]). + `limit` is 2 by default, at most 5. Use it before writing one the first time, or after a + `not defined in the schema` validation error. | `camel_validate_source` | Validates Camel YAML DSL or `.properties` source without writing: the YAML DSL schema (a misspelled option diff --git a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java index f63629556da2..a84ae421bbb0 100644 --- a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java +++ b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java @@ -103,16 +103,23 @@ public final class AuthoringTools { })); registry.accept(tool("camel_catalog_sample", - "A validated YAML DSL sample of an EIP or file entry (onException, aggregate, split, rest, beans) from " - + "the docs, with where it goes (a top-level entry or a step). Use before " - + "writing an EIP the first time or after a 'not defined in the schema' error.") - .param("name", "string", "EIP or entry name, or what to do (read file, call service, retry, batch)", true) + "A validated YAML DSL sample of an EIP or file entry (onException, aggregate, split, rest, beans), a " + + "component (kafka, file), a data format (csv) or a language (jq) from the " + + "docs, with where it goes (a top-level entry, a step, an endpoint uri, a " + + "marshal step, an expression). Use before writing one the first time or " + + "after a 'not defined in the schema' error.") + .param("name", "string", + "EIP, component, data format or language name, or what to do (read file, call service, retry, batch)", + true) + .param("kind", "string", "eip, component, dataformat or language; needed only when a name is in several " + + "(avro, file)", + false) .param("limit", "integer", "Maximum samples to return (default 2, max 5)", false) .param("camelVersion", "string", VERSION_DESC, false) .core(true) .executor((ctx, args) -> { applyVersion(ctx, args); - return CatalogSamples.sample(ctx.catalog(), args.get("name"), + return CatalogSamples.sample(ctx.catalog(), args.get("kind"), args.get("name"), integer(args, "limit", CatalogSamples.DEFAULT_LIMIT)); })); diff --git a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamples.java b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamples.java index 71c4e4f20cf0..a8648bfb113c 100644 --- a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamples.java +++ b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamples.java @@ -20,27 +20,37 @@ import java.io.InputStream; import java.io.InputStreamReader; import java.nio.charset.StandardCharsets; import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Map; import java.util.Set; import java.util.TreeMap; +import java.util.concurrent.ConcurrentHashMap; import java.util.regex.Matcher; import java.util.regex.Pattern; +import java.util.stream.Collectors; import org.apache.camel.catalog.CamelCatalog; +import org.apache.camel.tooling.model.ComponentModel; +import org.apache.camel.tooling.model.DataFormatModel; import org.apache.camel.util.json.JsonArray; import org.apache.camel.util.json.JsonObject; import org.apache.camel.util.json.Jsoner; /** * Validated Camel YAML DSL samples per EIP (and per top-level entry such as onException, rest or routeConfiguration), - * taken from the YAML examples in the Camel documentation, with where each goes in a file. + * per component, data format and language, taken from the YAML examples in the Camel documentation, with where each + * goes in a file. * <p> - * The EIP samples are read from the catalog's documentation at lookup time, so they follow the Camel version in use. - * The shipped {@code eip-samples.json} is the fallback (the user manual pages the catalog does not bundle, and the EIP - * pages when the catalog has none): the build generates it from the documentation with the - * {@code camel-yaml-dsl-validator:generate-doc-samples} goal, which fails when an example does not validate. + * The samples are read from the catalog's documentation at lookup time, so they follow the Camel version in use: the + * EIP page, or the component page and its sub-pages (the consumer, producer or examples pages of a component), the data + * format page or the language page. The shipped {@code eip-samples.json} is the fallback for the EIPs and the file + * entries (the user manual pages the catalog does not bundle, and the EIP pages when the catalog has none): the build + * generates it from the documentation with the {@code camel-yaml-dsl-validator:generate-doc-samples} goal, which fails + * when an example does not validate. The component, data format and language examples have no shipped fallback: they + * come from the catalog in use only, which is the point of asking for a version. */ public final class CatalogSamples { @@ -48,6 +58,14 @@ public final class CatalogSamples { public static final int DEFAULT_LIMIT = 2; public static final int MAX_LIMIT = 5; + /** The kinds a sample can be asked for; eip covers the file entries (onException, rest, beans) too. */ + static final List<String> KINDS = List.of("eip", "component", "dataformat", "language"); + + private static final List<String> CATALOG_KINDS = List.of("component", "dataformat", "language"); + + /** The suffixes of the catalog pages of the four families; the other pages are sub-pages or manual pages. */ + private static final List<String> FAMILY_SUFFIXES = List.of("-component", "-dataformat", "-language", "-eip"); + /** The entries that go at the top of a YAML file, as list items next to the route, not inside it. */ static final Set<String> TOP_LEVEL = Set.of("beans", "dataFormats", "errorHandler", "from", "intercept", "interceptFrom", "interceptSendToEndpoint", "onCompletion", "onException", "rest", "restConfiguration", @@ -73,6 +91,7 @@ public final class CatalogSamples { * What a request is about, in the words a person or a model uses, to the EIP that does it. The names and aliases of * the EIPs themselves (fan-out, dedup, rate-limit) come from the catalog's EIP models, so only what is not an alias * is here: a task (read file, call service), a wording of the outcome (retry, batch) or a technology (json, cron). + * A protocol or a product (mqtt, s3) is what camel_catalog_find turns into a component, so it is not here. */ static final Map<String, String> INTENTS = Map.ofEntries( Map.entry("read file", "poll"), Map.entry("readfile", "poll"), Map.entry("load file", "poll"), @@ -87,7 +106,7 @@ public final class CatalogSamples { Map.entry("parallel", "multicast"), Map.entry("resilience", "circuitBreaker"), Map.entry("rest api", "rest"), Map.entry("http server", "rest"), Map.entry("endpoint", "rest"), Map.entry("convert", "convertBodyTo"), Map.entry("json", "marshal"), - Map.entry("timer", "from"), Map.entry("schedule", "from"), Map.entry("cron", "from")); + Map.entry("schedule", "from"), Map.entry("cron", "from")); private static volatile Map<String, List<Map<String, String>>> samples; @@ -125,7 +144,7 @@ public final class CatalogSamples { return answer; } - /** The names with samples, sorted. */ + /** The names with shipped samples (the EIPs and the file entries), sorted. */ public static List<String> names() { return new ArrayList<>(samples().keySet()); } @@ -134,57 +153,210 @@ public final class CatalogSamples { * Samples for the given EIP or entry name (camelCase, kebab-case or any case), with where it goes. */ public static JsonObject sample(String name, int limit) { - return sample(null, name, limit); + return sample(null, null, name, limit); + } + + /** + * Samples for the given name of any kind, with where it goes; from the catalog's documentation for the Camel + * version in use when it has the page, else the shipped set. + */ + public static JsonObject sample(CamelCatalog catalog, String name, int limit) { + return sample(catalog, null, name, limit); } private static final Pattern YAML_BLOCK = Pattern.compile("\\[source,yaml\\]\\s*\\n----\\n(.*?)\\n----", Pattern.DOTALL); - private static final Map<String, List<Map<String, String>>> DOC_CACHE = new java.util.concurrent.ConcurrentHashMap<>(); + private static final Map<String, Found> DOC_CACHE = new ConcurrentHashMap<>(); + + /** The validated examples of a page family, and for a component whether any of them uses its endpoint. */ + record Found(List<Map<String, String>> samples, boolean endpoint) { + static final Found NONE = new Found(List.of(), false); + } /** - * The validated YAML examples of the EIP page in the catalog, or an empty list when the catalog has no such page or - * none of its examples validate. Cached per name for the default catalog. + * The validated YAML examples of the documentation of the given kind and name in the catalog, or none when the + * catalog has no such page or none of its examples validate. For an EIP the page named after it + * (circuitBreaker-eip, pollEnrich-eip; a few are kebab-case, and the pattern pages have no suffix: + * dead-letter-channel, intercept); for a component its page and sub-pages, the examples using its endpoint first; + * for a data format or a language its page. Cached per kind and name for the default catalog. */ - static List<Map<String, String>> fromCatalog(CamelCatalog catalog, String key) { + static Found fromCatalog(CamelCatalog catalog, String kind, String key) { if (catalog == null) { - return List.of(); + return Found.NONE; } - // the catalog names the pages after the EIP (circuitBreaker-eip, pollEnrich-eip); a few are kebab-case, and the - // pattern pages have no suffix (dead-letter-channel, intercept) boolean cacheable = catalog.getCatalogVersion() != null && catalog.getLoadedVersion() == null; - if (cacheable && DOC_CACHE.containsKey(key)) { - return DOC_CACHE.get(key); + String cacheKey = kind + ":" + key; + if (cacheable) { + Found cached = DOC_CACHE.get(cacheKey); + if (cached != null) { + return cached; + } } - List<Map<String, String>> answer = new ArrayList<>(); + Found found; try { - String page = null; - String doc = null; - for (String candidate : List.of(key + "-eip", kebab(key) + "-eip", kebab(key), key)) { - doc = catalog.asciiDoc(candidate); - if (doc != null) { - page = candidate; + found = switch (kind) { + case "component" -> componentSamples(catalog, key); + case "dataformat" -> pageSamples(catalog, dataFormatPages(catalog, key), null); + case "language" -> pageSamples(catalog, List.of(key + "-language"), null); + default -> eipSamples(catalog, key); + }; + } catch (Exception e) { + // a page that cannot be read or validated: the shipped samples are the fallback + found = Found.NONE; + } + if (cacheable) { + DOC_CACHE.put(cacheKey, found); + } + return found; + } + + private static Found eipSamples(CamelCatalog catalog, String key) { + for (String candidate : List.of(key + "-eip", kebab(key) + "-eip", kebab(key), key)) { + if (catalog.asciiDoc(candidate) != null) { + return pageSamples(catalog, List.of(candidate), null); + } + } + return Found.NONE; + } + + private static Found componentSamples(CamelCatalog catalog, String name) { + String page = componentPage(catalog, name); + if (page == null) { + return Found.NONE; + } + List<String> pages = new ArrayList<>(); + pages.add(page + "-component"); + pages.addAll(subPages(catalog, page)); + return pageSamples(catalog, pages, schemePattern(schemesOf(catalog, name, page))); + } + + /** + * The validated route examples of the pages, in page order; with a scheme pattern the examples that use the + * endpoint come first. + */ + private static Found pageSamples(CamelCatalog catalog, List<String> pages, Pattern scheme) { + List<Map<String, String>> uses = new ArrayList<>(); + List<Map<String, String>> others = new ArrayList<>(); + for (String page : pages) { + String doc = catalog.asciiDoc(page); + if (doc == null) { + continue; + } + Matcher m = YAML_BLOCK.matcher(doc); + while (m.find()) { + String yaml = m.group(1).stripTrailing() + "\n"; + if (!yaml.stripLeading().startsWith("- ")) { + // a fragment (an option list, a manifest), not a route file + continue; + } + if (!SourceValidator.validateYamlSchema(yaml, catalog).isEmpty()) { + continue; + } + Map<String, String> sample + = Map.of("source", page + ".adoc (Camel " + catalog.getCatalogVersion() + ")", "yaml", yaml); + if (scheme != null && scheme.matcher(yaml).find()) { + uses.add(sample); + } else { + others.add(sample); + } + } + } + List<Map<String, String>> all = new ArrayList<>(uses); + all.addAll(others); + return new Found(List.copyOf(all), !uses.isEmpty()); + } + + /** + * The base name of the documentation page of a component: its own name, or the page of its artifact when the + * component shares a page with its siblings (smtp, imap and pop3 are on the mail page, https on the http page). + */ + static String componentPage(CamelCatalog catalog, String name) { + if (catalog.asciiDoc(name + "-component") != null) { + return name; + } + ComponentModel model = componentModel(catalog, name); + String artifactId = model != null ? model.getArtifactId() : null; + if (artifactId != null && artifactId.startsWith("camel-")) { + String page = artifactId.substring("camel-".length()); + if (catalog.asciiDoc(page + "-component") != null) { + return page; + } + } + return null; + } + + /** + * The sub-pages of a component page (aws2-s3-consumer-examples, salesforce-streaming, platform-http-vertx): the + * catalog pages named with the component's prefix that are not a page of one of the four families, nor of another + * component with a longer name (mina-sftp-authentication is not a page of mina). + */ + static List<String> subPages(CamelCatalog catalog, String page) { + List<String> out = new ArrayList<>(); + String prefix = page + "-"; + List<String> components = catalog.findComponentNames(); + for (String doc : catalog.findDocNames()) { + if (!doc.startsWith(prefix) || FAMILY_SUFFIXES.stream().anyMatch(doc::endsWith)) { + continue; + } + boolean other = false; + for (String c : components) { + if (c.length() > page.length() && c.startsWith(prefix) && (doc.equals(c) || doc.startsWith(c + "-"))) { + other = true; break; } } - if (doc != null) { - Matcher m = YAML_BLOCK.matcher(doc); - while (m.find()) { - String yaml = m.group(1).stripTrailing() + "\n"; - if (!yaml.stripLeading().startsWith("- ")) { - continue; - } - if (SourceValidator.validateYamlSchema(yaml, catalog).isEmpty()) { - answer.add(Map.of("source", page + ".adoc (Camel " + catalog.getCatalogVersion() + ")", "yaml", yaml)); - } + if (!other) { + out.add(doc); + } + } + Collections.sort(out); + return out; + } + + /** + * The schemes an example of the component would use in an endpoint uri: the component's own and its alternatives + * (smtp, smtps, imap...), or for a page that groups components the schemes of the components of its artifact. + */ + private static Set<String> schemesOf(CamelCatalog catalog, String name, String page) { + Set<String> schemes = new LinkedHashSet<>(); + ComponentModel model = componentModel(catalog, name); + if (model != null) { + addSchemes(schemes, model); + } else { + for (String c : catalog.findComponentNames()) { + ComponentModel m = catalog.componentModel(c); + if (m != null && ("camel-" + page).equals(m.getArtifactId())) { + addSchemes(schemes, m); } } - } catch (Exception e) { - // a page that cannot be read or validated: the shipped samples are the fallback } - List<Map<String, String>> result = List.copyOf(answer); - if (cacheable) { - DOC_CACHE.put(key, result); + if (schemes.isEmpty()) { + schemes.add(name); } - return result; + return schemes; + } + + private static void addSchemes(Set<String> schemes, ComponentModel model) { + if (model.getScheme() != null) { + schemes.add(model.getScheme()); + } + String alternatives = model.getAlternativeSchemes(); + if (alternatives != null) { + for (String s : alternatives.split(",")) { + if (!s.isBlank()) { + schemes.add(s.trim()); + } + } + } + } + + /** Matches an endpoint uri of one of the schemes: {@code uri: kafka:...}, {@code uri: "kafka:..."}. */ + private static Pattern schemePattern(Set<String> schemes) { + String any = schemes.stream().map(Pattern::quote).collect(Collectors.joining("|")); + return Pattern.compile("(uri:\\s*[\"']?|[\"'])(" + any + "):"); + } + + private static ComponentModel componentModel(CamelCatalog catalog, String name) { + return catalog.findComponentNames().contains(name) ? catalog.componentModel(name) : null; } static String kebab(String camel) { @@ -200,35 +372,99 @@ public final class CatalogSamples { } /** - * Samples for the given EIP or entry name (camelCase, kebab-case, any case, or what to do), with where it goes; - * from the catalog's documentation for the Camel version in use when it has the page, else the shipped set. + * Samples for the given name, with where it goes: an EIP or entry name (camelCase, kebab-case, any case, or what to + * do), a component, a data format or a language. Without a kind the EIPs and file entries are looked up first, then + * the catalog's components, data formats and languages, then the aliases of the EIPs; a name that is in several + * kinds with samples returns the kinds to choose from. The samples come from the catalog's documentation for the + * Camel version in use when it has the page, else (EIPs and file entries) the shipped set. + * + * @param catalog the catalog of the Camel version to answer for, or null for the shipped samples only + * @param kind eip, component, dataformat or language, or null to resolve the name across the kinds + * @param name the name, or what to do + * @param limit the maximum number of samples, at most {@link #MAX_LIMIT} */ - public static JsonObject sample(CamelCatalog catalog, String name, int limit) { + public static JsonObject sample(CamelCatalog catalog, String kind, String name, int limit) { JsonObject answer = new JsonObject(); String given = name != null ? name.trim() : ""; if (given.isEmpty()) { - answer.put("error", "name is required, e.g. onException, aggregate, circuitBreaker, split, rest"); + answer.put("error", "name is required, e.g. onException, aggregate, circuitBreaker, split, rest, kafka, csv, jq"); + return answer; + } + String wanted = normalizeKind(kind); + if (wanted != null && !KINDS.contains(wanted)) { + answer.put("error", "kind must be one of " + String.join(", ", KINDS) + ", got '" + kind + "'"); return answer; } int max = Math.max(1, Math.min(MAX_LIMIT, limit <= 0 ? DEFAULT_LIMIT : limit)); - String key = resolve(catalog, given); - if (key == null) { - answer.put("error", "No sample for '" + given + "'"); - answer.put("suggestions", new JsonArray(suggest(normalize(given)))); - // the name may still be an EIP (dedup is idempotentConsumer) whose page has no YAML example - List<String> eips = catalog != null ? catalog.suggestEipNames(given, 1) : List.of(); - if (!eips.isEmpty()) { - answer.put("eip", eips.get(0)); - answer.put("hint", "'" + given + "' is the " + eips.get(0) + " EIP, which has no YAML sample; " - + "camel_catalog_doc gives its options"); - } else { - answer.put("hint", "camel_catalog_doc gives the options of an EIP; the sample tool covers the EIPs and file " - + "entries documented with YAML examples"); + boolean eips = wanted == null || wanted.equals("eip"); + + // an EIP or a file entry by its name, a part of it, or what to do + if (eips) { + String key = resolveExact(given); + if (key != null) { + return eipAnswer(catalog, given, key, max, wanted == null ? alsoKinds(catalog, given) : List.of()); } - return answer; } + // a component, data format or language by its name + if (catalog != null && !"eip".equals(wanted)) { + Map<String, String> kinds = new TreeMap<>(); + for (String k : wanted != null ? List.of(wanted) : CATALOG_KINDS) { + String canonical = canonical(catalog, k, given); + if (canonical != null) { + kinds.put(k, canonical); + } + } + if (kinds.size() > 1) { + // the kinds with samples decide (file is a component with examples and a language without) + Map<String, String> withSamples = new TreeMap<>(); + kinds.forEach((k, canonical) -> { + if (!fromCatalog(catalog, k, canonical).samples().isEmpty()) { + withSamples.put(k, canonical); + } + }); + if (!withSamples.isEmpty()) { + kinds = withSamples; + } + } + if (kinds.size() > 1) { + answer.put("name", given); + answer.put("kinds", new JsonArray(new ArrayList<>(kinds.keySet()))); + answer.put("hint", "'" + given + "' is a " + String.join(" and a ", kinds.keySet()) + + "; ask again with kind"); + return answer; + } + if (kinds.size() == 1) { + Map.Entry<String, String> only = kinds.entrySet().iterator().next(); + return catalogAnswer(catalog, only.getKey(), only.getValue(), max); + } + } + // an alias of an EIP model (fan-out, dedup, rate-limit) or a word of its title + if (eips) { + String key = resolveAlias(catalog, given); + if (key != null) { + return eipAnswer(catalog, given, key, max, List.of()); + } + } + answer.put("error", "No sample for '" + given + "'" + (wanted != null ? " as a " + wanted : "")); + answer.put("suggestions", new JsonArray(suggestions(catalog, wanted, given))); + // the name may still be an EIP (dedup is idempotentConsumer) whose page has no YAML example + List<String> eipNames = catalog != null && eips ? catalog.suggestEipNames(given, 1) : List.of(); + if (!eipNames.isEmpty()) { + answer.put("eip", eipNames.get(0)); + answer.put("hint", "'" + given + "' is the " + eipNames.get(0) + " EIP, which has no YAML sample; " + + "camel_catalog_doc gives its options"); + } else { + answer.put("hint", "camel_catalog_doc gives the options of an EIP, component, data format or language; " + + "the sample tool covers what the documentation shows with YAML examples"); + } + return answer; + } + + private static JsonObject eipAnswer(CamelCatalog catalog, String given, String key, int max, List<String> also) { + JsonObject answer = new JsonObject(); String partOf = PART_OF.get(normalize(given)); answer.put("name", key); + answer.put("kind", "eip"); if (partOf != null) { answer.put("partOf", partOf); answer.put("note", given + " is a part of " + partOf + "; the sample shows it in place"); @@ -236,10 +472,37 @@ public final class CatalogSamples { answer.put("note", "'" + given + "' is done with the " + key + " EIP"); } answer.put("placement", placement(key)); - List<Map<String, String>> list = fromCatalog(catalog, key); + List<Map<String, String>> list = fromCatalog(catalog, "eip", key).samples(); if (list.isEmpty()) { list = samples().getOrDefault(key, List.of()); } + putSamples(answer, list, max); + if (!also.isEmpty()) { + answer.put("also", new JsonArray(also)); + answer.put("hint", "'" + given + "' is also a " + String.join(" and a ", also) + + "; ask with kind for that sample"); + } + return answer; + } + + private static JsonObject catalogAnswer(CamelCatalog catalog, String kind, String name, int max) { + JsonObject answer = new JsonObject(); + answer.put("name", name); + answer.put("kind", kind); + answer.put("placement", placement(catalog, kind, name)); + Found found = fromCatalog(catalog, kind, name); + putSamples(answer, found.samples(), max); + if (found.samples().isEmpty()) { + answer.put("hint", "the documentation of " + name + " has no YAML route example; camel_catalog_doc gives " + + "its options" + ("component".equals(kind) ? " and the uri syntax" : "")); + } else if ("component".equals(kind) && !found.endpoint()) { + answer.put("note", "none of the examples of the documentation uses a " + name + " endpoint; they show the " + + "component in another way (a properties function, a policy, a converter)"); + } + return answer; + } + + private static void putSamples(JsonObject answer, List<Map<String, String>> list, int max) { JsonArray arr = new JsonArray(); for (Map<String, String> s : list.subList(0, Math.min(max, list.size()))) { JsonObject jo = new JsonObject(); @@ -249,7 +512,91 @@ public final class CatalogSamples { } answer.put("samples", arr); answer.put("count", list.size()); - return answer; + } + + /** The catalog kinds, other than the EIPs, the name has samples in. */ + private static List<String> alsoKinds(CamelCatalog catalog, String given) { + List<String> also = new ArrayList<>(); + if (catalog != null) { + for (String k : CATALOG_KINDS) { + String canonical = canonical(catalog, k, given); + if (canonical != null && !fromCatalog(catalog, k, canonical).samples().isEmpty()) { + also.add(k); + } + } + } + return also; + } + + /** + * The catalog name of the given kind the name stands for, when it has a documentation page: the name itself, or the + * name in another case; for a component also a page that groups components (mail). + */ + static String canonical(CamelCatalog catalog, String kind, String given) { + List<String> names = switch (kind) { + case "component" -> catalog.findComponentNames(); + case "dataformat" -> catalog.findDataFormatNames(); + case "language" -> catalog.findLanguageNames(); + default -> List.of(); + }; + String name = names.contains(given) ? given : null; + if (name == null) { + for (String n : names) { + if (n.equalsIgnoreCase(given)) { + name = n; + break; + } + } + } + if ("component".equals(kind)) { + if (name != null) { + return componentPage(catalog, name) != null ? name : null; + } + String page = given.toLowerCase(Locale.ROOT); + return catalog.asciiDoc(page + "-component") != null ? page : null; + } + if ("dataformat".equals(kind)) { + return name != null && !dataFormatPages(catalog, name).isEmpty() ? name : null; + } + return name != null && catalog.asciiDoc(name + "-" + kind) != null ? name : null; + } + + /** + * The documentation pages of a data format: its own, the page of its artifact (bindyCsv, bindyFixed and bindyKvp + * are on the bindy page), or the pages of its variants by major version (jackson2 and jackson3 for jackson). + */ + static List<String> dataFormatPages(CamelCatalog catalog, String name) { + if (catalog.asciiDoc(name + "-dataformat") != null) { + return List.of(name + "-dataformat"); + } + DataFormatModel model = catalog.findDataFormatNames().contains(name) ? catalog.dataFormatModel(name) : null; + String artifactId = model != null ? model.getArtifactId() : null; + if (artifactId != null && artifactId.startsWith("camel-")) { + String page = artifactId.substring("camel-".length()) + "-dataformat"; + if (catalog.asciiDoc(page) != null) { + return List.of(page); + } + } + List<String> variants = new ArrayList<>(); + Pattern variant = Pattern.compile(Pattern.quote(name) + "\\d+-dataformat"); + for (String doc : catalog.findDocNames()) { + if (variant.matcher(doc).matches()) { + variants.add(doc); + } + } + Collections.sort(variants); + return variants; + } + + static String normalizeKind(String kind) { + if (kind == null || kind.isBlank()) { + return null; + } + String k = kind.trim().toLowerCase(Locale.ROOT).replace("-", "").replace(" ", "").replace("_", ""); + if (k.endsWith("s")) { + k = k.substring(0, k.length() - 1); + } + return k; } static String placement(String key) { @@ -259,16 +606,32 @@ public final class CatalogSamples { return "a step inside the steps of a route (or of another EIP)"; } - /** Resolves a user given name to a sample key: exact, case-insensitive, kebab-case, or a part of another EIP. */ - static String resolve(String given) { - return resolve(null, given); + static String placement(CamelCatalog catalog, String kind, String name) { + switch (kind) { + case "component": { + ComponentModel model = componentModel(catalog, name); + if (model != null && model.isConsumerOnly()) { + return "an endpoint uri in from: only (a consumer; it cannot be used in to:)"; + } + if (model != null && model.isProducerOnly()) { + return "an endpoint uri in to: only (a producer; it cannot be used in from:)"; + } + return "an endpoint uri in from: (consumer) or to: (producer)"; + } + case "dataformat": + return "a marshal or unmarshal step inside the steps of a route, with the data format and its options" + + " under it"; + case "language": + return "the expression of a step (setBody, setHeader, filter, when, split, ...), under expression:"; + default: + return placement(name); + } } /** - * Resolves a user given name to a sample key: exact, case-insensitive, kebab-case, a part of another EIP, an - * intent, or with a catalog an alias of an EIP model (fan-out, dedup, rate-limit) or a word of its title. + * Resolves a user given name to a shipped sample key: exact, an intent, a part of another EIP, or case-insensitive. */ - static String resolve(CamelCatalog catalog, String given) { + static String resolveExact(String given) { String n = normalize(given); if (samples().containsKey(n)) { return n; @@ -286,16 +649,24 @@ public final class CatalogSamples { return k; } } - if (catalog != null) { - // the aliases of the EIP models, then the EIP an aliased part belongs to (fallback is a part of circuitBreaker) - for (String eip : catalog.suggestEipNames(given, 3)) { - if (samples().containsKey(eip)) { - return eip; - } - String whole = PART_OF.get(eip); - if (whole != null && samples().containsKey(whole)) { - return whole; - } + return null; + } + + /** + * Resolves a user given name through the catalog's EIP models: an alias (fan-out, dedup, rate-limit) or a word of + * the title, then the EIP an aliased part belongs to (fallback is a part of circuitBreaker). + */ + static String resolveAlias(CamelCatalog catalog, String given) { + if (catalog == null) { + return null; + } + for (String eip : catalog.suggestEipNames(given, 3)) { + if (samples().containsKey(eip)) { + return eip; + } + String whole = PART_OF.get(eip); + if (whole != null && samples().containsKey(whole)) { + return whole; } } return null; @@ -319,6 +690,37 @@ public final class CatalogSamples { return n; } + /** + * Names close to the given one: the shipped keys, and with a catalog its components, data formats and languages. + */ + private static List<String> suggestions(CamelCatalog catalog, String wanted, String given) { + List<String> out = wanted == null || wanted.equals("eip") ? suggest(normalize(given)) : new ArrayList<>(); + if (catalog != null) { + for (String kind : CATALOG_KINDS) { + if (wanted == null || wanted.equals(kind)) { + suggestCatalog(out, catalog, kind, given, kind.equals("component") ? 3 : 2); + } + } + } + return out; + } + + /** Adds the catalog names of the kind close to the given one, only those with a page to come back for. */ + private static void suggestCatalog(List<String> out, CamelCatalog catalog, String kind, String given, int max) { + List<String> found = switch (kind) { + case "component" -> catalog.suggestComponentNames(given, max * 3); + case "dataformat" -> catalog.suggestDataFormatNames(given, max * 3); + default -> catalog.suggestLanguageNames(given, max * 3); + }; + int n = 0; + for (String s : found) { + if (n < max && canonical(catalog, kind, s) != null) { + out.add(s + " (" + kind + ")"); + n++; + } + } + } + static List<String> suggest(String n) { List<String> out = new ArrayList<>(); String lower = n.toLowerCase(Locale.ROOT); diff --git a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamplesTest.java b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamplesTest.java index 6e4d0f503e02..8899f5644d75 100644 --- a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamplesTest.java +++ b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogSamplesTest.java @@ -17,7 +17,11 @@ package org.apache.camel.dsl.jbang.core.commands.ai; import java.util.List; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.apache.camel.catalog.CamelCatalog; +import org.apache.camel.catalog.DefaultCamelCatalog; import org.apache.camel.util.json.JsonArray; import org.apache.camel.util.json.JsonObject; import org.junit.jupiter.api.Test; @@ -61,12 +65,14 @@ public class CatalogSamplesTest { @Test void eipAliasesResolveThroughTheCatalog() { // the aliases of the EIP models, so the intent map need not repeat them - org.apache.camel.catalog.CamelCatalog catalog = new org.apache.camel.catalog.DefaultCamelCatalog(); + CamelCatalog catalog = new DefaultCamelCatalog(); JsonObject o = CatalogSamples.sample(catalog, "fan-out", 1); assertThat(o.getString("name")).isEqualTo("multicast"); assertThat(o.getString("note")).contains("multicast EIP"); assertThat(CatalogSamples.sample(catalog, "broadcast", 1).getString("name")).isEqualTo("multicast"); - assertThat(CatalogSamples.sample(catalog, "chunk", 1).getString("name")).isEqualTo("split"); + // an exact component name (chunk is the Chunk templating component) beats an alias of an EIP, unless asked + assertThat(CatalogSamples.sample(catalog, "chunk", 1).getString("kind")).isEqualTo("component"); + assertThat(CatalogSamples.sample(catalog, "eip", "chunk", 1).getString("name")).isEqualTo("split"); // an alias of an EIP whose page has no YAML example still names the EIP JsonObject dedup = CatalogSamples.sample(catalog, "dedup", 1); assertThat(dedup.getString("error")).contains("dedup"); @@ -105,7 +111,7 @@ public class CatalogSamplesTest { @Test void eipSamplesComeFromTheCatalogDocsAndTheRestFromTheShippedSet() { - org.apache.camel.catalog.CamelCatalog catalog = new org.apache.camel.catalog.DefaultCamelCatalog(); + CamelCatalog catalog = new DefaultCamelCatalog(); JsonObject o = CatalogSamples.sample(catalog, "poll", 2); JsonArray samples = (JsonArray) o.get("samples"); assertThat(samples).isNotEmpty(); @@ -158,10 +164,161 @@ public class CatalogSamplesTest { @Test void patternPagesWithoutTheEipSuffixAreReadFromTheCatalog() { - org.apache.camel.catalog.CamelCatalog catalog = new org.apache.camel.catalog.DefaultCamelCatalog(); + CamelCatalog catalog = new DefaultCamelCatalog(); JsonArray samples = (JsonArray) CatalogSamples.sample(catalog, "deadLetterChannel", 1).get("samples"); assertThat(((JsonObject) samples.get(0)).getString("source")).startsWith("dead-letter-channel.adoc (Camel "); samples = (JsonArray) CatalogSamples.sample(catalog, "keyValueRepository", 1).get("samples"); assertThat(((JsonObject) samples.get(0)).getString("source")).startsWith("keyValueRepository.adoc (Camel "); } + + // CAMEL-24720: components, data formats and languages from their documentation + + private static String yaml(JsonObject answer, int i) { + return ((JsonObject) ((JsonArray) answer.get("samples")).get(i)).getString("yaml"); + } + + private static String source(JsonObject answer, int i) { + return ((JsonObject) ((JsonArray) answer.get("samples")).get(i)).getString("source"); + } + + @Test + void componentSamplesComeFromTheComponentPageWithItsEndpointFirst() { + CamelCatalog catalog = new DefaultCamelCatalog(); + JsonObject o = CatalogSamples.sample(catalog, "kafka", 2); + assertThat(o.getString("name")).isEqualTo("kafka"); + assertThat(o.getString("kind")).isEqualTo("component"); + assertThat(o.getString("placement")).contains("from:").contains("to:"); + assertThat((JsonArray) o.get("samples")).hasSize(2); + // the first example of the kafka page is an idempotent consumer with a SQL repository: a kafka endpoint first + assertThat(yaml(o, 0)).contains("kafka:"); + assertThat(source(o, 0)).startsWith("kafka-component.adoc (Camel "); + assertThat(o.get("note")).isNull(); + assertThat((Integer) o.get("count")).isGreaterThan(5); + // any case, and the kind when given + assertThat(CatalogSamples.sample(catalog, "component", "Kafka", 1).getString("name")).isEqualTo("kafka"); + // without a catalog there is no component page to read + assertThat(CatalogSamples.sample("kafka", 1).getString("error")).contains("kafka"); + } + + @Test + void theSubPagesOfAComponentBelongToIt() { + CamelCatalog catalog = new DefaultCamelCatalog(); + assertThat(CatalogSamples.subPages(catalog, "aws2-s3")) + .contains("aws2-s3-consumer-examples", "aws2-s3-producer-operations", "aws2-s3-streaming"); + // mina-sftp is a component of its own: its pages are not mina's + assertThat(CatalogSamples.subPages(catalog, "mina")).isEmpty(); + assertThat(CatalogSamples.subPages(catalog, "mina-sftp")).contains("mina-sftp-authentication"); + // the examples of the sub-pages count, after the ones of the main page + Matcher m = Pattern.compile("\\[source,yaml\\]").matcher(catalog.asciiDoc("aws2-s3-component")); + int onMainPage = 0; + while (m.find()) { + onMainPage++; + } + JsonObject o = CatalogSamples.sample(catalog, "aws2-s3", 5); + assertThat((Integer) o.get("count")).isGreaterThan(onMainPage); + assertThat(source(o, 0)).startsWith("aws2-s3-component.adoc"); + } + + @Test + void aConsumerOrProducerOnlyComponentSaysItsSide() { + CamelCatalog catalog = new DefaultCamelCatalog(); + JsonObject timer = CatalogSamples.sample(catalog, "timer", 1); + assertThat(timer.getString("placement")).contains("from: only"); + assertThat(yaml(timer, 0)).contains("timer:"); + JsonObject log = CatalogSamples.sample(catalog, "component", "log", 1); + assertThat(log.getString("kind")).isEqualTo("component"); + assertThat(log.getString("placement")).contains("to: only"); + // without a kind the log EIP answers first, and says the component is there too + JsonObject eip = CatalogSamples.sample(catalog, "log", 1); + assertThat(eip.getString("kind")).isEqualTo("eip"); + assertThat((JsonArray) eip.get("also")).containsExactly("component"); + assertThat(eip.getString("hint")).contains("kind"); + // an EIP that is no component says nothing + assertThat(CatalogSamples.sample(catalog, "split", 1).get("also")).isNull(); + } + + @Test + void dataFormatAndLanguageSamplesShowTheStepAndTheExpression() { + CamelCatalog catalog = new DefaultCamelCatalog(); + JsonObject csv = CatalogSamples.sample(catalog, "csv", 1); + assertThat(csv.getString("kind")).isEqualTo("dataformat"); + assertThat(csv.getString("placement")).contains("marshal"); + assertThat(yaml(csv, 0)).contains("csv:").containsAnyOf("marshal:", "unmarshal:"); + assertThat(source(csv, 0)).startsWith("csv-dataformat.adoc (Camel "); + JsonObject jq = CatalogSamples.sample(catalog, "jq", 1); + assertThat(jq.getString("kind")).isEqualTo("language"); + assertThat(jq.getString("placement")).contains("expression"); + assertThat(yaml(jq, 0)).contains("jq:"); + assertThat(source(jq, 0)).startsWith("jq-language.adoc (Camel "); + } + + @Test + void aNameInSeveralKindsReturnsTheChoiceUnlessOnlyOneHasSamples() { + CamelCatalog catalog = new DefaultCamelCatalog(); + JsonObject avro = CatalogSamples.sample(catalog, "avro", 1); + assertThat(avro.get("samples")).isNull(); + assertThat((JsonArray) avro.get("kinds")).containsExactly("component", "dataformat"); + assertThat(avro.getString("hint")).contains("kind"); + assertThat(CatalogSamples.sample(catalog, "dataformat", "avro", 1).getString("kind")).isEqualTo("dataformat"); + assertThat(CatalogSamples.sample(catalog, "Data-Formats", "avro", 1).getString("kind")).isEqualTo("dataformat"); + assertThat(yaml(CatalogSamples.sample(catalog, "component", "avro", 1), 0)).contains("avro:"); + // file is a component and a language, and only the component page has examples + JsonObject file = CatalogSamples.sample(catalog, "file", 1); + assertThat(file.getString("kind")).isEqualTo("component"); + assertThat(yaml(file, 0)).contains("file:"); + assertThat(CatalogSamples.sample(catalog, "steps", "file", 1).getString("error")).contains("kind must be one of"); + } + + @Test + void componentsSharingAPageAndPagesWithoutAnEndpointExample() { + CamelCatalog catalog = new DefaultCamelCatalog(); + // smtp, imap and pop3 are documented on the mail page + JsonObject smtp = CatalogSamples.sample(catalog, "smtp", 1); + assertThat(smtp.getString("kind")).isEqualTo("component"); + assertThat(source(smtp, 0)).startsWith("mail-component.adoc (Camel "); + assertThat(yaml(smtp, 0)).containsAnyOf("smtp:", "imap:", "pop3:"); + assertThat(smtp.get("note")).isNull(); + JsonObject mail = CatalogSamples.sample(catalog, "mail", 1); + assertThat(mail.getString("name")).isEqualTo("mail"); + assertThat(mail.get("note")).isNull(); + // mapstruct is used through convertBodyTo, no mapstruct: endpoint in its examples + JsonObject mapstruct = CatalogSamples.sample(catalog, "mapstruct", 1); + assertThat((Integer) mapstruct.get("count")).isPositive(); + assertThat(mapstruct.getString("note")).contains("none of the examples"); + // knative shows Kubernetes manifests, no route example + JsonObject knative = CatalogSamples.sample(catalog, "knative", 1); + assertThat((Integer) knative.get("count")).isZero(); + assertThat(knative.getString("hint")).contains("no YAML route example"); + } + + @Test + void anUnknownNameGetsTheCatalogSuggestionsToo() { + CamelCatalog catalog = new DefaultCamelCatalog(); + JsonObject o = CatalogSamples.sample(catalog, "mqtt", 1); + assertThat(o.getString("error")).contains("mqtt"); + assertThat(String.valueOf(o.get("suggestions"))).contains("(component)"); + // a suggestion is a name with a page to come back for, and no EIP is guessed for another kind + JsonObject typo = CatalogSamples.sample(catalog, "dataformat", "univocity", 1); + assertThat(typo.getString("error")).contains("as a dataformat"); + assertThat(String.valueOf(typo.get("suggestions"))).contains("univocityCsv (dataformat)"); + assertThat(typo.get("eip")).isNull(); + } + + @Test + void aDataFormatDocumentedUnderAnotherPageNameStillAnswers() { + CamelCatalog catalog = new DefaultCamelCatalog(); + // jackson is on the jackson2 and jackson3 pages, one example each + assertThat(CatalogSamples.dataFormatPages(catalog, "jackson")) + .containsExactly("jackson2-dataformat", "jackson3-dataformat"); + JsonObject jackson = CatalogSamples.sample(catalog, "jackson", 5); + assertThat(jackson.getString("kind")).isEqualTo("dataformat"); + assertThat(source(jackson, 0)).startsWith("jackson2-dataformat.adoc"); + assertThat((Integer) jackson.get("count")).isEqualTo(2); + // bindyCsv is on the bindy page, which has no YAML example + assertThat(CatalogSamples.dataFormatPages(catalog, "bindyCsv")).containsExactly("bindy-dataformat"); + JsonObject bindy = CatalogSamples.sample(catalog, "bindyCsv", 1); + assertThat(bindy.getString("kind")).isEqualTo("dataformat"); + assertThat((Integer) bindy.get("count")).isZero(); + assertThat(bindy.getString("hint")).contains("no YAML route example"); + } } diff --git a/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringTools.java b/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringTools.java index 64ba31dfa8b5..44e21e2fe274 100644 --- a/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringTools.java +++ b/dsl/camel-jbang/camel-jbang-mcp/src/main/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringTools.java @@ -93,15 +93,20 @@ public class AuthoringTools { } @Tool(annotations = @Tool.Annotations(readOnlyHint = true, destructiveHint = false, openWorldHint = false), - description = "A validated YAML DSL sample of an EIP or file entry (onException, aggregate, split, rest, beans) " - + "from the docs, with where it goes (a top-level entry or a step). Use before writing an EIP the " - + "first time or after a 'not defined in the schema' error.") + description = "A validated YAML DSL sample of an EIP or file entry (onException, aggregate, split, rest, beans), " + + "a component (kafka, file), a data format (csv) or a language (jq) from the docs, with where " + + "it goes (a top-level entry, a step, an endpoint uri, a marshal step, an expression). Use " + + "before writing one the first time or after a 'not defined in the schema' error.") public JsonObject camel_catalog_sample( - @ToolArg(description = "EIP or entry name, or what to do (read file, call service, retry, batch)", + @ToolArg(description = "EIP, component, data format or language name, or what to do (read file, call " + + "service, retry, batch)", required = true) String name, + @ToolArg(description = "eip, component, dataformat or language; needed only when a name is in several " + + "(avro, file)", + required = false) String kind, @ToolArg(description = "Maximum samples to return (default 2, max 5)", required = false) Integer limit, @ToolArg(description = VERSION_DESC, required = false) String camelVersion) { - return call("camel_catalog_sample", args("name", name, "limit", limit, "camelVersion", camelVersion)); + return call("camel_catalog_sample", args("name", name, "kind", kind, "limit", limit, "camelVersion", camelVersion)); } @Tool(annotations = @Tool.Annotations(readOnlyHint = true, destructiveHint = false, openWorldHint = false),
