This is an automated email from the ASF dual-hosted git repository. luigidemasi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/camel.git
commit 1df7bd315b5a0b7c9d6d7e85d969e695e1291cfc Author: Luigi De Masi <[email protected]> AuthorDate: Wed Oct 14 22:26:55 2026 +0200 CAMEL-25669: Add semantic media input and audit input types Normalize explicit image input while preserving replayable bodies and expert input contracts. Validate shared state once per expert group before any inference, and retain input type in private audit records. Co-authored-by: OpenAI Codex <[email protected]> Signed-off-by: Luigi De Masi <[email protected]> --- .../camel/catalog/docs/semantic-language.adoc | 43 +++++- .../camel/catalog/docs/typesafe-ai-component.adoc | 8 ++ .../apache/camel/catalog/docs/wolf-defender.adoc | 4 +- .../src/main/docs/semantic-language.adoc | 43 +++++- .../camel/language/semantic/SemanticLanguage.java | 14 +- .../org/apache/camel/semantic/SemanticAdapter.java | 9 ++ .../apache/camel/semantic/SemanticAuditRecord.java | 10 +- .../camel/semantic/SemanticCapabilities.java | 17 ++- .../org/apache/camel/semantic/SemanticExpert.java | 3 +- .../org/apache/camel/semantic/SemanticInput.java | 152 +++++++++++++++++++++ .../camel/semantic/SemanticMetadataConsole.java | 3 + .../apache/camel/semantic/SemanticOperation.java | 3 + .../semantic/internal/SemanticAuditExtensions.java | 5 +- .../internal/SemanticAuditInputSnapshot.java | 58 ++++++++ .../semantic/internal/SemanticAuditService.java | 12 +- .../camel/semantic/SemanticAuditInputTest.java | 26 ++++ .../camel/semantic/SemanticAuditReviewTest.java | 36 +++++ .../apache/camel/semantic/SemanticInputTest.java | 122 +++++++++++++++++ .../src/main/docs/typesafe-ai-component.adoc | 8 ++ .../camel/component/typesafeai/TypeSafeAiJson.java | 15 +- .../typesafeai/TypeSafeAiSemanticAdapter.java | 10 ++ .../typesafeai/TypeSafeAiSemanticAdapterTest.java | 13 ++ .../src/main/docs/wolf-defender.adoc | 4 +- .../WolfDefenderSemanticAdapterTest.java | 5 +- design/semantic-audit.adoc | 12 ++ .../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 22 +++ 26 files changed, 631 insertions(+), 26 deletions(-) diff --git a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/semantic-language.adoc b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/semantic-language.adoc index d977fad977ba..1cb7caea48c1 100644 --- a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/semantic-language.adoc +++ b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/semantic-language.adoc @@ -169,6 +169,46 @@ when no labels apply. Use collections in predicates, for example The category and the meaning of a Boolean verdict must come from the selected expert's contract. A negative detection does not establish general safety or grant authorization. +=== Text and image input + +Experts declare supported modalities in their operation contract. `IMAGE` allows explicit image +parts; it does not imply that every expert accepts images. TypeSafe AI accepts text and its structured state, and Wolf Defender accepts text. Unsupported input fails +before inference, including before any expert in a mixed batch runs. + +For `$\{body}`, an image `Content-Type` such as `image/png` identifies a `byte[]`, `InputStream`, +`StreamCache` or base64 `String` body as an image. Streams are replaced by replayable Camel +stream caches and reset after reading; their bytes remain available downstream. A self-describing +`data:image/...;base64,...` string is also image input. Other strings remain text. +Content-Type handling applies only to the exact `$\{body}` selector after placeholder resolution. +Other expressions, including `$\{bodyAs(byte[])}`, use the generic selected-state path: raw binary +values are rejected. Select `$\{body}` for image bytes, or select an explicit media envelope or +self-describing image data URL. An image header does not reclassify separately selected headers or variables. +With `$\{body}`, `audio/*` and `video/*` content types are rejected, including for a `String` body; +no expert currently supports these modalities. Select an explicitly textual field when evaluating +text that accompanies such a body. + +An explicit JSON-compatible envelope supports ordered mixed input: + +[source,json] +---- +{"$semanticInput":[ + {"type":"text","text":"Inspect this package"}, + {"type":"image","data":"data:image/png;base64,..."} +]} +---- + +Replace the abbreviated image data with actual base64. `SemanticInput.ofParts`, `text` and +`image` provide Java helpers. The reserved `$semanticInput` key distinguishes media from an +ordinary application map. Format-specific validation belongs to the expert. Camel does not +fetch URLs, read image paths from route messages, or invoke transcription/OCR services. + +Audit records expose a bounded `inputType` extension independently of payload capture: +`text`, `image`, `text-and-images`, `structured`, `unsupported` or `unknown` when classification exceeds its bound. This classification remains +available when input capture is disabled. The extension is separate from monitoring events. +Image capture remains opt-in and subject to the existing redactor and size budget. When an +image exceeds the budget, a bounded descriptor may retain its MIME type and +`dataOmitted=snapshot_limit` instead of the image bytes. Such input cannot be replayed. + === Structured input and specialised experts A specialised moderation expert can accept a map containing a user prompt, an assistant response, @@ -874,7 +914,8 @@ Provider discovery descriptors are not used. The language checks the static operation contract and then calls `validate` for operation-specific parameter combinations and bean configuration before traffic. `validateInput` checks detailed -message requirements before evaluation. All input checks in a batch run before any expert is invoked. +message requirements before evaluation. `validateState` checks shared input once per expert group; +use it for expensive checks such as image encoding. Its default implementation does nothing. All input checks in a batch run before any expert is invoked. `evaluate` returns a typed `SemanticResult` after applying the expert's policy. A probability-only Boolean result is invalid: the expert must supply its Boolean verdict. Providers should also validate when their SPI is called directly. `evaluate` must be thread-safe, bound time and resource consumption, honor interruption and propagate diff --git a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/typesafe-ai-component.adoc b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/typesafe-ai-component.adoc index b367bf81f601..87d2621f0118 100644 --- a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/typesafe-ai-component.adoc +++ b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/typesafe-ai-component.adoc @@ -306,3 +306,11 @@ Only submit data permitted for the configured service. Test model decisions and against your application; structured output does not establish correctness. include::spring-boot:partial$starter.adoc[] + +== Semantic media input validation + +The Semantic adapter accepts text and TypeSafe AI's supported structured application state. +It rejects explicit `$semanticInput` image/media envelopes, including nested media envelopes, +and binary state before initializing transport. Selecting a body with an image `Content-Type` +or a self-describing image data URL does not make this text/structured expert multimodal. +Use an image-capable expert for images. diff --git a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/wolf-defender.adoc b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/wolf-defender.adoc index 85365d295470..226933110a25 100644 --- a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/wolf-defender.adoc +++ b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/wolf-defender.adoc @@ -125,7 +125,9 @@ Configure properties before starting the service. Changes take effect on the nex == Input, probability and document limits -The input must be a nonblank String. Missing input, structured objects and oversized input are errors; +The input must be a nonblank String. Image/media envelopes, image data URLs and binary bodies +are rejected before inference. An image `Content-Type` on the selected body identifies image +input; this expert does not transcribe, decode or stringify it. Missing input, structured objects and oversized input are errors; select or convert a field explicitly in the route when necessary. No implicit `toString()` conversion is performed. The expert adds the pinned tokenizer's special tokens, disables truncation and padding, and supplies INT64 `input_ids` and `attention_mask` tensors. Each evaluation runs one unpadded window. diff --git a/components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc b/components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc index d977fad977ba..1cb7caea48c1 100644 --- a/components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc +++ b/components/camel-ai/camel-semantic/src/main/docs/semantic-language.adoc @@ -169,6 +169,46 @@ when no labels apply. Use collections in predicates, for example The category and the meaning of a Boolean verdict must come from the selected expert's contract. A negative detection does not establish general safety or grant authorization. +=== Text and image input + +Experts declare supported modalities in their operation contract. `IMAGE` allows explicit image +parts; it does not imply that every expert accepts images. TypeSafe AI accepts text and its structured state, and Wolf Defender accepts text. Unsupported input fails +before inference, including before any expert in a mixed batch runs. + +For `$\{body}`, an image `Content-Type` such as `image/png` identifies a `byte[]`, `InputStream`, +`StreamCache` or base64 `String` body as an image. Streams are replaced by replayable Camel +stream caches and reset after reading; their bytes remain available downstream. A self-describing +`data:image/...;base64,...` string is also image input. Other strings remain text. +Content-Type handling applies only to the exact `$\{body}` selector after placeholder resolution. +Other expressions, including `$\{bodyAs(byte[])}`, use the generic selected-state path: raw binary +values are rejected. Select `$\{body}` for image bytes, or select an explicit media envelope or +self-describing image data URL. An image header does not reclassify separately selected headers or variables. +With `$\{body}`, `audio/*` and `video/*` content types are rejected, including for a `String` body; +no expert currently supports these modalities. Select an explicitly textual field when evaluating +text that accompanies such a body. + +An explicit JSON-compatible envelope supports ordered mixed input: + +[source,json] +---- +{"$semanticInput":[ + {"type":"text","text":"Inspect this package"}, + {"type":"image","data":"data:image/png;base64,..."} +]} +---- + +Replace the abbreviated image data with actual base64. `SemanticInput.ofParts`, `text` and +`image` provide Java helpers. The reserved `$semanticInput` key distinguishes media from an +ordinary application map. Format-specific validation belongs to the expert. Camel does not +fetch URLs, read image paths from route messages, or invoke transcription/OCR services. + +Audit records expose a bounded `inputType` extension independently of payload capture: +`text`, `image`, `text-and-images`, `structured`, `unsupported` or `unknown` when classification exceeds its bound. This classification remains +available when input capture is disabled. The extension is separate from monitoring events. +Image capture remains opt-in and subject to the existing redactor and size budget. When an +image exceeds the budget, a bounded descriptor may retain its MIME type and +`dataOmitted=snapshot_limit` instead of the image bytes. Such input cannot be replayed. + === Structured input and specialised experts A specialised moderation expert can accept a map containing a user prompt, an assistant response, @@ -874,7 +914,8 @@ Provider discovery descriptors are not used. The language checks the static operation contract and then calls `validate` for operation-specific parameter combinations and bean configuration before traffic. `validateInput` checks detailed -message requirements before evaluation. All input checks in a batch run before any expert is invoked. +message requirements before evaluation. `validateState` checks shared input once per expert group; +use it for expensive checks such as image encoding. Its default implementation does nothing. All input checks in a batch run before any expert is invoked. `evaluate` returns a typed `SemanticResult` after applying the expert's policy. A probability-only Boolean result is invalid: the expert must supply its Boolean verdict. Providers should also validate when their SPI is called directly. `evaluate` must be thread-safe, bound time and resource consumption, honor interruption and propagate diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/language/semantic/SemanticLanguage.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/language/semantic/SemanticLanguage.java index e27ecda4f592..19b7edbf8ef2 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/language/semantic/SemanticLanguage.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/language/semantic/SemanticLanguage.java @@ -38,6 +38,7 @@ import org.apache.camel.semantic.SemanticCapabilities.Operation; import org.apache.camel.semantic.SemanticEvaluation; import org.apache.camel.semantic.SemanticEvaluations; import org.apache.camel.semantic.SemanticExpert.ResultType; +import org.apache.camel.semantic.SemanticInput; import org.apache.camel.semantic.SemanticResult; import org.apache.camel.semantic.internal.SemanticAuditService; import org.apache.camel.spi.Metadata; @@ -110,9 +111,11 @@ public class SemanticLanguage extends LanguageSupport { invocation = audit.begin(null, null, evaluation, operation, resolved.reference, resolved.capabilities.getProvider(), null, request); reason = "invalid_input"; + input = SemanticInput.normalize(input); invocation.captureInput(input); operation.validateInput(input); resolved.provider.validateInput(evaluation, input); + resolved.provider.validateState(input); if (Thread.currentThread().isInterrupted()) { throw new InterruptedException("Semantic evaluation interrupted"); } @@ -383,7 +386,7 @@ public class SemanticLanguage extends LanguageSupport { Expression state = getCamelContext().resolveLanguage("simple").createExpression(selector); state.init(getCamelContext()); return new Compiled( - selected, Map.copyOf(operations), state, groups.stream() + selected, Map.copyOf(operations), state, "${body}".equals(selector), groups.stream() .map(group -> new Group(group.expert, Collections.unmodifiableMap(group.evaluations))).toList()); } @@ -438,7 +441,9 @@ public class SemanticLanguage extends LanguageSupport { } active.addAll(invocations.values()); failureReason = "invalid_input"; - Object state = current.state.evaluate(exchange, Object.class); + Object state = current.body + ? SemanticInput.body(exchange) + : SemanticInput.normalize(current.state.evaluate(exchange, Object.class)); invocations.values().forEach(invocation -> invocation.captureInput(state)); // A state selector can itself evaluate another semantic expression. exchange.removeProperty(RESULT); @@ -460,6 +465,9 @@ public class SemanticLanguage extends LanguageSupport { current.operations.get(entry.getKey()).validateInput(state); group.expert.provider.validateInput(entry.getValue(), state); } + active.clear(); + group.evaluations.keySet().forEach(name -> active.add(invocations.get(name))); + group.expert.provider.validateState(state); } catch (IllegalArgumentException invalid) { throw new IllegalArgumentException( "Semantic evaluations " + group.evaluations.keySet() + ", expert '" + group.expert.reference @@ -591,6 +599,6 @@ public class SemanticLanguage extends LanguageSupport { } private record Compiled(Map<String, SemanticEvaluation> evaluations, Map<String, Operation> operations, - Expression state, List<Group> groups) { + Expression state, boolean body, List<Group> groups) { } } diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAdapter.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAdapter.java index 135b0b606533..d3158424a767 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAdapter.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAdapter.java @@ -36,6 +36,15 @@ public interface SemanticAdapter { default void validateInput(SemanticEvaluation evaluation, Object state) { } + /** + * Validate requirements shared by all operations against the selected state. The language calls this once per + * expert group, after its per-evaluation input checks and before any expert performs inference. Use this hook for + * expensive checks independent of the question, such as validating image encoding. It must not contact services. + * Providers must also validate requests made through direct evaluate/evaluateBatch calls. + */ + default void validateState(Object state) { + } + /** Synchronous, potentially blocking evaluation. Operational errors must be thrown, never returned as decisions. */ SemanticResult evaluate(SemanticEvaluation evaluation, Object state) throws Exception; diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAuditRecord.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAuditRecord.java index a6fe764459a4..fad8b1bd69ae 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAuditRecord.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticAuditRecord.java @@ -32,7 +32,8 @@ import org.apache.camel.semantic.internal.SemanticAuditInputSnapshot; */ public final class SemanticAuditRecord { /** Additive field groups; destinations explicitly opt in before receiving them. */ - public static final Set<String> EXTENSIONS = Set.of("interpretation", "findings", "executingRouteId", "identity"); + public static final Set<String> EXTENSIONS + = Set.of("interpretation", "findings", "executingRouteId", "identity", "inputType"); private final Map<String, Object> fields; private final Instant timestamp; @@ -48,7 +49,7 @@ public final class SemanticAuditRecord { "status", "reasonCode", "durationNanos", "resultOmitted", "result", "action", "namespace", "correlationId", "policyId", "policyVersion", "rule", "evidence", "startedAt", "provider", "model", "revision", "input", "inputOmitted", "inputRedacted", "breadcrumbId", "executingRouteId", "interpretation", "interpretationOmitted", - "findings", "findingsOmitted", "taskId", "taskVersion"); + "findings", "findingsOmitted", "taskId", "taskVersion", "inputType"); if (!keys.containsAll(data.keySet()) || !(data.get("schemaVersion") instanceof Number version) || version.doubleValue() != 1 || !(data.get("eventId") instanceof String id) || id.isBlank() || id.length() > 256 @@ -69,6 +70,11 @@ public final class SemanticAuditRecord { || duration.longValue() < 0 || duration.doubleValue() != duration.longValue())) { throw new IllegalArgumentException("Invalid audit duration"); } + if (data.containsKey("inputType") && (!"evaluation".equals(category) + || !Set.of("text", "image", "text-and-images", "structured", "unsupported", "unknown") + .contains(data.get("inputType")))) { + throw new IllegalArgumentException("Invalid audit input type"); + } if (data.containsKey("startedAt")) { if (!(data.get("startedAt") instanceof String startedAt)) { throw new IllegalArgumentException("Invalid audit start timestamp"); diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticCapabilities.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticCapabilities.java index b43c544410fd..50c8fe68e6c8 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticCapabilities.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticCapabilities.java @@ -167,6 +167,14 @@ public final class SemanticCapabilities { || declaration.resultType() != ResultType.CHOICE && declaration.resultType() != ResultType.SCORE)) { throw new IllegalArgumentException("Complete probabilities require a CHOICE or SCORE distribution"); } + if (!declaration.scoreDescriptionsParameter().isEmpty()) { + Parameter descriptions = parameters.get(declaration.scoreDescriptionsParameter()); + if (declaration.scoreLevelsParameter().isEmpty() || descriptions == null + || descriptions.getType() != Map.class || descriptions.getItemType() != String.class) { + throw new IllegalArgumentException( + "Score descriptions require score levels and a Map<String, String> parameter"); + } + } if (!declaration.scoreLevelsParameter().isEmpty()) { Parameter levels = parameters.get(declaration.scoreLevelsParameter()); if (declaration.resultType() != ResultType.SCORE || levels == null @@ -246,6 +254,10 @@ public final class SemanticCapabilities { return declaration.scoreLevelsParameter(); } + public String getScoreDescriptionsParameter() { + return declaration.scoreDescriptionsParameter(); + } + public double getMinimum() { return declaration.minimum(); } @@ -295,10 +307,7 @@ public final class SemanticCapabilities { } public void validateInput(Object state) { - if (!(state instanceof String && inputTypes.contains(InputType.TEXT) - || (state instanceof Map<?, ?> || state instanceof List<?>) && inputTypes.contains(InputType.STRUCTURED))) { - throw new IllegalArgumentException("Unsupported selected state; accepts " + inputTypes + " input"); - } + SemanticInput.validate(state, inputTypes); } /** Validate the typed answer; decision policy has already been applied by the expert. */ diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticExpert.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticExpert.java index 1afa5a8023c7..e3c5fbbc90b9 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticExpert.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticExpert.java @@ -36,7 +36,8 @@ import java.lang.annotation.Target; public @interface SemanticExpert { enum InputType { TEXT, - STRUCTURED + STRUCTURED, + IMAGE } enum ResultType { diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticInput.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticInput.java new file mode 100644 index 000000000000..b237100bdbae --- /dev/null +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticInput.java @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.semantic; + +import java.io.InputStream; +import java.util.Base64; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; + +import org.apache.camel.Exchange; +import org.apache.camel.NoTypeConversionAvailableException; +import org.apache.camel.StreamCache; +import org.apache.camel.semantic.SemanticExpert.InputType; + +/** + * Explicit, JSON-compatible input envelope for ordered text and image parts. The reserved {@value #PARTS} key + * distinguishes media from ordinary application state. Values are data: no file or URL is resolved by this class. + */ +public final class SemanticInput { + public static final String PARTS = "$semanticInput"; + + private SemanticInput() { + } + + public static Map<String, Object> ofParts(List<Map<String, Object>> parts) { + Map<String, Object> input = Map.of(PARTS, parts.stream().map(Map::copyOf).toList()); + validate(input, Set.of(InputType.TEXT, InputType.IMAGE)); + return input; + } + + public static Map<String, Object> text(String text) { + return Map.of("type", "text", "text", text); + } + + /** An image data URL, with an optional expert-specific detail setting supplied separately in the part. */ + public static Map<String, Object> image(String data) { + return Map.of("type", "image", "data", data); + } + + /** Binary image part with an explicit MIME type. No image interpretation or inference is performed. */ + public static Map<String, Object> image(byte[] data, String mediaType) { + return image("data:" + mediaType + ";base64," + Base64.getEncoder().encodeToString(data)); + } + + /** Normalize a self-describing image string; ordinary strings remain text. */ + public static Object normalize(Object input) { + return input instanceof String text && text.startsWith("data:image/") + ? ofParts(List.of(image(text))) : input; + } + + /** + * Select an image body using Content-Type. Streams become replayable Camel stream caches before reading, so later + * route steps can still consume the body. Headers do not affect explicitly selected headers or variables. + */ + public static Object body(Exchange exchange) throws NoTypeConversionAvailableException { + Object input = exchange.getMessage().getBody(); + String contentType = exchange.getMessage().getHeader(Exchange.CONTENT_TYPE, String.class); + String mediaType = contentType == null ? "" : contentType.split(";", 2)[0].trim().toLowerCase(Locale.ROOT); + if (mediaType.startsWith("audio/") || mediaType.startsWith("video/")) { + throw new IllegalArgumentException("Audio and video bodies are not supported by the selected input contract"); + } + if (!mediaType.startsWith("image/")) { + return normalize(input); + } + if (!mediaType.matches("image/[a-z0-9.+-]{1,100}")) { + throw new IllegalArgumentException("Invalid image Content-Type"); + } + if (input instanceof InputStream && !(input instanceof StreamCache)) { + input = exchange.getContext().getTypeConverter().mandatoryConvertTo(StreamCache.class, exchange, input); + exchange.getMessage().setBody(input); + } + if (input instanceof StreamCache cache) { + try { + cache.reset(); + byte[] data = exchange.getContext().getTypeConverter().mandatoryConvertTo(byte[].class, exchange, cache); + return ofParts(List.of(image(data, mediaType))); + } finally { + cache.reset(); + } + } + if (input instanceof byte[] data) { + return ofParts(List.of(image(data, mediaType))); + } + if (input instanceof String text) { + return text.startsWith("data:image/") + ? normalize(text) + : ofParts(List.of(image("data:" + mediaType + ";base64," + text))); + } + if (isMedia(input)) { + return input; + } + throw new IllegalArgumentException("An image body requires bytes, an InputStream, or a base64 string"); + } + + public static boolean isMedia(Object input) { + return input instanceof Map<?, ?> map && map.containsKey(PARTS); + } + + /** Read an explicit envelope. Validate against the selected expert before using its parts. */ + public static List<?> parts(Object input) { + if (!(input instanceof Map<?, ?> map) || map.size() != 1 || !(map.get(PARTS) instanceof List<?> parts) + || parts.isEmpty()) { + throw new IllegalArgumentException("Media input requires a nonempty $semanticInput part list"); + } + return parts; + } + + public static void validate(Object input, Set<InputType> accepted) { + input = normalize(input); + if (isMedia(input)) { + for (Object item : parts(input)) { + if (!(item instanceof Map<?, ?> part)) { + throw new IllegalArgumentException("Media input parts must be objects"); + } + if ("text".equals(part.get("type")) && accepted.contains(InputType.TEXT) + && part.keySet().equals(Set.of("type", "text")) && part.get("text") instanceof String) { + continue; + } + if ("image".equals(part.get("type")) && accepted.contains(InputType.IMAGE) + && Set.of("type", "data", "detail").containsAll(part.keySet()) + && part.get("data") instanceof String data && !data.isBlank() + && (!part.containsKey("detail") || part.get("detail") instanceof String)) { + continue; + } + throw new IllegalArgumentException("Unsupported media part or modality; expert accepts " + accepted); + } + } else if (input instanceof String && accepted.contains(InputType.TEXT)) { + // After normalizing data:image/ URLs above, ordinary strings (including URLs and JSON) remain text. + } else if ((input instanceof Map<?, ?> || input instanceof List<?>) && accepted.contains(InputType.STRUCTURED)) { + // The expert validates its application-state schema; STRUCTURED does not imply a JSON-only contract. + } else { + throw new IllegalArgumentException("Unsupported selected state; accepts " + accepted + " input"); + } + } + +} diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticMetadataConsole.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticMetadataConsole.java index 6e53ced2e730..3b84ab61f114 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticMetadataConsole.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticMetadataConsole.java @@ -148,6 +148,9 @@ public class SemanticMetadataConsole extends AbstractDevConsole { contract.put("resultMeaning", op.getResultMeaning()); if (!op.getScoreLevelsParameter().isEmpty()) { contract.put("scoreLevelsParameter", op.getScoreLevelsParameter()); + if (!op.getScoreDescriptionsParameter().isEmpty()) { + contract.put("scoreDescriptionsParameter", op.getScoreDescriptionsParameter()); + } } if (!op.getChoiceLabelsParameter().isEmpty()) { contract.put("choiceLabelsParameter", op.getChoiceLabelsParameter()); diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticOperation.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticOperation.java index 585fb3533652..fc8d7d9dd31a 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticOperation.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/SemanticOperation.java @@ -59,6 +59,9 @@ public @interface SemanticOperation { */ String scoreLevelsParameter() default ""; + /** Optional Map<String, String> parameter describing labels in the ordered score levels parameter. */ + String scoreDescriptionsParameter() default ""; + /** Map<String, String> parameter declaring choice keys and their descriptions. */ String choiceLabelsParameter() default ""; diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditExtensions.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditExtensions.java index 63f24cb11419..3b6dadf541a4 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditExtensions.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditExtensions.java @@ -32,7 +32,7 @@ public final class SemanticAuditExtensions { private static final Set<String> INTERPRETATION_FIELDS = Set.of( "source", "meaning", "probabilityMeaning", "confidenceMeaning", "scoreLevels", "choices", "booleanLabels", "noMatchDeclared", "noMatchLabel", - "probabilitiesComplete", "scoreProbabilityKeys"); + "probabilitiesComplete", "scoreProbabilityKeys", "scoreDescriptions"); private SemanticAuditExtensions() { } @@ -51,6 +51,7 @@ public final class SemanticAuditExtensions { } Map<String, Object> parameters = evaluation.getParameters(); copyParameter(fields, "scoreLevels", parameters, operation.getScoreLevelsParameter()); + copyParameter(fields, "scoreDescriptions", parameters, operation.getScoreDescriptionsParameter()); copyParameter(fields, "choices", parameters, operation.getChoiceLabelsParameter()); copyParameter(fields, "booleanLabels", parameters, operation.getBooleanLabelsParameter()); if (!fields.containsKey("choices") && !operation.getLabels().isEmpty()) { @@ -95,7 +96,7 @@ public final class SemanticAuditExtensions { } copy.put("scoreLevels", List.copyOf(levels)); } - for (String name : List.of("choices", "booleanLabels")) { + for (String name : List.of("choices", "booleanLabels", "scoreDescriptions")) { if (fields.containsKey(name)) { if (!(fields.get(name) instanceof Map<?, ?> labels)) { throw new IllegalArgumentException("Invalid interpretation labels"); diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditInputSnapshot.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditInputSnapshot.java index 460a6bd090f2..b11ebbc9af1c 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditInputSnapshot.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditInputSnapshot.java @@ -24,6 +24,8 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import org.apache.camel.semantic.SemanticInput; + /** Bounded JSON values only: never converts arbitrary objects or consumes streams. */ public final class SemanticAuditInputSnapshot { private int remaining; @@ -37,6 +39,62 @@ public final class SemanticAuditInputSnapshot { return new SemanticAuditInputSnapshot(maxChars).value(input, 0); } + /** Classify without reading streams, decoding content, or exposing values in an audit summary. */ + public static String inputType(Object input) { + if (SemanticInput.isMedia(input)) { + boolean text = false; + boolean image = false; + List<?> parts = SemanticInput.parts(input); + if (parts.size() > 1000) { + return "unknown"; + } + for (Object item : parts) { + if (!(item instanceof Map<?, ?> part)) { + return "unsupported"; + } + if ("text".equals(part.get("type"))) { + text = true; + } else if ("image".equals(part.get("type"))) { + image = true; + } else { + return "unsupported"; + } + } + return image ? text ? "text-and-images" : "image" : "text"; + } + return input instanceof String ? "text" + : input instanceof Map<?, ?> || input instanceof List<?> ? "structured" : "unsupported"; + } + + /** Retain bounded image descriptors when the configured capture budget cannot hold the image payload. */ + public static Object capture(Object input, int maxChars) { + try { + return copy(input, maxChars); + } catch (IllegalArgumentException limit) { + if (!SemanticInput.isMedia(input) || SemanticInput.parts(input).size() > 128) { + throw limit; + } + List<Object> parts = new ArrayList<>(); + for (Object item : SemanticInput.parts(input)) { + if (item instanceof Map<?, ?> part && "image".equals(part.get("type"))) { + Map<String, Object> descriptor = new LinkedHashMap<>(); + descriptor.put("type", "image"); + descriptor.put("dataOmitted", "snapshot_limit"); + if (part.get("data") instanceof String data && data.startsWith("data:image/")) { + int end = data.indexOf(";base64,"); + if (end > 5 && end < 120 && data.substring(5, end).matches("image/[a-z0-9.+-]+")) { + descriptor.put("mediaType", data.substring(5, end)); + } + } + parts.add(descriptor); + } else { + parts.add(item); + } + } + return copy(Map.of(SemanticInput.PARTS, parts), maxChars); + } + } + private Object value(Object value, int depth) { if (--nodes < 0 || depth > 8) { throw new IllegalArgumentException("Input snapshot limit"); diff --git a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditService.java b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditService.java index b9c887c960e1..fa1cf5d1672b 100644 --- a/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditService.java +++ b/components/camel-ai/camel-semantic/src/main/java/org/apache/camel/semantic/internal/SemanticAuditService.java @@ -452,20 +452,28 @@ public final class SemanticAuditService extends ServiceSupport { /** Snapshot the effective selected state before a provider can mutate it. Observers never receive this data. */ public void captureInput(Object input) { - if (fields == null || completed || inputCaptured || inputFields.isEmpty()) { + if (fields == null || completed || inputCaptured || !enabled) { return; } inputCaptured = true; SemanticAuditInputConfiguration policy = configuration.getInput((String) fields.get("expert")); SemanticAuditInputRedactor redactor = redactors.get(fields.get("expert")); inputFields.clear(); + try { + inputFields.put("inputType", SemanticAuditInputSnapshot.inputType(input)); + } catch (Exception | AssertionError | LinkageError failure) { + inputFields.put("inputType", "unsupported"); + } + if (!policy.isEnabled()) { + return; + } if (policy.getRedactor() != null && redactor == null) { inputFields.put("inputOmitted", "redaction_failed"); return; } Object snapshot; try { - snapshot = SemanticAuditInputSnapshot.copy(input, policy.getMaxChars()); + snapshot = SemanticAuditInputSnapshot.capture(input, policy.getMaxChars()); } catch (Exception | AssertionError | LinkageError failure) { // Isolate audit snapshot failures from evaluation; fatal VM errors still propagate. inputFields.put("inputOmitted", "snapshot_limit_or_unsupported_type"); diff --git a/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditInputTest.java b/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditInputTest.java index fdf777419320..bc7be3abbc3c 100644 --- a/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditInputTest.java +++ b/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditInputTest.java @@ -23,6 +23,7 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Properties; +import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; @@ -51,6 +52,31 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; class SemanticAuditInputTest { + @ParameterizedTest + @ValueSource(booleans = { true, false }) + void imageInputTypeSurvivesCapturePolicyAndPayloadLimits(boolean capture) throws Exception { + try (var context = new DefaultCamelContext()) { + var audit = SemanticAuditService.get(context); + audit.configure("test", configuration(true, new SemanticAuditInputConfiguration(capture, 256, null))); + audit.activate(); + context.start(); + var input = SemanticInput.ofParts(List.of(SemanticInput.image("data:image/png;base64," + "A".repeat(1000)))); + var invocation = audit.begin(null, null, evaluation("security"), null, "security", null, null, null); + invocation.captureInput(input); + invocation.complete("success", "evaluation_completed", new SemanticResult(true, null, null, null, Map.of())); + audit.stop(); + var record = records(audit).get(0); + assertThat(record.toMap()).containsEntry("inputType", "image"); + assertThat(record.forExtensions(Set.of()).toMap()).doesNotContainKey("inputType"); + if (capture) { + assertThat(record.toMap().get("input")).isEqualTo(Map.of(SemanticInput.PARTS, + List.of(Map.of("type", "image", "mediaType", "image/png", "dataOmitted", "snapshot_limit")))); + } else { + assertThat(record.toMap()).doesNotContainKeys("input", "inputOmitted"); + } + } + } + @Test void captureIsPerExpertAndNeverSentToObservers() throws Exception { try (var context = new DefaultCamelContext()) { diff --git a/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditReviewTest.java b/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditReviewTest.java index 6fe15ebc1d42..a1d6f18d5938 100644 --- a/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditReviewTest.java +++ b/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticAuditReviewTest.java @@ -92,6 +92,42 @@ class SemanticAuditReviewTest { } } + @Test + void sharedInputValidationRunsOncePerExpertAndPreventsAllInferenceOnFailure() throws Exception { + try (var context = new DefaultCamelContext()) { + var checks = new AtomicInteger(); + var first = new FixedSemanticExpert() { + @Override + public void validateState(Object state) { + checks.incrementAndGet(); + } + }; + var second = new FixedSemanticExpert() { + @Override + public void validateState(Object state) { + throw new IllegalArgumentException("invalid shared input"); + } + }; + context.getRegistry().bind("security", first); + context.getRegistry().bind("second", second); + var audit = configured(context); + SemanticEvaluations.get(context).replace("defs", Map.of("a", evaluation(0.1), "b", evaluation(0.2), + "c", new SemanticEvaluation("boolean", "second", "${body}", Map.of()))); + context.start(); + var exchange = new DefaultExchange(context); + exchange.getMessage().setBody("text"); + assertThatThrownBy( + () -> context.resolveLanguage("semantic").createExpression("refs:a,b,c").evaluate(exchange, Map.class)) + .hasMessageContaining("invalid shared input"); + assertThat(checks.get()).isEqualTo(1); + assertThat(first.calls).isZero(); + assertThat(second.calls).isZero(); + audit.stop(); + assertThat(rows(audit)).filteredOn(r -> List.of("a", "b").contains(r.text("definition"))) + .hasSize(2).allSatisfy(r -> assertThat(r.getStatus()).isEqualTo("not_executed")); + } + } + @ParameterizedTest @ValueSource(booleans = { true, false }) void expertOnlyCaptureRecordsEvaluationsWithoutRouteAnnotations(boolean enabled) throws Exception { diff --git a/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticInputTest.java b/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticInputTest.java new file mode 100644 index 000000000000..3e9c28170232 --- /dev/null +++ b/components/camel-ai/camel-semantic/src/test/java/org/apache/camel/semantic/SemanticInputTest.java @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.semantic; + +import java.io.ByteArrayInputStream; +import java.io.FilterInputStream; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.apache.camel.Exchange; +import org.apache.camel.StreamCache; +import org.apache.camel.impl.DefaultCamelContext; +import org.apache.camel.semantic.SemanticExpert.InputType; +import org.apache.camel.support.DefaultExchange; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class SemanticInputTest { + @Test + void imageBodiesNormalizeWithoutLosingTheRoutePayload() throws Exception { + byte[] bytes = new byte[] { 1, 2, 3 }; + try (var context = new DefaultCamelContext()) { + context.start(); + for (Object body : List.of(bytes, "AQID", new ByteArrayInputStream(bytes), + new FilterInputStream(new ByteArrayInputStream(bytes)) { + @Override + public boolean markSupported() { + return false; + } + })) { + var exchange = new DefaultExchange(context); + exchange.getMessage().setBody(body); + exchange.getMessage().setHeader(Exchange.CONTENT_TYPE, "image/png"); + assertThat(SemanticInput.body(exchange)) + .isEqualTo(SemanticInput.ofParts(List.of(SemanticInput.image("data:image/png;base64,AQID")))); + if (exchange.getMessage().getBody() instanceof StreamCache cache) { + assertThat(exchange.getMessage().getBody(byte[].class)).containsExactly(bytes); + cache.reset(); + assertThat(SemanticInput.body(exchange)) + .isEqualTo(SemanticInput.ofParts(List.of(SemanticInput.image(bytes, "image/png")))); + } else { + assertThat(exchange.getMessage().getBody()).isSameAs(body); + } + } + } + } + + @Test + void plainBase64IsTextUnlessAnImageIsExplicit() throws Exception { + try (var context = new DefaultCamelContext()) { + var exchange = new DefaultExchange(context); + exchange.getMessage().setBody("AQID"); + assertThat(SemanticInput.body(exchange)).isEqualTo("AQID"); + exchange.getMessage().setBody("data:image/png;base64,AQID"); + assertThat(SemanticInput.isMedia(SemanticInput.body(exchange))).isTrue(); + } + } + + @Test + void audioAndVideoContentTypesRejectEvenStringBodies() throws Exception { + try (var context = new DefaultCamelContext()) { + var exchange = new DefaultExchange(context); + exchange.getMessage().setBody("AQID"); + for (String type : List.of("audio/wav", "video/mp4")) { + exchange.getMessage().setHeader(Exchange.CONTENT_TYPE, type); + assertThatThrownBy(() -> SemanticInput.body(exchange)).isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("not supported"); + } + } + } + + @Test + void validatesEveryExplicitPartAgainstExpertCapabilities() { + var input = SemanticInput + .ofParts(List.of(SemanticInput.text("Inspect this"), SemanticInput.image("data:image/png;base64,AA=="))); + assertThatCode(() -> SemanticInput.validate(input, Set.of(InputType.TEXT, InputType.IMAGE))).doesNotThrowAnyException(); + assertThatThrownBy(() -> SemanticInput.validate(input, Set.of(InputType.TEXT, InputType.STRUCTURED))) + .isInstanceOf(IllegalArgumentException.class).hasMessageContaining("Unsupported media"); + assertThatThrownBy(() -> SemanticInput.validate(input, Set.of(InputType.IMAGE))) + .isInstanceOf(IllegalArgumentException.class); + assertThat(SemanticInput.parts(input)).hasSize(2); + } + + @Test + void preservesOrdinaryApplicationStateAndText() { + assertThatCode(() -> SemanticInput.validate(Map.of("custom", new Object()), Set.of(InputType.STRUCTURED))) + .doesNotThrowAnyException(); + assertThatCode(() -> SemanticInput.validate(Map.of("type", "image", "url", "https://example.com";), + Set.of(InputType.STRUCTURED))).doesNotThrowAnyException(); + assertThatCode(() -> SemanticInput.validate("data:audio/wav;base64,AA==", Set.of(InputType.TEXT))) + .doesNotThrowAnyException(); + assertThatCode(() -> SemanticInput.validate(List.of(Map.of("value", true), 0.5, "text"), Set.of(InputType.STRUCTURED))) + .doesNotThrowAnyException(); + } + + @Test + void rejectsUnknownPartsAndAmbiguousEnvelopes() { + for (Object input : List.of(Map.of(SemanticInput.PARTS, List.of(Map.of("type", "audio", "data", "raw"))), + Map.of(SemanticInput.PARTS, List.of(SemanticInput.text("text")), "extra", "value"))) { + assertThatThrownBy(() -> SemanticInput.validate(input, Set.of(InputType.TEXT, InputType.IMAGE))) + .isInstanceOf(IllegalArgumentException.class); + } + } +} diff --git a/components/camel-ai/camel-typesafe-ai/src/main/docs/typesafe-ai-component.adoc b/components/camel-ai/camel-typesafe-ai/src/main/docs/typesafe-ai-component.adoc index b367bf81f601..87d2621f0118 100644 --- a/components/camel-ai/camel-typesafe-ai/src/main/docs/typesafe-ai-component.adoc +++ b/components/camel-ai/camel-typesafe-ai/src/main/docs/typesafe-ai-component.adoc @@ -306,3 +306,11 @@ Only submit data permitted for the configured service. Test model decisions and against your application; structured output does not establish correctness. include::spring-boot:partial$starter.adoc[] + +== Semantic media input validation + +The Semantic adapter accepts text and TypeSafe AI's supported structured application state. +It rejects explicit `$semanticInput` image/media envelopes, including nested media envelopes, +and binary state before initializing transport. Selecting a body with an image `Content-Type` +or a self-describing image data URL does not make this text/structured expert multimodal. +Use an image-capable expert for images. diff --git a/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiJson.java b/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiJson.java index 2c9e0f72da4a..e32e50341e50 100644 --- a/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiJson.java +++ b/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiJson.java @@ -37,7 +37,7 @@ final class TypeSafeAiJson { Map<String, Object> request = new LinkedHashMap<>(input); request.putIfAbsent("model", model); text(request.get("model")); - content(request.get("state")); + validateState(request.get("state")); validateQuestions(request.get("questions")); jsonValue(request); // Detach nested criteria so later caller mutations cannot change response validation. @@ -66,7 +66,7 @@ final class TypeSafeAiJson { private static void question(Map<?, ?> question) { if (question.get("instructions") != null) { - content(question.get("instructions")); + validateState(question.get("instructions")); } Object type = question.get("type"); if ("noul".equals(type)) { @@ -74,7 +74,7 @@ final class TypeSafeAiJson { object(question.get("criteria")).forEach((key, value) -> { require("true".equals(key) || "false".equals(key), "Noul criteria keys must be true or false"); if (value != null) { - content(value); + validateState(value); } }); } @@ -84,14 +84,14 @@ final class TypeSafeAiJson { criteria.forEach((key, value) -> { text(key); if (value != null) { - content(value); + validateState(value); } }); } else if ("score".equals(type)) { require(question.get("criteria") instanceof List<?>, "Score criteria must be a list"); List<?> criteria = (List<?>) question.get("criteria"); require(!criteria.isEmpty() && criteria.size() <= 10, "Score requires 1 to 10 levels"); - criteria.forEach(TypeSafeAiJson::content); + criteria.forEach(TypeSafeAiJson::validateState); } else { throw new IllegalArgumentException("Question type must be noul, choice or score"); } @@ -140,7 +140,7 @@ final class TypeSafeAiJson { require(probabilities.keySet().equals(levels), "Score probabilities must cover the requested levels"); Map<?, ?> legend = object(answer.get("legend")); require(legend.keySet().equals(levels), "Score legend must cover the requested levels"); - legend.values().forEach(TypeSafeAiJson::content); + legend.values().forEach(TypeSafeAiJson::validateState); } } @@ -165,7 +165,7 @@ final class TypeSafeAiJson { require(value instanceof String text && !text.isBlank(), "Expected a nonblank string"); } - private static void content(Object value) { + static void validateState(Object value) { require(value instanceof String || value instanceof Map<?, ?> || value instanceof List<?>, "Content must be a string, map or list"); jsonValue(value); @@ -173,6 +173,7 @@ final class TypeSafeAiJson { private static void jsonValue(Object value) { if (value instanceof Map<?, ?> map) { + require(!map.containsKey("$semanticInput"), "TypeSafe AI does not support media input envelopes"); map.forEach((key, child) -> { require(key instanceof String, "JSON object keys must be strings"); jsonValue(child); diff --git a/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapter.java b/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapter.java index f52bcbd29f8d..2089a5275162 100644 --- a/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapter.java +++ b/components/camel-ai/camel-typesafe-ai/src/main/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapter.java @@ -130,6 +130,12 @@ public class TypeSafeAiSemanticAdapter implements SemanticAdapter, CamelContextA } } + @Override + public void validateInput(SemanticEvaluation evaluation, Object state) { + SemanticCapabilities.from(getClass()).operation(evaluation.getOperation()).validateInput(state); + TypeSafeAiJson.validateState(state); + } + private TypeSafeAiEndpoint endpoint() { if (endpoint == null) { synchronized (this) { @@ -151,6 +157,10 @@ public class TypeSafeAiSemanticAdapter implements SemanticAdapter, CamelContextA @Override public Map<String, SemanticResult> evaluateBatch(Map<String, SemanticEvaluation> questions, Object state) throws Exception { + questions.values().forEach(question -> { + validate(question); + validateInput(question, state); + }); Map<String, Object> definitions = new LinkedHashMap<>(); questions.forEach((name, question) -> definitions.put(name, definition(question))); JsonObject response = endpoint().evaluate(Map.of("state", state, "questions", definitions)); diff --git a/components/camel-ai/camel-typesafe-ai/src/test/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapterTest.java b/components/camel-ai/camel-typesafe-ai/src/test/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapterTest.java index 808077ad3a17..bfc2651412d5 100644 --- a/components/camel-ai/camel-typesafe-ai/src/test/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapterTest.java +++ b/components/camel-ai/camel-typesafe-ai/src/test/java/org/apache/camel/component/typesafeai/TypeSafeAiSemanticAdapterTest.java @@ -43,6 +43,19 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; class TypeSafeAiSemanticAdapterTest extends TypeSafeAiTestSupport { + @Test + void rejectsImagesAndBinaryStateBeforeInitializingTransport() { + var adapter = new TypeSafeAiSemanticAdapter(); + var question = new SemanticEvaluation("boolean", "decisions", null, Map.of("instructions", "Assess")); + var image = Map.of("$semanticInput", List.of(Map.of("type", "image", "data", "data:image/png;base64,AA=="))); + for (Object input : List.of(new byte[] { 1 }, image, Map.of("nested", image), "data:image/png;base64,AA==")) { + assertThatThrownBy(() -> adapter.validateInput(question, input)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> adapter.evaluateBatch(Map.of("q", question), input)) + .isInstanceOf(IllegalArgumentException.class); + } + assertThat(requests).isEmpty(); + } + @BeforeEach void registerExpert() { var expert = new TypeSafeAiSemanticAdapter(); diff --git a/components/camel-ai/camel-wolf-defender/src/main/docs/wolf-defender.adoc b/components/camel-ai/camel-wolf-defender/src/main/docs/wolf-defender.adoc index 85365d295470..226933110a25 100644 --- a/components/camel-ai/camel-wolf-defender/src/main/docs/wolf-defender.adoc +++ b/components/camel-ai/camel-wolf-defender/src/main/docs/wolf-defender.adoc @@ -125,7 +125,9 @@ Configure properties before starting the service. Changes take effect on the nex == Input, probability and document limits -The input must be a nonblank String. Missing input, structured objects and oversized input are errors; +The input must be a nonblank String. Image/media envelopes, image data URLs and binary bodies +are rejected before inference. An image `Content-Type` on the selected body identifies image +input; this expert does not transcribe, decode or stringify it. Missing input, structured objects and oversized input are errors; select or convert a field explicitly in the route when necessary. No implicit `toString()` conversion is performed. The expert adds the pinned tokenizer's special tokens, disables truncation and padding, and supplies INT64 `input_ids` and `attention_mask` tensors. Each evaluation runs one unpadded window. diff --git a/components/camel-ai/camel-wolf-defender/src/test/java/org/apache/camel/component/wolfdefender/WolfDefenderSemanticAdapterTest.java b/components/camel-ai/camel-wolf-defender/src/test/java/org/apache/camel/component/wolfdefender/WolfDefenderSemanticAdapterTest.java index 9c9bd00c02ed..084ac006ca98 100644 --- a/components/camel-ai/camel-wolf-defender/src/test/java/org/apache/camel/component/wolfdefender/WolfDefenderSemanticAdapterTest.java +++ b/components/camel-ai/camel-wolf-defender/src/test/java/org/apache/camel/component/wolfdefender/WolfDefenderSemanticAdapterTest.java @@ -184,7 +184,10 @@ class WolfDefenderSemanticAdapterTest { expert.setMaxCharacters(4); expert.start(); try { - for (Object state : new Object[] { null, Map.of("text", "x"), 42, new StringBuilder("text"), "", " ", "12345" }) { + for (Object state : new Object[] { + null, Map.of("text", "x"), 42, new StringBuilder("text"), "", " ", "12345", + new byte[] { 1 }, "data:image/png;base64,AA==", + Map.of("$semanticInput", List.of(Map.of("type", "image", "data", "data:image/png;base64,AA=="))) }) { assertThatThrownBy(() -> expert.evaluate(evaluation(), state)).isInstanceOf(RuntimeException.class) .satisfies(e -> assertThat(e.getMessage()).doesNotContain("12345")); } diff --git a/design/semantic-audit.adoc b/design/semantic-audit.adoc index 6041c6223a54..44ef1d5a7e3e 100644 --- a/design/semantic-audit.adoc +++ b/design/semantic-audit.adoc @@ -76,6 +76,18 @@ retention and durability. Audit delivery is best effort and independent of manag SemanticObserver remains available for captureContext and detached tracing completion; its established bounded projection and callback ordering remain compatible. +== Media input + +The private audit copy classifies input independently of content capture using the optional +`inputType` extension. Values are text, image, text-and-images, structured, unsupported or unknown +when bounded classification cannot finish. Older destinations receive their existing projection. +Neither image bytes nor this classification are added to the frozen monitoring/event payload. + +Ordered media input uses the provider-independent `$semanticInput` envelope. Per-expert input +capture remains opt-in, bounded and redacted. An image too large for the configured snapshot +may be replaced by a bounded descriptor (`type=image`, optional mediaType, dataOmitted=snapshot_limit). +The redactor still processes that detached snapshot. A descriptor cannot recreate missing data. + == Read-only access The reader SPI has no evidence-specific limit on getAll. Explicit console callers limit requests to diff --git a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc index 68e31b6ceb25..e535071478c6 100644 --- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc +++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc @@ -723,3 +723,25 @@ before Camel 3.13), for example a file from a second uploader, or from a client visible later: GridFS sets the upload date on the client. Use `TimeStampAndFileAttribute` or `PersistentTimestampAndFileAttribute` if such files must be routed: these strategies keep the start date in their query, as before, and mark each file so that it is routed once. + +=== Semantic image input and audit output + +Semantic evaluations now recognize a selected `$\{body}` with an image `Content-Type`, and +self-describing `data:image/...` strings, as image input. Text-only experts reject these inputs +instead of treating encoded images as text. For binary image streams, Camel preserves the bytes +using a replayable stream cache; downstream code must not rely on the original stream class. +For the exact `$\{body}` selector, `audio/*` and `video/*` content types now fail before inference, +even for a `String` body. Select an explicitly textual field to evaluate text accompanying audio +or video. Content-Type normalization does not apply to other selectors such as `$\{bodyAs(byte[])}`; +use `$\{body}` for image bytes or select a media envelope or image data URL explicitly. +Ordinary strings without a media content type remain text. +The `$semanticInput` key is reserved for explicit media envelopes. TypeSafe AI rejects that +key in structured state, including nested objects; rename an application-state key that conflicts with it. + +`SemanticAdapter` has an additive default `validateState(Object)` hook for validation shared by +all questions for one expert. Existing per-evaluation `validateInput` hooks still run; both finish +for every expert before any provider performs inference. A shared validation failure marks that +expert's group as invalid input in the audit. + +Semantic audit backends can opt into the additive `inputType` extension. Readers that do not +support it keep their existing projection.
