Luigi De Masi created CAMEL-25310:
-------------------------------------
Summary: camel-semantic: Add experts, capability discovery and
catalog metadata
Key: CAMEL-25310
URL: https://issues.apache.org/jira/browse/CAMEL-25310
Project: Camel
Issue Type: Improvement
Components: camel-catalog, tooling, camel-ai
Reporter: Luigi De Masi
Assignee: Luigi De Masi
h2. Problem and motivation
Extend camel-semantic so that general-purpose semantic evaluators and
specialised classifiers can be used through the same named-evaluation contract,
with explicit expert selection, discoverable capabilities, and Camel Catalog
metadata.
Today, SemanticQuestion requires nonblank instructions, SemanticLanguage
selects one adapter for the language, and SemanticAdapter offers validate(...)
and evaluate(...) but no machine-readable capability descriptor. The generated
semantic-adapter service descriptor identifies the implementation class without
describing its supported inputs or results.
These assumptions work for an instruction-driven provider such as Jev. They are
awkward for a fixed-purpose classifier such as Wolf-Defender, which receives
text and classifies it as BENIGN or INJECTION without accepting a question or
configurable instructions. Its injection probability can already fit
SemanticResult's boolean probability field. It does not provide arbitrary
Choice categories or an application-defined Score rubric.
The gap is in declaring, selecting, and describing evaluations, rather than in
requiring a different routing language or inventing unsupported model
behaviour. Applications should be able to use both kinds of expert in the same
CamelContext while keeping provider-specific formats and inference libraries
outside the route logic.
h2. Goals and terminology
* Use *expert* as the user-facing term and the proposed declaration key. An
expert is a configured implementation of the semantic evaluation SPI; it is not
an autonomous agent.
* Retain the semantic language, named declarations, and ref:name /
refs:name1,name2 expressions.
* Let each expert support a subset of the common result contracts: BOOLEAN,
CHOICE, and SCORE.
* Support both instruction-driven evaluations and fixed evaluations whose
meaning is supplied by the configured expert.
* Make capabilities discoverable at runtime and through the Camel Catalog, with
one authoritative definition of static metadata.
* Keep camel-semantic independent of LangChain4j and concrete inference
runtimes. Provider dependencies, models, tokenizers, and transport resources
belong to optional provider modules.
The syntax and API names below are proposals for review. They are not claims
about existing supported options. Using expert in the DSL does not require an
incompatible rename of the existing SemanticAdapter SPI.
h2. Expert selection
Add an optional expert reference to each named evaluation. Resolve it
deterministically:
# If the declaration explicitly names an expert, use that configured instance.
# Otherwise, use an explicitly configured default expert.
# Otherwise, if exactly one eligible semantic expert is registered/discovered,
select it automatically.
# If there are no eligible experts, fail initialization with an actionable
error.
# If there are multiple eligible experts and no default, fail initialization
and list the available names.
An unknown explicit expert or invalid default is an error; it must not trigger
fallback. Do not select the first discovered implementation or infer selection
from supported result type. Two BOOLEAN experts may evaluate entirely different
properties of the input.
Expert references identify configured application instances, not merely catalog
entries. The same implementation may have multiple differently configured
instances. Catalog availability must not cause a provider to be selected or
instantiated automatically.
Example diagnostic:
{noformat}
Semantic evaluation 'injection' has no expert configured.
Available experts: security, general.
Specify 'expert' or configure a default expert.
{noformat}
h2. Capabilities and validation
Introduce a machine-readable capability description, for example
SemanticCapabilities exposed through capabilities(). Keep
validate(SemanticQuestion) for checking an individual declaration against the
selected expert.
The descriptor should express the supported contract, including:
* Accepted input kinds, such as text and structured state.
* Supported result types.
* Whether instructions are required, optional, or unsupported.
* Whether the expert accepts caller-defined criteria/rubrics, or has fixed
output semantics.
* Availability and meaning of optional probability/confidence fields, per
supported result type where necessary.
* Descriptions of fixed semantics, especially what a positive BOOLEAN result
means.
* Provider identity, owning artifact, and any statically known limits needed
for tooling.
Capabilities describe what is possible; validation checks the actual
declaration, including category/scale restrictions, instruction requirements,
and decision-policy compatibility. Static metadata must not imply that every
possible question of a supported type is meaningful for that expert.
Validate declarations before the associated route handles traffic, and validate
changed declarations when they are reloaded. Validation must not perform
inference or contact a model service. Dynamic input-shape and provider-response
validation remains necessary during evaluation.
For example, binding a CHOICE evaluation to a Wolf boolean expert must produce
an initialization error:
{noformat}
Semantic evaluation 'department' requires CHOICE.
Expert 'security' supports BOOLEAN for prompt-injection detection.
{noformat}
Do not silently convert CHOICE or SCORE into BOOLEAN, ignore unsupported
instructions, or reroute to another expert.
h2. Optional instructions and result semantics
Allow instructions to be absent in the common declaration model. Move the
requirement to the relevant expert validation: the TypeSafe/Jev expert still
requires instructions, while a fixed-purpose Wolf expert accepts a declaration
without them and rejects instruction/criteria combinations it cannot honour.
The Wolf-specific integration owns model/tokenizer selection, input
preparation, document windowing and aggregation, inference lifecycle, and the
mapping of class scores to the common result. A new task keyword is not
required for a dedicated fixed-purpose expert.
For the injection evaluation:
* The selected text is passed as data to the classifier; no synthetic question
is prepended.
* SemanticResult.probability represents the score assigned to INJECTION, not
the confidence of whichever class happened to win.
* The existing threshold and uncertainty policy determine the boolean decision.
* BENIGN means no injection detected by this evaluation; it is not a general
safety or authorization guarantee.
* A classification probability must not be presented as a SCORE severity rating
or as confidence metadata with a different meaning.
* Missing optional metadata remains absent. Invalid outputs and operational
failures remain errors rather than false decisions.
Actions such as continue, quarantine, or review remain application policy
expressed through Camel EIPs. A binary model can drive several policy branches
without pretending to support arbitrary Choice classification. Preserve
model/provider identity in diagnostic metadata where available.
h2. Illustrative declarations
The following assumes two configured expert instances: security (a fixed
injection detector) and general (an instruction-driven provider). The expert
field and omission of instructions for injection are proposed extensions.
{code:yaml}
- semantic:
question:
injection:
expert: security
type: boolean
state: "${body}"
threshold: "{{security.injection.threshold}}"
uncertainty: "{{security.injection.uncertainty}}"
uncertaintyPolicy: fail
department:
expert: general
type: choice
state: "${body}"
instructions: Which department should handle this message?
criteria:
billing: Invoices, payments and refunds
technical: Bugs, outages and technical problems
other: Other requests
{code}
Routes continue to reference ref:injection or ref:department through the
semantic language. Keep threshold configuration explicit and preserve
placeholders through declaration parsing and tooling until normal runtime
resolution.
h2. Batching and lifecycle
Expert selection must work consistently for both single references and refs:
batches. For a batch spanning experts, group evaluations by the resolved expert
instance, delegate each group through the existing batch contract, and return
one result map keyed by the original evaluation names.
Preserve existing requirements for compatible state selectors, a consistent
declaration snapshot, and complete result validation before publishing
decisions/diagnostics. Do not expose partial results if any group fails.
Grouping by implementation class alone is insufficient because two instances
may have different models or configuration. A mixed-expert batch must not be
described as one inference request.
Preserve managed lifecycle ownership, bounded resources, interruption, and
cleanup for discovered and explicitly registered providers. Keep resource/model
loading separate from catalog inspection and declaration validation.
h2. Camel Catalog integration
Publish static expert capabilities as generated catalog metadata. A dedicated
semantic-experts collection is one possible design; agree the catalog kind and
schema during implementation rather than storing provider capability data in
the generic semantic language expression options.
Define the static contract once, for example with an expert annotation or
another declarative descriptor. Generate the provider-packaged metadata and
catalog entry from that source. The runtime capabilities implementation should
reuse the same static data so runtime declarations, catalog metadata, and
documentation do not drift.
An illustrative entry could contain:
{code:json}
{
"kind": "semantic-expert",
"name": "wolf-defender",
"artifactId": "camel-wolf-defender",
"inputTypes": ["string"],
"resultTypes": ["boolean"],
"instructions": "unsupported",
"booleanProbability": true,
"trueMeaning": "Prompt injection or jailbreak detected"
}
{code}
Include normal artifact/version identity and implementation linkage in the
final schema. A provider artifact may advertise more than one expert. Catalog
retrieval should support listing experts and retrieving their capability
models, with API names such as findSemanticExpertNames() and
semanticExpertModel(name) subject to review.
The catalog describes expert implementations or documented configurations. It
does not enumerate application bean names, prove that a provider is installed,
or establish the effective capabilities of an arbitrary configured model.
Runtime capabilities and validate(...) remain authoritative. Tooling should
distinguish a definite incompatibility from configuration it cannot resolve
offline.
This metadata should enable IDEs, visual designers, CLI/TUI and MCP tooling to
discover provider artifacts, explain supported operations, and check
declarations once those consumers adopt the new catalog contract. Do not
require model downloads, provider instantiation, credentials, or native
inference libraries to read catalog entries.
h2. Provider packaging and scope
Wolf-Defender is the concrete motivating use case for a narrow expert. Its
runtime integration should live in an optional provider module, provisionally
camel-wolf-defender, alongside camel-typesafe-ai. The common semantic and
catalog contracts must not depend on Wolf-specific libraries or model assets.
This issue tracks the reusable expert/capability/catalog support and the
contract needed by that provider. The concrete Wolf inference backend and full
provider implementation can be delivered as a linked follow-up if needed.
Validate the common design with both the existing instruction-driven provider
and a deterministic fixed BOOLEAN expert, without requiring network inference
in ordinary tests.
h2. Compatibility and implementation areas
* Preserve existing declarations, default expert selection for applications
with one provider, ref: expressions, batch result keys, and the current global
adapter configuration or a documented compatible alias.
* Introduce capability reporting without forcing existing third-party adapters
to implement a new abstract method immediately. Unknown legacy capabilities
must not be represented as support for every operation; existing validation
remains usable.
* Keep Java, YAML and XML declarations aligned, including expert references,
optional instructions, placeholders, validation errors, resource reload, and
generated schemas/documentation.
* Reuse the existing SemanticResult representation where possible; retain the
distinction between the normalized decision, probabilities, provider
confidence, and operational errors.
Relevant code areas:
* components/camel-ai/camel-semantic: SemanticAdapter, SemanticQuestion,
SemanticQuestionBuilder, SemanticResult, SemanticLanguage, and declaration
loaders.
* components/camel-ai/camel-typesafe-ai: TypeSafeAiSemanticAdapter and provider
discovery metadata.
* tooling/spi-annotations, camel-tooling-model, and camel-package-maven-plugin
for generated expert metadata.
* catalog/camel-catalog and its aggregation, lookup, and documentation support.
h2. Acceptance criteria
# Existing semantic applications and the TypeSafe provider retain their
behaviour and compatibility.
# A fixed BOOLEAN expert works without instructions and supplies an injection
probability through the common result contract.
# Explicit expert, configured default, sole-provider discovery, missing
provider, unknown reference, and ambiguous selection are covered.
# Incompatible result types, unsupported criteria/instructions, and invalid
decision policies fail before inference with errors identifying the evaluation
and expert.
# BOOLEAN threshold boundaries, uncertainty handling, positive-class mapping,
and provider failures remain distinct and are tested.
# Single-expert and mixed-expert batches preserve names, instance identity,
state selection, and all-or-error result publication.
# Java, YAML and XML support the same declarations and validation behaviour,
including reload.
# Static capability metadata is generated from one definition, packaged with
the provider, discoverable through the catalog, and consistent with runtime
reporting.
# Catalog access works offline without constructing providers or loading
model/runtime dependencies; unresolved application configuration is not
misreported as a known capability.
# Documentation explains expert selection, fixed versus instruction-driven
evaluations, probability versus rubric score, and static catalog metadata
versus configured runtime instances.
h2. Related work and references
* CAMEL-24977: provider-independent semantic evaluation.
* CAMEL-25049: batching named questions.
* CAMEL-25138: Java and XML semantic declarations.
* CAMEL-25259 and CAMEL-25257: related DSL-extension/model/discovery work;
coordinate declaration and generation changes rather than introduce parallel
mechanisms. Expert capability metadata describes providers and is distinct from
the schema of the semantic declaration DSL.
* [Semantic language
documentation|https://camel.apache.org/components/next/languages/semantic-language.html]
* [Camel Catalog
documentation|https://camel.apache.org/manual/camel-catalog.html]
* [Wolf-Defender model
card|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small]
* [Wolf-Defender v2
article|https://patronus.studio/en/posts/wolf-defender-v2-prompt-injection-detection-on-device]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
