davsclaus commented on code in PR #28457:
URL: https://github.com/apache/camel/pull/28457#discussion_r4376814542
##########
design/payload-mode.adoc:
##########
@@ -0,0 +1,351 @@
+---
+title: Payload mode
+authors:
+ - "@davsclaus"
+reviewers: []
+approvers: []
+creation-date: 2026-10-25
+last-updated: 2026-10-25
+status: proposed
+see-also:
+ - "CAMEL-25334"
+ - "CAMEL-24844"
+ - "CAMEL-25330"
+ - "CAMEL-25329"
+ - "CAMEL-25746"
+ - "CAMEL-25749"
+replaces: []
+superseded-by: []
+---
+
+== Summary
+
+A route carries the body as the Java type that each component and each step
happens to produce: a `GenericFile`, a
+`byte[]`, an `InputStream`, a `Map`, a `String`. Users who think of the
payload as JSON (or CSV, or XML) write their
+mappings for that shape, and the gap between the two shows up as runtime
errors.
+
+This proposal adds an opt-in *payload mode*. With `payloadMode=json` the body
inside the route is JSON data (a
+`JsonObject` or a `JsonArray`), whatever the component or the step before
gave, and at the edges of the route it is
+the JSON text. The payload mode is an SPI: JSON is the first plugin, and other
text and binary formats (CSV, XML, CBOR,
+Avro, ...) can follow as plugins.
+
+The same route with both modes, as a visual route editor could show it with
the body types of the catalog:
+
+image::payload-mode-route-diagram.svg[The same five-step route twice. With
payloadMode default the body is a GenericFile, then a LinkedHashMap, then an
InputStream, and the groovy mapping, the http call and the header sent to JMS
are marked with warnings. With payloadMode json every arrow carries JSON data
(JsonObject), and the callouts say the file is parsed, the http call gets the
JSON text, and the header is sent as JSON text.]
+
+== Motivation
+
+Low-code users, and AI models writing routes, think of the payload as JSON and
write data mappings in Groovy and simple
+accordingly. Camel carries the native Java type of each component instead, and
that type changes during routing and
+from component to component. Examples:
+
+- Groovy `body.find { it.sku == headers.sku }` on JSON that was not
unmarshalled fails with
+ `No such property: sku for class: java.lang.Byte`.
+- simple `${body[amount]}` on the body of the file consumer fails with
+ `Key: amount not found in bean: GenericFile[...]`.
+- A mapping that returns a `Map`, sent to an http producer, fails: there is no
type converter from `LinkedHashMap` to
+ `InputStream`.
+- A jsonpath result stored in a header as a `LinkedList` disappears when the
message crosses a JMS broker.
+
+Raymond Meester described the same from the Fluxygen low-code platform built
on Camel, where users write Groovy and
+simple without knowing Java (Zulip thread "Text-orientated vs.
Object-orientated in integration"). In the local-model
+benchmark, 19 of 156 attempts in one series failed on such a body type
collision.
+
+The hints of CAMEL-25330, CAMEL-25746 and CAMEL-25749 now say what to do in
each of these cases ("unmarshal it
+first"), but nothing does it: a user still has to know which step gives which
Java type, and where to unmarshal and
+marshal.
+
+== Goals
+
+- A route can opt in to work on the payload as data of one format: JSON first.
+- Inside such a route, the body has one predictable shape: Groovy `body.sku`,
simple `${body.sku}` and
+ `${body[sku]}`, split and aggregate work on it without unmarshal and marshal
steps.
+- At the edges (producers, header values, what users see in logs and tooling)
the payload is its text, or its bytes
+ for a binary format.
+- Formats are plugins of one SPI, so CSV, XML and binary formats can be added
without changing the engine.
+- Routes that do not opt in behave exactly as today.
+
+== Non-goals
+
+- Changing the default behaviour of Camel. The mode is opt-in, per
`CamelContext` and per route.
+- The lowest overhead. The mode parses and writes payloads that a hand-written
route might pass through untouched; it
+ is for routes where predictability matters more than raw speed.
+- Mapping arbitrary Java objects (SDK objects, entities) to JSON. They stay as
they are; a JSON writer SPI can come
+ later.
+- Schemas and validation of the payload.
+
+== Current design
+
+- The body is whatever the last step set. Consumers and producers document
their body types in the catalog
+ (CAMEL-24961), and the validator follows them through a route (CAMEL-24844,
CAMEL-25928), but at runtime nothing
+ normalizes the body.
+- Reading fields needs an `unmarshal` step first. When it is missing, the
Groovy language, the bean OGNL of simple,
+ jsonpath, jq and several producers say so in their error (`BodyFacts`,
CAMEL-25749).
+- Some JSON support already exists in core without Jackson: camel-util-json
(`JsonObject` extends `LinkedHashMap`,
+ `JsonArray` extends `ArrayList`, `Jsoner` parses and writes), type
converters from a `String`, `byte[]`,
+ `InputStream`, `File` and `WrappedFile` to `JsonObject` / `JsonArray` in
camel-support (`JsonConverter`; there are
+ none in the other direction), and the `${simpleJsonpath(...)}` function of
simple.
+- Stream caching is the precedent for an opt-in mode that changes the body: it
is enabled per `CamelContext` and per
+ route (`RuntimeConfiguration#setStreamCaching`, the `streamCache` option of
a route), and works through an advice
+ where the consumer hands the message to the route (`CamelInternalProcessor`)
and around every step
+ (`DefaultChannel`).
+
+== Proposal
+
+=== The option
+
+`payloadMode` is a new option of the `CamelContext` and of a route, like
stream caching:
+
+- `camel.main.payloadMode = json` for all the routes, and a route can set its
own `payloadMode` to opt in or out.
+- The value is the name of a payload mode plugin. `default` (the default)
means no payload mode: today's behaviour.
+- A route has one payload mode. The value is a single name, so a list
(`json,csv`) can be added later without
+ breaking it.
+
+=== The SPI
+
+A payload mode is a plugin found by its name, like data formats and languages
+(`META-INF/services/org/apache/camel/payload-mode/<name>`), or bound in the
registry. It is marked preview in its first
+release, so the interface can still change when the second plugin arrives.
+
+[source,java]
+----
+public interface PayloadMode extends Service, CamelContextAware {
+
+ /** The name, such as json, csv, cbor. */
+ String getName();
+
+ /**
+ * The content types of the payloads of this mode, such as
application/cbor and application/x-cbor. The first is
+ * the one this mode writes.
+ */
+ List<String> getContentTypes();
+
+ /** Whether the value is data of this mode (for json: a JsonObject or a
JsonArray). */
+ boolean isData(Object value);
+
+ /** Whether the body is a payload of this mode, which toData turns into
data. */
+ boolean isPayload(Exchange exchange, Object body);
+
+ /**
+ * The payload as data of this mode. Only called when isPayload is true;
throws when the payload does not parse.
+ */
+ Object toData(Exchange exchange, Object body) throws Exception;
+
+ /** A copy of the data, so the branches of an EIP that copies the exchange
do not share it. */
+ Object copy(Object data);
+
+ /** The wire form of the data: a String for a text format, a byte[] for a
binary one. */
+ Object toPayload(Exchange exchange, Object data);
+
+ /** The data as readable text, for logs, tracing, tooling and header
values. */
+ String toDisplay(Exchange exchange, Object data);
+}
+----
+
+A payload mode is a service with the lifecycle of a data format: it is started
with the routes that use it and
+stopped with them, and it can have options (an Avro or Protobuf schema),
configured like a data format. A plugin with
+nothing to start, such as json, gets the lifecycle from the base class.
+
+A base class in camel-support holds the rule that is the same for every
format, so a plugin declares its content
+types and file extensions, and brings its parser, its writer and its copy:
+
+- `isPayload` reads the `Content-Type` header (and its `charset`), the file
name through `MimeTypeHelper` when there is
+ no `Content-Type`, and a sniff of the content that a plugin can override
(the first character for a text format,
+ magic bytes for a binary one)
+- the engine calls `toData` only when `isPayload` is true, so there is no "not
mine" result to check. When it throws,
+ the engine leaves the body as it is and logs at DEBUG: the mode never fails
a message
+- a stream that stream caching has spooled to disk is not a payload: the spool
threshold is the size limit, so the
+ mode has no size option of its own
+
+=== The engine
+
+The engine part does not depend on the format.
+
+*Which bodies become data.* Where the consumer hands the message to the route,
and after every step, the body is
+given to the route's mode (where stream caching works today): when it `isData`
already it is left alone, and when it
+`isPayload` it is turned into data with `toData`. So one rule covers the body
of a consumer, the reply of a producer and the result of
+any step, including a language, `process`, `bean`, split and aggregate, and a
route that stays in its data costs a
+type check per step.
+
+The steps of the error paths are steps too: `onException`, `doTry` / `doCatch`
/ `doFinally` and the error handler
+have a channel like any other step, so the hook runs there as well. A body
that is not a payload, such as an exception
+or an error text, is left as it is.
+
+*Copies of the exchange.* Multicast, recipient list, wire tap and split give
each branch a copy of the exchange, and
+the copies would share the same mutable data, so a branch that changes a field
would change it for the others. Where
+these EIPs copy a `StreamCache` body for each branch today
(`MulticastProcessor`, `WireTapProcessor`), they call
+`copy` of the payload mode for data, so each branch gets its own.
+
+*The edges.* Producers get the payload through the type converter, not by
replacing the body before them:
+
+- a fallback type converter gives `toPayload` (and `toDisplay` for a `String`
from a binary format) when the target is
+ a `String`, `byte[]` or `InputStream` and the value is data of the
exchange's payload mode. So http, file, kafka and
+ the other producers that read text, bytes or a stream get the payload.
+- producers that take a `Map` as their API use the data: sql named parameters
(`:#sku`, `SqlHelper` converts the body
+ to a `Map`), mongodb documents.
+- producers that would turn a `Map` into something that is not the payload are
fixed to ask the payload mode first:
+ JMS and SJMS send a `Map` as a `MapMessage` (`JmsBinding`), and send data of
a payload mode as a `TextMessage`
+ (`BytesMessage` for a binary mode). Other producers of this kind are found
by a survey of the components.
+- before a remote producer (`Endpoint#isRemote()`), header values that are
data of the mode, a `Map` or a `List`
+ become their `toDisplay` text (not the `Camel*` headers, some of which are
maps on purpose), so they survive a JMS
+ or Kafka hop; and the `Content-Type` header is set to the first of the
mode's content types unless it is already
+ one of them.
+- local endpoints (`direct`, `seda`) pass the data as it is to the next route.
+
+*What users see.* `MessageHelper` (the body in logs, tracing, the backlog
tracer and the dev consoles, so also Camel
+TUI and the MCP tools) shows `toDisplay` of data of a payload mode, so users
see the JSON text, not
+`{sku=CAMEL-MUG, qty=3}`.
+
+=== Type converters, data formats and body types
+
+A payload mode is a stage of its own, not a type converter, and it does not
replace the type converters or the data
+formats:
+
+- Inside the route the engine hook turns payloads into data, at the spots of
the stream caching advice. A type
+ converter only runs when a step asks for a type, so it cannot keep the body
as data from one step to the next; the
+ hook can.
+- At the edges the mode adds one fallback type converter, from data of the
mode to a `String`, `byte[]` or
+ `InputStream` (see the edges above). It only acts on exchanges of a route
with a payload mode, and the other type
+ converters are left as they are.
+- A conversion inside the route uses the type converters as today:
`convertBodyTo`, `${bodyAs(...)}` and the
+ parameter binding of a bean. Data of the json mode is a `Map` or a `List`,
so a conversion to `Map` gives it as it
+ is and a conversion to `String` gives the JSON text; a conversion to another
type goes through the type converters
Review Comment:
Good point, and it led to a better answer: instead of a fallback converter
of the json mode, `JsonObject` and `JsonArray` now get regular type converters
to their JSON text (`String`, `byte[]`, `InputStream`) in camel-support, in
every route: [CAMEL-26324](https://issues.apache.org/jira/browse/CAMEL-26324),
#28520. Today they give the `toString` of the map, which is not JSON, and no
bytes or stream at all.
In 106b1404d24c the bullet now says the `String` conversion goes through
those converters (not the `toString` of a `Map`); the edges and the
compatibility section use them; the fallback converter stays only for a plugin
whose data type has no regular converters. There is a new open question:
regular converters run before fallback ones, so a binary plugin (CBOR) that
shares the JSON data model needs either the converter to hand the conversion to
the route's mode, or a data type of its own.
_Claude Code on behalf of davsclaus_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
