[
https://issues.apache.org/jira/browse/CAMEL-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25334:
--------------------------------
Description:
Low-code users, and AI models writing routes, think of the payload as JSON and
write data mappings in Groovy and simple accordingly. Camel carries the native
Java type of each component instead - a GenericFile, a byte[], an InputStream,
a Map - and that type changes during routing and from component to component.
The gap shows up as runtime errors:
* Groovy {{body.find \{ it.sku == headers.sku \}}} on JSON that was not
unmarshalled: {{No such property: sku for class: java.lang.Byte}}
* simple {{$\{body[amount]\}}} on the file consumer's body: {{Key: amount not
found in bean: GenericFile[...]}}
* a mapping that returns a Map, sent to an http producer: no type converter
from LinkedHashMap to InputStream
* a jsonpath result stored in a header as a LinkedList, which disappears when
the message crosses a JMS broker
Since then the hints of CAMEL-25330, CAMEL-25746 and CAMEL-25749 say what to do
in each of these cases, but nothing does it: on main {{$\{body.sku\}}},
{{$\{body[sku]\}}} and Groovy {{body.sku}} on JSON text all fail with
"unmarshal it first".
Raymond Meester described the same from the Fluxygen low-code platform on
Camel, where users write Groovy and simple without knowing Java:
[Text-orientated vs. Object-orientated in
integration|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/.E2.9C.94.20Text-orientated.20vs.2E.20Object-orientated.20in.20integration/with/568024059].
In the local-model benchmark, 19 of 156 attempts in one series died on such a
body type collision.
h3. Design: payloadMode
An opt-in mode where the route works on the payload as JSON, instead of the
various Java types that change during routing or from different components. It
is for low-code users and AI-built routes; classic routes keep the native body
types, and the mode is not meant for routes that want the lowest overhead.
*The switch*, like stream caching: {{payloadMode}} with the values {{default}}
(today's behaviour) and {{json}}, on the CamelContext ({{camel.main.payloadMode
= json}}) and per route, so a route can opt in or out. An enum leaves room for
other formats (XML) later.
*Inside the route the body is JSON data*: a {{JsonObject}} or {{JsonArray}} of
camel-util-json ({{JsonObject}} is a LinkedHashMap, so the field order is kept;
numbers are BigDecimal). So with no change to the languages or EIPs:
* Groovy: {{body.sku}}, {{body.items.each \{ ... \}}}, {{body.find \{ ... \}}}
work
* simple: {{$\{body.sku\}}} and {{$\{body[sku]\}}} work (the bean OGNL reads a
key of a Map)
* split of a JsonArray gives its elements, each a JsonObject; aggregating them
gives a List, which becomes a JsonArray
*Which bodies become JSON data*: the mode works where stream caching does, when
the consumer hands the message to the route and after every step, so one rule
covers the consumer's body, the reply of a producer and the result of any step:
||body||in json mode||
|Map, List (Groovy, process, sql, split, aggregate)|becomes JsonObject /
JsonArray|
|String, byte[], stream, file with a Content-Type of application/json or
*+json|parsed|
|the same without a Content-Type|parsed when the file name ends with .json
(MimeTypeHelper) or the text starts with \{ or [|
|another Content-Type (text/csv, application/xml, image/png ...)|as it is|
|JSON-looking text that does not parse ({{[INFO] started}})|as it is (logged at
DEBUG): the mode never fails a message|
|a stream that stream caching spooled to disk|as it is: the spool threshold is
the size limit, no new option|
|other Java objects (SDK objects, entities, JsonNode)|as they are (a JSON
writer SPI, for example from camel-jackson, can come later)|
A step that produces JSON text, including {{convertBodyTo: String}}, has it
parsed back into data; the text inside the route is {{$\{bodyAs(String)\}}}, or
{{body.toJson()}} in Groovy.
*At the edges it is JSON text*:
* new type converters from JsonObject / JsonArray to String, byte[] and
InputStream, so http, file, kafka and the other producers that read text, bytes
or a stream get the JSON text
* producers that take a Map as their API use the data: sql named parameters
({{:#sku}}), mongodb documents
* producers that would turn a Map into something that is not the JSON text are
fixed: JMS / SJMS send a JsonObject as a TextMessage, not a MapMessage (other
producers to be surveyed)
* header values that are a JsonObject, JsonArray, Map or List become JSON text
before a producer (not the Camel* headers), so they survive a JMS or Kafka hop
* direct / seda to another route pass the data as it is
*What users see is JSON text*: logs, tracing, the backlog tracer, the dev
consoles, Camel TUI and the MCP tools show the JSON text of the body (in
MessageHelper, or through a small display SPI).
h3. Plan
# camel-core: the payloadMode option (camel-api, camel-main, the model and the
YAML / XML DSL), the hook at the consumer and after every step with the rule
above, the converters to text, the header values before a producer, the
display, a user manual page, and tests that run the cases of the problem list
in both modes.
# components: JMS / SJMS send a JsonObject as a TextMessage, and a survey of
the producers that turn a Map into something else.
# tooling: the body flow of the validator, the catalog and Camel TUI / MCP know
payloadMode (in json mode the body of a file consumer is JSON data).
# Run the local-model benchmark with the mode off and on, and compare the body
type collisions and the pass rate.
Related: CAMEL-24844 (body type flow), CAMEL-25330 (the Groovy hint this
replaces with behaviour in json mode), CAMEL-25329 (marshal json writes JSON
text as it is).
_Claude Code on behalf of davsclaus_
was:
Low-code users, and AI models writing routes, think of the payload as JSON (or
XML, CSV, text) and write data mappings in Groovy and simple accordingly. Camel
carries the native Java type of each component instead - a GenericFile, a
byte[], an InputStream, a Map - and the gap shows up as runtime errors:
* Groovy {{body.find { it.sku == headers.sku } }} on JSON that was not
unmarshalled: {{No such property: sku for class: java.lang.Byte}}
* simple {{$\{body[amount]\}}} on the file consumer's body: {{Key: amount not
found in bean: GenericFile[...]}}
* a mapping that returns a Map, sent to an http producer: no type converter
from LinkedHashMap to InputStream
* a jsonpath result stored in a header as a LinkedList, which disappears when
the message crosses a JMS broker
Raymond Meester described the same from the Fluxygen low-code platform on
Camel, where users write Groovy and simple without knowing Java:
[Text-orientated vs. Object-orientated in
integration|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/.E2.9C.94.20Text-orientated.20vs.2E.20Object-orientated.20in.20integration/with/568024059].
In the local-model benchmark, 19 of 156 attempts in one series died on such a
body type collision.
h3. Proposal: a content-type aware mode
Not "everything is text", and not a second Camel: the payload keeps its content
type, the body stays the raw text or bytes between steps, and each language
reads it the way it expects:
||who||what it sees when the content type is JSON||
|jsonpath, jq|the raw JSON text, as today|
|Groovy|parsed data (Map / List), so {{body.sku}}, {{body.find \{ it.sku == ...
\}}}, {{body.items.each \{ ... \}}} work; the text stays available (for example
as {{bodyText}})|
|simple {{$\{body[sku]\}}}, {{$\{body.sku\}}}|the field, read from the parsed
JSON|
|producers|the text, unchanged|
|marshal json|the text, passed through (CAMEL-25329)|
The return side matters as much as the read side: a structured result (Map,
List, JsonNode) that a language stores with setBody or setHeader becomes JSON
text again, so the next producer, broker or mapping gets text and headers
survive a JMS hop.
Where the content type comes from: the Content-Type header, which HTTP
consumers and Jackson's marshal already set; for the file consumer, the file
name through MimeTypeHelper (probeContentType); otherwise the first character
of the text. Binary content stays binary, and a size threshold (as stream
caching has) keeps large payloads as streams.
Groovy needs no new dependency: camel-groovy already depends on groovy-json and
groovy-xml (JsonSlurper gives Map/List, XmlSlurper GPath for XML). Simple
already compares text numbers numerically (ObjectHelper.typeCoerceCompare), so
{{$\{header.qty\} > 5}} keeps working with text headers.
h3. Prototype and measurement
# Groovy, behind an option: a JSON body is bound as Map/List, the text as
bodyText, a Map/List result goes back to JSON text.
# simple: {{$\{body[x]\}}} and {{$\{body.x\}}} read fields from JSON text.
# Headers: a structured result from any language becomes JSON text when the
mode is on.
# Run the local-model benchmark ladder with the mode off and on and compare the
body type collisions and pass rate before deciding on the switch (one
camel.main property, or per language) and on defaults.
Classic routes keep the native body types; the mode is opt-in.
Related: CAMEL-24844 (body type flow), CAMEL-25330 (the Groovy hint this would
replace with behaviour), CAMEL-25329 (marshal passes JSON text through).
_Claude Code on behalf of davsclaus_
Summary: payloadMode=json for low-code: inside the route the body is
JSON data (JsonObject / JsonArray), at the edges JSON text (was: Content-type
aware payloads for low-code: the body stays text with its content type, Groovy
and simple read JSON fields from it, structured results go back to text)
> payloadMode=json for low-code: inside the route the body is JSON data
> (JsonObject / JsonArray), at the edges JSON text
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25334
> URL: https://issues.apache.org/jira/browse/CAMEL-25334
> Project: Camel
> Issue Type: New Feature
> Components: camel-core, camel-groovy
> Reporter: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> Low-code users, and AI models writing routes, think of the payload as JSON
> and write data mappings in Groovy and simple accordingly. Camel carries the
> native Java type of each component instead - a GenericFile, a byte[], an
> InputStream, a Map - and that type changes during routing and from component
> to component. The gap shows up as runtime errors:
> * Groovy {{body.find \{ it.sku == headers.sku \}}} on JSON that was not
> unmarshalled: {{No such property: sku for class: java.lang.Byte}}
> * simple {{$\{body[amount]\}}} on the file consumer's body: {{Key: amount not
> found in bean: GenericFile[...]}}
> * a mapping that returns a Map, sent to an http producer: no type converter
> from LinkedHashMap to InputStream
> * a jsonpath result stored in a header as a LinkedList, which disappears when
> the message crosses a JMS broker
> Since then the hints of CAMEL-25330, CAMEL-25746 and CAMEL-25749 say what to
> do in each of these cases, but nothing does it: on main {{$\{body.sku\}}},
> {{$\{body[sku]\}}} and Groovy {{body.sku}} on JSON text all fail with
> "unmarshal it first".
> Raymond Meester described the same from the Fluxygen low-code platform on
> Camel, where users write Groovy and simple without knowing Java:
> [Text-orientated vs. Object-orientated in
> integration|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/.E2.9C.94.20Text-orientated.20vs.2E.20Object-orientated.20in.20integration/with/568024059].
> In the local-model benchmark, 19 of 156 attempts in one series died on such
> a body type collision.
> h3. Design: payloadMode
> An opt-in mode where the route works on the payload as JSON, instead of the
> various Java types that change during routing or from different components.
> It is for low-code users and AI-built routes; classic routes keep the native
> body types, and the mode is not meant for routes that want the lowest
> overhead.
> *The switch*, like stream caching: {{payloadMode}} with the values
> {{default}} (today's behaviour) and {{json}}, on the CamelContext
> ({{camel.main.payloadMode = json}}) and per route, so a route can opt in or
> out. An enum leaves room for other formats (XML) later.
> *Inside the route the body is JSON data*: a {{JsonObject}} or {{JsonArray}}
> of camel-util-json ({{JsonObject}} is a LinkedHashMap, so the field order is
> kept; numbers are BigDecimal). So with no change to the languages or EIPs:
> * Groovy: {{body.sku}}, {{body.items.each \{ ... \}}}, {{body.find \{ ...
> \}}} work
> * simple: {{$\{body.sku\}}} and {{$\{body[sku]\}}} work (the bean OGNL reads
> a key of a Map)
> * split of a JsonArray gives its elements, each a JsonObject; aggregating
> them gives a List, which becomes a JsonArray
> *Which bodies become JSON data*: the mode works where stream caching does,
> when the consumer hands the message to the route and after every step, so one
> rule covers the consumer's body, the reply of a producer and the result of
> any step:
> ||body||in json mode||
> |Map, List (Groovy, process, sql, split, aggregate)|becomes JsonObject /
> JsonArray|
> |String, byte[], stream, file with a Content-Type of application/json or
> *+json|parsed|
> |the same without a Content-Type|parsed when the file name ends with .json
> (MimeTypeHelper) or the text starts with \{ or [|
> |another Content-Type (text/csv, application/xml, image/png ...)|as it is|
> |JSON-looking text that does not parse ({{[INFO] started}})|as it is (logged
> at DEBUG): the mode never fails a message|
> |a stream that stream caching spooled to disk|as it is: the spool threshold
> is the size limit, no new option|
> |other Java objects (SDK objects, entities, JsonNode)|as they are (a JSON
> writer SPI, for example from camel-jackson, can come later)|
> A step that produces JSON text, including {{convertBodyTo: String}}, has it
> parsed back into data; the text inside the route is {{$\{bodyAs(String)\}}},
> or {{body.toJson()}} in Groovy.
> *At the edges it is JSON text*:
> * new type converters from JsonObject / JsonArray to String, byte[] and
> InputStream, so http, file, kafka and the other producers that read text,
> bytes or a stream get the JSON text
> * producers that take a Map as their API use the data: sql named parameters
> ({{:#sku}}), mongodb documents
> * producers that would turn a Map into something that is not the JSON text
> are fixed: JMS / SJMS send a JsonObject as a TextMessage, not a MapMessage
> (other producers to be surveyed)
> * header values that are a JsonObject, JsonArray, Map or List become JSON
> text before a producer (not the Camel* headers), so they survive a JMS or
> Kafka hop
> * direct / seda to another route pass the data as it is
> *What users see is JSON text*: logs, tracing, the backlog tracer, the dev
> consoles, Camel TUI and the MCP tools show the JSON text of the body (in
> MessageHelper, or through a small display SPI).
> h3. Plan
> # camel-core: the payloadMode option (camel-api, camel-main, the model and
> the YAML / XML DSL), the hook at the consumer and after every step with the
> rule above, the converters to text, the header values before a producer, the
> display, a user manual page, and tests that run the cases of the problem list
> in both modes.
> # components: JMS / SJMS send a JsonObject as a TextMessage, and a survey of
> the producers that turn a Map into something else.
> # tooling: the body flow of the validator, the catalog and Camel TUI / MCP
> know payloadMode (in json mode the body of a file consumer is JSON data).
> # Run the local-model benchmark with the mode off and on, and compare the
> body type collisions and the pass rate.
> Related: CAMEL-24844 (body type flow), CAMEL-25330 (the Groovy hint this
> replaces with behaviour in json mode), CAMEL-25329 (marshal json writes JSON
> text as it is).
> _Claude Code on behalf of davsclaus_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
