This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new d4111cefbbd4 CAMEL-25324: camel-dataweave - fix common DataWeave
construct conversion bugs
d4111cefbbd4 is described below
commit d4111cefbbd476e269e003274608299ed1d6e017
Author: Guillaume Nodet <[email protected]>
AuthorDate: Thu Oct 8 22:10:06 2026 +0200
CAMEL-25324: camel-dataweave - fix common DataWeave construct conversion
bugs
Fix conversion bugs across the DataWeave lexer, parser and converter.
The lexer now recognizes @, ? and $$ instead of dropping or splitting
them. The parser keeps header var/fun declarations, skips type
annotations on parameters, and handles attribute access (.@), existence
checks (?) and the $$ reduce accumulator. The converter no longer
double-escapes quotes, maps $(..) string interpolation to DataSonnet
%(..), emits c.multiValue for multi-value selectors (*), stringifies
groupBy keys, and uses block comments for TODO markers, with */ escaped
so it can't break the generated Jsonnet.
Closes #27535
Co-authored-by: Claude Sonnet 4.5 <[email protected]>
---
.../camel/catalog/docs/datasonnet-language.adoc | 9 +-
.../src/main/docs/datasonnet-language.adoc | 9 +-
.../camel/component/dataweave/DataWeaveAst.java | 12 +
.../component/dataweave/DataWeaveConverter.java | 179 ++++++++++++++-
.../camel/component/dataweave/DataWeaveLexer.java | 16 +-
.../camel/component/dataweave/DataWeaveParser.java | 151 ++++++++++++-
.../dataweave/DataWeaveConverterTest.java | 246 +++++++++++++++++++++
.../component/dataweave/DataWeaveLexerTest.java | 36 ++-
8 files changed, 636 insertions(+), 22 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc
index 5734f50880d5..539b17aa9027 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/datasonnet-language.adoc
@@ -518,11 +518,16 @@ The transpiler automatically maps DataWeave constructs to
their DataSonnet equiv
- `payload` to `body`, `vars` to `cml.variable()`, `attributes.headers` to
`cml.header()`
- Operators: `++` to `+`, `default` to `cml.defaultVal()`, `as
Number/String/Boolean` to `cml.toInteger()/std.toString()/cml.toBoolean()`
-- Collection operations: `map`, `filter`, `reduce`, `flatMap`, `groupBy`,
`orderBy`, `distinctBy`
+- Collection operations: `map`, `filter`, `reduce` (including `$$`/`$`
shorthand form), `flatMap`, `groupBy`, `orderBy`, `distinctBy`
- String operations: `contains`, `startsWith`, `endsWith`, `splitBy`,
`joinBy`, `replace`
- Functions: `sizeOf`, `upper`, `lower`, `trim`, `now`, `uuid`, `abs`,
`round`, `sqrt`
+- XML selectors: multi-value selector `.*field` (maps all `field` values from
a collection), attribute access `.@attr` (maps to `["@attr"]`)
+- Key-present selector `expr?` (maps to `std.objectHas()` for field and
attribute expressions)
+- Header `var` and `fun` declarations above `---` are converted to Jsonnet
`local` bindings
+- Type annotations (`: Type`, including generics and union types) in function
parameters are stripped
+- String literals from single-quoted DataWeave strings are properly re-escaped
for Jsonnet
-Constructs that cannot be automatically converted (such as `match`
expressions) are emitted as `null` for manual review.
+Constructs that cannot be automatically converted (such as `match`
expressions, `mapObject`, `pluck`, and `..` descendant selectors) are emitted
as a TODO comment and `null` for manual review.
== Dependencies
diff --git a/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc
b/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc
index 5734f50880d5..539b17aa9027 100644
--- a/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc
+++ b/components/camel-datasonnet/src/main/docs/datasonnet-language.adoc
@@ -518,11 +518,16 @@ The transpiler automatically maps DataWeave constructs to
their DataSonnet equiv
- `payload` to `body`, `vars` to `cml.variable()`, `attributes.headers` to
`cml.header()`
- Operators: `++` to `+`, `default` to `cml.defaultVal()`, `as
Number/String/Boolean` to `cml.toInteger()/std.toString()/cml.toBoolean()`
-- Collection operations: `map`, `filter`, `reduce`, `flatMap`, `groupBy`,
`orderBy`, `distinctBy`
+- Collection operations: `map`, `filter`, `reduce` (including `$$`/`$`
shorthand form), `flatMap`, `groupBy`, `orderBy`, `distinctBy`
- String operations: `contains`, `startsWith`, `endsWith`, `splitBy`,
`joinBy`, `replace`
- Functions: `sizeOf`, `upper`, `lower`, `trim`, `now`, `uuid`, `abs`,
`round`, `sqrt`
+- XML selectors: multi-value selector `.*field` (maps all `field` values from
a collection), attribute access `.@attr` (maps to `["@attr"]`)
+- Key-present selector `expr?` (maps to `std.objectHas()` for field and
attribute expressions)
+- Header `var` and `fun` declarations above `---` are converted to Jsonnet
`local` bindings
+- Type annotations (`: Type`, including generics and union types) in function
parameters are stripped
+- String literals from single-quoted DataWeave strings are properly re-escaped
for Jsonnet
-Constructs that cannot be automatically converted (such as `match`
expressions) are emitted as `null` for manual review.
+Constructs that cannot be automatically converted (such as `match`
expressions, `mapObject`, `pluck`, and `..` descendant selectors) are emitted
as a TODO comment and `null` for manual review.
== Dependencies
diff --git
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java
index f1ef34b9b40b..cac36093e0ca 100644
---
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java
+++
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveAst.java
@@ -58,6 +58,18 @@ public sealed interface DataWeaveAst {
record MultiValueSelector(DataWeaveAst object, String field) implements
DataWeaveAst {
}
+ // XML attribute access: expr.@attrName
+ record AttributeAccess(DataWeaveAst object, String attribute) implements
DataWeaveAst {
+ }
+
+ // Existence check: expr?
+ record ExistenceCheck(DataWeaveAst expr) implements DataWeaveAst {
+ }
+
+ // $$ (reduce accumulator) reference inside lambda body
+ record DoubleDollar() implements DataWeaveAst {
+ }
+
record ObjectLit(List<ObjectEntry> entries) implements DataWeaveAst {
}
diff --git
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java
index 6bc12820c8ea..5545deaa13c9 100644
---
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java
+++
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveConverter.java
@@ -149,6 +149,17 @@ public class DataWeaveConverter {
return emitNode(ia.object()) + "[" + emitNode(ia.index()) + "]";
} else if (node instanceof MultiValueSelector mv) {
return emitMultiValueSelector(mv);
+ } else if (node instanceof AttributeAccess aa) {
+ return emitAttributeAccess(aa);
+ } else if (node instanceof ExistenceCheck ec) {
+ return emitExistenceCheck(ec);
+ } else if (node instanceof DoubleDollar) {
+ // $$ means accumulator only inside reduce. In
map/filter/mapObject/pluck it means index or key.
+ // Outside the reduce-shorthand emitter this context is unknown,
so emit a TODO.
+ todoCount++;
+ return includeComments
+ ? "/* TODO: manual conversion needed -- $$ outside reduce
context */" + "\nnull"
+ : "null";
} else if (node instanceof ObjectLit obj) {
return emitObjectLit(obj);
} else if (node instanceof ArrayLit arr) {
@@ -214,8 +225,24 @@ public class DataWeaveConverter {
}
private String emitStringLit(StringLit s) {
- // The lexer preserves escape sequences as-is, so don't double-escape
- return "\"" + s.value().replace("\"", "\\\"") + "\"";
+ String value = s.value();
+ // Jsonnet has no in-string interpolation. DataWeave uses "Hello
$(expr)" but there is
+ // no faithful automated conversion (the sub-expression must be
re-emitted through the AST).
+ // Emit as a TODO comment and a placeholder so the converter output is
syntactically valid.
+ if (value.contains("$(")) {
+ todoCount++;
+ String escapedInterp = value.replaceAll("(?<!\\\\)\"", "\\\\\"");
+ return includeComments
+ ? "/* TODO: manual conversion needed -- string
interpolation: \"" + value.replace("*/", "* /") + "\"*/\n\""
+ + escapedInterp + "\""
+ : "\"" + escapedInterp + "\"";
+ }
+ // The lexer preserves escape sequences verbatim (e.g. \" is stored as
\").
+ // When the source was a single-quoted DataWeave string, bare
double-quote characters
+ // in the value would make the output invalid Jsonnet. Re-escape any
unescaped " chars.
+ // Strategy: replace every " that is NOT already preceded by a
backslash with \".
+ String escaped = value.replaceAll("(?<!\\\\)\"", "\\\\\"");
+ return "\"" + escaped + "\"";
}
private String emitIdentifier(Identifier id) {
@@ -254,9 +281,36 @@ public class DataWeaveConverter {
private String emitMultiValueSelector(MultiValueSelector mv) {
String collection = emitNode(mv.object());
+ // DataWeave .*field collects all values for key 'field' from each
element of the collection.
+ // In DataSonnet/Jsonnet: std.map(function(x) x.<field>, collection).
+ // Note: camel.libsonnet has no multiValue helper, so we emit directly.
return "std.map(function(x) x." + mv.field() + ", " + collection + ")";
}
+ private String emitAttributeAccess(AttributeAccess aa) {
+ // DataWeave .@attr accesses an XML attribute; in DataSonnet XML
attributes are exposed
+ // as object keys prefixed with '@', e.g. body.Order['@id'].
+ return emitNode(aa.object()) + "[\"@" + aa.attribute() + "\"]";
+ }
+
+ private String emitExistenceCheck(ExistenceCheck ec) {
+ // DataWeave expr? (key-present selector) returns true/false.
+ // In Jsonnet, accessing a missing field raises an error, so we cannot
simply wrap the expression.
+ // For FieldAccess and AttributeAccess we can use std.objectHas(obj,
"key") safely.
+ // For other shapes the semantics cannot be faithfully reproduced
without a helper; emit as TODO.
+ if (ec.expr() instanceof FieldAccess fa) {
+ return "std.objectHas(" + emitNode(fa.object()) + ", \"" +
fa.field() + "\")";
+ }
+ if (ec.expr() instanceof AttributeAccess aa) {
+ return "std.objectHas(" + emitNode(aa.object()) + ", \"@" +
aa.attribute() + "\")";
+ }
+ todoCount++;
+ return includeComments
+ ? "/* TODO: manual conversion needed -- existence check on
non-field expression: "
+ + emitNode(ec.expr()).replace("*/", "* /") + " */\nfalse"
+ : "false";
+ }
+
private String emitObjectLit(ObjectLit obj) {
if (obj.entries().isEmpty()) {
return "{}";
@@ -352,7 +406,7 @@ public class DataWeaveConverter {
case "Boolean" -> "cml.toBoolean(" + expr + ")";
default -> {
todoCount++;
- yield expr + (includeComments ? " // TODO: manual conversion
needed -- as " + tc.type() : "");
+ yield expr + (includeComments ? " /* TODO: manual conversion
needed -- as " + tc.type() + " */" : "");
}
};
}
@@ -410,9 +464,9 @@ public class DataWeaveConverter {
yield "c.max(" + argStr + ")";
}
case "read" -> "std.parseJson(" + argStr + ")"
- + (includeComments ? " // NOTE: assumes JSON input
-- DW read() supports multiple formats" : "");
+ + (includeComments ? " /* NOTE: assumes JSON input
-- DW read() supports multiple formats */" : "");
case "write" -> "std.manifestJsonEx(" + argStr + ", \" \")"
- + (includeComments ? " // NOTE: outputs JSON -- DW
write() supports multiple formats" : "");
+ + (includeComments ? " /* NOTE: outputs JSON -- DW
write() supports multiple formats */" : "");
default -> fc.name() + "(" + argStr + ")";
};
}
@@ -479,7 +533,110 @@ public class DataWeaveConverter {
+ collection + ", " + init + ")";
}
}
- return "std.foldl(" + emitNode(re.lambda()) + ", " + collection + ",
null)";
+ // Shorthand form: payload.items reduce ($$ + $.price)
+ // $$ is the accumulator ($$ -> acc) and $ is the current item ($ ->
item).
+ // DataWeave shorthand without an explicit initial value uses the
first element
+ // as the starting accumulator: std.foldl(function(acc, item) body,
arr[1:], arr[0]).
+ String body = emitReduceShorthandBody(re.lambda());
+ return "(local _arr = " + collection + ";\n"
+ + "if std.length(_arr) == 0 then null else
std.foldl(function(acc, item) " + body + ", _arr[1:], _arr[0]))";
+ }
+
+ /**
+ * Emit a reduce shorthand body, rewriting {@code $$} to {@code acc} and
{@code $} (optionally with field access) to
+ * {@code item} or {@code item.field}. Falls back to normal {@code
emitNode} for any sub-expression that doesn't
+ * contain shorthand references.
+ */
+ private String emitReduceShorthandBody(DataWeaveAst node) {
+ if (node instanceof DoubleDollar) {
+ return "acc";
+ }
+ if (node instanceof LambdaShorthand ls) {
+ if (ls.fields().isEmpty()) {
+ return "item";
+ }
+ return "item." + String.join(".", ls.fields());
+ }
+ if (node instanceof BinaryOp op) {
+ String left = emitReduceShorthandBody(op.left());
+ String right = emitReduceShorthandBody(op.right());
+ return switch (op.op()) {
+ case "++" -> left + " + " + right;
+ case "and" -> left + " && " + right;
+ case "or" -> left + " || " + right;
+ default -> left + " " + op.op() + " " + right;
+ };
+ }
+ if (node instanceof Parens p) {
+ return "(" + emitReduceShorthandBody(p.expr()) + ")";
+ }
+ if (node instanceof FieldAccess fa) {
+ return emitReduceShorthandBody(fa.object()) + "." + fa.field();
+ }
+ if (node instanceof IndexAccess ia) {
+ return emitReduceShorthandBody(ia.object()) + "[" +
emitReduceShorthandBody(ia.index()) + "]";
+ }
+ if (node instanceof UnaryOp op) {
+ return switch (op.op()) {
+ case "not" -> "!" + emitReduceShorthandBody(op.operand());
+ default -> op.op() + emitReduceShorthandBody(op.operand());
+ };
+ }
+ // For anything else: if the sub-expression contains a shorthand
reference ($ or $$)
+ // that emitNode cannot rewrite, emit a TODO to avoid silently
producing wrong code.
+ // Pure literals and identifiers without shorthand references are safe
to emit normally.
+ if (containsShorthand(node)) {
+ todoCount++;
+ return includeComments
+ ? "/* TODO: manual conversion needed -- reduce shorthand
in unsupported context: "
+ + node.getClass().getSimpleName() + " */\nnull"
+ : "null";
+ }
+ return emitNode(node);
+ }
+
+ /**
+ * Returns true if the given AST node or any of its children contain a
LambdaShorthand ($) or DoubleDollar ($$) that
+ * would be emitted incorrectly by the normal emitNode path in a reduce
shorthand context.
+ */
+ private boolean containsShorthand(DataWeaveAst node) {
+ if (node == null) {
+ return false;
+ }
+ if (node instanceof LambdaShorthand || node instanceof DoubleDollar) {
+ return true;
+ }
+ if (node instanceof BinaryOp op) {
+ return containsShorthand(op.left()) ||
containsShorthand(op.right());
+ }
+ if (node instanceof UnaryOp op) {
+ return containsShorthand(op.operand());
+ }
+ if (node instanceof Parens p) {
+ return containsShorthand(p.expr());
+ }
+ if (node instanceof FieldAccess fa) {
+ return containsShorthand(fa.object());
+ }
+ if (node instanceof FunctionCall fc) {
+ return fc.args().stream().anyMatch(this::containsShorthand);
+ }
+ if (node instanceof DefaultExpr def) {
+ return containsShorthand(def.expr()) ||
containsShorthand(def.fallback());
+ }
+ if (node instanceof IfElse ie) {
+ return containsShorthand(ie.condition()) ||
containsShorthand(ie.thenExpr()) || containsShorthand(ie.elseExpr());
+ }
+ if (node instanceof IndexAccess ia) {
+ return containsShorthand(ia.object()) ||
containsShorthand(ia.index());
+ }
+ // Unknown node type: default to true (conservative — prevents silent
wrong output).
+ // Only return false for nodes known to never contain shorthand
(literals, Identifier).
+ if (node instanceof NumberLit || node instanceof StringLit || node
instanceof BooleanLit
+ || node instanceof NullLit || node instanceof Identifier) {
+ return false;
+ }
+ return true;
}
private String emitFlatMap(FlatMapExpr fme) {
@@ -510,9 +667,12 @@ public class DataWeaveConverter {
if (gbe.lambda() instanceof Lambda lam) {
List<String> paramNames = lambdaParamNames(lam);
String body = emitNode(lam.body());
- return "c.groupBy(" + collection + ", function(" +
paramNames.get(0) + ") " + body + ")";
+ // Jsonnet object keys must be strings. DataWeave allows any type
as a groupBy key,
+ // so we wrap with std.toString() unconditionally to ensure valid
Jsonnet output.
+ // If the key expression is already a string, std.toString() is a
no-op.
+ return "c.groupBy(" + collection + ", function(" +
paramNames.get(0) + ") std.toString(" + body + "))";
}
- return "c.groupBy(" + collection + ", " + emitNode(gbe.lambda()) + ")";
+ return "c.groupBy(" + collection + ", function(x) std.toString((" +
emitNode(gbe.lambda()) + ")(x)))";
}
private String emitOrderBy(OrderByExpr obe) {
@@ -593,7 +753,8 @@ public class DataWeaveConverter {
todoCount++;
convertedCount--;
return includeComments
- ? "// TODO: manual conversion needed -- " + u.reason() + ": "
+ u.originalText() + "\nnull"
+ ? "/* TODO: manual conversion needed -- " +
u.reason().replace("*/", "* /") + ": "
+ + u.originalText().replace("*/", "* /") + " */\nnull"
: "null";
}
diff --git
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java
index 703424acaeb6..73f89c93aeab 100644
---
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java
+++
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveLexer.java
@@ -61,9 +61,13 @@ public class DataWeaveLexer {
LBRACKET,
RBRACKET,
DOLLAR,
+ DOLLAR_DOLLAR, // $$ (reduce accumulator shorthand)
+ AT, // @ (XML attribute selector)
+ QUESTION, // ? (existence check / key filter)
// Special
HEADER_SEPARATOR, // ---
PERCENT, // %
+ PIPE, // | (type union separator)
TILDE, // ~
EOF
}
@@ -303,7 +307,17 @@ public class DataWeaveLexer {
case '}' -> new Token(TokenType.RBRACE, "}", startLine, startCol);
case '[' -> new Token(TokenType.LBRACKET, "[", startLine,
startCol);
case ']' -> new Token(TokenType.RBRACKET, "]", startLine,
startCol);
- case '$' -> new Token(TokenType.DOLLAR, "$", startLine, startCol);
+ case '$' -> {
+ // Check for $$ (reduce accumulator shorthand)
+ if (pos < input.length() && input.charAt(pos) == '$') {
+ advance();
+ yield new Token(TokenType.DOLLAR_DOLLAR, "$$", startLine,
startCol);
+ }
+ yield new Token(TokenType.DOLLAR, "$", startLine, startCol);
+ }
+ case '@' -> new Token(TokenType.AT, "@", startLine, startCol);
+ case '?' -> new Token(TokenType.QUESTION, "?", startLine,
startCol);
+ case '|' -> new Token(TokenType.PIPE, "|", startLine, startCol);
case '%' -> new Token(TokenType.PERCENT, "%", startLine, startCol);
case '~' -> {
if (pos < input.length() && input.charAt(pos) == '=') {
diff --git
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java
index e91187d23f17..b0e7ea48e8df 100644
---
a/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java
+++
b/components/camel-dataweave/src/main/java/org/apache/camel/component/dataweave/DataWeaveParser.java
@@ -36,9 +36,13 @@ public class DataWeaveParser {
}
public DataWeaveAst parse() {
- DataWeaveAst.Header header = parseHeader();
+ HeaderResult hr = parseHeader();
DataWeaveAst body = parseExpression();
- return new DataWeaveAst.Script(header, body);
+ // Wrap header var/fun declarations around the body if present
+ if (!hr.declarations().isEmpty()) {
+ body = new DataWeaveAst.Block(hr.declarations(), body);
+ }
+ return new DataWeaveAst.Script(hr.header(), body);
}
public DataWeaveAst parseExpressionOnly() {
@@ -47,10 +51,15 @@ public class DataWeaveParser {
// -- Header parsing --
- private DataWeaveAst.Header parseHeader() {
+ /** Internal record carrying both the header directives and any var/fun
declarations found in the header section. */
+ private record HeaderResult(DataWeaveAst.Header header, List<DataWeaveAst>
declarations) {
+ }
+
+ private HeaderResult parseHeader() {
String version = "2.0";
String outputType = null;
List<DataWeaveAst.InputDecl> inputs = new ArrayList<>();
+ List<DataWeaveAst> declarations = new ArrayList<>();
// Only parse header if it starts with %dw or a known header directive
boolean hasHeader = (check(TokenType.PERCENT) && peekAhead(1) != null
&& "dw".equals(peekAhead(1).value()))
@@ -58,7 +67,7 @@ public class DataWeaveParser {
if (!hasHeader) {
// No header section -- skip directly to body
- return new DataWeaveAst.Header(version, null, inputs);
+ return new HeaderResult(new DataWeaveAst.Header(version, null,
inputs), declarations);
}
// Check for %dw directive
@@ -85,9 +94,52 @@ public class DataWeaveParser {
} else if (checkIdentifier("import")) {
// Skip import directives
while (!check(TokenType.EOF) && !checkIdentifier("output") &&
!checkIdentifier("input")
+ && !checkIdentifier("var") && !checkIdentifier("fun")
&& !check(TokenType.HEADER_SEPARATOR)) {
advance();
}
+ } else if (checkIdentifier("var")) {
+ // var declaration in header: emit as local binding in body
+ advance(); // var
+ String name = current().value();
+ advance(); // name
+ // Skip optional type annotation: var rate: Number = 0.08
+ if (check(TokenType.COLON)) {
+ advance(); // :
+ skipTypeExpression();
+ }
+ expect(TokenType.ASSIGN); // =
+ DataWeaveAst value = parseExpression();
+ declarations.add(new DataWeaveAst.VarDecl(name, value, null));
+ } else if (checkIdentifier("fun")) {
+ // fun declaration in header: emit as local function in body
+ advance(); // fun
+ String name = current().value();
+ advance(); // name
+ expect(TokenType.LPAREN);
+ List<String> params = new ArrayList<>();
+ while (!check(TokenType.RPAREN) && !check(TokenType.EOF)) {
+ String paramName = current().value();
+ advance();
+ // Skip type annotation: fun f(a: Number) or (a:
Array<Number>) -> skip to next param
+ if (check(TokenType.COLON)) {
+ advance(); // :
+ skipTypeExpression(); // type expression (simple,
generic, or union)
+ }
+ params.add(paramName);
+ if (check(TokenType.COMMA)) {
+ advance();
+ }
+ }
+ expect(TokenType.RPAREN);
+ // Optional return type annotation: fun f(a): Number = ...
+ if (check(TokenType.COLON)) {
+ advance(); // :
+ skipTypeExpression(); // return type
+ }
+ expect(TokenType.ASSIGN); // =
+ DataWeaveAst funBody = parseExpression();
+ declarations.add(new DataWeaveAst.FunDecl(name, params,
funBody, null));
} else {
advance(); // skip unknown header tokens
}
@@ -97,7 +149,7 @@ public class DataWeaveParser {
advance(); // ---
}
- return new DataWeaveAst.Header(version, outputType, inputs);
+ return new HeaderResult(new DataWeaveAst.Header(version, outputType,
inputs), declarations);
}
private String parseMediaType() {
@@ -159,13 +211,24 @@ public class DataWeaveParser {
expect(TokenType.LPAREN);
List<String> params = new ArrayList<>();
while (!check(TokenType.RPAREN) && !check(TokenType.EOF)) {
- params.add(current().value());
+ String paramName = current().value();
advance();
+ // Skip type annotation: fun f(a: Number) or (a: Array<Number>) ->
skip to next param
+ if (check(TokenType.COLON)) {
+ advance(); // :
+ skipTypeExpression(); // type expression (simple, generic, or
union)
+ }
+ params.add(paramName);
if (check(TokenType.COMMA)) {
advance();
}
}
expect(TokenType.RPAREN);
+ // Optional return type annotation: fun f(a): Number = ...
+ if (check(TokenType.COLON)) {
+ advance(); // :
+ skipTypeExpression(); // return type
+ }
expect(TokenType.ASSIGN); // =
DataWeaveAst funBody = parseExpression();
DataWeaveAst next = null;
@@ -353,11 +416,21 @@ public class DataWeaveParser {
String field = current().value();
advance();
expr = new DataWeaveAst.MultiValueSelector(expr, field);
+ } else if (check(TokenType.AT)) {
+ // .@attrName — XML attribute selector
+ advance(); // @
+ String attr = current().value();
+ advance();
+ expr = new DataWeaveAst.AttributeAccess(expr, attr);
} else if (check(TokenType.IDENTIFIER)) {
String field = current().value();
advance();
expr = new DataWeaveAst.FieldAccess(expr, field);
}
+ } else if (check(TokenType.QUESTION)) {
+ // expr? — existence check
+ advance(); // ?
+ expr = new DataWeaveAst.ExistenceCheck(expr);
} else if (check(TokenType.LBRACKET)) {
advance(); // [
DataWeaveAst index = parseExpression();
@@ -489,6 +562,10 @@ public class DataWeaveParser {
if (check(TokenType.DOLLAR)) {
return parseDollarShorthand();
}
+ if (check(TokenType.DOLLAR_DOLLAR)) {
+ advance(); // $$
+ return new DataWeaveAst.DoubleDollar();
+ }
return parsePrimary();
}
@@ -503,8 +580,17 @@ public class DataWeaveParser {
List<DataWeaveAst.LambdaParam> params = new ArrayList<>();
while (!check(TokenType.RPAREN) && !check(TokenType.ARROW) &&
!check(TokenType.EOF)) {
+ // Only accept identifiers as parameter names; if we see something
else, this is not a lambda
+ if (!check(TokenType.IDENTIFIER)) {
+ throw new IllegalStateException("Expected identifier for
lambda parameter, got: " + current());
+ }
String paramName = current().value();
advance();
+ // Skip type annotation: (a: Number) or (a: Array<Number>) -> skip
to next param
+ if (check(TokenType.COLON)) {
+ advance(); // :
+ skipTypeExpression(); // type expression (simple, generic, or
union)
+ }
DataWeaveAst defaultValue = null;
if (check(TokenType.ASSIGN)) {
advance();
@@ -567,6 +653,11 @@ public class DataWeaveParser {
return parseDollarShorthand();
}
+ if (check(TokenType.DOLLAR_DOLLAR)) {
+ advance(); // $$
+ return new DataWeaveAst.DoubleDollar();
+ }
+
if (check(TokenType.LPAREN)) {
advance(); // (
DataWeaveAst expr = parseExpression();
@@ -719,6 +810,54 @@ public class DataWeaveParser {
// -- Token helpers --
+ /**
+ * Skip a DataWeave type expression after a colon. Handles: - Simple:
{@code Number} - Generic:
+ * {@code Array<Number>}, {@code Array<String | Null>} - Union: {@code
String | Null} - Object type: {@code {name:
+ * String}} Stops when it encounters a {@code ,}, {@code )}, {@code =} or
EOF at depth 0.
+ */
+ private void skipTypeExpression() {
+ // Skip the leading type name (or opening brace/bracket for object
types)
+ if (!check(TokenType.EOF)) {
+ if (check(TokenType.LBRACE)) {
+ // Object type: {name: String} — skip balanced braces
+ int depth = 1;
+ advance(); // {
+ while (depth > 0 && !check(TokenType.EOF)) {
+ if (check(TokenType.LBRACE)) {
+ depth++;
+ } else if (check(TokenType.RBRACE)) {
+ depth--;
+ }
+ advance();
+ }
+ return; // object type is self-contained, no trailing < or |
+ }
+ advance(); // consume base type name / '['
+ }
+ // Now handle trailing generic parameters '<...>' and union '|'
+ while (!check(TokenType.EOF)) {
+ if (check(TokenType.LT)) {
+ // Generic type params: skip balanced < ... >
+ int depth = 1;
+ advance(); // <
+ while (depth > 0 && !check(TokenType.EOF)) {
+ if (check(TokenType.LT)) {
+ depth++;
+ } else if (check(TokenType.GT)) {
+ depth--;
+ }
+ advance();
+ }
+ } else if (check(TokenType.PIPE)) {
+ // Union type: skip '|' and the next type expression
+ advance(); // |
+ skipTypeExpression();
+ } else {
+ break;
+ }
+ }
+ }
+
private Token current() {
return pos < tokens.size() ? tokens.get(pos) :
tokens.get(tokens.size() - 1);
}
diff --git
a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java
b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java
index 97441b844b19..5ea0590561a5 100644
---
a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java
+++
b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveConverterTest.java
@@ -473,6 +473,7 @@ class DataWeaveConverterTest {
String result = converter.convertExpression("payload.items.*name");
assertEquals("std.map(function(x) x.name, body.items)", result);
assertEquals(0, converter.getTodoCount());
+ assertFalse(converter.needsCamelLib());
}
// -- Escape handling --
@@ -483,8 +484,253 @@ class DataWeaveConverterTest {
assertTrue(result.contains("\"\\n\""), "Newline escape should be
preserved, got: " + result);
}
+ // -- CAMEL-25324 fixes --
+
+ @Test
+ void testDoubleQuoteNoDoubleEscape() {
+ // DW: "say \"hi\"" -- the lexer stores the backslash-quote verbatim;
do NOT double-escape on emit
+ String result = converter.convertExpression("\"say \\\"hi\\\"\"");
+ assertEquals("\"say \\\"hi\\\"\"", result);
+ }
+
+ @Test
+ void testStringInterpolation() {
+ // DW: "Hello $(payload.name)" -- Jsonnet has no string interpolation,
emit as TODO
+ String result = converter.convertExpression("\"Hello
$(payload.name)\"");
+ assertTrue(result.contains("TODO"), "String interpolation should be a
TODO, got: " + result);
+ assertTrue(converter.getTodoCount() > 0);
+ }
+
+ @Test
+ void testAttributeAccess() {
+ // DW: payload.Order.@id -> DS: body.Order["@id"]
+ String result = converter.convertExpression("payload.Order.@id");
+ assertEquals("body.Order[\"@id\"]", result);
+ }
+
+ @Test
+ void testExistenceCheck() {
+ // DW: payload.a? -> DS: std.objectHas(body, "a") -- using
std.objectHas for FieldAccess
+ String result = converter.convertExpression("payload.a?");
+ assertEquals("std.objectHas(body, \"a\")", result);
+ assertFalse(converter.needsCamelLib());
+ }
+
+ @Test
+ void testDoubleDollarInReduce() {
+ // DW: payload.items reduce ((item, acc = 0) -> acc + item.price) --
explicit lambda
+ String result = converter.convertExpression("payload.items reduce
((item, acc = 0) -> acc + item.price)");
+ assertTrue(result.contains("std.foldl"), "Should use std.foldl, got: "
+ result);
+ assertTrue(result.contains("function(acc, item)"), "acc and item
should be swapped for foldl, got: " + result);
+ }
+
+ @Test
+ void testDoubleDollarOutsideReduce() {
+ // $$ outside a reduce shorthand context is ambiguous (index in map,
key in mapObject).
+ // The converter emits a TODO rather than silently binding 'acc'.
+ String result = converter.convertExpression("$$");
+ assertTrue(result.contains("null"), "Should emit null placeholder,
got: " + result);
+ assertEquals(1, converter.getTodoCount(), "$$ outside reduce should
count as TODO");
+ }
+
+ @Test
+ void testVarDeclarationInHeader() {
+ // DW header var declarations must survive as local bindings in the
body
+ String dw = """
+ %dw 2.0
+ output application/json
+ var rate = 0.08
+ ---
+ payload.price * rate
+ """;
+ String result = converter.convert(dw);
+ assertTrue(result.contains("local rate = 0.08"), "var rate must emit
as local rate, got: " + result);
+ assertTrue(result.contains("body.price * rate"), "body expression must
reference rate, got: " + result);
+ }
+
+ @Test
+ void testFunDeclarationInHeader() {
+ // DW header fun declarations must survive as local functions in the
body
+ String dw = """
+ %dw 2.0
+ output application/json
+ fun double(x) = x * 2
+ ---
+ double(payload.value)
+ """;
+ String result = converter.convert(dw);
+ assertTrue(result.contains("local double(x) ="), "fun double must emit
as local function, got: " + result);
+ assertTrue(result.contains("double(body.value)"), "body expression
must call double, got: " + result);
+ }
+
+ @Test
+ void testTypedFunParams() {
+ // DW: fun f(a: Number): Number = a * 2 -- type annotations must be
stripped
+ String result = converter.convertExpression("fun f(a: Number) = a *
2\nf(payload.x)");
+ assertTrue(result.contains("local f(a) ="), "typed param should be
stripped, got: " + result);
+ assertFalse(result.contains("Number"), "type annotation must not
appear in output, got: " + result);
+ }
+
+ @Test
+ void testGroupByKeyStringified() {
+ // DW: payload.items groupBy ((i) -> i.qty) -- groupBy key must be
stringified
+ String result = converter.convertExpression("payload.items groupBy
((i) -> i.qty)");
+ assertTrue(result.contains("c.groupBy("), "Should use c.groupBy, got:
" + result);
+ assertTrue(result.contains("std.toString("), "groupBy key must be
stringified, got: " + result);
+ }
+
+ @Test
+ void testGroupByShorthand() {
+ // DW: payload.items groupBy $.qty -- shorthand (non-Lambda) branch
+ // The emitted lambda function must be parenthesized before being
called with (x),
+ // otherwise (x) binds to the field access rather than invoking the
whole function.
+ String result = converter.convertExpression("payload.items groupBy
$.qty");
+ assertTrue(result.contains("c.groupBy("), "Should use c.groupBy, got:
" + result);
+ assertTrue(result.contains("std.toString("), "groupBy key must be
stringified, got: " + result);
+ // The function must be called on x: (function(x) ...)(x), not
function(x) ...(x)
+ assertTrue(result.contains(")(x)"), "Emitted lambda must be
parenthesized before (x) call, got: " + result);
+ }
+
+ @Test
+ void testMultiValueSelectorXmlChildren() {
+ // DW: payload.Order.Items.*Item -> DS: std.map(function(x) x.Item,
body.Order.Items)
+ String result =
converter.convertExpression("payload.Order.Items.*Item");
+ assertEquals("std.map(function(x) x.Item, body.Order.Items)", result);
+ assertFalse(converter.needsCamelLib());
+ }
+
// -- Helpers --
+ @Test
+ void testDoubleQuotedStringWithEscapedQuote() {
+ // DW: "say \"hi\"" (double-quoted string with escaped quotes inside)
+ // The lexer stores the value with the escapes resolved: say "hi"
+ // emitStringLit must re-escape the " to produce valid Jsonnet: "say
\"hi\""
+ DataWeaveConverter c2 = new DataWeaveConverter();
+ // Pass a double-quoted DW string with escaped quotes to verify no
double-escape
+ String result = c2.convertExpression("\"say \\\"hi\\\"\"");
+ assertEquals("\"say \\\"hi\\\"\"", result, "Double-quoted string with
\\\" must not double-escape");
+ }
+
+ @Test
+ void testSingleQuotedStringWithDoubleQuote() {
+ // DW: 'say "hi"' (single-quoted string containing a bare double-quote)
+ // The lexer stores the value without surrounding quotes: say "hi"
(with bare ")
+ // emitStringLit must re-escape the bare " to produce valid Jsonnet:
"say \"hi\""
+ DataWeaveConverter c2 = new DataWeaveConverter();
+ String result = c2.convertExpression("'say \"hi\"'");
+ assertEquals("\"say \\\"hi\\\"\"", result, "Single-quoted string: bare
\" must be escaped to \\\" in output");
+ }
+
+ @Test
+ void testExistenceCheckOnAttribute() {
+ // DW: payload.Order.@id? -> DS: std.objectHas(body.Order, "@id")
+ String result = converter.convertExpression("payload.Order.@id?");
+ assertEquals("std.objectHas(body.Order, \"@id\")", result);
+ assertFalse(converter.needsCamelLib());
+ }
+
+ @Test
+ void testTypedFunParamsCompound() {
+ // DW: fun f(a: Array<Number>): Number = a[0] -- compound type
annotations must be stripped
+ String result = converter.convertExpression("fun f(a: Array<Number>) =
a[0]\nf(payload.x)");
+ assertTrue(result.contains("local f(a) ="), "typed compound param
should be stripped, got: " + result);
+ assertFalse(result.contains("Array"), "compound type annotation must
not appear in output, got: " + result);
+ assertFalse(result.contains("Number"), "type name must not appear in
output, got: " + result);
+ }
+
+ @Test
+ void testTypedFunParamsUnion() {
+ // DW: fun f(a: String | Null) = a -- union type annotation with |
must be stripped
+ // Before this fix, '|' was silently dropped by the lexer and 'Null'
leaked into the body.
+ String result = converter.convertExpression("fun f(a: String | Null) =
a\nf(payload.x)");
+ assertTrue(result.contains("local f(a) ="), "union type param should
be stripped, got: " + result);
+ // 'Null' must not appear as a null-literal emitted into the body
+ assertFalse(result.contains("null\n"), "union type token 'Null' must
not leak into body, got: " + result);
+ }
+
+ @Test
+ void testReduceShorthandAccumulatorPlusField() {
+ // DW: payload.items reduce ($$ + $.price)
+ // Shorthand: $$ = accumulator, $ = current item (optionally .field)
+ // Expected: std.foldl using first element as initial accumulator
+ String result = converter.convertExpression("payload.items reduce ($$
+ $.price)");
+ assertTrue(result.contains("std.foldl("), "Should use std.foldl, got:
" + result);
+ assertTrue(result.contains("function(acc, item)"), "Should have acc,
item params, got: " + result);
+ assertTrue(result.contains("acc + item.price"),
+ "Body should rewrite $$ -> acc and $.price -> item.price, got:
" + result);
+ assertEquals(0, converter.getTodoCount(), "Reduce shorthand should not
produce TODOs, got: " + result);
+ }
+
+ @Test
+ void testReduceShorthandConcatItems() {
+ // DW: payload.items reduce ($$ ++ $)
+ // $$ = accumulator, $ = whole item (no field access)
+ String result = converter.convertExpression("payload.items reduce ($$
++ $)");
+ assertTrue(result.contains("std.foldl("), "Should use std.foldl, got:
" + result);
+ assertTrue(result.contains("function(acc, item)"), "Should have acc,
item params, got: " + result);
+ assertTrue(result.contains("acc + item"), "Body should rewrite $$ ->
acc and $ -> item, got: " + result);
+ assertEquals(0, converter.getTodoCount(), "Reduce shorthand should not
produce TODOs, got: " + result);
+ }
+
+ @Test
+ void testReduceShorthandEmptyArrayGuard() {
+ // DW: payload.items reduce ($$ + $.price) -- the emitted foldl must
guard against empty arrays.
+ // DataWeave's reduce without an initial value on an empty list
returns null;
+ // Jsonnet's _arr[0] would throw "Array index 0 out of bounds".
+ String result = converter.convertExpression("payload.items reduce ($$
+ $.price)");
+ assertTrue(result.contains("std.length(_arr) == 0"), "Empty-array
guard missing, got: " + result);
+ assertTrue(result.contains("then null"), "Null fallback for empty
array missing, got: " + result);
+ }
+
+ @Test
+ void testReduceShorthandIndexAccess() {
+ // DW: payload.items reduce ($$ + $[0]) -- shorthand body with index
access on $
+ // containsShorthand must recurse into IndexAccess, and
emitReduceShorthandBody must handle it.
+ // Expected body: acc + item[0] (not emitNode's function(x) x[0])
+ String result = converter.convertExpression("payload.items reduce ($$
+ $[0])");
+ assertTrue(result.contains("std.foldl("), "Should use std.foldl, got:
" + result);
+ assertTrue(result.contains("item[0]"), "IndexAccess shorthand body
must emit item[0], got: " + result);
+ assertFalse(result.contains("function(x)"),
+ "emitNode fallback must not be used for shorthand IndexAccess,
got: " + result);
+ assertEquals(0, converter.getTodoCount(), "Reduce shorthand index
access should not produce TODOs, got: " + result);
+ }
+
+ @Test
+ void testHeaderVarComplexValue() {
+ // DW header var with a complex value (if/else) — parseExpression()
handles if/else,
+ // parseOr() does not. Before the fix, the if/else would be truncated.
+ String dw = """
+ %dw 2.0
+ output application/json
+ var label = if (true) "yes" else "no"
+ ---
+ label
+ """;
+ String result = converter.convert(dw);
+ assertTrue(result.contains("local label ="), "var label must emit as
local binding, got: " + result);
+ // The if/else must appear in the var value, not be cut off
+ assertTrue(result.contains("\"yes\"") && result.contains("\"no\""),
+ "Complex var value (if/else) must be fully emitted, got: " +
result);
+ }
+
+ @Test
+ void testHeaderFunComplexBody() {
+ // DW header fun with an if/else body — parseExpression() handles
if/else, parseOr() does not.
+ String dw = """
+ %dw 2.0
+ output application/json
+ fun toUpper(x) = if (x != null) upper(x) else ""
+ ---
+ toUpper(payload.name)
+ """;
+ String result = converter.convert(dw);
+ assertTrue(result.contains("local toUpper(x) ="), "fun toUpper must
emit as local function, got: " + result);
+ // The if/else body must be present in the output
+ assertTrue(result.contains("\"\"") || result.contains("else"),
+ "Complex fun body (if/else) must be fully emitted, got: " +
result);
+ }
+
private String loadResource(String path) throws IOException {
try (InputStream is =
getClass().getClassLoader().getResourceAsStream(path)) {
assertNotNull(is, "Resource not found: " + path);
diff --git
a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveLexerTest.java
b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveLexerTest.java
index 6d3b844babe1..a8cd85f0a43c 100644
---
a/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveLexerTest.java
+++
b/components/camel-dataweave/src/test/java/org/apache/camel/component/dataweave/DataWeaveLexerTest.java
@@ -145,9 +145,41 @@ class DataWeaveLexerTest {
@Test
void shouldSkipUnknownCharactersInsteadOfFailing() {
- // '@' is not a recognized token; the lexer drops it and keeps going
+ // '@' is now a recognized AT token; only truly unknown characters are
silently dropped
assertEquals(
- List.of(TokenType.IDENTIFIER, TokenType.IDENTIFIER,
TokenType.EOF),
+ List.of(TokenType.IDENTIFIER, TokenType.AT,
TokenType.IDENTIFIER, TokenType.EOF),
types("a @ b"));
}
+
+ // -- CAMEL-25324: new tokens --
+
+ @Test
+ void shouldTokenizeDoubleDollarAsDoubleDollarToken() {
+ // $$ must be a single DOLLAR_DOLLAR token, not two DOLLAR tokens
+ List<Token> tokens = tokenize("$$");
+ assertEquals(2, tokens.size()); // DOLLAR_DOLLAR + EOF
+ assertEquals(TokenType.DOLLAR_DOLLAR, tokens.get(0).type());
+ assertEquals("$$", tokens.get(0).value());
+ }
+
+ @Test
+ void shouldDistinguishSingleAndDoubleDollar() {
+ assertEquals(
+ List.of(TokenType.DOLLAR, TokenType.DOLLAR_DOLLAR,
TokenType.EOF),
+ types("$ $$"));
+ }
+
+ @Test
+ void shouldTokenizeAtAsAtToken() {
+ assertEquals(
+ List.of(TokenType.AT, TokenType.EOF),
+ types("@"));
+ }
+
+ @Test
+ void shouldTokenizeQuestionMarkAsQuestionToken() {
+ assertEquals(
+ List.of(TokenType.IDENTIFIER, TokenType.QUESTION,
TokenType.EOF),
+ types("payload?"));
+ }
}
