davsclaus opened a new pull request, #28801: URL: https://github.com/apache/camel/pull/28801
Fixes [CAMEL-25015](https://issues.apache.org/jira/browse/CAMEL-25015). The tokenize language has a `regex` option (default `false`), but the token was always compiled as a regular expression: `ExchangeHelper.getScanner` hands it to `Scanner`, which calls `Pattern.compile`. So common delimiters that are regex metacharacters split wrongly or failed: | route | before | now | |---|---|---| | `tokenize("\|")` of `a\|b\|c` | `[a, \|, b, \|, c]` | `[a, b, c]` | | `tokenize(".")` of `a.b.c` | `[, , , ]` | `[a, b, c]` | | `tokenize("$")` of `a$b$c` | `[a$b$c]` | `[a, b, c]` | | `tokenize("*")` of `a*b*c` | `PatternSyntaxException` | `[a, b, c]` | | `tokenizePair("(", ")")` of `(a)(b)` | `PatternSyntaxException` | `[a, b]` | | `tokenizePair("[rec]", "END.")` of `[rec]1 ENDS HERE END.[rec]2END.` | `[1 ]` | `[1 ENDS HERE , 2]` | **What changes:** - `ExpressionBuilder.tokenizeExpression` quotes the token (after evaluating a Simple function in it) with `Pattern.quote`. `regexTokenizeExpression` (used for `regex=true`) is unchanged. - The pair mode quotes the end token, which replaces the escaping of a leading `[` and a trailing `]` by hand. The start token was already matched as text. - `GroupTokenIterator` unquotes the delimiter it reads back from the scanner for a token with a Simple function, so the parts are joined with the token and not with `\Q|\E`. - `TokenizerExpressionReifier`: for a token that is not a regular expression, the escapes `\n`, `\r` and `\t` are turned into the characters, so XML/YAML `token="\r\n"` (four characters, which worked only because it was compiled as a regex) keeps working. Before, only a leading `\n` was turned into a new line; that stays as is for `regex=true`. - The tokenize language page and the 4.24 upgrade guide say the token is literal text, and that a route relying on a regex token such as `\s+` must set `regex=true`. **Compatibility:** routes that use a regular expression as token without `regex=true` change. In this repository only `TokenizerTest.testTokenizeHeaderRegEx` did that (`(\W+)\s*`); it now sets `regex=true`. Option 1 of the ticket, chosen by Claus. **Tests:** new `TokenizeLiteralTokenTest` (16 tests: the metacharacter delimiters, `ValueBuilder.tokenize`, group with a plain and a Simple token, regex still working, the `\r\n` and `\t` escapes, the pair cases). The tokenize, tokenizer, split, sort and `ExpressionBuilder` tests of camel-core pass (389). _Claude Code on behalf of davsclaus (Claus Ibsen)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012nJsFsjqqFiwroeHRWaKbS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
