shashank created CAMEL-25015:
--------------------------------
Summary: Tokenize language: a token with regex=false is still
compiled as a regular expression, so tokenize("|") splits every character and
tokenize("*") fails
Key: CAMEL-25015
URL: https://issues.apache.org/jira/browse/CAMEL-25015
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
The tokenize language has a {{regex}} option ("If the token is a regular
expression pattern", default {{false}}), but a token is always compiled as a
regular expression. {{TokenizeLanguage.createExpression}} picks
{{tokenizeExpression}} or {{regexTokenizeExpression}}, and both end in
{{ExchangeHelper.getScanner(exchange, value, token)}}, which passes the token
unchanged to {{org.apache.camel.util.Scanner}}, which calls
{{Pattern.compile}}. With {{regex=false}} the only difference is that a Simple
function in the token is evaluated.
So common delimiters that are regex metacharacters give wrong splits, or fail,
with no hint that the token is a regex:
{noformat}
split().tokenize("|") of "a|b|c" -> [a, |, b, |, c] (one message per
character)
split().tokenize(".") of "a.b.c" -> [, , , ]
split().tokenize("$") of "a$b$c" -> [a$b$c]
split().tokenize("*") of "a*b*c" -> PatternSyntaxException: Dangling meta
character '*'
split().tokenize("|", false, 2) of "a|b|c|d" -> [a||, b||, c||, d]
tokenizePair("(", ")") of "(a)(b)" -> PatternSyntaxException: Unmatched
closing ')'
tokenizePair("[rec]", "END.") of "[rec]1 ENDS HERE END.[rec]2END." -> [1 ]
(expected [1 ENDS HERE , 2])
controls: tokenize(newline), tokenize(";"), tokenize(escaped pipe, regex=true)
-> [a, b, c]
{noformat}
Pipe-delimited files are the most visible case. This has been the behaviour
since at least 2.25.
*The fix has a compatibility question, so I would like to ask before sending a
PR.* The obvious fix is to quote the token ({{Pattern.quote}}) when
{{regex=false}} (and the end token of {{tokenizePair}}, and to use the unquoted
token as the group delimiter). Tokens without regex metacharacters would behave
exactly as before. But routes that rely on a regex token without {{regex=true}}
would change. For example, in the XML or YAML DSL a token of backslash-r
backslash-n is four characters, and it only works today because it is compiled
as a regex ({{TokenizerExpressionReifier}} only converts a leading
backslash-n); a whitespace pattern such as backslash-s-plus works the same way.
Options I can see:
# quote the token when {{regex=false}}, and document it in the upgrade guide
(routes with regex tokens must set {{regex=true}}); maybe also convert the
backslash-r and backslash-t escapes in the reifier as it already does for
backslash-n;
# keep compiling the token, but quote it when it is a single character (the
common delimiter case: pipe, dot, dollar, star, plus), which changes no
multi-character token;
# keep the behaviour, and document that the token is always a regular
expression (and what {{regex}} really means).
Which would the committers prefer? I can send a PR with tests for either.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
