davsclaus opened a new pull request, #26923: URL: https://github.com/apache/camel/pull/26923
Fixes [CAMEL-25043](https://issues.apache.org/jira/browse/CAMEL-25043): bugs in the tokenize language found in a deep review. Each fix has a test that fails without it (`TokenizeEdgeCasesTest`, `RecordableInputStreamTest`). 1. **A message without a body fails with a `NullPointerException`.** `split().tokenize(",")` of a message without a body (or a `source` with no value) failed with `NullPointerException: source`. It now has no tokens, like an empty body. This is a regression from CAMEL-12688 (2018). 2. **`skipFirst` on an empty body fails with `NoSuchElementException`.** Without `group`, skip first called `next()` without `hasNext()`. The group path already handled it (CAMEL-12607). 3. **Pair mode ignores `source`.** `token("[").endToken("]").source("header:data")` always read the body; `tokenizePairExpression(source, ...)` dropped the source. 4. **`skipFirst` is ignored in xml mode and pair mode.** 5. **Pair mode with `group` puts the start token between the pairs**: `<a>1</a><a><a>2</a>`. The pairs are now joined without a delimiter, as in xml mode, unless `groupDelimiter` is set. 6. **A pair with the same start and end token silently has no tokens.** The end token is the scanner delimiter, so the start token is never found. It is now refused when the route is created. 7. **xml wrap mode (`inheritNamespaceTagName="*"`) builds broken XML when a comment or DOCTYPE comes before the root.** Each part ended with `</!-->`. 8. **xml wrap mode fails on a multi-byte character before the first token.** The head was cut by a character offset from the recorded bytes: `StringIndexOutOfBoundsException` for `<root name="æøå">`. 9. **`ValueBuilder.tokenize(token, null, skipFirst)` fails at route startup** ("expression must be specified"), also in `MockValueBuilder`. Also: the group delimiter is encoded in the exchange charset (it was decoded with it, but encoded with the platform charset), and the recording stream of xml wrap mode no longer drops a NUL byte. **Not changed** - A token with `regex=false` is still compiled as a regular expression, and the end token of a pair is not quoted: that is CAMEL-25015, which is waiting for the committers to pick a compatible fix. - `SplitterTest.testEmptyBody` asserted that a request without a body has no out message, which only held because the exchange failed with the `NullPointerException` in 1. It now asserts that the exchange does not fail and has no parts. The behaviour changes (1, 4, 5, 6) are in the 4.23 upgrade guide. The whole `camel-core` test suite passes (the only failure, `FileScanReloadTest`, is a known flaky test that passed on rerun). _Claude Code on behalf of Claus Ibsen_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
