allthingssecurity opened a new pull request, #28594: URL: https://github.com/apache/camel/pull/28594
# Description [CAMEL-26086](https://issues.apache.org/jira/browse/CAMEL-26086) `XMLTokenExpressionIterator` (the `xtokenize` language) builds the token of a mode from the raw text of the matched element. Two of these steps read that text in a way an XML parser does not, so valid XML gives wrong tokens without an error: - Mode `t` ("extracting the text content of the specified element") copies the characters between each `>` and the next `<`. Entity and character references are not decoded and the content of a CDATA section is lost: `<order>AT&T <5></order>` gives `AT&T <5>` instead of `AT&T <5>`, and `<order><![CDATA[a<b]]></order>` gives an empty string. With a `group` above 1 the `<group>` element could even get a stray `]]>`, so it was not XML. - Modes `i` (the default) and `u` take the start tag up to the first `>`, which can be inside a quoted attribute value (legal XML). In mode `i` the inherited namespace declarations are then inserted inside the attribute: `<orders xmlns='urn:orders'><order note='qty > 5'>1</order></orders>` with `xtokenize("//o:order")` gives `<order note='qty xmlns="urn:orders"> 5'>1</order>`, which parses, but the element has lost its namespace and `note` holds another value. In mode `u` the content starts with the rest of the start tag (` 5'><id>1</id>`). Business XML often declares a default namespace and has attributes such as `condition="amount > 100"` or `label="a -> b"`; text such as company names (`Smith & Sons`) and URLs with `&` is common. Changes: - Mode `t` reads the text of the token with a StAX reader: the characters, CDATA and whitespace events, with entity and character references decoded; comments and processing instructions are left out. The reader factory comes from `StaxConverter.createXMLInputFactory(false)` (no DTD, no external entities), with coalescing on. It is not namespace aware, as the token may use prefixes that its ancestors declare. It is created once and only creates readers after that (Woodstox and the JDK factory create a new reader per call). An entity reference that the token does not declare (Woodstox reports it without a replacement text) is kept as written, as before. This relies on Woodstox, which is a compile dependency of camel-stax and the provider `XMLInputFactory.newInstance()` finds unless another one is set: with the JDK reader, such a token now fails with `The entity "nbsp" was referenced, but not declared.` (checked with the JDK 21 factory), where main passed the text through. - Cost: mode `t` creates one StAX reader per token (the factory is shared). Woodstox allocates its input buffers per reader, so a very high number of small tokens costs some more allocation than the old character scan; I did not optimise it. - With a `group` above 1 in mode `t`, the texts are escaped as XML text (`EscapeHelper.xml`) inside `<group>`, so the group stays well-formed XML with the same text content. - Modes `i` and `u` find the end of the start tag with a quote-aware scan. - Upgrade guide 4.24: `camel-stax - the text mode of xtokenize gives the text as an XML parser reads it` (a route that decoded the entities itself must stop). Modes `i` and `u` gave a corrupt token before, so there is nothing to migrate for them. - Not in this PR: a document that starts with a comment or a doctype and has no XML declaration fails in every mode, and mode `w` with a descendant path whose matches are at different depths throws `IndexOutOfBoundsException` or gives unbalanced wrappers. Both are in CAMEL-26086 (separate PR, same class; the two branches merge cleanly). - Backport: camel-4.23.x, 4.22.x, 4.18.x and 4.14.x have the same bug (read, not built). The text mode output changes, so it could stay on main. The fix uses `org.apache.camel.util.EscapeHelper.xml`, added by CAMEL-24802 (1bff11f8c92c); camel-4.23.x has it, but camel-4.22.x, 4.18.x and 4.14.x do not, so a cherry-pick there would not compile and needs another way to escape the text. Tests: - New `XMLTokenizeTextAndStartTagTest`, 6 tests, JUnit assertions as in the module. All 6 fail on main: - `textModeDecodesEntityReferences`: `Expected: <AT&T <5>> but was: <AT&T <5>>`; - `textModeKeepsCdataContent`: `Expected: <a<b> but was: <>`; - `textModeGroupIsXml` (group 2, text and CDATA): the group does not parse on main (`The character sequence "]]>" must not appear in content`); with the fix its text content is `AT&T<5>`; - `injectModeKeepsAnAttributeValueWithGreaterThan`: `<order note='qty xmlns="urn:orders"> 5'>1</order> ==> expected: <urn:orders> but was: <null>`; - `injectModeKeepsAnEmptyElementWithGreaterThanInAnAttributeValue` (`<order note="a -> b"/>`): the token does not parse on main; - `unwrapModeWithGreaterThanInAnAttributeValue`: `Expected: <<id>1</id>> but was: < 5'><id>1</id>>`. - Unit tests of camel-stax with the fix: 116 run, 0 failures, 2 skipped (both skipped on main too). - A differential run against a namespace-aware DOM parser on 20,000 generated well-formed documents (entities, CDATA, comments, `>` in attribute values, namespaces), with this PR and the other xtokenize PR merged: modes `i`, `t` and `u` give the same elements, namespaces, attributes and text as the parser on every document for both `/*:root/*:order` and `//*:order`. The only differences left are mode `w` keeping the preceding siblings, which is by design (CAMEL-17466). - The formatter and import-sort plugins ran. I built offline with `-Djackson3-version=3.2.3` (3.2.4 is not in my local repository) and `-Dasm-version=9.11`. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26086](https://issues.apache.org/jira/browse/CAMEL-26086). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-stax and ran its tests, including the formatter and import-sort plugins. No generated file changes. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
