allthingssecurity opened a new pull request, #28593: URL: https://github.com/apache/camel/pull/28593
# Description [CAMEL-26085](https://issues.apache.org/jira/browse/CAMEL-26085) When no `header` is set (or `headerDisabled=true`, which drops it), `CsvMarshaller.DynamicColumnsMarshaller` writes each map of a `List<Map>` body with `map.values()`, so in the iteration order of that map. Within one body the column of a key then depends on each row: - a row whose map has its keys in another order puts its values in other columns: `[{id=1, name=Alice, city=Oslo}, {name=Bob, city=Rome, id=2}]` gives `1,Alice,Oslo` and `Bob,Rome,2`; - a row without one of the keys moves the following values one column to the left: `[{id=1, name=Alice, city=Oslo}, {id=2, city=Rome}]` gives `2,Rome`, so `Rome` is under `name`. There is no error and no header line to notice it by. It happens when JSON objects or documents are marshalled to CSV (`unmarshal().json()` then `marshal().csv()`, MongoDB documents, Jackson with `NON_NULL` leaving out empty fields): the system that imports the file reads values in the wrong columns. The class comment said the marshaller "adapts the columns but always keep them in the same order". Camel 2.15 kept a set of columns in the marshaller; commit bbc905eb6f (2.16, "always use the record map to define the columns. This allows the marshaller to be reused for asymmetric records") removed it, because that set was shared by all exchanges, and the alignment inside one body went with it. camel-univocity-parsers aligns by key name for the same body. Changes: - `CsvMarshaller.marshal` asks a new protected method, `createMapRecordValues()`, once per body for the function that turns a map row into record values. The default is `getMapRecordValues`, so a custom marshaller from a `CsvMarshallerFactory` works as before. - `DynamicColumnsMarshaller` returns a function that keeps the columns of the body: the keys of the first map, then the keys of later maps that the first one does not have, in the order they first appear. Each row writes `map.get(column)` for every column, so a missing key is a null value (an empty field, or `nullString` when it is set). The rows written before a new key appears have fewer fields; they cannot be changed once written. Nothing is kept across bodies, so different messages still have their own columns (what bbc905eb6f was for; `CsvMarshalTest.shouldMarshalDifferentDynamicColumns` passes unchanged). - A single `Map` body and `List` rows are written as before. With `header` set (and `headerDisabled` false), `FixedColumnsMarshaller` is unchanged. - The "Marshalling a List<Map> to CSV" section of the data format page says how the columns are chosen (and the catalog copy of the page). - Upgrade guide 4.24: `camel-csv - the maps of a list are written in the columns of their keys`. Users who reordered the maps themselves can stop; users who read such files with the shifted columns get the columns of the keys now. - Backport: camel-4.23.x, 4.22.x, 4.18.x and 4.14.x have the same marshaller (read, not built). It changes the output, so it could stay on main; I leave that to you. Tests: - New `CsvMarshalMapRowsColumnOrderTest`, 5 tests, JUnit assertions as in the module. 4 fail on main: - `rowsWithTheSameKeysInAnotherOrderStayAligned`: `array contents differ at index [1], expected: <2,Bob,Rome> but was: <Bob,Rome,2>`; - `rowWithAMissingKeyDoesNotShiftTheFollowingColumns`: `expected: <2,,Rome> but was: <2,Rome>`; - `rowWithAMissingKeyGetsTheNullString` (`nullString=NULL`): `expected: <2,NULL,Rome> but was: <2,Rome>`; - `keyThatTheFirstRowDoesNotHaveGetsTheNextColumn`: `expected: <2,,Rome> but was: <Rome,2>`. - `eachBodyHasItsOwnColumns` is a control and passes on main: the second message's columns do not depend on the first message. - Unit tests of camel-csv with the fix: 98 run, 0 failures. In the hunt that found it, a differential run over generated `List<Map>` bodies (rows in random key order, with missing keys) gave 1,815 and 1,799 misaligned bodies of 2,000 on main for two kinds of bodies, none with the same column logic, and none with camel-univocity-parsers. - The formatter and import-sort plugins ran. I built offline with `-Djackson3-version=3.2.3` (3.2.4 is not in my local repository) and `-Dasm-version=9.11`. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26085](https://issues.apache.org/jira/browse/CAMEL-26085). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-csv and ran its tests, including the formatter and import-sort plugins. The only generated change is the catalog copy of the data format page, copied from the source page. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
