This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new a6092f243dd1 CAMEL-26085: camel-csv - marshal the maps of a list in
the columns of their keys (#28593)
a6092f243dd1 is described below
commit a6092f243dd15ca854eac320e23b3463df251d8c
Author: allthingssecurity <[email protected]>
AuthorDate: Wed Oct 28 21:34:26 2026 +0530
CAMEL-26085: camel-csv - marshal the maps of a list in the columns of their
keys (#28593)
The maps of a list are marshalled in the columns of their keys, with the
columns of all rows, instead of the order of the values of each map. A null key
and a sorted map are handled. The change in output for mixed record types and
the cost of the column scan are documented in the data format page and the 4.24
upgrade guide, with List rows or a custom marshallerFactory as the way back.
Co-Authored-By: Claude <[email protected]>
---
.../apache/camel/catalog/docs/csv-dataformat.adoc | 20 +++
.../camel-csv/src/main/docs/csv-dataformat.adoc | 20 +++
.../apache/camel/dataformat/csv/CsvMarshaller.java | 68 +++++++-
.../csv/CsvMarshalMapRowsColumnOrderTest.java | 184 +++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 19 +++
5 files changed, 302 insertions(+), 9 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/csv-dataformat.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/csv-dataformat.adoc
index f1b977cc7e13..c8fe3efceb67 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/csv-dataformat.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/csv-dataformat.adoc
@@ -182,6 +182,26 @@ format, you can now store the message payload as a
`List<Map<String, Object>>` object where the list contains a Map for
each row.
+When `header` is set (and `headerDisabled` is false), its names are the
+columns, in that order. Otherwise, the columns are the keys of the first Map,
+followed by the keys of later Maps that the first one does not have, in the
+order they first appear (the rows written before such a key appears have
+fewer fields). A Map without one of these keys gets a null value in that
+column, and no header line is written.
+
+So a body that mixes record types, such as a header row `{type=H, date=d}`,
+detail rows `{type=D, sku=s, qty=q}` and a trailer row `{type=T, count=n}`,
+is written as `H,d`, `D,,s,q` and `T,,,,n`. To write each row with only its
+own fields, in the order of its Map, convert the rows to Lists first, or set
+a custom `marshallerFactory` (`marshallerFactoryRef` in XML and YAML) whose
+`CsvMarshaller` returns the values of each Map from `getMapRecordValues`.
+
+Every row is written with all the columns seen so far, so the output grows
+with the number of rows times the number of distinct keys. A body of rows
+that each have keys of their own (such as sparse or pivoted data) gives
+about N²/2 fields for N rows; for such data, convert the rows to Lists, or
+set `header` to only the columns you need.
+
=== File Poller of CSV, then unmarshaling
Given a bean which can handle the incoming data...
diff --git a/components/camel-csv/src/main/docs/csv-dataformat.adoc
b/components/camel-csv/src/main/docs/csv-dataformat.adoc
index f1b977cc7e13..c8fe3efceb67 100644
--- a/components/camel-csv/src/main/docs/csv-dataformat.adoc
+++ b/components/camel-csv/src/main/docs/csv-dataformat.adoc
@@ -182,6 +182,26 @@ format, you can now store the message payload as a
`List<Map<String, Object>>` object where the list contains a Map for
each row.
+When `header` is set (and `headerDisabled` is false), its names are the
+columns, in that order. Otherwise, the columns are the keys of the first Map,
+followed by the keys of later Maps that the first one does not have, in the
+order they first appear (the rows written before such a key appears have
+fewer fields). A Map without one of these keys gets a null value in that
+column, and no header line is written.
+
+So a body that mixes record types, such as a header row `{type=H, date=d}`,
+detail rows `{type=D, sku=s, qty=q}` and a trailer row `{type=T, count=n}`,
+is written as `H,d`, `D,,s,q` and `T,,,,n`. To write each row with only its
+own fields, in the order of its Map, convert the rows to Lists first, or set
+a custom `marshallerFactory` (`marshallerFactoryRef` in XML and YAML) whose
+`CsvMarshaller` returns the values of each Map from `getMapRecordValues`.
+
+Every row is written with all the columns seen so far, so the output grows
+with the number of rows times the number of distinct keys. A body of rows
+that each have keys of their own (such as sparse or pivoted data) gives
+about N²/2 fields for N rows; for such data, convert the rows to Lists, or
+set `header` to only the columns you need.
+
=== File Poller of CSV, then unmarshaling
Given a bean which can handle the incoming data...
diff --git
a/components/camel-csv/src/main/java/org/apache/camel/dataformat/csv/CsvMarshaller.java
b/components/camel-csv/src/main/java/org/apache/camel/dataformat/csv/CsvMarshaller.java
index 6cef2740940e..13079072c991 100644
---
a/components/camel-csv/src/main/java/org/apache/camel/dataformat/csv/CsvMarshaller.java
+++
b/components/camel-csv/src/main/java/org/apache/camel/dataformat/csv/CsvMarshaller.java
@@ -22,8 +22,11 @@ import java.io.OutputStreamWriter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
import org.apache.camel.Exchange;
import org.apache.camel.NoTypeConversionAvailableException;
@@ -80,13 +83,14 @@ public abstract class CsvMarshaller {
throws NoTypeConversionAvailableException, IOException {
CSVPrinter printer = createPrinter(exchange, outputStream);
try {
+ Function<Map<?, ?>, Iterable<?>> mapRecordValues =
createMapRecordValues();
if (object instanceof Map map) {
- printer.printRecord(getMapRecordValues(map));
+ printer.printRecord(mapRecordValues.apply(map));
} else {
Iterator it = ObjectHelper.createIterator(object);
while (it.hasNext()) {
Object child = it.next();
- printer.printRecord(getRecordValues(exchange, child));
+ printer.printRecord(getRecordValues(exchange, child,
mapRecordValues));
}
}
} finally {
@@ -107,11 +111,12 @@ public abstract class CsvMarshaller {
return new CSVPrinter(new OutputStreamWriter(outputStream,
ExchangeHelper.getCharsetName(exchange)), format);
}
- private Iterable<?> getRecordValues(Exchange exchange, Object data) throws
NoTypeConversionAvailableException {
+ private Iterable<?> getRecordValues(Exchange exchange, Object data,
Function<Map<?, ?>, Iterable<?>> mapRecordValues)
+ throws NoTypeConversionAvailableException {
// each row must be a map or list based
Map<?, ?> map =
exchange.getContext().getTypeConverter().tryConvertTo(Map.class, exchange,
data);
if (map != null) {
- return getMapRecordValues(map);
+ return mapRecordValues.apply(map);
}
return ExchangeHelper.convertToMandatoryType(exchange, List.class,
data);
}
@@ -124,6 +129,17 @@ public abstract class CsvMarshaller {
*/
protected abstract Iterable<?> getMapRecordValues(Map<?, ?> map);
+ /**
+ * Creates the function that gives the CSV record values of the map rows
of one marshalled body. It is created for
+ * each body, so it can keep the columns of the rows already written.
+ *
+ * @return the function that gives the CSV record values of a map row
+ * @since 4.24
+ */
+ protected Function<Map<?, ?>, Iterable<?>> createMapRecordValues() {
+ return this::getMapRecordValues;
+ }
+
//region Implementations
/**
@@ -148,7 +164,9 @@ public abstract class CsvMarshaller {
}
/**
- * This marshaller adapts the columns but always keep them in the same
order
+ * This marshaller takes the columns from the keys of the map rows. Within
one body each key keeps its column: the
+ * keys of the first row, then the keys of later rows in the order they
are first seen. A row without one of these
+ * keys gets a null value in its column.
*/
private static final class DynamicColumnsMarshaller extends CsvMarshaller {
private DynamicColumnsMarshaller(CSVFormat format) {
@@ -156,11 +174,43 @@ public abstract class CsvMarshaller {
}
@Override
- protected Iterable<?> getMapRecordValues(Map<?, ?> map) {
- List<Object> result = new ArrayList<>(map.size());
- result.addAll(map.values());
+ protected Function<Map<?, ?>, Iterable<?>> createMapRecordValues() {
+ // the columns of one body, so each key keeps its column in every
row, whatever the iteration order of the
+ // row map; a new body starts with no columns, as the rows of
another message may have other keys
+ Set<Object> columns = new LinkedHashSet<>();
+ return map -> {
+ columns.addAll(map.keySet());
+ List<Object> result = new ArrayList<>(columns.size());
+ for (Object column : columns) {
+ result.add(valueOf(map, column));
+ }
+ return result;
+ };
+ }
- return result;
+ private static Object valueOf(Map<?, ?> map, Object column) {
+ if (column == null) {
+ // a null key of an earlier row: maps such as Map.of(...) do
not permit get(null)
+ for (Map.Entry<?, ?> entry : map.entrySet()) {
+ if (entry.getKey() == null) {
+ return entry.getValue();
+ }
+ }
+ return null;
+ }
+ try {
+ return map.get(column);
+ } catch (ClassCastException e) {
+ // a sorted map, such as a TreeMap, cannot compare a key of an
earlier row of another type, so it
+ // has no value for it
+ return null;
+ }
+ }
+
+ @Override
+ protected Iterable<?> getMapRecordValues(Map<?, ?> map) {
+ // a map on its own: its keys are the columns
+ return createMapRecordValues().apply(map);
}
}
//endregion
diff --git
a/components/camel-csv/src/test/java/org/apache/camel/dataformat/csv/CsvMarshalMapRowsColumnOrderTest.java
b/components/camel-csv/src/test/java/org/apache/camel/dataformat/csv/CsvMarshalMapRowsColumnOrderTest.java
new file mode 100644
index 000000000000..9ef7d0984993
--- /dev/null
+++
b/components/camel-csv/src/test/java/org/apache/camel/dataformat/csv/CsvMarshalMapRowsColumnOrderTest.java
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.dataformat.csv;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.camel.EndpointInject;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+
+/**
+ * A List of Maps without a configured header: every key must stay in one
column, whatever the iteration order of each
+ * row map, and a key missing in a row must not shift the following values
into the wrong column. Each body has its own
+ * columns.
+ */
+class CsvMarshalMapRowsColumnOrderTest extends CamelTestSupport {
+ @EndpointInject("mock:output")
+ MockEndpoint output;
+
+ @Test
+ void rowsWithTheSameKeysInAnotherOrderStayAligned() throws Exception {
+ output.expectedMessageCount(1);
+
+ template.sendBody("direct:default", List.of(
+ row("id", "1", "name", "Alice", "city", "Oslo"),
+ row("name", "Bob", "city", "Rome", "id", "2")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "1,Alice,Oslo", "2,Bob,Rome" },
readOutputLines(0));
+ }
+
+ @Test
+ void rowWithAMissingKeyDoesNotShiftTheFollowingColumns() throws Exception {
+ output.expectedMessageCount(1);
+
+ template.sendBody("direct:default", List.of(
+ row("id", "1", "name", "Alice", "city", "Oslo"),
+ row("id", "2", "city", "Rome")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "1,Alice,Oslo", "2,,Rome" },
readOutputLines(0));
+ }
+
+ @Test
+ void rowWithAMissingKeyGetsTheNullString() throws Exception {
+ output.expectedMessageCount(1);
+
+ template.sendBody("direct:nullString", List.of(
+ row("id", "1", "name", "Alice", "city", "Oslo"),
+ row("id", "2", "city", "Rome")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "1,Alice,Oslo", "2,NULL,Rome" },
readOutputLines(0));
+ }
+
+ @Test
+ void keyThatTheFirstRowDoesNotHaveGetsTheNextColumn() throws Exception {
+ output.expectedMessageCount(1);
+
+ template.sendBody("direct:default", List.of(
+ row("id", "1", "name", "Alice"),
+ row("city", "Rome", "id", "2"),
+ row("name", "Carol", "city", "Bergen", "id", "3")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "1,Alice", "2,,Rome",
"3,Carol,Bergen" }, readOutputLines(0));
+ }
+
+ @Test
+ void rowsOfDifferentRecordTypesGetTheColumnsOfAllRows() throws Exception {
+ output.expectedMessageCount(1);
+
+ template.sendBody("direct:default", List.of(
+ row("type", "H", "date", "20261028"),
+ row("type", "D", "sku", "A1", "qty", "2"),
+ row("type", "T", "count", "1")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "H,20261028", "D,,A1,2", "T,,,,1" },
readOutputLines(0));
+ }
+
+ @Test
+ void listRowsOfDifferentRecordTypesKeepTheirOwnFields() throws Exception {
+ output.expectedMessageCount(1);
+
+ template.sendBody("direct:default", List.of(
+ List.of("H", "20261028"),
+ List.of("D", "A1", "2"),
+ List.of("T", "1")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "H,20261028", "D,A1,2", "T,1" },
readOutputLines(0));
+ }
+
+ @Test
+ void nullKeyOfAnEarlierRowWithAMapThatDoesNotPermitNullKeys() throws
Exception {
+ output.expectedMessageCount(1);
+
+ Map<String, Object> withNullKey = new LinkedHashMap<>();
+ withNullKey.put("id", "1");
+ withNullKey.put(null, "x");
+ // Map.of(...) throws a NullPointerException on get(null)
+ template.sendBody("direct:default", List.of(withNullKey, Map.of("id",
"2")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "1,x", "2," }, readOutputLines(0));
+ }
+
+ @Test
+ void keyOfAnotherTypeOfAnEarlierRowWithASortedMap() throws Exception {
+ output.expectedMessageCount(1);
+
+ Map<Object, Object> first = new LinkedHashMap<>();
+ first.put(1, "x");
+ first.put("id", "1");
+ // a TreeMap of String keys throws a ClassCastException on get(1)
+ Map<Object, Object> sorted = new TreeMap<>(Map.of("id", "2"));
+ template.sendBody("direct:default", List.of(first, sorted));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "x,1", ",2" }, readOutputLines(0));
+ }
+
+ @Test
+ void eachBodyHasItsOwnColumns() throws Exception {
+ output.expectedMessageCount(2);
+
+ template.sendBody("direct:default", List.of(row("id", "1", "name",
"Alice")));
+ template.sendBody("direct:default", List.of(row("city", "Rome", "id",
"2")));
+ output.assertIsSatisfied();
+
+ assertArrayEquals(new String[] { "1,Alice" }, readOutputLines(0));
+ assertArrayEquals(new String[] { "Rome,2" }, readOutputLines(1));
+ }
+
+ private static Map<String, Object> row(String... keyValues) {
+ Map<String, Object> map = new LinkedHashMap<>();
+ for (int i = 0; i < keyValues.length; i += 2) {
+ map.put(keyValues[i], keyValues[i + 1]);
+ }
+ return map;
+ }
+
+ @Override
+ protected RouteBuilder createRouteBuilder() {
+ return new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("direct:default")
+ .marshal(new CsvDataFormat())
+ .to("mock:output");
+
+ from("direct:nullString")
+ .marshal(new CsvDataFormat().setNullString("NULL"))
+ .to("mock:output");
+ }
+ };
+ }
+
+ private String[] readOutputLines(int index) {
+ return
output.getExchanges().get(index).getIn().getBody(String.class).split("\r\n|\r|\n");
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 205a36a7675b..7767b53870a0 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -1460,6 +1460,25 @@ file is written from the column names, not from the
first map). Every map of the
`ignoreFirstRecord` only applies to unmarshal. A route that set
`ignoreFirstRecord=false` to work around this is not
affected; a route that added a dummy first map now gets it in the output too.
+=== camel-csv - the maps of a list are written in the columns of their keys
+
+When no `header` is set (or `headerDisabled=true`), marshalling a
`List<Map<String, Object>>` wrote the values of
+each map in the iteration order of that map, so a row whose map had its keys
in another order, or had no entry for
+one of the keys, put its values in the wrong columns. Within one message body,
each key now keeps its column: the
+columns are the keys of the first map, followed by the keys of later maps that
the first one does not have, in the
+order they first appear (the rows written before such a key appears have fewer
fields). A map without one of these
+keys gets a null value in its column (an empty field, or the `nullString` when
it is set), where before the following
+values moved one column to the left. Each message body starts again with no
columns, as before. Set `header` (with
+`headerDisabled=false`) to get a fixed set and order of columns.
+
+This also changes the output of a body that mixes record types on purpose,
such as a header row
+`{type=H, date=d}`, detail rows `{type=D, sku=s, qty=q}` and a trailer row
`{type=T, count=n}` (common in flat-file
+exchanges): it was written as `H,d`, `D,s,q` and `T,n`, and is now written as
`H,d`, `D,,s,q` and `T,,,,n`. To keep
+the previous output, convert the rows to `List` before marshalling, or set a
custom `marshallerFactory`
+(`marshallerFactoryRef` in XML and YAML) whose `CsvMarshaller` returns the
values of each map. Every row
+now has all the columns seen so far in the body, so rows that each have keys
of their own (sparse or pivoted data)
+give about N²/2 fields for N rows, where the output was linear in the input
before.
+
=== camel-github2 - the commit consumer routes the commits beyond the newest
100
The `commit` consumer read only the 100 newest commits of the branch at each
poll, so when more than 100 commits were
