[
https://issues.apache.org/jira/browse/CAMEL-25904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shashank updated CAMEL-25904:
-----------------------------
Description:
{{BeanIOSplitter.evaluate}} (main 74ee085f7d64, lines 79-107) creates the
BeanIO reader like this:
{code:java}
if (body instanceof WrappedFile wrappedfile) {
body = wrappedfile.getFile();
}
if (body instanceof File file) {
beanReader = factory.createReader(getStreamName(), file);
}
if (beanReader == null) {
Reader reader = msg.getMandatoryBody(Reader.class);
beanReader = factory.createReader(getStreamName(), reader);
}
{code}
BeanIO's {{StreamFactory.createReader(String, File)}} (beanio 3.2.1) opens the
file with a {{FileReader}}, that is with the default charset of the JVM. Other
bodies (byte array, stream) are converted to a {{Reader}} with the charset of
the exchange, else the default charset. The {{encoding}} option of the splitter
({{setEncoding}}, kept in its {{BeanIOConfiguration}}) is never read.
The usual route for big files is a file consumer and the streaming splitter,
and the file body is always a {{File}}:
{code:java}
from("file:in?charset=ISO-8859-1")
.split(beanIOSplitter).streaming()
.to("direct:record");
{code}
The {{charset}} of the file consumer is ignored, and so is
{{beanIOSplitter.setEncoding(StandardCharsets.ISO_8859_1)}}: an ISO-8859-1 or
windows-1252 file (the typical encoding of the legacy fixed-length and CSV
files BeanIO is used for) is decoded as UTF-8 on Java 18+, and every non-ASCII
character of the records becomes U+FFFD ({{Jörg Müller}} -> {{J?rg M?ller}}).
The split completes without an error. CAMEL-12284 fixed the same for
{{BeanIODataFormat}} (its encoding option did not work); the splitter was added
later (CAMEL-9843) and has never used it.
*Fix*: {{BeanIOSplitter}} decodes a file, stream or byte array body itself,
with the charset of the exchange ({{CamelCharsetName}} header or property, as
set by the {{charset}} option of the file consumer), else with its {{encoding}}
(default: the default charset of the JVM, as before). A {{String}} or
{{Reader}} body is read as before, and other text ({{CharSequence}}, such as a
{{StringBuilder}}, which has no converter to a {{Reader}} and failed on main)
is read as text, without a round trip through bytes. The reader is closed if
BeanIO cannot create its reader (an unknown stream name) or if reading the
first record fails. Upgrade note in the 4.24 guide.
*Tests*: new {{BeanIOSplitterEncodingTest}} (5 tests, ISO-8859-1 data with ö,
ü, é, ß). On main, 3 fail with {{expected: <[Jörg Müller Développeur, Renée
Straße Architect]> but was: <[J?rg M?ller D?veloppeur, Ren?e Stra?e
Architect]>}} (U+FFFD shown as ?): {{fileIsReadWithTheEncodingOfTheSplitter}}
(a {{File}} body, {{setEncoding(ISO_8859_1)}}),
{{fileFromTheFileConsumerIsReadWithItsCharset}} (a real file consumer with
{{charset=ISO-8859-1}}), {{bytesAreReadWithTheEncodingOfTheSplitter}}.
{{bytesAreReadWithTheCharsetOfTheExchange}} passes on main and with the fix
(control: that path already used the exchange charset). {{textIsReadAsItIs}} (a
{{StringBuilder}} body, splitter encoding UTF-16BE) fails on main with {{No
type converter available to convert from type: java.lang.StringBuilder to the
required type: java.io.Reader}} and passes with the fix. camel-beanio unit
tests: 26 run, 0 failures.
*How it was found*: a sweep of the data formats and their streaming splitters
for options that one code path does not use (here the {{File}} branch hands the
file to a library method without a charset). The JUnit tests above are the
evidence.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
(the same {{createReader(getStreamName(), file)}}; read, not run).
Not changed, possible follow-up: {{BeanIOIterator}} reads the first record in
its constructor, before {{evaluate}} sets the error handler on the reader, so
the {{ignoreInvalidRecords}} / {{ignoreUnexpectedRecords}} / custom
{{BeanReaderErrorHandler}} options do not apply to the first record (an invalid
first record fails the split, loudly).
_Filed with Claude Code on behalf of allthingssecurity._
was:
{{BeanIOSplitter.evaluate}} (main 74ee085f7d64, lines 79-107) creates the
BeanIO reader like this:
{code:java}
if (body instanceof WrappedFile wrappedfile) {
body = wrappedfile.getFile();
}
if (body instanceof File file) {
beanReader = factory.createReader(getStreamName(), file);
}
if (beanReader == null) {
Reader reader = msg.getMandatoryBody(Reader.class);
beanReader = factory.createReader(getStreamName(), reader);
}
{code}
BeanIO's {{StreamFactory.createReader(String, File)}} (beanio 3.2.1) opens the
file with a {{FileReader}}, that is with the default charset of the JVM. Other
bodies (byte array, stream) are converted to a {{Reader}} with the charset of
the exchange, else the default charset. The {{encoding}} option of the splitter
({{setEncoding}}, kept in its {{BeanIOConfiguration}}) is never read.
The usual route for big files is a file consumer and the streaming splitter,
and the file body is always a {{File}}:
{code:java}
from("file:in?charset=ISO-8859-1")
.split(beanIOSplitter).streaming()
.to("direct:record");
{code}
The {{charset}} of the file consumer is ignored, and so is
{{beanIOSplitter.setEncoding(StandardCharsets.ISO_8859_1)}}: an ISO-8859-1 or
windows-1252 file (the typical encoding of the legacy fixed-length and CSV
files BeanIO is used for) is decoded as UTF-8 on Java 18+, and every non-ASCII
character of the records becomes U+FFFD ({{Jörg Müller}} -> {{J?rg M?ller}}).
The split completes without an error. CAMEL-12284 fixed the same for
{{BeanIODataFormat}} (its encoding option did not work); the splitter was added
later (CAMEL-9843) and has never used it.
*Fix*: {{BeanIOSplitter}} decodes a file, stream or byte array body itself,
with the charset of the exchange ({{CamelCharsetName}} header or property, as
set by the {{charset}} option of the file consumer), else with its {{encoding}}
(default: the default charset of the JVM, as before). A {{String}} or
{{Reader}} body is read as before. The reader is closed if BeanIO cannot create
its reader (an unknown stream name). Upgrade note in the 4.24 guide.
*Tests*: new {{BeanIOSplitterEncodingTest}} (4 tests, ISO-8859-1 data with ö,
ü, é, ß). On main, 3 fail with {{expected: <[Jörg Müller Développeur, Renée
Straße Architect]> but was: <[J?rg M?ller D?veloppeur, Ren?e Stra?e
Architect]>}} (U+FFFD shown as ?): {{fileIsReadWithTheEncodingOfTheSplitter}}
(a {{File}} body, {{setEncoding(ISO_8859_1)}}),
{{fileFromTheFileConsumerIsReadWithItsCharset}} (a real file consumer with
{{charset=ISO-8859-1}}), {{bytesAreReadWithTheEncodingOfTheSplitter}}.
{{bytesAreReadWithTheCharsetOfTheExchange}} passes on main and with the fix
(control: that path already used the exchange charset). camel-beanio unit
tests: 25 run, 0 failures.
*How it was found*: a sweep of the data formats and their streaming splitters
for options that one code path does not use (here the {{File}} branch hands the
file to a library method without a charset). The JUnit tests above are the
evidence.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
(the same {{createReader(getStreamName(), file)}}; read, not run).
Not changed, possible follow-up: {{BeanIOIterator}} reads the first record in
its constructor, before {{evaluate}} sets the error handler on the reader, so
the {{ignoreInvalidRecords}} / {{ignoreUnexpectedRecords}} / custom
{{BeanReaderErrorHandler}} options do not apply to the first record (an invalid
first record fails the split, loudly).
_Filed with Claude Code on behalf of allthingssecurity._
> camel-beanio - BeanIOSplitter reads a file with the default charset of the
> JVM and ignores its encoding, so a file consumed with charset=ISO-8859-1 is
> split with its non-ASCII characters replaced
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25904
> URL: https://issues.apache.org/jira/browse/CAMEL-25904
> Project: Camel
> Issue Type: Bug
> Components: camel-beanio
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> {{BeanIOSplitter.evaluate}} (main 74ee085f7d64, lines 79-107) creates the
> BeanIO reader like this:
> {code:java}
> if (body instanceof WrappedFile wrappedfile) {
> body = wrappedfile.getFile();
> }
> if (body instanceof File file) {
> beanReader = factory.createReader(getStreamName(), file);
> }
> if (beanReader == null) {
> Reader reader = msg.getMandatoryBody(Reader.class);
> beanReader = factory.createReader(getStreamName(), reader);
> }
> {code}
> BeanIO's {{StreamFactory.createReader(String, File)}} (beanio 3.2.1) opens
> the file with a {{FileReader}}, that is with the default charset of the JVM.
> Other bodies (byte array, stream) are converted to a {{Reader}} with the
> charset of the exchange, else the default charset. The {{encoding}} option of
> the splitter ({{setEncoding}}, kept in its {{BeanIOConfiguration}}) is never
> read.
> The usual route for big files is a file consumer and the streaming splitter,
> and the file body is always a {{File}}:
> {code:java}
> from("file:in?charset=ISO-8859-1")
> .split(beanIOSplitter).streaming()
> .to("direct:record");
> {code}
> The {{charset}} of the file consumer is ignored, and so is
> {{beanIOSplitter.setEncoding(StandardCharsets.ISO_8859_1)}}: an ISO-8859-1 or
> windows-1252 file (the typical encoding of the legacy fixed-length and CSV
> files BeanIO is used for) is decoded as UTF-8 on Java 18+, and every
> non-ASCII character of the records becomes U+FFFD ({{Jörg Müller}} -> {{J?rg
> M?ller}}). The split completes without an error. CAMEL-12284 fixed the same
> for {{BeanIODataFormat}} (its encoding option did not work); the splitter was
> added later (CAMEL-9843) and has never used it.
> *Fix*: {{BeanIOSplitter}} decodes a file, stream or byte array body itself,
> with the charset of the exchange ({{CamelCharsetName}} header or property, as
> set by the {{charset}} option of the file consumer), else with its
> {{encoding}} (default: the default charset of the JVM, as before). A
> {{String}} or {{Reader}} body is read as before, and other text
> ({{CharSequence}}, such as a {{StringBuilder}}, which has no converter to a
> {{Reader}} and failed on main) is read as text, without a round trip through
> bytes. The reader is closed if BeanIO cannot create its reader (an unknown
> stream name) or if reading the first record fails. Upgrade note in the 4.24
> guide.
> *Tests*: new {{BeanIOSplitterEncodingTest}} (5 tests, ISO-8859-1 data with ö,
> ü, é, ß). On main, 3 fail with {{expected: <[Jörg Müller Développeur, Renée
> Straße Architect]> but was: <[J?rg M?ller D?veloppeur, Ren?e Stra?e
> Architect]>}} (U+FFFD shown as ?): {{fileIsReadWithTheEncodingOfTheSplitter}}
> (a {{File}} body, {{setEncoding(ISO_8859_1)}}),
> {{fileFromTheFileConsumerIsReadWithItsCharset}} (a real file consumer with
> {{charset=ISO-8859-1}}), {{bytesAreReadWithTheEncodingOfTheSplitter}}.
> {{bytesAreReadWithTheCharsetOfTheExchange}} passes on main and with the fix
> (control: that path already used the exchange charset). {{textIsReadAsItIs}}
> (a {{StringBuilder}} body, splitter encoding UTF-16BE) fails on main with
> {{No type converter available to convert from type: java.lang.StringBuilder
> to the required type: java.io.Reader}} and passes with the fix. camel-beanio
> unit tests: 26 run, 0 failures.
> *How it was found*: a sweep of the data formats and their streaming splitters
> for options that one code path does not use (here the {{File}} branch hands
> the file to a library method without a charset). The JUnit tests above are
> the evidence.
> *Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
> (the same {{createReader(getStreamName(), file)}}; read, not run).
> Not changed, possible follow-up: {{BeanIOIterator}} reads the first record in
> its constructor, before {{evaluate}} sets the error handler on the reader, so
> the {{ignoreInvalidRecords}} / {{ignoreUnexpectedRecords}} / custom
> {{BeanReaderErrorHandler}} options do not apply to the first record (an
> invalid first record fails the split, loudly).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
