Federico Mariani created CAMEL-26996:
----------------------------------------
Summary: camel-jdbc / camel-sql - outputType=StreamList: once
reading a row has failed, the iterator looks finished, so a redelivered split
succeeds with part of the rows
Key: CAMEL-26996
URL: https://issues.apache.org/jira/browse/CAMEL-26996
Project: Camel
Issue Type: Bug
Components: camel-jdbc, camel-sql
Affects Versions: 4.14.9, 4.18.5, 4.23.0
Reporter: Federico Mariani
With {{outputType=StreamList}} the body is a {{ResultSetIterator}} over the
open {{ResultSet}}. When reading a row fails, {{next()}} closes the iterator
and throws, and {{hasNext()}} is {{!closed}}, so after the failure the iterator
says that there are no more rows, as when it has read the last one.
camel-jdbc
({{components/camel-jdbc/src/main/java/org/apache/camel/component/jdbc/ResultSetIterator.java}},
lines 72-97 on main):
{code:java}
public boolean hasNext() {
return !closed.get();
}
...
} catch (SQLException e) {
close();
throw new RuntimeCamelException("Cannot process result", e);
}
{code}
camel-sql
({{components/camel-sql/src/main/java/org/apache/camel/component/sql/ResultSetIterator.java}},
lines 54-72) has the same code.
The first pass is right: the split fails with {{RuntimeCamelException: Cannot
process result}}. But the failure comes from the iteration of the splitter
itself, not from a sub-exchange, so it is not marked as exhausted
({{MulticastProcessor.MulticastTask.doFailed}} calls {{doDone}} with
{{forceExhaust=false}}), and the error handler of the route redelivers to the
splitter with the same body: the same iterator, which now has "no more rows".
The split completes and the exchange succeeds. The rows after the failure are
lost without an error; the only trace is the DEBUG log of the redelivery.
A typical cause is a connection lost in the middle of the stream (with a fetch
size the rows come from a server-side cursor, a batch at a time), but any
{{SQLException}} while reading a row does it. Found with fault-injection tests
(Toxiproxy in front of a real PostgreSQL: connection reset, backend restart,
blackhole).
h3. Steps to reproduce
{code:java}
errorHandler(defaultErrorHandler().maximumRedeliveries(3).redeliveryDelay(0));
from("direct:stream")
.setBody(constant("select id from items order by id"))
.to("jdbc:ds?outputType=StreamList") // or sql:select id from items order
by id?outputType=StreamList
.split(body()).streaming()
.to("mock:row")
.end();
{code}
30 rows, and a data source whose result set fails to fetch a row in the middle
of the stream (the database session ends while the route handles row 12).
h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)
* Without error handler: the exchange fails (correct).
* With the redelivery policy, camel-jdbc and camel-sql: the exchange succeeds,
with 12 of the 30 rows routed.
h3. Expected
The exchange fails once the redeliveries are exhausted: a redelivery cannot
read the rest of a result set that has failed, and must not turn the failure
into a success.
h3. Suggested fix
Keep the failure in {{ResultSetIterator}}: once reading a row has failed,
{{hasNext()}} and {{next()}} throw again (a {{RuntimeCamelException}} with the
first {{SQLException}} as cause), also after {{close()}}, so whatever reads the
iterator again gets an error, not an end of data. {{IOHelper.closeIterator}}
does the same for a {{Scanner}}: it rethrows the {{IOException}} the scanner
swallowed, so a failed read does not look like the end of the input.
Not redelivering a split over an iterator (in the splitter or the error
handler) would also change iterators that can go on after a failure; only the
iterator knows that it cannot.
h3. Related
* CAMEL-7447 ({{StreamList}} in camel-jdbc), CAMEL-9849 ({{StreamList}} in
camel-sql).
* The camel-jdbc issue about {{StreamList}} with a fetch size on PostgreSQL
(the producer commits before the stream is read), which with this issue and a
redelivery policy truncates the result with no fault at all.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)