[ 
https://issues.apache.org/jira/browse/CAMEL-26996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Federico Mariani updated CAMEL-26996:
-------------------------------------
    Fix Version/s: 4.24.0
      Description: 
With {{outputType=StreamList}} the body is a {{ResultSetIterator}} over the 
open {{ResultSet}}. When reading a row fails, {{next()}} closes the iterator 
and throws, and {{hasNext()}} is {{!closed}}, so after the failure the iterator 
says that there are no more rows, as when it has read the last one.

camel-jdbc 
({{components/camel-jdbc/src/main/java/org/apache/camel/component/jdbc/ResultSetIterator.java}},
 lines 72-97 on main):
{code:java}
public boolean hasNext() {
    return !closed.get();
}
...
} catch (SQLException e) {
    close();
    throw new RuntimeCamelException("Cannot process result", e);
}
{code}
camel-sql 
({{components/camel-sql/src/main/java/org/apache/camel/component/sql/ResultSetIterator.java}},
 lines 54-72) has the same code.

The first pass is right: the split fails with {{RuntimeCamelException: Cannot 
process result}}. But the failure comes from the iteration of the splitter 
itself, not from a sub-exchange, so it is not marked as exhausted 
({{MulticastProcessor.MulticastTask.doFailed}} calls {{doDone}} with 
{{forceExhaust=false}}), and the error handler of the route redelivers to the 
splitter with the same body: the same iterator, which now has "no more rows". 
The split completes and the exchange succeeds. The rows after the failure are 
lost without an error; the only trace is the DEBUG log of the redelivery.

A typical cause is a connection lost in the middle of the stream (with a fetch 
size the rows come from a server-side cursor, a batch at a time), but any 
{{SQLException}} while reading a row does it. Found with fault-injection tests 
(Toxiproxy in front of a real PostgreSQL: connection reset, backend restart, 
blackhole).

h3. Steps to reproduce

{code:java}
errorHandler(defaultErrorHandler().maximumRedeliveries(3).redeliveryDelay(0));

from("direct:stream")
    .setBody(constant("select id from items order by id"))
    .to("jdbc:ds?outputType=StreamList")   // or sql:select id from items order 
by id?outputType=StreamList
    .split(body()).streaming()
        .to("mock:row")
    .end();
{code}

30 rows, and a data source whose result set fails to fetch a row in the middle 
of the stream (the database session ends while the route handles row 12).

h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)

* Without error handler: the exchange fails (correct).
* With the redelivery policy, camel-jdbc and camel-sql: the exchange succeeds, 
with 12 of the 30 rows routed.

h3. Expected

The exchange fails once the redeliveries are exhausted: a redelivery cannot 
read the rest of a result set that has failed, and must not turn the failure 
into a success.

h3. Suggested fix

Keep the failure in {{ResultSetIterator}}: once reading a row has failed, 
{{hasNext()}} and {{next()}} throw again (a {{RuntimeCamelException}} with the 
first {{SQLException}} as cause), also after {{close()}}, so whatever reads the 
iterator again gets an error, not an end of data. {{IOHelper.closeIterator}} 
does the same for a {{Scanner}}: it rethrows the {{IOException}} the scanner 
swallowed, so a failed read does not look like the end of the input.

Not redelivering a split over an iterator (in the splitter or the error 
handler) would also change iterators that can go on after a failure; only the 
iterator knows that it cannot.

h3. Related

* CAMEL-7447 ({{StreamList}} in camel-jdbc), CAMEL-9849 ({{StreamList}} in 
camel-sql).
* CAMEL-26997: {{StreamList}} with a fetch size on PostgreSQL (the producer 
commits before the stream is read), which with this issue and a redelivery 
policy truncates the result with no fault at all.

_Claude Code on behalf of Croway_


  was:
With {{outputType=StreamList}} the body is a {{ResultSetIterator}} over the 
open {{ResultSet}}. When reading a row fails, {{next()}} closes the iterator 
and throws, and {{hasNext()}} is {{!closed}}, so after the failure the iterator 
says that there are no more rows, as when it has read the last one.

camel-jdbc 
({{components/camel-jdbc/src/main/java/org/apache/camel/component/jdbc/ResultSetIterator.java}},
 lines 72-97 on main):
{code:java}
public boolean hasNext() {
    return !closed.get();
}
...
} catch (SQLException e) {
    close();
    throw new RuntimeCamelException("Cannot process result", e);
}
{code}
camel-sql 
({{components/camel-sql/src/main/java/org/apache/camel/component/sql/ResultSetIterator.java}},
 lines 54-72) has the same code.

The first pass is right: the split fails with {{RuntimeCamelException: Cannot 
process result}}. But the failure comes from the iteration of the splitter 
itself, not from a sub-exchange, so it is not marked as exhausted 
({{MulticastProcessor.MulticastTask.doFailed}} calls {{doDone}} with 
{{forceExhaust=false}}), and the error handler of the route redelivers to the 
splitter with the same body: the same iterator, which now has "no more rows". 
The split completes and the exchange succeeds. The rows after the failure are 
lost without an error; the only trace is the DEBUG log of the redelivery.

A typical cause is a connection lost in the middle of the stream (with a fetch 
size the rows come from a server-side cursor, a batch at a time), but any 
{{SQLException}} while reading a row does it. Found with fault-injection tests 
(Toxiproxy in front of a real PostgreSQL: connection reset, backend restart, 
blackhole).

h3. Steps to reproduce

{code:java}
errorHandler(defaultErrorHandler().maximumRedeliveries(3).redeliveryDelay(0));

from("direct:stream")
    .setBody(constant("select id from items order by id"))
    .to("jdbc:ds?outputType=StreamList")   // or sql:select id from items order 
by id?outputType=StreamList
    .split(body()).streaming()
        .to("mock:row")
    .end();
{code}

30 rows, and a data source whose result set fails to fetch a row in the middle 
of the stream (the database session ends while the route handles row 12).

h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)

* Without error handler: the exchange fails (correct).
* With the redelivery policy, camel-jdbc and camel-sql: the exchange succeeds, 
with 12 of the 30 rows routed.

h3. Expected

The exchange fails once the redeliveries are exhausted: a redelivery cannot 
read the rest of a result set that has failed, and must not turn the failure 
into a success.

h3. Suggested fix

Keep the failure in {{ResultSetIterator}}: once reading a row has failed, 
{{hasNext()}} and {{next()}} throw again (a {{RuntimeCamelException}} with the 
first {{SQLException}} as cause), also after {{close()}}, so whatever reads the 
iterator again gets an error, not an end of data. {{IOHelper.closeIterator}} 
does the same for a {{Scanner}}: it rethrows the {{IOException}} the scanner 
swallowed, so a failed read does not look like the end of the input.

Not redelivering a split over an iterator (in the splitter or the error 
handler) would also change iterators that can go on after a failure; only the 
iterator knows that it cannot.

h3. Related

* CAMEL-7447 ({{StreamList}} in camel-jdbc), CAMEL-9849 ({{StreamList}} in 
camel-sql).
* The camel-jdbc issue about {{StreamList}} with a fetch size on PostgreSQL 
(the producer commits before the stream is read), which with this issue and a 
redelivery policy truncates the result with no fault at all.

_Claude Code on behalf of Croway_



> camel-jdbc / camel-sql - outputType=StreamList: once reading a row has 
> failed, the iterator looks finished, so a redelivered split succeeds with 
> part of the rows
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-26996
>                 URL: https://issues.apache.org/jira/browse/CAMEL-26996
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-jdbc, camel-sql
>    Affects Versions: 4.14.9, 4.18.5, 4.23.0
>            Reporter: Federico Mariani
>            Priority: Major
>             Fix For: 4.24.0
>
>
> With {{outputType=StreamList}} the body is a {{ResultSetIterator}} over the 
> open {{ResultSet}}. When reading a row fails, {{next()}} closes the iterator 
> and throws, and {{hasNext()}} is {{!closed}}, so after the failure the 
> iterator says that there are no more rows, as when it has read the last one.
> camel-jdbc 
> ({{components/camel-jdbc/src/main/java/org/apache/camel/component/jdbc/ResultSetIterator.java}},
>  lines 72-97 on main):
> {code:java}
> public boolean hasNext() {
>     return !closed.get();
> }
> ...
> } catch (SQLException e) {
>     close();
>     throw new RuntimeCamelException("Cannot process result", e);
> }
> {code}
> camel-sql 
> ({{components/camel-sql/src/main/java/org/apache/camel/component/sql/ResultSetIterator.java}},
>  lines 54-72) has the same code.
> The first pass is right: the split fails with {{RuntimeCamelException: Cannot 
> process result}}. But the failure comes from the iteration of the splitter 
> itself, not from a sub-exchange, so it is not marked as exhausted 
> ({{MulticastProcessor.MulticastTask.doFailed}} calls {{doDone}} with 
> {{forceExhaust=false}}), and the error handler of the route redelivers to the 
> splitter with the same body: the same iterator, which now has "no more rows". 
> The split completes and the exchange succeeds. The rows after the failure are 
> lost without an error; the only trace is the DEBUG log of the redelivery.
> A typical cause is a connection lost in the middle of the stream (with a 
> fetch size the rows come from a server-side cursor, a batch at a time), but 
> any {{SQLException}} while reading a row does it. Found with fault-injection 
> tests (Toxiproxy in front of a real PostgreSQL: connection reset, backend 
> restart, blackhole).
> h3. Steps to reproduce
> {code:java}
> errorHandler(defaultErrorHandler().maximumRedeliveries(3).redeliveryDelay(0));
> from("direct:stream")
>     .setBody(constant("select id from items order by id"))
>     .to("jdbc:ds?outputType=StreamList")   // or sql:select id from items 
> order by id?outputType=StreamList
>     .split(body()).streaming()
>         .to("mock:row")
>     .end();
> {code}
> 30 rows, and a data source whose result set fails to fetch a row in the 
> middle of the stream (the database session ends while the route handles row 
> 12).
> h3. Actual (4.23.0 and 4.24.0-SNAPSHOT)
> * Without error handler: the exchange fails (correct).
> * With the redelivery policy, camel-jdbc and camel-sql: the exchange 
> succeeds, with 12 of the 30 rows routed.
> h3. Expected
> The exchange fails once the redeliveries are exhausted: a redelivery cannot 
> read the rest of a result set that has failed, and must not turn the failure 
> into a success.
> h3. Suggested fix
> Keep the failure in {{ResultSetIterator}}: once reading a row has failed, 
> {{hasNext()}} and {{next()}} throw again (a {{RuntimeCamelException}} with 
> the first {{SQLException}} as cause), also after {{close()}}, so whatever 
> reads the iterator again gets an error, not an end of data. 
> {{IOHelper.closeIterator}} does the same for a {{Scanner}}: it rethrows the 
> {{IOException}} the scanner swallowed, so a failed read does not look like 
> the end of the input.
> Not redelivering a split over an iterator (in the splitter or the error 
> handler) would also change iterators that can go on after a failure; only the 
> iterator knows that it cannot.
> h3. Related
> * CAMEL-7447 ({{StreamList}} in camel-jdbc), CAMEL-9849 ({{StreamList}} in 
> camel-sql).
> * CAMEL-26997: {{StreamList}} with a fetch size on PostgreSQL (the producer 
> commits before the stream is read), which with this issue and a redelivery 
> policy truncates the result with no fault at all.
> _Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to