Re: Strange issue with JDBC add Postgres
Hi Tim, Can you possible provide me a snippet of what your route actually looks like with the workaround? I ran into same problem where my fetchsize is not working and I want to have a work around while still using camel. Thanks -- View this message in context: http://camel.465427.n5.nabble.com/Strange-issue-with-JDBC-add-Postgres-tp5758548p5782983.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Strange issue with JDBC add Postgres
There is a resetAutoCommit option, you can try set that to false
On Mon, Nov 10, 2014 at 1:54 PM, Tim Dudgeon tdudgeon...@gmail.com wrote:
I dug a bit deeper here and I think that Camel is doing something with the
connection, possibly related to autoCommit.
I worked around the problem by writing my own Processor that sets autoCommit
to false, sets maxRows on the statement and then iterates through the
ResultSet and a cursor does seem to be used as expected.
But trying to do the same with the Camel JDBC component fails for reasons
described earlier.
Tim
On 06/11/2014 18:57, Claus Ibsen wrote:
So maybe its your maxRows=1000 that is the problem, so when you are
trying to get rows 1000 then it throws that exception.
What is your use case? You want to select * from a table, but only the
first 1000 rows?
On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com
wrote:
On 05/11/2014 19:07, Claus Ibsen wrote:
maxRowSize ??? do you mean fetchSize?
Sorry, I mean maxRows (as in the example here:
http://camel.apache.org/jdbc.html).
So something like:
.to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000')
Also which version of Camel do you use?
2.14
Tim
--
Claus Ibsen
-
Red Hat, Inc.
Email: cib...@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Strange issue with JDBC add Postgres
I already did. Described here:
http://camel.465427.n5.nabble.com/Strange-issue-with-JDBC-add-Postgres-tp5758548p5758657.html
Tim
On 12/11/2014 19:26, Claus Ibsen wrote:
There is a resetAutoCommit option, you can try set that to false
On Mon, Nov 10, 2014 at 1:54 PM, Tim Dudgeon tdudgeon...@gmail.com wrote:
I dug a bit deeper here and I think that Camel is doing something with the
connection, possibly related to autoCommit.
I worked around the problem by writing my own Processor that sets autoCommit
to false, sets maxRows on the statement and then iterates through the
ResultSet and a cursor does seem to be used as expected.
But trying to do the same with the Camel JDBC component fails for reasons
described earlier.
Tim
On 06/11/2014 18:57, Claus Ibsen wrote:
So maybe its your maxRows=1000 that is the problem, so when you are
trying to get rows 1000 then it throws that exception.
What is your use case? You want to select * from a table, but only the
first 1000 rows?
On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com
wrote:
On 05/11/2014 19:07, Claus Ibsen wrote:
maxRowSize ??? do you mean fetchSize?
Sorry, I mean maxRows (as in the example here:
http://camel.apache.org/jdbc.html).
So something like:
.to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000')
Also which version of Camel do you use?
2.14
Tim
Re: Strange issue with JDBC add Postgres
I dug a bit deeper here and I think that Camel is doing something with
the connection, possibly related to autoCommit.
I worked around the problem by writing my own Processor that sets
autoCommit to false, sets maxRows on the statement and then iterates
through the ResultSet and a cursor does seem to be used as expected.
But trying to do the same with the Camel JDBC component fails for
reasons described earlier.
Tim
On 06/11/2014 18:57, Claus Ibsen wrote:
So maybe its your maxRows=1000 that is the problem, so when you are
trying to get rows 1000 then it throws that exception.
What is your use case? You want to select * from a table, but only the
first 1000 rows?
On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote:
On 05/11/2014 19:07, Claus Ibsen wrote:
maxRowSize ??? do you mean fetchSize?
Sorry, I mean maxRows (as in the example here:
http://camel.apache.org/jdbc.html).
So something like:
.to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000')
Also which version of Camel do you use?
2.14
Tim
Re: Strange issue with JDBC add Postgres
I tracked it down a bit. Its something to do with auto commit.
There's something about it here:
http://www.postgresql.org/message-id/083dbfcc-8eac-4a5f-bd61-3a82dfc45...@visualdistortion.org
In my case I tried using the resetAutoCommit=false property on the JDBC
component, and thought it had worked, but it seems that what happened is
that it now runs with autoCommit=true which means the maxRows property
is ignored and the whole ResultSet gets loaded and I then get OOM
errors. So back to square 1 :-(
Seems like Camel is doing something with autoCommit which Postgresql is
not liking.
Tim
On 06/11/2014 18:57, Claus Ibsen wrote:
So maybe its your maxRows=1000 that is the problem, so when you are
trying to get rows 1000 then it throws that exception.
What is your use case? You want to select * from a table, but only the
first 1000 rows?
On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote:
On 05/11/2014 19:07, Claus Ibsen wrote:
maxRowSize ??? do you mean fetchSize?
Sorry, I mean maxRows (as in the example here:
http://camel.apache.org/jdbc.html).
So something like:
.to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000')
Also which version of Camel do you use?
2.14
Tim
Re: Strange issue with JDBC add Postgres
So maybe its your maxRows=1000 that is the problem, so when you are
trying to get rows 1000 then it throws that exception.
What is your use case? You want to select * from a table, but only the
first 1000 rows?
On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote:
On 05/11/2014 19:07, Claus Ibsen wrote:
maxRowSize ??? do you mean fetchSize?
Sorry, I mean maxRows (as in the example here:
http://camel.apache.org/jdbc.html).
So something like:
.to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000')
Also which version of Camel do you use?
2.14
Tim
--
Claus Ibsen
-
Red Hat, Inc.
Email: cib...@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Strange issue with JDBC add Postgres
On 06/11/2014 18:57, Claus Ibsen wrote:
So maybe its your maxRows=1000 that is the problem, so when you are
trying to get rows 1000 then it throws that exception.
What is your use case? You want to select * from a table, but only the
first 1000 rows?
No, it happens without maxRows. I was just experimenting with maxRows to
confirm that the statement.XXX bit was working.
My use case is to select all rows from a very large table and process
them one by one. e.g. something like this:
from('direct:databasequery')
.to('jdbc:myDataSource?outputType=StreamListstatement.fetchSize=100')
.split(body()).streaming()
.log('Processing row')
Without fetchSize Postgresql tries to load the whole table into memory
so it blows up with OOM exception.
With fetchSize in plain Java it works fine.
With fetchSize plus Camel it seems to blow up with that wierd Postgresql
exception.
Tim
Re: Strange issue with JDBC add Postgres
On 05/11/2014 19:07, Claus Ibsen wrote:
maxRowSize ??? do you mean fetchSize?
Sorry, I mean maxRows (as in the example here:
http://camel.apache.org/jdbc.html).
So something like:
.to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000')
Also which version of Camel do you use?
2.14
Tim
