Re: Best way to poll a database and process in chunks

Hi David, I think you will find that binding a bigger number of parameters won't lead to better performance, both with missing statement cache due to fluctuating number of parameters and due to not using indexes and performing full table scans. But you should run SQL explain and gather statistics

Re: Best way to poll a database and process in chunks

Hi Maybe you can try and let us see what it goes. I am not so sure we can make this in a way to make it not complicated. How would you specify what those IN keys are from a select statement that can return basically anything. On Thu, Apr 6, 2017 at 6:59 PM, dvlato wrote: >

Re: Best way to poll a database and process in chunks

Hello Claus, Ibsen: Thank you very much for the thorough responses. We don't really need to use the JPA component, and I think it makes more sense to use sql/jdbc for the kind of bulk operations we want to do. What I think it would be really useful is to allow for the "onConsumeBatchComplete"

Re: Best way to poll a database and process in chunks

I didn't see the original message from dvlato. We are doing pretty much exactly what you describe. We are using blueprint.xml to describe our routes. We start with a timer Then we have some intermediate code using a bean that determines status we are going to look for - in our case records

Re: Best way to poll a database and process in chunks

Yeah The other sql components are likely a bit better at this, such as mybatis or the sql component. For the latter you can set useIterator=false to get the List. And you can set that maxMessagesPerPoll to 10 but that is after the poll, so try to set it on the jdbc driver, eg on

Best way to poll a database and process in chunks

Hello, I have just started working with Camel and I have what I believe is a pretty common requirement: poll the database to fetch all the records with a certain status, and process them in chunks (let's 10 rows at a time). I though I could leverage the existing components to do this easily but I