Re: Looping in routes using Direct Component - Performance impact?

[Vitalii Tymchyshyn]

I think you would get into a recusion this way and end up with stack
overflow.
You should try http://camel.apache.org/dynamic-router.html

Best regards, Vitalii Tymchyshyn

13 жовт. 2016 р. 18:31 "Goyal, Arpit"  пише:

> Hi,
>
> We currently are on 2.16.3 version and can't use the loop function (part
> of 2.17). So we have the following routes based on which we create sort of
> loop with switch case. Now we are worried about the performance (when we
> see stacktrace of camel processing)
>
> Route 1:  from("direct:s1).to("read one page of data").process("process
> one page").to("direct:s2");
>
> Route 2:  from("direct:s2").toD("go to external source").when("are more
> pages left").to("direct:s1").otherwise().process("end processing");
>
> Now if there are thousands of pages, it is possible that this loop creates
> lot of objects in memory & can create performance issue? Is there a way for
> Camel to be told, before going to route 1 again, forget all the past? Is
> disabling HISTORY would solve that?
>
> Regards,
> Arpit.
>

Re: Looping in routes using Direct Component - Performance impact?

[Claus Ibsen]

Hi

Yes if you loop 1000 times (or alot) then message history will take up
memory on the exchange as each step in the routes is recored by
default. You can turn this off
http://camel.apache.org/message-history.html

But as Brad says its not ideal to keep looping in Camel routes as they
are not intended to run forever/very long routes.

You can have higher performance from a java bean in a regular for loop
and then if needed use producer template to call a Camel route.





On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit  wrote:
> Hi,
>
> We currently are on 2.16.3 version and can't use the loop function (part of 
> 2.17). So we have the following routes based on which we create sort of loop 
> with switch case. Now we are worried about the performance (when we see 
> stacktrace of camel processing)
>
> Route 1:  from("direct:s1).to("read one page of data").process("process one 
> page").to("direct:s2");
>
> Route 2:  from("direct:s2").toD("go to external source").when("are more pages 
> left").to("direct:s1").otherwise().process("end processing");
>
> Now if there are thousands of pages, it is possible that this loop creates 
> lot of objects in memory & can create performance issue? Is there a way for 
> Camel to be told, before going to route 1 again, forget all the past? Is 
> disabling HISTORY would solve that?
>
> Regards,
> Arpit.



-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Looping in routes using Direct Component - Performance impact?

[Brad Johnson]

I didn't think you were appending the data.  But realize that when you call
direct or direct-vm it is a request/response style of route.  So you are
calling route 1 which calls route 2 which calls route 1 and are not
returning as far as I know.  So you are essentially making the equivalent
of a recursive method call.  You should end up with a stack overflow that
way eventually not because of the size of the individual messages but due
to the number of calls from 1 to 2 and 2 to 1 and 1 to 2 and so on.
Essentially you keep filling up the stack with each new call.

There is a chapter on the POJO approach in the Camel in Action book with
the same sort of recommendation.  All of chapter 4 is about it and one
section, for example, is called "using beans the easy way" and another
covers annotations. Hopefully in the new book (V2) they'll emphasize it a
bit more. Most of the time you don't need to specify the annotations if you
are passing a bean in the body that Camel can identify as the type and
method to invoke.  Otherwise you can annotate the method call and type with
@Body and if you need to pass in a header you use @Header.  If you want to
specify the method to call I think it's @Handler (can't be sure as I don't
use it most of the time).

http://camel.apache.org/bean.html
http://camel.apache.org/parameter-binding-annotations.html

On Fri, Oct 14, 2016 at 4:07 PM, Goyal, Arpit <arpit.go...@sap.com> wrote:

> Hi Brad,
>
> TOPIC 1:
> ---
> "StackOverflowError" - this is what happened with the current Route 2
> calling Route 1 for LARGE # OF PAGES. It was not about the data, because
> the body is not getting appended and is always limited to one page (at
> least in most cases).
>
> JVM/stack-size exploded around about 200 calls to the loop :) But yeah a
> good learning of what not to do :) Moved the Loop outside the route
> completely and based on Route 2 response, triggered the Route 1 again.
>
> Route 1 to Route 2 linkage was done only for modularization purpose.
>
> TOPIC 2:
> --
> Very interesting point you have raised here about using POJO and not
> Processors. I actually wanted to know is there some explanation of what you
> said in CAMEL IN ACTION book. Just got hand of hard-copy and see if I can
> understand that better.
>
> I did use split and bean binding, but not sure if that is the right POJO
> approach? I guess I should use @ExchangeProperty
> - public List splitMessage(@Body Object body, Exchange
> exchange)
>
> Current use cases which processor is doing:
> - set / get exchange headers (multiple)
> - set / get camel exchange property (so far single)
> - saw annotation ExchangeProperty, but it has String type return
> value. Whereas on exchange we are setting Objects.
>
> Regards,
> Arpit.
>
>
> -Original Message-
> From: Brad Johnson [mailto:brad.john...@mediadriver.com]
> Sent: Friday, October 14, 2016 8:54 AM
> To: users@camel.apache.org
> Subject: Re: Looping in routes using Direct Component - Performance impact?
>
> Have you looked at splitter/tokenizer/streaming for this instead of having
> route 1 call route 2 which calls route 1?  Interesting route setup as I've
> never used routes in a recursive fashion like that. When using direct the
> return doesn't happen until the end of the last route in the chain.  In
> this case it appears that you are continually delaying the return from the
> route via this iterative approach.  I'd have to check that via testing to
> see what's really happening in that case but I've never done it like that
> and would assume it would eventually run out of memory for large amounts of
> data.  As far as I know you can't just erase the history though you could
> set the body to null or empty string.  But I wouldn't do that.
>
> Just unmarshal your data and run it through the splitter in a streaming
> fashion.  If you don't need to process these in serial order you could then
> drop them on a SEDA queue with multiple consumer threads. You can then also
> set the queue size to limit how many documents are coming through at a
> time.
>
> Also, unless you have a compelling reason don't use Processors as they
> couple you to the framework. Just use a POJO and let Camel use reflection
> to select the correct method for processing or use annotations.  As a
> general rule of thumb I use, if you don't know exactly why you need to use
> a Processor you don't need to use it.
>
>
> On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <arpit.go...@sap.com> wrote:
>
> > I can't compromise my business logic and there is hardly any data; but
> > each time camel Route 2 starts Route 1, the previous route processing
> goes
> > onto stack. I only have limited stack after that any operati

RE: Looping in routes using Direct Component - Performance impact?

[Goyal, Arpit]

Hi Brad,

TOPIC 1: 
---
"StackOverflowError" - this is what happened with the current Route 2 calling 
Route 1 for LARGE # OF PAGES. It was not about the data, because the body is 
not getting appended and is always limited to one page (at least in most cases).

JVM/stack-size exploded around about 200 calls to the loop :) But yeah a good 
learning of what not to do :) Moved the Loop outside the route completely and 
based on Route 2 response, triggered the Route 1 again. 

Route 1 to Route 2 linkage was done only for modularization purpose. 

TOPIC 2:
--
Very interesting point you have raised here about using POJO and not 
Processors. I actually wanted to know is there some explanation of what you 
said in CAMEL IN ACTION book. Just got hand of hard-copy and see if I can 
understand that better. 

I did use split and bean binding, but not sure if that is the right POJO 
approach? I guess I should use @ExchangeProperty 
- public List splitMessage(@Body Object body, Exchange 
exchange)

Current use cases which processor is doing: 
- set / get exchange headers (multiple)
- set / get camel exchange property (so far single)
- saw annotation ExchangeProperty, but it has String type return value. 
Whereas on exchange we are setting Objects. 

Regards,
Arpit.


-Original Message-
From: Brad Johnson [mailto:brad.john...@mediadriver.com] 
Sent: Friday, October 14, 2016 8:54 AM
To: users@camel.apache.org
Subject: Re: Looping in routes using Direct Component - Performance impact?

Have you looked at splitter/tokenizer/streaming for this instead of having
route 1 call route 2 which calls route 1?  Interesting route setup as I've
never used routes in a recursive fashion like that. When using direct the
return doesn't happen until the end of the last route in the chain.  In
this case it appears that you are continually delaying the return from the
route via this iterative approach.  I'd have to check that via testing to
see what's really happening in that case but I've never done it like that
and would assume it would eventually run out of memory for large amounts of
data.  As far as I know you can't just erase the history though you could
set the body to null or empty string.  But I wouldn't do that.

Just unmarshal your data and run it through the splitter in a streaming
fashion.  If you don't need to process these in serial order you could then
drop them on a SEDA queue with multiple consumer threads. You can then also
set the queue size to limit how many documents are coming through at a time.

Also, unless you have a compelling reason don't use Processors as they
couple you to the framework. Just use a POJO and let Camel use reflection
to select the correct method for processing or use annotations.  As a
general rule of thumb I use, if you don't know exactly why you need to use
a Processor you don't need to use it.


On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <arpit.go...@sap.com> wrote:

> I can't compromise my business logic and there is hardly any data; but
> each time camel Route 2 starts Route 1, the previous route processing goes
> onto stack. I only have limited stack after that any operation can fail.
>
> To me below approach seems risky...I need to go async or remove the
> looping out of the route overall.
>
> Regards,
> Arpit.
>
> -Original Message-
> From: souciance [mailto:souciance.eqdam.ras...@gmail.com]
> Sent: Thursday, October 13, 2016 3:45 PM
> To: users@camel.apache.org
> Subject: Re: Looping in routes using Direct Component - Performance impact?
>
> I think you should first run your route setup with a few pages, 10s of
> pages, hundreds of pages and then thousands of pages and then do
> performance tests to see where what is eating memory and optimise from
> there.
>
> On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] <
> ml-node+s465427n5788744...@n5.nabble.com> wrote:
>
> > Hi,
> >
> > We currently are on 2.16.3 version and can't use the loop function (part
> > of 2.17). So we have the following routes based on which we create sort
> of
> > loop with switch case. Now we are worried about the performance (when we
> > see stacktrace of camel processing)
> >
> > Route 1:  from("direct:s1).to("read one page of data").process("process
> > one page").to("direct:s2");
> >
> > Route 2:  from("direct:s2").toD("go to external source").when("are more
> > pages left").to("direct:s1").otherwise().process("end processing");
> >
> > Now if there are thousands of pages, it is possible that this loop
> creates
> > lot of objects in memory & can create performance issue? Is there a way
> for
> > Camel to be told, before going to route 1 again, forget all the pas

Re: Looping in routes using Direct Component - Performance impact?

[souciance]

Also, use seda for async processing and you can shutoff some camel stuff
like jmx, original message etc for increased performance. But for
performance you need to do good tests and make a change and repeat those
tests and measure.

On Fri, Oct 14, 2016 at 5:54 PM, Brad Johnson [via Camel] <
ml-node+s465427n578881...@n5.nabble.com> wrote:

> Have you looked at splitter/tokenizer/streaming for this instead of having
> route 1 call route 2 which calls route 1?  Interesting route setup as I've
> never used routes in a recursive fashion like that. When using direct the
> return doesn't happen until the end of the last route in the chain.  In
> this case it appears that you are continually delaying the return from the
> route via this iterative approach.  I'd have to check that via testing to
> see what's really happening in that case but I've never done it like that
> and would assume it would eventually run out of memory for large amounts
> of
> data.  As far as I know you can't just erase the history though you could
> set the body to null or empty string.  But I wouldn't do that.
>
> Just unmarshal your data and run it through the splitter in a streaming
> fashion.  If you don't need to process these in serial order you could
> then
> drop them on a SEDA queue with multiple consumer threads. You can then
> also
> set the queue size to limit how many documents are coming through at a
> time.
>
> Also, unless you have a compelling reason don't use Processors as they
> couple you to the framework. Just use a POJO and let Camel use reflection
> to select the correct method for processing or use annotations.  As a
> general rule of thumb I use, if you don't know exactly why you need to use
> a Processor you don't need to use it.
>
>
> On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <[hidden email]
> <http:///user/SendEmail.jtp?type=node=5788811=0>> wrote:
>
> > I can't compromise my business logic and there is hardly any data; but
> > each time camel Route 2 starts Route 1, the previous route processing
> goes
> > onto stack. I only have limited stack after that any operation can fail.
> >
> > To me below approach seems risky...I need to go async or remove the
> > looping out of the route overall.
> >
> > Regards,
> > Arpit.
> >
> > -Original Message-
> > From: souciance [mailto:[hidden email]
> <http:///user/SendEmail.jtp?type=node=5788811=1>]
> > Sent: Thursday, October 13, 2016 3:45 PM
> > To: [hidden email]
> <http:///user/SendEmail.jtp?type=node=5788811=2>
> > Subject: Re: Looping in routes using Direct Component - Performance
> impact?
> >
> > I think you should first run your route setup with a few pages, 10s of
> > pages, hundreds of pages and then thousands of pages and then do
> > performance tests to see where what is eating memory and optimise from
> > there.
> >
> > On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] <
> > [hidden email] <http:///user/SendEmail.jtp?type=node=5788811=3>>
> wrote:
> >
> > > Hi,
> > >
> > > We currently are on 2.16.3 version and can't use the loop function
> (part
> > > of 2.17). So we have the following routes based on which we create
> sort
> > of
> > > loop with switch case. Now we are worried about the performance (when
> we
> > > see stacktrace of camel processing)
> > >
> > > Route 1:  from("direct:s1).to("read one page of
> data").process("process
> > > one page").to("direct:s2");
> > >
> > > Route 2:  from("direct:s2").toD("go to external source").when("are
> more
> > > pages left").to("direct:s1").otherwise().process("end processing");
> > >
> > > Now if there are thousands of pages, it is possible that this loop
> > creates
> > > lot of objects in memory & can create performance issue? Is there a
> way
> > for
> > > Camel to be told, before going to route 1 again, forget all the past?
> Is
> > > disabling HISTORY would solve that?
> > >
> > > Regards,
> > > Arpit.
> > >
> > >
> > > --
> > > If you reply to this email, your message will be added to the
> discussion
> > > below:
> > > http://camel.465427.n5.nabble.com/Looping-in-routes-using-
> > > Direct-Component-Performance-impact-tp5788744.html
> > > To start a new topic under Camel - Users, email
> > > [hidden email] <http:///user/SendEmail.jtp?type=node=5788811=4>
> > > To unsubs

Re: Looping in routes using Direct Component - Performance impact?

[Brad Johnson]

Have you looked at splitter/tokenizer/streaming for this instead of having
route 1 call route 2 which calls route 1?  Interesting route setup as I've
never used routes in a recursive fashion like that. When using direct the
return doesn't happen until the end of the last route in the chain.  In
this case it appears that you are continually delaying the return from the
route via this iterative approach.  I'd have to check that via testing to
see what's really happening in that case but I've never done it like that
and would assume it would eventually run out of memory for large amounts of
data.  As far as I know you can't just erase the history though you could
set the body to null or empty string.  But I wouldn't do that.

Just unmarshal your data and run it through the splitter in a streaming
fashion.  If you don't need to process these in serial order you could then
drop them on a SEDA queue with multiple consumer threads. You can then also
set the queue size to limit how many documents are coming through at a time.

Also, unless you have a compelling reason don't use Processors as they
couple you to the framework. Just use a POJO and let Camel use reflection
to select the correct method for processing or use annotations.  As a
general rule of thumb I use, if you don't know exactly why you need to use
a Processor you don't need to use it.


On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <arpit.go...@sap.com> wrote:

> I can't compromise my business logic and there is hardly any data; but
> each time camel Route 2 starts Route 1, the previous route processing goes
> onto stack. I only have limited stack after that any operation can fail.
>
> To me below approach seems risky...I need to go async or remove the
> looping out of the route overall.
>
> Regards,
> Arpit.
>
> -Original Message-
> From: souciance [mailto:souciance.eqdam.ras...@gmail.com]
> Sent: Thursday, October 13, 2016 3:45 PM
> To: users@camel.apache.org
> Subject: Re: Looping in routes using Direct Component - Performance impact?
>
> I think you should first run your route setup with a few pages, 10s of
> pages, hundreds of pages and then thousands of pages and then do
> performance tests to see where what is eating memory and optimise from
> there.
>
> On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] <
> ml-node+s465427n5788744...@n5.nabble.com> wrote:
>
> > Hi,
> >
> > We currently are on 2.16.3 version and can't use the loop function (part
> > of 2.17). So we have the following routes based on which we create sort
> of
> > loop with switch case. Now we are worried about the performance (when we
> > see stacktrace of camel processing)
> >
> > Route 1:  from("direct:s1).to("read one page of data").process("process
> > one page").to("direct:s2");
> >
> > Route 2:  from("direct:s2").toD("go to external source").when("are more
> > pages left").to("direct:s1").otherwise().process("end processing");
> >
> > Now if there are thousands of pages, it is possible that this loop
> creates
> > lot of objects in memory & can create performance issue? Is there a way
> for
> > Camel to be told, before going to route 1 again, forget all the past? Is
> > disabling HISTORY would solve that?
> >
> > Regards,
> > Arpit.
> >
> >
> > --
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://camel.465427.n5.nabble.com/Looping-in-routes-using-
> > Direct-Component-Performance-impact-tp5788744.html
> > To start a new topic under Camel - Users, email
> > ml-node+s465427n465428...@n5.nabble.com
> > To unsubscribe from Camel - Users, click here
> > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=
> unsubscribe_by_code=465428=c291Y2lhbmNlLmVxZGFtLnJhc2h0aU
> BnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> > .
> > NAML
> > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_
> viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/Looping-in-routes-using-Direct-Component-Performance-
> impact-tp5788744p5788745.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

RE: Looping in routes using Direct Component - Performance impact?

[Goyal, Arpit]

I can't compromise my business logic and there is hardly any data; but each 
time camel Route 2 starts Route 1, the previous route processing goes onto 
stack. I only have limited stack after that any operation can fail. 

To me below approach seems risky...I need to go async or remove the looping out 
of the route overall. 

Regards,
Arpit.

-Original Message-
From: souciance [mailto:souciance.eqdam.ras...@gmail.com] 
Sent: Thursday, October 13, 2016 3:45 PM
To: users@camel.apache.org
Subject: Re: Looping in routes using Direct Component - Performance impact?

I think you should first run your route setup with a few pages, 10s of
pages, hundreds of pages and then thousands of pages and then do
performance tests to see where what is eating memory and optimise from
there.

On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] <
ml-node+s465427n5788744...@n5.nabble.com> wrote:

> Hi,
>
> We currently are on 2.16.3 version and can't use the loop function (part
> of 2.17). So we have the following routes based on which we create sort of
> loop with switch case. Now we are worried about the performance (when we
> see stacktrace of camel processing)
>
> Route 1:  from("direct:s1).to("read one page of data").process("process
> one page").to("direct:s2");
>
> Route 2:  from("direct:s2").toD("go to external source").when("are more
> pages left").to("direct:s1").otherwise().process("end processing");
>
> Now if there are thousands of pages, it is possible that this loop creates
> lot of objects in memory & can create performance issue? Is there a way for
> Camel to be told, before going to route 1 again, forget all the past? Is
> disabling HISTORY would solve that?
>
> Regards,
> Arpit.
>
>
> ------
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Looping-in-routes-using-
> Direct-Component-Performance-impact-tp5788744.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428...@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=465428=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: 
http://camel.465427.n5.nabble.com/Looping-in-routes-using-Direct-Component-Performance-impact-tp5788744p5788745.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Looping in routes using Direct Component - Performance impact?

[souciance]

I think you should first run your route setup with a few pages, 10s of
pages, hundreds of pages and then thousands of pages and then do
performance tests to see where what is eating memory and optimise from
there.

On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] <
ml-node+s465427n5788744...@n5.nabble.com> wrote:

> Hi,
>
> We currently are on 2.16.3 version and can't use the loop function (part
> of 2.17). So we have the following routes based on which we create sort of
> loop with switch case. Now we are worried about the performance (when we
> see stacktrace of camel processing)
>
> Route 1:  from("direct:s1).to("read one page of data").process("process
> one page").to("direct:s2");
>
> Route 2:  from("direct:s2").toD("go to external source").when("are more
> pages left").to("direct:s1").otherwise().process("end processing");
>
> Now if there are thousands of pages, it is possible that this loop creates
> lot of objects in memory & can create performance issue? Is there a way for
> Camel to be told, before going to route 1 again, forget all the past? Is
> disabling HISTORY would solve that?
>
> Regards,
> Arpit.
>
>
> ------
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Looping-in-routes-using-
> Direct-Component-Performance-impact-tp5788744.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428...@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=465428=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: 
http://camel.465427.n5.nabble.com/Looping-in-routes-using-Direct-Component-Performance-impact-tp5788744p5788745.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Looping in routes using Direct Component - Performance impact?

[Goyal, Arpit]

Hi,

We currently are on 2.16.3 version and can't use the loop function (part of 
2.17). So we have the following routes based on which we create sort of loop 
with switch case. Now we are worried about the performance (when we see 
stacktrace of camel processing)

Route 1:  from("direct:s1).to("read one page of data").process("process one 
page").to("direct:s2");

Route 2:  from("direct:s2").toD("go to external source").when("are more pages 
left").to("direct:s1").otherwise().process("end processing");

Now if there are thousands of pages, it is possible that this loop creates lot 
of objects in memory & can create performance issue? Is there a way for Camel 
to be told, before going to route 1 again, forget all the past? Is disabling 
HISTORY would solve that?

Regards,
Arpit.