Hi Brad, TOPIC 1: ----------- "StackOverflowError" - this is what happened with the current Route 2 calling Route 1 for LARGE # OF PAGES. It was not about the data, because the body is not getting appended and is always limited to one page (at least in most cases).
JVM/stack-size exploded around about 200 calls to the loop :) But yeah a good learning of what not to do :) Moved the Loop outside the route completely and based on Route 2 response, triggered the Route 1 again. Route 1 to Route 2 linkage was done only for modularization purpose. TOPIC 2: ---------- Very interesting point you have raised here about using POJO and not Processors. I actually wanted to know is there some explanation of what you said in CAMEL IN ACTION book. Just got hand of hard-copy and see if I can understand that better. I did use split and bean binding, but not sure if that is the right POJO approach? I guess I should use @ExchangeProperty - public List<Message> splitMessage(@Body Object body, Exchange exchange) Current use cases which processor is doing: - set / get exchange headers (multiple) - set / get camel exchange property (so far single) - saw annotation ExchangeProperty, but it has String type return value. Whereas on exchange we are setting Objects. Regards, Arpit. -----Original Message----- From: Brad Johnson [mailto:brad.john...@mediadriver.com] Sent: Friday, October 14, 2016 8:54 AM To: email@example.com Subject: Re: Looping in routes using Direct Component - Performance impact? Have you looked at splitter/tokenizer/streaming for this instead of having route 1 call route 2 which calls route 1? Interesting route setup as I've never used routes in a recursive fashion like that. When using direct the return doesn't happen until the end of the last route in the chain. In this case it appears that you are continually delaying the return from the route via this iterative approach. I'd have to check that via testing to see what's really happening in that case but I've never done it like that and would assume it would eventually run out of memory for large amounts of data. As far as I know you can't just erase the history though you could set the body to null or empty string. But I wouldn't do that. Just unmarshal your data and run it through the splitter in a streaming fashion. If you don't need to process these in serial order you could then drop them on a SEDA queue with multiple consumer threads. You can then also set the queue size to limit how many documents are coming through at a time. Also, unless you have a compelling reason don't use Processors as they couple you to the framework. Just use a POJO and let Camel use reflection to select the correct method for processing or use annotations. As a general rule of thumb I use, if you don't know exactly why you need to use a Processor you don't need to use it. On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <arpit.go...@sap.com> wrote: > I can't compromise my business logic and there is hardly any data; but > each time camel Route 2 starts Route 1, the previous route processing goes > onto stack. I only have limited stack after that any operation can fail. > > To me below approach seems risky...I need to go async or remove the > looping out of the route overall. > > Regards, > Arpit. > > -----Original Message----- > From: souciance [mailto:souciance.eqdam.ras...@gmail.com] > Sent: Thursday, October 13, 2016 3:45 PM > To: firstname.lastname@example.org > Subject: Re: Looping in routes using Direct Component - Performance impact? > > I think you should first run your route setup with a few pages, 10s of > pages, hundreds of pages and then thousands of pages and then do > performance tests to see where what is eating memory and optimise from > there. > > On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] < > ml-node+s465427n5788744...@n5.nabble.com> wrote: > > > Hi, > > > > We currently are on 2.16.3 version and can't use the loop function (part > > of 2.17). So we have the following routes based on which we create sort > of > > loop with switch case. Now we are worried about the performance (when we > > see stacktrace of camel processing) > > > > Route 1: from("direct:s1).to("read one page of data").process("process > > one page").to("direct:s2"); > > > > Route 2: from("direct:s2").toD("go to external source").when("are more > > pages left").to("direct:s1").otherwise().process("end processing"); > > > > Now if there are thousands of pages, it is possible that this loop > creates > > lot of objects in memory & can create performance issue? Is there a way > for > > Camel to be told, before going to route 1 again, forget all the past? Is > > disabling HISTORY would solve that? > > > > Regards, > > Arpit. > > > > > > ------------------------------ > > If you reply to this email, your message will be added to the discussion > > below: > > http://camel.465427.n5.nabble.com/Looping-in-routes-using- > > Direct-Component-Performance-impact-tp5788744.html > > To start a new topic under Camel - Users, email > > ml-node+s465427n465428...@n5.nabble.com > > To unsubscribe from Camel - Users, click here > > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro= > unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aU > BnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=> > > . > > NAML > > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_ > viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.view.web.template.NodeNamespace&breadcrumbs= > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > -- > View this message in context: http://camel.465427.n5.nabble. > com/Looping-in-routes-using-Direct-Component-Performance- > impact-tp5788744p5788745.html > Sent from the Camel - Users mailing list archive at Nabble.com. >