Hi, I might be misusing the ConsumerTemplate. Given that its a polling consumer,and with File Endpoint it will keep polling for a directory(and not a single file) once started. My requirement is to consume a single file,and a large number of those too.
I might want to use the conventional Java file reading here. Thanks a lot for you help. Chirag Dewan ________________________________ From: Chirag Dewan <chirag.dewa...@yahoo.in> To: "users@camel.apache.org" <users@camel.apache.org> Sent: Thursday, 20 February 2014 4:06 PM Subject: Re: OOM issue due to MemoryIdempotentRepository Hi, After setting the max cache for consumer template the threads are now in WAITING state instead of TIMED_WAITING state. Actually I want to consume a single file with a filename in my application. So once a file is consumed,that endpoint cannot be reused and for every file a new endpoint is created. Is there a better way of doing this? And also the memory consumed by MemoryIdempotentRepository,could this be the reason for that? Thanks Chirag ________________________________ From: Claus Ibsen <claus.ib...@gmail.com> To: "users@camel.apache.org" <users@camel.apache.org> Sent: Thursday, 20 February 2014 3:46 PM Subject: Re: OOM issue due to MemoryIdempotentRepository Hi That is because the endpoint uri, is unique, eg you put the index in there. Then the endpoint cannot be reused for next poll. See this FAQ how you can lower the pool size http://camel.apache.org/how-do-i-configure-the-default-maximum-cache-size-for-producercache-or-producertemplate.html You can set it to 1 or 0 to not have any pooling. On Thu, Feb 20, 2014 at 10:53 AM, Chirag Dewan <chirag.dewa...@yahoo.in> wrote: > Hi Claus, > > Sorry for the late reply. > > I wrote a simple code to test this behaviour: > > CamelContext con = new DefaultCamelContext(); > > ConsumerTemplate template = con.createConsumerTemplate(); > > template.start(); > > int i = 0; > > while (i < 100) { > > > Exchange exchange = template.receive( > "file:/sourceDir?fileName=abc (" + i > + "th copy).txt&delete=true", 10000); > > System.out.println(exchange + " " + i); > > > template.doneUoW(exchange); > > i++; > > > > } > > template.stop(); > > Even on profiling this I see a thread being created for every file consumed > by the template. And the memory continous to grow. > > Thanks. > > Chirag Dewan > > > > > > > ________________________________ > From: Claus Ibsen <claus.ib...@gmail.com> > To: "users@camel.apache.org" <users@camel.apache.org> > Sent: Wednesday, 19 February 2014 2:59 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > Hi > > What is the exact url you use in the consumer template? > And are you 100% sure you only create and use exactly one instance of > the consumer template? > > On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <chirag.dewa...@yahoo.in> wrote: >> >> >> Hi Claus, >> >> I had this implementation >> before. With this I saw an unusual behaviour. A thread was created for >> every file consumed by the template and even though I called >> template.doneUoW the thread is not stopped. So if I call doneUoW(and not >> stop) the thread should be destroyed? And also on doneUoW the >> inprogressRepository cache is cleared. Which is not getting cleared at >> the moment(the reason I believe is causing the OOM) . Is there a possibility >> that doneUoW is not working as it should? >> >> Thanks . >> >> Chirag >> >> ________________________________ >> From: Chirag Dewan <chirag.dewa...@yahoo.in> >> To: "users@camel.apache.org" <users@camel.apache.org> >> Sent: Monday, 17 February 2014 1:53 PM >> Subject: Re: OOM issue due to MemoryIdempotentRepository >> >> >> Hi Claus, >> >> I had this implementation before. With this I saw an unusual behaviour. A >> thread was created for every file consumed by the template and even though I >> called template.doneUoW the thread is not stopped. So if I call doneUoW(and >> not stop) the thread should be destroyed? And also on doneUoW the >> inprogressRepository cache is cleared. Which is not getting cleared at the >> moment(the reason I believe is causing the . Is there a possibility that >> doneUoW is not working as it should? >> >> Thanks . >> >> Chirag >> >> >> >> >> ________________________________ >> >> From: Claus Ibsen <claus.ib...@gmail.com> >> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan >> <chirag.dewa...@yahoo.in> >> Sent: Monday, 17 February 2014 1:28 PM >> Subject: Re: OOM issue due to MemoryIdempotentRepository >> >> >> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <chirag.dewa...@yahoo.in> >> wrote: >>> Hi Claus, >>> >>> So I dont need to start/stop the template for every file? >>> >>> Right now I create a consumerTemplate at satrtup and then on every method >>> call(responsible for consuming file) I start the template,consume the >>> file(template.receive) and stop the template. >>> >>> Is this the right approach? >>> >> >> No, only start|stop the template once on startup and shutdown of your app. >> And then call doneUoW for each file you consume. >> >> >>> Thanks. >>> >>> Chirag >>> >>> >>> >>> >>> ________________________________ >>> From: Claus Ibsen <claus.ib...@gmail.com> >>> To: "users@camel.apache.org" <users@camel.apache.org> >>> Sent: Monday, 17 February 2014 12:33 PM >>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>> >>> >>> Hi >>> >>> You should likely reuse the consumer template instead. As this FAQ >>> applies to it too >>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html >>> >>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <chirag.dewa...@yahoo.in> >>> wrote: >>>> Hi All, >>>> >>>> Any views on this? >>>> >>>> Actually my previous mail is not understandable. Let me rephrase the issue. >>>> >>>> I am using consumer template to consume single file using fileName >>>> property. I have a single instance of consumerTemplate and I call doneUoW >>>> and stop after consuming the file. >>>> >>>> I am running a test case where 10000 files are consumed by >>>> consumerTemplate(one file at a time) . After a while JVM goes OOM and >>>> crashes. In heap dump,I can see large number of MemoryIdempotentRepository >>>> instances. I am not using idempotent=true. I figured out that >>>> inProgressRepository also uses this cache. >>>> >>>> Is there something I am missing here? >>>> >>>> Thanks in advance. >>>> >>>> Chirag Dewan >>>> >>>> >>>> >>>> >>>> ________________________________ >>>> From: Chirag Dewan <chirag.dewa...@yahoo.in> >>>> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan >>>> <chirag.dewa...@yahoo.in> >>>> Sent: Friday, 14 February 2014 3:22 PM >>>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>>> >>>> >>>> Hi All, >>>> >>>> I have been looking around why a large number of heap is being consumed by >>>> MemoryIdempotentRepository. I saw one JIRA as well regarding >>>> inProgressRepository not getting cleared. Can that be the reason here? >>>> >>>> Also,I am not using consumerTemplate to consume 1000o files. Does that >>>> also use inProgressRepository? And when is it cleared from the memory? >>>> >>>> Any help is much appreciated. >>>> >>>> Thanks! >>>> >>>> Chirag Dewan >>>> >>>> >>>> >>>> >>>> ________________________________ >>>> >>>> From: Chirag Dewan <chirag.dewa...@yahoo.in> >>>> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan >>>> <chirag.dewa...@yahoo.in> >>>> Sent: Wednesday, 12 February 2014 6:31 PM >>>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>>> >>>> >>>> Hi, >>>> >>>> And I am using readLock=changed. So that can be a reason too? >>>> >>>> Chirag Dewan >>>> >>>> >>>> >>>> >>>> ________________________________ >>>> >>>> From: Chirag Dewan <chirag.dewa...@yahoo.in> >>>> To: "users@camel.apache.org" <users@camel.apache.org> >>>> Sent: Wednesday, 12 February 2014 6:07 PM >>>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>>> >>>> >>>> Hi Claus, >>>> >>>> Thanks for the quick reply. >>>> >>>> It might well be. But there is one thing I would like to get some insight >>>> into. >>>> >>>> If idempotent=false(default) is used,will the LRU Cache still store the >>>> file referrence? Because that is what I can see in my memory dump. >>>> >>>> org.apache.camel.util.LRUCache >>>> >>>>>org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap >>>> >>>> I maybe doing something wrong here. Is there anything else I need to do to >>>> make sure that file is deleted after it has been processed and not stored >>>> in the cache? >>>> >>>> Thanks. >>>> >>>> Chirag Dewan >>>> >>>> >>>> >>>> >>>> >>>> ________________________________ >>>> >>>> From: Claus Ibsen <claus.ib...@gmail.com> >>>> To: "users@camel.apache.org" <users@camel.apache.org> >>>> Sent: Wednesday, 12 February 2014 5:42 PM >>>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>>> >>>> >>>> Hi >>>> >>>> Some days ago someone else posted about a OOME issue when using Storm. >>>> It smells like a Storm + Camel issue somewhere. >>>> >>>> >>>> >>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <chirag.dewa...@yahoo.in> >>>> wrote: >>>>> Hi All, >>>>> >>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in >>>>> my application. >>>>> >>>>> >>>>> Now I intent to delete the file after consuming it. So I have delete=true >>>>> in my route. After consuming around 6000 files(out of 10000,as my test >>>>> case) JVM goes Out of memory. >>>>> >>>>> >>>>> In my heap dump I can see the following trace as the major memory >>>>> consumers: >>>>> >>>>> java.util.ArrayList >>>>> >java.lang.Object >>>>> >>>>> >>>>>>org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService >>>>> >>>>> >org.apache.camel.component.file.FileEndpoint >>>>> >>>>> >>>>>>org.apache.camel.processor.idempotent.MemoryIdempotentRepository >>>>> >org.apache.camel.util.LRUCache >>>>> >>>>>>org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap >>>>> >>>>> >>>>> So if I am not using indempotent,should it still store the files in the >>>>> memory? Is this memory repository used,even if I am deleting the files >>>>> and not storing in memory? Is there a way I can avoid that? >>>>> >>>>> I am testing this scenario under Storm as execution environment and with >>>>> 2GB max heap space. >>>>> >>>>> Thanks in advance >>>>> >>>>> Chirag Dewan >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> ----------------- >>>> Red Hat, Inc. >>>> Email: cib...@redhat.com >>>> Twitter: davsclaus >>>> Blog: http://davsclaus.com >>>> Author of Camel in Action: http://www.manning.com/ibsen >>>> Make your Camel applications look hawt, try: http://hawt.io > >> >>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> Red Hat, Inc. >>> Email: cib...@redhat.com >>> Twitter: davsclaus >>> Blog: http://davsclaus.com >>> Author of Camel in Action: http://www.manning.com/ibsen >>> Make your Camel applications look hawt, try: http://hawt.io >> >> >> >> -- >> Claus Ibsen >> ----------------- >> Red Hat, Inc. >> Email: cib...@redhat.com >> Twitter: davsclaus >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> Make your Camel applications look hawt, try: http://hawt.io > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: cib...@redhat.com > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen > Make your Camel applications look hawt, try: http://hawt.io -- Claus Ibsen ----------------- Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen Make your Camel applications look hawt, try: http://hawt.io
