thanks
On Thu, Jan 11, 2018 at 3:09 PM, Claus Ibsen <claus.ib...@gmail.com> wrote:
> Hi
>
> Yes its a little bug, its easier to reproduce if you make the
> processing slower, and speedup the file polling consumer to scan
> faster.
>
>
>
> On Tue, Jan 9, 2018 at 8:26 AM, Claus Ibsen <claus.ib...@gmail.com> wrote:
> > Hi
> >
> > Have patience, we will take a look at this later when we get time.
> >
> >
> >
> > On Tue, Jan 9, 2018 at 1:52 AM, Bhuvan Gupta <bhuva...@gmail.com> wrote:
> >> Humble request to validate my findings and confirming that it is a bug
> >>
> >> On 06-Jan-2018 5:06 PM, "Bhuvan Gupta" <bhuva...@gmail.com> wrote:
> >>
> >> Camel Users,
> >>
> >> I have found the issue
> >> First lets go over the code routines that are executed for a route like
> >>
> >> from("file:stageDummy")
> >> .threads(1)
> >> .process(myProcess)
> >>
> >> [1] In the above route there will be one polling thread (Let call it as
> "
> >> *pollingThread*",) which poll the route directory and fetch all the
> files
> >> in it. (link
> >> <https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152
> e7d26eb60b/camel-core/src/main/java/org/apache/camel/
> component/file/FileConsumer.java#L75>
> >> )
> >> [2] Then fo reach file *pollingThread *does a
> GenericFileConsumer.isValid
> >> check , inside isValid it checks if the file is processed using
> LRUcache of
> >> size 1000 (link)
> >> <https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a
> 7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/
> GenericFileConsumer.java#L541>
> >> [3] If file is Valid then *pollingThread *submit the task to
> >> executorService having 1 thread(Lets Name is it CamelThread#1)
> >> [4] Inside CamelThread#1 does following things
> >> [1] Call the processor
> >> [1] moving the file to .camel
> >> [2] clear up the .camelLock file
> >> [3] Remove the entry from LRU cache ----- IMPORTANT Point (link
> >> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4
> 610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/
> GenericFileOnCompletion.java#L95>
> >> )
> >>
> >>
> >> Now if the two thread execution in the following timeline fashion THEN
> we
> >> will end up with the same file been processed twice.
> >>
> >> [1] *pollingThread*: poll and fetch all the files in it let say we get
> >> [File1]
> >> [2] *pollingThread*: send the file for isValid Check
> >> [3] *pollingThread*: isValid Check passes for File1 as it is not
> present in
> >> LRU cahce
> >> [4] *pollingThread*: Submit the File1 exchange to CamelThread#1
> >> [5] *pollingThread*: poll and fetch all the files in it let say we get
> >> [File1, File1.camelLock]
> >> [6] *CamelThread#1*: call processor for File1 and then remove File1
> entry
> >> from LRU cache
> >> [7] *pollingThread*: send the file for isValid Check
> >> [8] *pollingThread*: isValid check passes for the File1 as it is not is
> LRU
> >> cache.
> >> NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
> >>
> >>
> >> Solution I see:
> >> [1] Dont remove entry from cache after onCompletion (link
> >> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4
> 610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/
> GenericFileOnCompletion.java#L95>
> >> )
> >>
> >> *Humble request to validate my findings and confirming it as a bug :)*
> >> Feel free to ask for more explanatory text if above text is not clean
> >> enough.
> >>
> >> Thanks
> >> Bhuvan
> >>
> >> On Wed, Jan 3, 2018 at 11:18 PM, <sim_yihts...@yahoo.com.invalid>
> wrote:
> >>
> >>> I suspect using “File consumer + threads(…) + default readLock” is like
> >>> having multiple File consumer watching the same folder - they’ll screw
> up
> >>> each other.
> >>>
> >>>
> >>> I offer this as a possible proof: if you add an `idempotent=true` to
> your
> >>> URI, the problem should go away.
> >>>
> >>>
> >>>
> >>> p/s: I think you’re trying to process your files parallel-ly. If that
> is
> >>> your _actual_ question, I have no idea if the above is what you’re
> supposed
> >>> to do even if it works.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: Claus Ibsen
> >>> Sent: Wednesday, 3 January, 2018 9:56 PM
> >>> To: users@camel.apache.org
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> We cannot reproduce your problem.
> >>>
> >>> There can be several reasons why a file cannot be renamed, but
> >>> unfortunately the file api just returns a boolean if it cannot rename
> >>> a file.
> >>> A typical problem is that your do some custom code where you read the
> >>> file or do something with a input stream you do not close properly.
> >>>
> >>> Also using threads(1) does not really make sense.
> >>>
> >>>
> >>>
> >>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bhuva...@gmail.com>
> wrote:
> >>> > Hello nice people,
> >>> >
> >>> > We are using camel 2.17 with java 1.7 in our sample project and we
> >>> defined
> >>> > our route as:
> >>> >
> >>> > from("file:stageDummy")
> >>> > .threads(1)
> >>> > .process(myProcess);
> >>> >
> >>> > We first add 1000 file in stageDummy folder, then we run the code
> and
> >>> we
> >>> > get exception as below
> >>> >
> >>> >
> >>> > org.apache.camel.component.file.GenericFileOperationFailedException:
> >>> Cannot
> >>> > rename file: GenericFile[newFile319] to:
> GenericFile[.camel/newFile319]
> >>> > at
> >>> > org.apache.camel.component.file.strategy.GenericFileProcessS
> >>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
> >>> > at
> >>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
> >>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
> >>> > at
> >>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
> >>> essStrategyCommit(GenericFileOnCompletion.java:127)
> >>> > at
> >>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
> >>> mpletion(GenericFileOnCompletion.java:83)
> >>> > at
> >>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
> >>> mplete(GenericFileOnCompletion.java:57)
> >>> > at
> >>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
> >>> UnitOfWorkHelper.java:104)
> >>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
> >>> rk.java:229)
> >>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
> >>> per.java:65)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> >>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
> >>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
> >>> llback.done(CamelInternalProcessor.java:239)
> >>> > at
> >>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
> >>> llback.done(CamelInternalProcessor.java:250)
> >>> > at
> >>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
> >>> eliveryErrorHandler.java:480)
> >>> > at
> >>> > org.apache.camel.management.InstrumentationProcessor$1.done(
> >>> InstrumentationProcessor.java:86)
> >>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
> >>> > at
> >>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
> >>> ThreadsProcessor.java:85)
> >>> > at java.util.concurrent.Executors$RunnableAdapter.call(
> >>> Executors.java:471)
> >>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>> > at
> >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> >>> Executor.java:1145)
> >>> > at
> >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> >>> lExecutor.java:615)
> >>> > at java.lang.Thread.run(Thread.java:745)
> >>> >
> >>> > As per our understanding camel picks up the file from the mentioned
> >>> > directory i.e in my case-stageDummy and processed by
> process-myProcess
> >>> > and moves it to .camel sub folder.We noticed that if we were having
> more
> >>> > than or around 1000 files in the folder -stageDummy ,there were some
> >>> > exceptions
> >>> > thrown
> >>> >
> >>> > This exception is thrown randomly for n number of files,where n
> changes
> >>> > everytime we execute our main.Note that we were not trying to access
> the
> >>> > exchange
> >>> > instead we kept our process simple.We could see that if there are
> 1000
> >>> files
> >>> > in our folder then the processor is getting called 1000+x times
> ,where x
> >>> > changes everytime example it gets called 1024 times,and we could see
> that
> >>> > camel is trying to process these x number of files again
> >>> >
> >>> > We are attaching the Main class as well as the project zip
> >>> >
> >>> > AND HAPPY NEW YEAR.
> >>> >
> >>> > hoping to resolve it soon
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> -----------------
> >>> http://davsclaus.com @davsclaus
> >>> Camel in Action 2: https://www.manning.com/ibsen2
> >>>
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
