Hi
Have patience, we will take a look at this later when we get time.
On Tue, Jan 9, 2018 at 1:52 AM, Bhuvan Gupta <bhuva...@gmail.com> wrote:
> Humble request to validate my findings and confirming that it is a bug
>
> On 06-Jan-2018 5:06 PM, "Bhuvan Gupta" <bhuva...@gmail.com> wrote:
>
> Camel Users,
>
> I have found the issue
> First lets go over the code routines that are executed for a route like
>
> from("file:stageDummy")
> .threads(1)
> .process(myProcess)
>
> [1] In the above route there will be one polling thread (Let call it as "
> *pollingThread*",) which poll the route directory and fetch all the files
> in it. (link
> <https://github.com/apache/camel/blob/20936b173f332090c6fc5903737152e7d26eb60b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java#L75>
> )
> [2] Then fo reach file *pollingThread *does a GenericFileConsumer.isValid
> check , inside isValid it checks if the file is processed using LRUcache of
> size 1000 (link)
> <https://github.com/apache/camel/blob/bb39dc44d9603c0e6fce2460ddf35a7b94ca30d4/camel-core/src/main/java/org/apache/camel/component/file/GenericFileConsumer.java#L541>
> [3] If file is Valid then *pollingThread *submit the task to
> executorService having 1 thread(Lets Name is it CamelThread#1)
> [4] Inside CamelThread#1 does following things
> [1] Call the processor
> [1] moving the file to .camel
> [2] clear up the .camelLock file
> [3] Remove the entry from LRU cache ----- IMPORTANT Point (link
> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
> )
>
>
> Now if the two thread execution in the following timeline fashion THEN we
> will end up with the same file been processed twice.
>
> [1] *pollingThread*: poll and fetch all the files in it let say we get
> [File1]
> [2] *pollingThread*: send the file for isValid Check
> [3] *pollingThread*: isValid Check passes for File1 as it is not present in
> LRU cahce
> [4] *pollingThread*: Submit the File1 exchange to CamelThread#1
> [5] *pollingThread*: poll and fetch all the files in it let say we get
> [File1, File1.camelLock]
> [6] *CamelThread#1*: call processor for File1 and then remove File1 entry
> from LRU cache
> [7] *pollingThread*: send the file for isValid Check
> [8] *pollingThread*: isValid check passes for the File1 as it is not is LRU
> cache.
> NOTE: isValid checked passed for File1 twice....HENCE THE EXCEPTION
>
>
> Solution I see:
> [1] Dont remove entry from cache after onCompletion (link
> <https://github.com/apache/camel/blob/f6ac3d81420d3f53b029a0f0c31dd4610810e2c2/camel-core/src/main/java/org/apache/camel/component/file/GenericFileOnCompletion.java#L95>
> )
>
> *Humble request to validate my findings and confirming it as a bug :)*
> Feel free to ask for more explanatory text if above text is not clean
> enough.
>
> Thanks
> Bhuvan
>
> On Wed, Jan 3, 2018 at 11:18 PM, <sim_yihts...@yahoo.com.invalid> wrote:
>
>> I suspect using “File consumer + threads(…) + default readLock” is like
>> having multiple File consumer watching the same folder - they’ll screw up
>> each other.
>>
>>
>> I offer this as a possible proof: if you add an `idempotent=true` to your
>> URI, the problem should go away.
>>
>>
>>
>> p/s: I think you’re trying to process your files parallel-ly. If that is
>> your _actual_ question, I have no idea if the above is what you’re supposed
>> to do even if it works.
>>
>>
>>
>>
>>
>>
>> From: Claus Ibsen
>> Sent: Wednesday, 3 January, 2018 9:56 PM
>> To: users@camel.apache.org
>>
>>
>>
>>
>>
>> We cannot reproduce your problem.
>>
>> There can be several reasons why a file cannot be renamed, but
>> unfortunately the file api just returns a boolean if it cannot rename
>> a file.
>> A typical problem is that your do some custom code where you read the
>> file or do something with a input stream you do not close properly.
>>
>> Also using threads(1) does not really make sense.
>>
>>
>>
>> On Tue, Jan 2, 2018 at 6:02 AM, Bhuvan Gupta <bhuva...@gmail.com> wrote:
>> > Hello nice people,
>> >
>> > We are using camel 2.17 with java 1.7 in our sample project and we
>> defined
>> > our route as:
>> >
>> > from("file:stageDummy")
>> > .threads(1)
>> > .process(myProcess);
>> >
>> > We first add 1000 file in stageDummy folder, then we run the code and
>> we
>> > get exception as below
>> >
>> >
>> > org.apache.camel.component.file.GenericFileOperationFailedException:
>> Cannot
>> > rename file: GenericFile[newFile319] to: GenericFile[.camel/newFile319]
>> > at
>> > org.apache.camel.component.file.strategy.GenericFileProcessS
>> trategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
>> > at
>> > org.apache.camel.component.file.strategy.GenericFileRenamePr
>> ocessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.proc
>> essStrategyCommit(GenericFileOnCompletion.java:127)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> mpletion(GenericFileOnCompletion.java:83)
>> > at
>> > org.apache.camel.component.file.GenericFileOnCompletion.onCo
>> mplete(GenericFileOnCompletion.java:57)
>> > at
>> > org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>> UnitOfWorkHelper.java:104)
>> > at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWo
>> rk.java:229)
>> > at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHel
>> per.java:65)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> ProcessorAdvice.after(CamelInternalProcessor.java:654)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$UnitOfWork
>> ProcessorAdvice.after(CamelInternalProcessor.java:609)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> llback.done(CamelInternalProcessor.java:239)
>> > at
>> > org.apache.camel.processor.CamelInternalProcessor$InternalCa
>> llback.done(CamelInternalProcessor.java:250)
>> > at
>> > org.apache.camel.processor.RedeliveryErrorHandler$1.done(Red
>> eliveryErrorHandler.java:480)
>> > at
>> > org.apache.camel.management.InstrumentationProcessor$1.done(
>> InstrumentationProcessor.java:86)
>> > at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:148)
>> > at
>> > org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(
>> ThreadsProcessor.java:85)
>> > at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > As per our understanding camel picks up the file from the mentioned
>> > directory i.e in my case-stageDummy and processed by process-myProcess
>> > and moves it to .camel sub folder.We noticed that if we were having more
>> > than or around 1000 files in the folder -stageDummy ,there were some
>> > exceptions
>> > thrown
>> >
>> > This exception is thrown randomly for n number of files,where n changes
>> > everytime we execute our main.Note that we were not trying to access the
>> > exchange
>> > instead we kept our process simple.We could see that if there are 1000
>> files
>> > in our folder then the processor is getting called 1000+x times ,where x
>> > changes everytime example it gets called 1024 times,and we could see that
>> > camel is trying to process these x number of files again
>> >
>> > We are attaching the Main class as well as the project zip
>> >
>> > AND HAPPY NEW YEAR.
>> >
>> > hoping to resolve it soon
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
