GenericFileConsumer fails to pick up all files in a directory if there are more
files than maxMessagePerPoll setting
--------------------------------------------------------------------------------------------------------------------
Key: CAMEL-1824
URL: https://issues.apache.org/activemq/browse/CAMEL-1824
Project: Apache Camel
Issue Type: Bug
Components: camel-core
Affects Versions: 2.0-M2
Reporter: Alexander Lee
Camel adds filenames it is processing to an in memory, in progress map. Once
it is finished processing it removes them. If we specific a maxMessagesPerPoll
number less then the files in a directory, it only processes up to the number
of files we specify for each poll. It then removes the rest from the in
progress map and tries them again next poll. This is the code (from
GenericFileConsumer:processBatch()) for the last part:
for (int index = 0; index < exchanges.size() && isRunAllowed(); index++)
{
GenericFileExchange<T> exchange = (GenericFileExchange<T>)
exchanges.poll();
String key = exchange.getGenericFile().getFileName();
endpoint.getInProgressRepository().remove(key);
}
Unfortunately, as you can see it uses exchanges.size() to determine how many
file names to remove (i.e. how many times to loop), however exchanges.poll()
removes one from the head of exchanges for each loop. This means that the
exchanges.size() reduces by one for each loop, which means it only cleans up
half of the filenames that are in exchanges, which means these files are never
picked up again as Camel thinks it is still processing them.
The fix is to replace the for loop with a while:
while ((exchanges.size() > 0) && isRunAllowed()) {
GenericFileExchange<T> exchange = (GenericFileExchange<T>)
exchanges.poll();
String key = exchange.getGenericFile().getFileName();
endpoint.getInProgressRepository().remove(key);
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
