[
https://issues.apache.org/jira/browse/CAMEL-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853245#comment-17853245
]
Timur commented on CAMEL-20850:
-------------------------------
May I ask why the priority was set to Minor? Maybe not critical, as dynamic
routes is not the most used case. Likely.
But at least where dynamic routes are needed, the issue leads to infinite loops
and might cause severe problems in other components.
At least if our hypothesis is correct.
> LRUCache evicts entries unexpectedly
> ------------------------------------
>
> Key: CAMEL-20850
> URL: https://issues.apache.org/jira/browse/CAMEL-20850
> Project: Camel
> Issue Type: Bug
> Affects Versions: 4.0.3, 4.2.0
> Reporter: Timur
> Assignee: Nicolas Filotto
> Priority: Minor
>
> h1. Summary
> We encountered an infinite loop while downloading files using the SFTP
> endpoint.
> During our investigation, we discovered the {{InProgressRepository}}
> utilizing the {{SimpleLRUCache}}. This cache incorporates an unconventional
> eviction implementation that involves a changes tracking queue.
> Any attempt to modify the cache, even when putting the same element, results
> in an increase in the cache size, potentially causing eviction to occur even
> for a single element.
> The code is as follows:
> {code:java}
> Map<Object, Object> lruCache = LRUCacheFactory.newLRUCache(1, 1);
> int numberOfInserts = 0;
> lruCache.put("key", "value");
> while (lruCache.size() > 0) {
> lruCache.put("key", "value");
> numberOfInserts++;
> }
> throw new IllegalStateException("We should never reach this point, we
> inserted the same element " + numberOfInserts + " times");
> {code}
> In the provided code, the failure occurs when the number of inserts reaches 2.
> h1. Expected Result
> Utilize the LRU Cache limited only by the maximum size passed into it, rather
> than by some internal implementation limits.
> h1. Proposed Solution
> Implement a standard LRUCache, say, by leveraging a {{LRUMap}} from Apache
> Commons.
> We may provide the PR later on.
> h1. The Background
> We need to use a dynamic SFTP route (the download parameters depend on some
> configuration passed from the outside). By default, this is impossible, so we
> will have to use {{pollEnrich}}:
> {code:java}
> .pollEnrich().simple(SFTP_DYNAMIC_URI)
> .timeout(RETURN_CURRENT_RESULT_WITHOUT_WAIT)
> {code}
> The pollEnrich takes only one element (see {{GenericFilePollingConsumer}}).
> The file consumer requests from the SFTP endpoint the list of all files, then
> takes the first, then gets the list again and takes the second file, and so
> on.
> The combination of {{InProgressRepository}} and idempotent consumer is used
> to avoid handling the same files again.
> However, every time the SFTP endpoint lists files, the endpoints and polling
> consumers adjust the list in the {{InProgressRepository}}, thus affecting the
> {{SimpleLRUCache}}.
> The number of elements in the cache doesn't grow, however, the number of
> changes grows with almost n^2 speed.
> With the limit of 50k files, the SimpleLRUCache is evicted already on ~600(!)
> files. This leads to an infinite loop while processing the files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
