[
https://issues.apache.org/jira/browse/CAMEL-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853771#comment-17853771
]
Timur commented on CAMEL-20850:
-------------------------------
Actually, the snippet provided in the summary can be used as a reproducer.
Regarding end-to-end scenarios, we tested the change and it works!
Thanks!
> LRUCache evicts entries unexpectedly
> ------------------------------------
>
> Key: CAMEL-20850
> URL: https://issues.apache.org/jira/browse/CAMEL-20850
> Project: Camel
> Issue Type: Bug
> Components: came-core
> Affects Versions: 4.0.3, 4.2.0
> Reporter: Timur
> Assignee: Nicolas Filotto
> Priority: Minor
>
> h1. Summary
> We encountered an infinite loop while downloading files using the SFTP
> endpoint.
> During our investigation, we discovered the {{InProgressRepository}}
> utilizing the {{SimpleLRUCache}}. This cache incorporates an unconventional
> eviction implementation that involves a changes tracking queue.
> Any attempt to modify the cache, even when putting the same element, results
> in an increase in the cache size, potentially causing eviction to occur even
> for a single element.
> The code is as follows:
> {code:java}
> Map<Object, Object> lruCache = LRUCacheFactory.newLRUCache(1, 1);
> int numberOfInserts = 0;
> lruCache.put("key", "value");
> while (lruCache.size() > 0) {
> lruCache.put("key", "value");
> numberOfInserts++;
> }
> throw new IllegalStateException("We should never reach this point, we
> inserted the same element " + numberOfInserts + " times");
> {code}
> In the provided code, the failure occurs when the number of inserts reaches 2.
> h1. Expected Result
> Utilize the LRU Cache limited only by the maximum size passed into it, rather
> than by some internal implementation limits.
> h1. Proposed Solution
> Implement a standard LRUCache, say, by leveraging a {{LRUMap}} from Apache
> Commons.
> We may provide the PR later on.
> h1. The Background
> We need to use a dynamic SFTP route (the download parameters depend on some
> configuration passed from the outside). By default, this is impossible, so we
> will have to use {{pollEnrich}}:
> {code:java}
> .pollEnrich().simple(SFTP_DYNAMIC_URI)
> .timeout(RETURN_CURRENT_RESULT_WITHOUT_WAIT)
> {code}
> The pollEnrich takes only one element (see {{GenericFilePollingConsumer}}).
> The file consumer requests from the SFTP endpoint the list of all files, then
> takes the first, then gets the list again and takes the second file, and so
> on.
> The combination of {{InProgressRepository}} and idempotent consumer is used
> to avoid handling the same files again.
> However, every time the SFTP endpoint lists files, the endpoints and polling
> consumers adjust the list in the {{InProgressRepository}}, thus affecting the
> {{SimpleLRUCache}}.
> The number of elements in the cache doesn't grow, however, the number of
> changes grows with almost n^2 speed.
> With the limit of 50k files, the SimpleLRUCache is evicted already on ~600(!)
> files. This leads to an infinite loop while processing the files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
