[
https://issues.apache.org/jira/browse/CAMEL-25856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25856:
--------------------------------
Fix Version/s: 4.24.0
> camel-file - the per-file lock of the file producer is evicted after 100
> other file names, so two exchanges can append to the same file at the same
> time and mix its content
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25856
> URL: https://issues.apache.org/jira/browse/CAMEL-25856
> Project: Camel
> Issue Type: Bug
> Components: camel-file
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> {{GenericFileProducer}} (main 20cdb309f7d0) takes a lock per target file name
> around the write, so that two exchanges do not write the same file at the
> same time (added by CAMEL-3584 for concurrent appends). The locks are kept in
> an LRU cache of 100 entries:
> {code:java}
> // assume writing to 100 different files concurrently at most for the same
> // file producer
> private final Map<String, Lock> locks = LRUCacheFactory.newLRUCache(100);
> ...
> Lock lock = locks.computeIfAbsent(target, f -> new ReentrantLock());
> lock.lock();
> try {
> processExchange(exchange, target);
> } finally {
> // do not remove as the locks cache has an upper bound
> lock.unlock();
> {code}
> The cache does not count the files being written at the same time: it keeps
> the 100 file names used last. While one exchange is still writing a file (a
> large body, a stream, a slow disk or share), any 100 other file names written
> by the same producer evict its lock. The next exchange for the same file then
> gets a new lock and writes at the same time as the first one. With
> {{fileExist=Append}} the two contents are mixed in the file; with the default
> {{fileExist=Override}} two writers truncate and write the same file at the
> same time. Nothing fails and nothing is logged.
> This needs one producer that is used by more than one thread (threads(),
> parallel split or multicast, concurrent consumers in front of it) and that
> writes many file names, such as one file per customer, order or day with
> {{fileName=...}}. Two threads are enough: one writes the large file, the
> other writes 100 small files and then the same large file again.
> *Fix*: keep a lock in a {{ConcurrentHashMap}} while an exchange holds or
> waits for it (a count of users, changed in {{compute}} /
> {{computeIfPresent}}) and remove it when the last one is done. The map then
> holds only the file names being written, so it stays bounded without evicting
> a lock in use. camel-smb uses the same producer code; the FTP/SFTP producers
> do not take this lock.
> *Tests*: new {{FileProducerSameFileLockTest}} in camel-file (2 tests, no
> route: the file producer is called from two threads). A first exchange
> appends {{AAAA}}, then its body stream blocks; meanwhile the producer writes
> other files; a second exchange appends {{BBBB}} to the same file. With 10
> other files the second exchange waits (passes on main). With 150 other files,
> on main: {{the two appends to same.txt are mixed ==> expected: <AAAAAAAABBBB>
> but was: <AAAABBBBAAAA>}}. Both pass with the fix. camel-core file component
> tests ({{org.apache.camel.component.file.**}}) with the fix: 474 run, 0
> failures, 0 errors, 12 skipped (manual tests).
> *How it was found*: a review of bounded caches ({{LRUCacheFactory}}) whose
> entries are in use when they are evicted, then the JUnit test above.
> *Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
> have the same LRU cache of 100 locks (read, not run).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
