shashank created CAMEL-25856:
--------------------------------
Summary: camel-file - the per-file lock of the file producer is
evicted after 100 other file names, so two exchanges can append to the same
file at the same time and mix its content
Key: CAMEL-25856
URL: https://issues.apache.org/jira/browse/CAMEL-25856
Project: Camel
Issue Type: Bug
Components: camel-file
Reporter: shashank
Assignee: shashank
{{GenericFileProducer}} (main 20cdb309f7d0) takes a lock per target file name
around the write, so that two exchanges do not write the same file at the same
time (added by CAMEL-3584 for concurrent appends). The locks are kept in an LRU
cache of 100 entries:
{code:java}
// assume writing to 100 different files concurrently at most for the same
// file producer
private final Map<String, Lock> locks = LRUCacheFactory.newLRUCache(100);
...
Lock lock = locks.computeIfAbsent(target, f -> new ReentrantLock());
lock.lock();
try {
processExchange(exchange, target);
} finally {
// do not remove as the locks cache has an upper bound
lock.unlock();
{code}
The cache does not count the files being written at the same time: it keeps the
100 file names used last. While one exchange is still writing a file (a large
body, a stream, a slow disk or share), any 100 other file names written by the
same producer evict its lock. The next exchange for the same file then gets a
new lock and writes at the same time as the first one. With
{{fileExist=Append}} the two contents are mixed in the file; with the default
{{fileExist=Override}} two writers truncate and write the same file at the same
time. Nothing fails and nothing is logged.
This needs one producer that is used by more than one thread (threads(),
parallel split or multicast, concurrent consumers in front of it) and that
writes many file names, such as one file per customer, order or day with
{{fileName=...}}. Two threads are enough: one writes the large file, the other
writes 100 small files and then the same large file again.
*Fix*: keep a lock in a {{ConcurrentHashMap}} while an exchange holds or waits
for it (a count of users, changed in {{compute}} / {{computeIfPresent}}) and
remove it when the last one is done. The map then holds only the file names
being written, so it stays bounded without evicting a lock in use. camel-smb
uses the same producer code; the FTP/SFTP producers do not take this lock.
*Tests*: new {{FileProducerSameFileLockTest}} in camel-file (2 tests, no route:
the file producer is called from two threads). A first exchange appends
{{AAAA}}, then its body stream blocks; meanwhile the producer writes other
files; a second exchange appends {{BBBB}} to the same file. With 10 other files
the second exchange waits (passes on main). With 150 other files, on main:
{{the two appends to same.txt are mixed ==> expected: <AAAAAAAABBBB> but was:
<AAAABBBBAAAA>}}. Both pass with the fix. camel-core file component tests
({{org.apache.camel.component.file.**}}) with the fix: 474 run, 0 failures, 0
errors, 12 skipped (manual tests).
*How it was found*: a review of bounded caches ({{LRUCacheFactory}}) whose
entries are in use when they are evicted, then the JUnit test above.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
have the same LRU cache of 100 locks (read, not run).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
