Federico Mariani created CAMEL-24089:
----------------------------------------
Summary: camel-file: file-to-file copy with readLock=fileLock
silently truncates files larger than 2 GB on Linux (single transferTo, return
value ignored)
Key: CAMEL-24089
URL: https://issues.apache.org/jira/browse/CAMEL-24089
Project: Camel
Issue Type: Bug
Components: camel-file
Affects Versions: 4.21.0
Reporter: Federico Mariani
Attachments: FileConsumerFileLockHugeFileTruncationIssueTest.java
h3. Problem
Route {{from("file:in?readLock=fileLock").to("file:out")}} keeps the read-lock
{{FileChannel}} in an exchange property; {{FileOperations.writeFileByFile}}
(FileOperations.java:515-532) then copies via:
{code:java}
channel.transferTo(0, channel.size(), out); // single call, return value
discarded
{code}
{{FileChannel.transferTo}} is documented to possibly transfer fewer bytes than
requested, and the JDK caps each call at {{Integer.MAX_VALUE}} bytes (~2 147
479 552 with sendfile on Linux). Any source file larger than ~2 GB produces a
silently truncated target while the producer reports success - the done file is
still written and the commit deletes/moves the source.
*Platform note:* the truncation manifests on Linux, where the JDK takes the
direct sendfile/copy_file_range path capped at {{Integer.MAX_VALUE}}-ish bytes
per call ({{sun.nio.ch.FileChannelImpl.transferTo}}: {{int icount = (int)
Math.min(count, nd.maxDirectTransferSize())}}). On macOS the direct transfer is
unsupported for file-to-file targets and the JDK falls back to
{{transferToTrustedChannel}}, an internal loop that copies the full count -
verified empirically (a single {{transferTo}} of a 3 GB file returned
3221225472 on macOS). Relying on that fallback is still a contract violation:
{{transferTo}} is documented to possibly transfer fewer bytes than requested.
h3. Failure scenario
{{readLock=fileLock}} consumer + file producer + source file > 2 GB on Linux:
target is truncated at ~2 GB, no exception, source is gone after commit. The
non-locked path ({{Files.copy}}) is unaffected.
h3. History
Introduced by CAMEL-13680 (2019). Notably the pre-2017 code this area evolved
from *did* loop: {{while (position < size) position += in.transferTo(position,
bufferSize, out);}} (removed in d8c0a53b6787).
h3. Suggested fix
Loop until {{channel.size()}} bytes are transferred (standard transferTo loop).
h3. Reproducer
Attached JUnit test (place in
{{core/camel-core/src/test/java/org/apache/camel/component/file/}}, where the
file component tests live). Uses a sparse 3 GB source file; needs ~3 GB free
transient disk space. Asserts target length equals source length. It FAILS on
Linux (target truncated to ~2147483647 bytes) and passes on macOS due to the
JDK fallback described above. The fix - the standard loop until all bytes are
transferred - is correct on all platforms.
----
_This issue was researched and filed by Claude Code on behalf of [~croway]
(GitHub: Croway), as part of a deep code review of camel-file and camel-ftp. A
JUnit reproducer is attached; it fails deterministically on Linux on current
main (4.22.0-SNAPSHOT)._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
