Croway commented on code in PR #29855:
URL: https://github.com/apache/camel/pull/29855#discussion_r4575424093
##########
components/camel-file/src/main/java/org/apache/camel/component/file/strategy/GenericFileRenameProcessStrategy.java:
##########
@@ -41,6 +41,11 @@ public boolean begin(
return false;
}
+ // skip the file before it is processed if it cannot be moved where it
would be moved to
+ if (!acceptMoveTargets(operations, endpoint, exchange, file,
beginRenamer, commitRenamer, failureRenamer)) {
Review Comment:
**A skipped file goes through the read lock on every poll**
`acceptMoveTargets` runs after `super.begin`, which takes the read lock.
With `readLock=changed&readLockCheckInterval=1000`, a consumer whose only file
is skipped here polled 3 times in 4 s. A consumer whose file is skipped when
listed (a link resolving outside) polled 39 times in the same 4 s. Other files
in the same poll wait behind the skipped one. Calling `acceptMoveTargets`
before `super.begin` (also in `GenericFileDeleteProcessStrategy`) would skip
the file without taking the lock. This is not a regression: on the merge-base
the same file is moved outside the starting directory.
_Claude Code on behalf of Federico Mariani_
##########
components/camel-file/src/main/java/org/apache/camel/component/file/FileOperations.java:
##########
@@ -270,25 +278,54 @@ public void releaseRetrievedFileResources(Exchange
exchange) throws GenericFileO
* before it creates missing parent directories of the target.
*/
void jailToStartingDirectory(File target) {
- if (!endpoint.isJailStartingDirectory()) {
- return;
- }
- Path startingDirectory = endpoint.getFile().toPath();
- if
(!target.toPath().toAbsolutePath().normalize().startsWith(startingDirectory.toAbsolutePath().normalize()))
{
- return;
- }
+ jailToStartingDirectory(target, "Cannot write file: ");
+ }
+
+ /**
+ * Checks a path before it is written, deleted, renamed onto or created as
a directory.
+ *
+ * @throws GenericFileOperationFailedException if the path is inside the
starting directory but resolves outside of
+ * it through a symbolic link
+ */
+ private void jailToStartingDirectory(File target, String failure) {
boolean within;
try {
- within =
GenericFileHelper.isWithinDirectoryResolvingLinks(target.toPath(),
startingDirectory);
+ within = isWithinStartingDirectoryResolvingLinks(target);
} catch (IOException e) {
throw new GenericFileOperationFailedException(
"Cannot verify file: " + target + " is within the starting
directory: " + endpoint.getFile(), e);
}
if (!within) {
throw new GenericFileOperationFailedException(
- "Cannot write file: " + target + " as it resolves outside
the starting directory: "
- +
endpoint.getFile());
+ failure + target + " as it resolves outside the starting
directory: " + endpoint.getFile());
+ }
+ }
+
+ /**
+ * Whether the path stays within the starting directory once symbolic
links are resolved. Always true when
+ * jailStartingDirectory is disabled, and for a path that is lexically
outside the starting directory (such as a
+ * move to ../archive the route author configured).
+ */
+ @Override
+ public boolean isWithinStartingDirectory(String name) {
+ try {
+ return isWithinStartingDirectoryResolvingLinks(new File(name));
+ } catch (IOException e) {
+ // such as a dangling symbolic link
+ LOG.debug("Cannot resolve file: {} against the starting directory
due to: {}", name, e.getMessage());
+ return false;
+ }
+ }
+
+ private boolean isWithinStartingDirectoryResolvingLinks(File target)
throws IOException {
+ if (endpoint == null || !endpoint.isJailStartingDirectory()) {
+ return true;
+ }
+ Path startingDirectory = endpoint.getFile().toPath();
+ if
(!target.toPath().toAbsolutePath().normalize().startsWith(startingDirectory.toAbsolutePath().normalize()))
{
+ return true;
}
+ return
GenericFileHelper.isWithinDirectoryResolvingLinks(target.toPath(),
startingDirectory);
Review Comment:
**Every consumed file costs about twice as much with the default options,
also without any link**
Measured on macOS with JDK 25: 3000 empty files,
`file:inbox?initialDelay=0&delay=10` to a processor that counts them, 5 rounds,
2 runs each.
| Commit | 3000 files | Per file |
|---|---|---|
| merge-base f07a0e80 | ~500 ms | 0.17 ms |
| round 1 (28d47042) | ~700 ms | 0.23 ms |
| 9e6be9db | ~1000 ms | 0.33 ms |
The cost per file is the same with 1000 and 6000 files. Each file is now
checked twice: in `begin`, and in `renameFile` when it is moved. Each check
resolves the target (about 22 µs here) and resolves the starting directory
again (about 14 µs). `begin` also copies the exchange, binds the file and
evaluates the expression. A route that reads the file sees a smaller relative
cost, but this applies to every file consumer with the default options, also in
4.18.6 and 4.22.3.
Two ideas:
- Resolve the starting directory once per poll, as
`FileConsumer.isWithinStartingDirectory` already does.
- Check first whether the path goes through a link at all, with one
`Files.isSymbolicLink` per segment below the starting directory, and call
`toRealPath` only when it does. Without a link, the lexical check is enough.
_Claude Code on behalf of Federico Mariani_
##########
docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc:
##########
@@ -2336,6 +2336,23 @@ which could cause the producer to create a spurious
directory with a backslash i
consumer to miss files in a backslash-named directory. On Windows, where
`java.io.File` already
normalizes both separators to `\`, this change has no effect.
+=== camel-file - the consumer skips a file it would move through a symbolic
link resolving outside the starting directory
+
+With `jailStartingDirectory` enabled (the default), the file consumer now
resolves symbolic links on the targets of
+`preMove`, `move` and `moveFailed`, including the default `.camel` folder. It
checks these targets before the file is
+processed. When a target is inside the starting directory but resolves outside
of it through a link, the consumer skips
+the file. It logs a WARN the first time. The file is not processed and not
moved, and it stays in the folder. Before,
+the existing target was deleted and the file was moved wherever the link
pointed. A target that depends on what the
+route sets can only be checked when the file is moved. Such a move fails, and
the file is consumed again on the next
+poll.
+
+The file producer now refuses to delete or rename onto a file through such a
link. This covers `fileExist=Move` with
+`moveExisting`, and the rename of a file written with `tempFileName`. The
write fails.
+
+Targets configured outside the starting directory (such as `move=../archive`)
and links resolving inside it work as
+before. A deployment that links these folders to another directory on purpose,
such as `.camel` on another volume, must
+set `jailStartingDirectory=false` to keep following those links.
Review Comment:
**`jailStartingDirectory=false` also turns off the check of CAMEL-25412**
For a deployment that links `.camel` to another volume on purpose,
`jailStartingDirectory=false` also lets the consumer read a link to any file
outside the starting directory. Reproduced on 9e6be9db, with `hello.txt`,
`.camel` linking to `volume1` and `evil.txt` linking to `outside/secret.txt` in
the inbox:
- `jailStartingDirectory=false`: the route received `[Hello World, Secret]`,
and `evil.txt` was moved into `volume1`.
- The default, with `move=<absolute path of volume2>/${file:name}`: the
route received `[Hello World]`, `hello.txt` was moved to `volume2`, and
`evil.txt` was skipped.
As a target configured outside the starting directory is not checked, I'd
recommend that first: point `move` (or `preMove`, `moveFailed`) at the folder
the link resolves to. `jailStartingDirectory=false` can stay as the
alternative, saying that the consumer then also reads links to files outside.
The same applies to the 4.18 and 4.22 guides and to the component doc.
The guides could also say what the component doc says about the extra
evaluation. With a counting bean in `move` and `moveFailed`, and one file
processed without error, the merge-base calls `move` once and `moveFailed`
never, and 9e6be9db calls `move` twice and `moveFailed` once. That matters for
an expression with side effects, such as a sequence.
_Claude Code on behalf of Federico Mariani_
##########
components/camel-file/src/main/java/org/apache/camel/component/file/strategy/GenericFileProcessStrategySupport.java:
##########
@@ -127,6 +128,12 @@ public void
setExclusiveReadLockStrategy(GenericFileExclusiveReadLockStrategy<T>
}
protected GenericFile<T> renameFile(GenericFileOperations<T> operations,
GenericFile<T> from, GenericFile<T> to) {
+ if (operations instanceof FileOperations fileOperations) {
+ // check before the existing target is deleted, its parent
directories are created and the file is renamed,
+ // as a symbolic link on the target path could otherwise move the
file outside the starting directory
+ fileOperations.jailMoveToStartingDirectory(new
File(to.getAbsoluteFilePath()));
Review Comment:
Verified on 9e6be9db: the file is skipped and not processed. 9 of the 12
cases of `FileConsumerMoveJailStartingDirectorySymlinkTest` fail on the
merge-base f07a0e80 (the route receives the file), and all 12 pass on 9e6be9db.
Thanks!
_Claude Code on behalf of Federico Mariani_
##########
docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_18.adoc:
##########
@@ -77,6 +77,16 @@ at `DEBUG` level with the message only. Checkpoint failures
triggered by batch s
The webhook consumer still logs the body it validates at `DEBUG`, but no
longer the signature from the `X-Signature` header, nor the signature it
calculates.
The `InvalidMessageSignatureException` of a failed check no longer carries the
calculated signature: `getCalculatedSignature()` returns `null` and is
deprecated, as is the constructor that takes it.
+=== camel-file - the consumer's move, preMove and moveFailed resolve symbolic
links for jailStartingDirectory
+
+With `jailStartingDirectory` enabled (the default), the file consumer now
resolves symbolic links on the target of
+`move`, `preMove` and `moveFailed`, including the default `.camel` folder,
when that target is inside the starting
+directory. A target that resolves outside the starting directory through a
link is not used: the move fails like any
+other failed move, so the file stays where it is and the error is passed to
the exception handler. Before, the existing
Review Comment:
Thanks, the entries now say clearly what happens to the file. One follow-up
on the `jailStartingDirectory=false` advice, in a new comment on the 4.24 guide.
_Claude Code on behalf of Federico Mariani_
##########
components/camel-file/src/main/java/org/apache/camel/component/file/strategy/GenericFileProcessStrategySupport.java:
##########
@@ -127,6 +128,12 @@ public void
setExclusiveReadLockStrategy(GenericFileExclusiveReadLockStrategy<T>
}
protected GenericFile<T> renameFile(GenericFileOperations<T> operations,
GenericFile<T> from, GenericFile<T> to) {
+ if (operations instanceof FileOperations fileOperations) {
Review Comment:
Verified on 9e6be9db: the 4 refusal cases of
`FileProducerJailStartingDirectorySymlinkMoveTest` fail on the merge-base
f07a0e80 and pass on 9e6be9db, and the files outside stay unchanged. Thanks for
doing it in this PR.
_Claude Code on behalf of Federico Mariani_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]