aicam opened a new pull request, #8832:
URL: https://github.com/apache/texera/pull/8832
### What changes were proposed in this PR?
A computing unit whose GeeseFS process has died can never mount that
repository again until the unit is recreated.
A dead FUSE mount keeps its entry in `/proc/mounts`, and the kernel can
still answer `stat()` on it from attributes it cached while the server was
alive. Every read fails with `Transport endpoint is not connected`. Two checks
mistake that for a live mount:
- **`RepositoryMountManager.isFuseMounted`** (in the pod) only asked whether
the mount point exists and is listed as `fuse`. A dead mount passes both, so
`ensureMounted` logs "already mounted" and never sends the mount request. The
operator then gets an unreadable directory. In our case the UDF reported `No
af3.bin in the mounted model` for a model that does contain `af3.bin`.
- **The mounter's `_responds`** is what lets `do_mount` replace a dead
mount, but it also used `stat()`. A dead mount that still stats is handed back
as live.
Both now treat a mount as live only if its directory can be listed. With
that, the next run re-requests a dead mount, and the mounter detaches it and
mounts it again, which `do_mount` already knows how to do.
Seen on a live deployment, on the same dead mount:
| where | `stat()` | listing |
| --- | --- | --- |
| computing-unit pod (engine user) | ok | fails |
| node (mounter, root) | fails for one mount, **ok** for the other | fails |
### Any related issues, documentation, discussions?
Follow-up to the repository-mount infrastructure in #6866 and #6897, part of
#6606.
### How was this PR tested?
New tests:
- `RepositoryMountManagerSpec`: `InPodMount.responds` accepts a listable
directory and rejects a missing path and a non-directory.
`InPodMount.isFuseMounted` rejects a plain directory.
- `test_mounter.py`: `_responds` accepts a listable directory, rejects a
missing path, and rejects a path that still stats but fails to list with
`ENOTCONN`. That last test fails against the old `stat()` check.
```
sbt "WorkflowCore/testOnly
org.apache.texera.amber.storage.RepositoryMountManagerSpec" # 10 passed
cd bin/mounter && python -m pytest tests/test_mounter.py
# 99 passed
sbt WorkflowCore/scalafmtCheck WorkflowCore/Test/scalafmtCheck
"WorkflowCore/scalafix --check" "WorkflowCore/Test/scalafix --check"
```
A real dead FUSE mount can't be created in a unit test. The table above
comes from running both checks against dead GeeseFS mounts on a live cluster.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
