Hello Alexey Serbin, Zoltan Martonka, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24681
to look at the new patch set (#4).
Change subject: [dist-test] Fix chrpath ETXTBSY race in dist slaves
......................................................................
[dist-test] Fix chrpath ETXTBSY race in dist slaves
chrpath rewrites ELF RPATH/RUNPATH entries in-place using an open() with
write permission. Linux returns ETXTBSY for that open() when the target
inode is currently being executed by some process.
Since KUDU-3788 all dist-test slaves share a single prebuilt thirdparty
installation. fixup_rpaths() is called at the start of every task
regardless of build type and walks the entire thirdparty tree. If an
ASAN or TSAN task on the same slave has already spawned llvm-symbolizer,
any subsequent task calling chrpath on that binary will fail with
ETXTBSY. The failure surfaces in whichever task runs fixup_rpaths()
concurrently, which can be a DEBUG/RELEASE task even though the cause is
always an ASAN/TSAN task holding llvm-symbolizer open for execution.
Multiple such errors ('open: Text file busy') were seen in dist-test
runs, all pointing to chrpath failing to rewrite llvm-symbolizer.
The rewritten RPATH is what makes a lock-free fix possible: fix_rpath()
rewrites each thirdparty component to '$ORIGIN/<relative path to
thirdparty>'. That value is anchored at $ORIGIN and is a pure function
of the binary's position within the extracted tree, so it does not
depend on the absolute extraction path, the build type, or the session.
Any two tasks that share a binary therefore compute the identical target
RPATH. This yields two properties that together remove the race without
any coordination:
- Idempotent short-circuit: if the computed RPATH already equals the
current one, fix_rpath() returns without opening the inode for
writing. The first task to relativize a shared binary makes it a
no-op for every later task, so chrpath is never invoked on a binary
another task is already executing (e.g. llvm-symbolizer, which ships
with '$ORIGIN/../lib' and is skipped outright). Since all sharers
target the same value, even a concurrent rewrite writes identical
bytes.
- ETXTBSY fallback: for the first rewrite of a binary that is genuinely
busy, attempt the in-place chrpath and, only if it fails, classify
the failure by probing the kernel errno directly (chrpath reports
ETXTBSY via the locale-dependent strerror text "Text file busy", so
matching on errno is robust). On ETXTBSY, copy the binary to a
sibling temp file, rewrite the RPATH on the copy, and atomically
replace the original with os.rename(). os.rename() only updates the
directory entry, so any process already executing the original inode
continues undisturbed. Any non-ETXTBSY failure is propagated.
This eliminates the ETXTBSY failure for all build types without
requiring any coordination between concurrent tasks.
Change-Id: I80cdc5b903f5ed0865ec5a92f905a46719f7e316
---
M build-support/run_dist_test.py
1 file changed, 61 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/81/24681/4
--
To view, visit http://gerrit.cloudera.org:8080/24681
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I80cdc5b903f5ed0865ec5a92f905a46719f7e316
Gerrit-Change-Number: 24681
Gerrit-PatchSet: 4
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
