Hi,
I'm using GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu),
on a CentOS Stream 9 machine.
I am unable to understand what I consider an unexpected behavior from a
bash script that does the following:
- use "set -e"
- install an EXIT trap that takes some time to execute
- install a TERM trap that just echoes and then re-kills the script
with TERM
- start a "sleep infinity" process
Both traps unset traps for EXIT and TERM on entry.
On reboot, the TERM signal is not caught immediately because of the
"sleep" subprocess, which
is killed and returns an error. Bash then could start the EXIT trap or
handle the TERM signal.
What I observe is that it "starts" the EXIT trap but is then
immediately interrupted by the TERM signal.
It does the TERM trap, which ends in the re-kill.
But then it continues with the EXIT trap for quite a number of seconds
(90 in the example below).
I would have expected the re-kill to prevent bash from doing anything
significant after that
kill invocation. It looks as if the TERM signal sent by the re-kill was
not received at all, waiting
for bash to be killed by a final KILL from reboot.
Note: the machine executing the script does only that: no other user
processes running, load close to 0.
Notice that the TERM trap is preceded by 3 "+" signs, which seems to
indicate that the trap was activated
while at level 2 (the EXIT trap) which did not have time to do/print
anything useful, but is resumed after
the TERM trap.
Looking for some insight to explain that behavior, which I cannot
reproduce without a reboot.
Killing with TERM on the process group outputs 'Terminated' (from sleep
begin killed I suppose)
and executes only the TERM trap.
I have also reproduced the issue with version 5.3.9 (commit
637f5c8696a6adc9b4519f1cd74aa78492266b7f),
although with a slightly different behavior. I could share these
outputs if relevant.
The script:
--------------- 8< ------------------
#!/usr/bin/env bash
set -ex
function errtrap() {
trap - EXIT TERM
echo "errtrap"
i=0
while (( ++i <= 100 )); do
echo "cleanup $i"
sleep 1
done
return 0
}
function termtrap() {
trap - EXIT TERM
echo "termtrap"
return 0
}
trap 'errtrap || true' EXIT
trap 'signal=SIGTERM; termtrap || true; kill -TERM $$' TERM
sleep infinity
--------------- 8< ------------------
An excerpt of the output:
--------------- 8< ------------------
+ trap 'errtrap || true' EXIT
+ trap 'signal=SIGTERM; termtrap || true; kill -TERM $$' TERM
+ sleep infinity
+++ signal=SIGTERM
+++ termtrap
+++ trap - EXIT TERM
+++ echo termtrap
termtrap
+++ return 0
+++ kill -TERM 3129
++ errtrap
++ trap - EXIT TERM
++ echo errtrap
errtrap
++ i=0
++ (( ++i <= 100 ))
++ echo 'cleanup 1'
cleanup 1
++ sleep 1
++ (( ++i <= 100 ))
++ echo 'cleanup 2'
cleanup 2
++ sleep 1
...
++ (( ++i <= 100 ))
++ echo 'cleanup 89'
cleanup 89
++ sleep 1
++ (( ++i <= 100 ))
++ echo 'cleanup 90'
cleanup 90
++ sleep 1
--------------- 8< ------------------
An excerpt of the strace output:
--------------- 8< ------------------
access("/usr/bin/sleep", R_OK) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], [], 8) = 0
lseek(255, -1, SEEK_CUR) = 371
clone(child_stack=NULL,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7fe431140a10) = 3130
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x55988f237fe0, sa_mask=[],
sa_flags=SA_RESTORER, sa_restorer=0x7fe430e3fc30},
{sa_handler=0x55988f261aa0, sa_m>
wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 3130
--- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=1, si_uid=0} ---
--- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=1, si_uid=0} ---
rt_sigreturn({mask=[CHLD]}) = 3130
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---
rt_sigreturn({mask=[CHLD]}) = 3130
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [CHLD], 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [CHLD], 8) = 0
write(2, "+++ signal=SIGTERM\n", 19) = 19
write(2, "+++ termtrap\n", 13) = 13
write(2, "+++ trap - EXIT TERM\n", 21) = 21
rt_sigaction(SIGTERM, {sa_handler=0x55988f261aa0, sa_mask=[],
sa_flags=SA_RESTORER, sa_restorer=0x7fe430e3fc30},
{sa_handler=0x55988f25cad0, sa_>
write(2, "+++ echo termtrap\n", 18) = 18
fstat(1, {st_mode=S_IFREG|0640, st_size=180, ...}) = 0
write(1, "termtrap\n", 9) = 9
write(2, "+++ return 0\n", 13) = 13
write(2, "+++ kill -TERM 3129\n", 20) = 20
kill(3129, SIGTERM) = 0
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=3129,
si_uid=1000} ---
rt_sigreturn({mask=[CHLD]}) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
write(2, "++ errtrap\n", 11) = 11
write(2, "++ trap - EXIT TERM\n", 20) = 20
write(2, "++ echo errtrap\n", 16) = 16
write(1, "errtrap\n", 8) = 8
write(2, "++ i=0\n", 7) = 7
write(2, "++ (( ++i <= 100 ))\n", 22) = 22
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
write(2, "++ echo 'cleanup 1'\n", 20) = 20
write(1, "cleanup 1\n", 10) = 10
write(2, "++ sleep 1\n", 11) = 11
rt_sigprocmask(SIG_BLOCK, NULL, [CHLD], 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], [CHLD], 8) = 0
clone(child_stack=NULL,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7fe431140a10) = 3207
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [CHLD], 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x55988f237fe0, sa_mask=[],
sa_flags=SA_RESTORER, sa_restorer=0x7fe430e3fc30},
{sa_handler=0x55988f237fe0, sa_m>
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3207
rt_sigaction(SIGINT, {sa_handler=0x55988f261aa0, sa_mask=[],
sa_flags=SA_RESTORER, sa_restorer=0x7fe430e3fc30},
{sa_handler=0x55988f237fe0, sa_m>
ioctl(2, TIOCGWINSZ, 0x7fffc25fff50) = -1 ENOTTY (Inappropriate
ioctl for device)
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
write(2, "++ (( ++i <= 100 ))\n", 22) = 22
...
--------------- 8< ------------------
--
Daniel Villeneuve
