Re: Exiting an unconditional juju debug-hooks session
John, Any will work: - ./hooks/$JUJU_HOOK_NAME && tmux kill-session -t $JUJU_UNIT_NAME - ./hooks/$JUJU_HOOK_NAME, C-a d - C-a 0, exit, ./hooks/$JUJU_HOOK_NAME, C-a exit This is because we have an `exec` on new session creation https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/client.go#L103 and on attachment to an existing session: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/client.go#L61 we have screen keys enabled for tmux so the prefix is C-a. However, it is interesting that on session detach I am still getting a message (defined in a function associated with the trap on EXIT) that says "Cleaning up the debug session" - shouldn't be there after the exec. Looking at the process tree, I can see 4 bash processes under an sshd process with the same script in base64: http://paste.ubuntu.com/24781121/ It seems like bash (pid 3391) was forked 4 times consecutively with the last process (3404) `exec`-ing `tmux attach-session -t {unit_name}` I am going to need to check why there are 4 of them but detaching like that is fine by me. Best Regards, Dmitrii Shcherbakov Field Software Engineer IRC (freenode): Dmitrii-Sh On Sun, Jun 4, 2017 at 6:56 PM, John Meinel wrote: > Doesn't the equivalent of ^A ^D (from screen) also work to just disconnect > all sessions? (http://www.dayid.org/comp/tm.html says it would be ^B d). Or > switching to session 0 and exiting that one first? > > I thought we had a quick way to disconnect, but its possible you have to > exit 2x and that fast firing hooks always catch a new window before you can > exit a second time. > > John > =:-> > > > On Sun, Jun 4, 2017 at 5:56 PM, Dmitrii Shcherbakov > wrote: >> >> Hi everybody, >> >> Currently if you do >> >> juju debug-hooks # no event (hook) in particular >> >> each time there is a new event you will get a new tmux window open and >> this will be done serially as there is no parallelism in hook >> execution on a given logical machine. This is all good and intentional >> but when you've observed the charm behavior and want to let it work >> without your interference again, you need to end your tmux session. >> This can be hard via `exit [status]` shell builtin when you get a lot >> of events (think of an OpenStack HA deployment) - each time you do >> >> ./hooks/$JUJU_HOOK_NAME && exit >> >> you are dropped into a session '0' and a new session is created for a >> queued event for which you have to manually execute a hook and exit >> again until you process the backlog. >> >> tmux list-windows >> 0: bash- (1 panes) [239x62] [layout bbde,239x62,0,0,1] @1 # <--- >> dropping here after `exit` >> 1: update-status* (1 panes) [239x62] [layout bbe0,239x62,0,0,3] @3 >> (active) >> >> >> https://jujucharms.com/docs/stable/authors-hook-debug#running-a-debug-session >> "Note: To allow Juju to continue processing events normally, you must >> exit the hook execution with a zero return code (using the exit >> command), otherwise all further events on that unit may be blocked >> indefinitely." >> >> My initial thought was something like this - send SIGTERM to a child >> of sshd which will terminate your ssh session: >> unset n ; p=`pgrep -f 'tmux attach-session.*'$JUJU_UNIT_NAME` ; while >> [ "$n" != "sshd" ] ; do pc=$p ; p=$(ps -o ppid= $p | tr -d ' ') ; echo >> $p ; n=`basename $(readlink /proc/$p/exe || echo -n none)` ; done && >> kill $pc >> >> as an agent waits for an SSH client to exit: >> >> https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/server.go#L53 >> >> After thinking about it some more, I thought it would be cleaner to >> just kill a specific tmux session: >> >> tmux list-sessions >> gluster/0: 2 windows (created Fri Jun 2 20:22:30 2017) [239x62] >> (attached) >> >> ./hooks/$JUJU_HOOK_NAME && tmux kill-session -t $JUJU_UNIT_NAME >> [exited] >> Cleaning up the debug session >> no server running on /tmp/tmux-0/default >> Connection to 10.10.101.77 closed. >> >> The cleanup message comes from debugHooksClientScript that simply sets >> up a bash trap on EXIT: >> >> https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/client.go#L51 >> >> Judging by the code, it should be pretty safe to do so - unless there >> is a debug session in a debug context for a particular unit, other >> hooks will be executed regularly by an agent instead of creating a new >> tmux window: >> >> https://github.com/juju/juju/blob/develop/worker/uniter/runner/runner.go#L225 >> debugctx := debug.NewHooksContext(runner.context.UnitName()) >> if session, _ := debugctx.FindSession(); session != nil && >> session.MatchHook(hookName) { >> logger.Infof("executing %s via debug-hooks", hookName) >> err = session.RunHook(hookName, runner.paths.GetCharmDir(), env) >> } else { >> err = runner.runCharmHook(hookName, env, charmLocation) >> } >> return runner.context.Flush(hookName, err) >> >> There are two scripts: >> >> - a client script executed via an ssh client when you run juju debug-hooks >> - a s
Re: Exiting an unconditional juju debug-hooks session
Doesn't the equivalent of ^A ^D (from screen) also work to just disconnect all sessions? (http://www.dayid.org/comp/tm.html says it would be ^B d). Or switching to session 0 and exiting that one first? I thought we had a quick way to disconnect, but its possible you have to exit 2x and that fast firing hooks always catch a new window before you can exit a second time. John =:-> On Sun, Jun 4, 2017 at 5:56 PM, Dmitrii Shcherbakov < dmitrii.shcherba...@canonical.com> wrote: > Hi everybody, > > Currently if you do > > juju debug-hooks # no event (hook) in particular > > each time there is a new event you will get a new tmux window open and > this will be done serially as there is no parallelism in hook > execution on a given logical machine. This is all good and intentional > but when you've observed the charm behavior and want to let it work > without your interference again, you need to end your tmux session. > This can be hard via `exit [status]` shell builtin when you get a lot > of events (think of an OpenStack HA deployment) - each time you do > > ./hooks/$JUJU_HOOK_NAME && exit > > you are dropped into a session '0' and a new session is created for a > queued event for which you have to manually execute a hook and exit > again until you process the backlog. > > tmux list-windows > 0: bash- (1 panes) [239x62] [layout bbde,239x62,0,0,1] @1 # <--- > dropping here after `exit` > 1: update-status* (1 panes) [239x62] [layout bbe0,239x62,0,0,3] @3 (active) > > https://jujucharms.com/docs/stable/authors-hook-debug# > running-a-debug-session > "Note: To allow Juju to continue processing events normally, you must > exit the hook execution with a zero return code (using the exit > command), otherwise all further events on that unit may be blocked > indefinitely." > > My initial thought was something like this - send SIGTERM to a child > of sshd which will terminate your ssh session: > unset n ; p=`pgrep -f 'tmux attach-session.*'$JUJU_UNIT_NAME` ; while > [ "$n" != "sshd" ] ; do pc=$p ; p=$(ps -o ppid= $p | tr -d ' ') ; echo > $p ; n=`basename $(readlink /proc/$p/exe || echo -n none)` ; done && > kill $pc > > as an agent waits for an SSH client to exit: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/server.go#L53 > > After thinking about it some more, I thought it would be cleaner to > just kill a specific tmux session: > > tmux list-sessions > gluster/0: 2 windows (created Fri Jun 2 20:22:30 2017) [239x62] (attached) > > ./hooks/$JUJU_HOOK_NAME && tmux kill-session -t $JUJU_UNIT_NAME > [exited] > Cleaning up the debug session > no server running on /tmp/tmux-0/default > Connection to 10.10.101.77 closed. > > The cleanup message comes from debugHooksClientScript that simply sets > up a bash trap on EXIT: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/client.go#L51 > > Judging by the code, it should be pretty safe to do so - unless there > is a debug session in a debug context for a particular unit, other > hooks will be executed regularly by an agent instead of creating a new > tmux window: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/runner.go#L225 > debugctx := debug.NewHooksContext(runner.context.UnitName()) > if session, _ := debugctx.FindSession(); session != nil && > session.MatchHook(hookName) { > logger.Infof("executing %s via debug-hooks", hookName) > err = session.RunHook(hookName, runner.paths.GetCharmDir(), env) > } else { > err = runner.runCharmHook(hookName, env, charmLocation) > } > return runner.context.Flush(hookName, err) > > There are two scripts: > > - a client script executed via an ssh client when you run juju debug-hooks > - a server script which is executed in the `RunHook` function by an > agent and creates a new window for an existing tmux session. > > client side: > https://github.com/juju/juju/blob/develop/cmd/juju/ > commands/debughooks.go#L137 > script := base64.StdEncoding.EncodeToString([]byte(unitdebug.ClientScript( > debugctx, > c.hooks))) > innercmd := fmt.Sprintf(`F=$(mktemp); echo %s | base64 -d > $F; . $F`, > script) > args := []string{fmt.Sprintf("sudo /bin/bash -c '%s'", innercmd)} > c.Args = args > return c.sshCommand.Run(ctx) > > Server script: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/server.go#L90 > Client script: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/client.go#L49 > > A worker waits until a client exists by monitoring a file lock at > ClientExitFileLock: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/server.go#L34 > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/server.go#L53 > A path of a lock itself for a particular session: > https://github.com/juju/juju/blob/develop/worker/uniter/ > runner/debug/common.go#L24 > > --- > > If this approach with killing a tmux session is fine then I could > create a PR for the doc repo and for the description in the > debugHooksServer
Exiting an unconditional juju debug-hooks session
Hi everybody, Currently if you do juju debug-hooks # no event (hook) in particular each time there is a new event you will get a new tmux window open and this will be done serially as there is no parallelism in hook execution on a given logical machine. This is all good and intentional but when you've observed the charm behavior and want to let it work without your interference again, you need to end your tmux session. This can be hard via `exit [status]` shell builtin when you get a lot of events (think of an OpenStack HA deployment) - each time you do ./hooks/$JUJU_HOOK_NAME && exit you are dropped into a session '0' and a new session is created for a queued event for which you have to manually execute a hook and exit again until you process the backlog. tmux list-windows 0: bash- (1 panes) [239x62] [layout bbde,239x62,0,0,1] @1 # <--- dropping here after `exit` 1: update-status* (1 panes) [239x62] [layout bbe0,239x62,0,0,3] @3 (active) https://jujucharms.com/docs/stable/authors-hook-debug#running-a-debug-session "Note: To allow Juju to continue processing events normally, you must exit the hook execution with a zero return code (using the exit command), otherwise all further events on that unit may be blocked indefinitely." My initial thought was something like this - send SIGTERM to a child of sshd which will terminate your ssh session: unset n ; p=`pgrep -f 'tmux attach-session.*'$JUJU_UNIT_NAME` ; while [ "$n" != "sshd" ] ; do pc=$p ; p=$(ps -o ppid= $p | tr -d ' ') ; echo $p ; n=`basename $(readlink /proc/$p/exe || echo -n none)` ; done && kill $pc as an agent waits for an SSH client to exit: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/server.go#L53 After thinking about it some more, I thought it would be cleaner to just kill a specific tmux session: tmux list-sessions gluster/0: 2 windows (created Fri Jun 2 20:22:30 2017) [239x62] (attached) ./hooks/$JUJU_HOOK_NAME && tmux kill-session -t $JUJU_UNIT_NAME [exited] Cleaning up the debug session no server running on /tmp/tmux-0/default Connection to 10.10.101.77 closed. The cleanup message comes from debugHooksClientScript that simply sets up a bash trap on EXIT: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/client.go#L51 Judging by the code, it should be pretty safe to do so - unless there is a debug session in a debug context for a particular unit, other hooks will be executed regularly by an agent instead of creating a new tmux window: https://github.com/juju/juju/blob/develop/worker/uniter/runner/runner.go#L225 debugctx := debug.NewHooksContext(runner.context.UnitName()) if session, _ := debugctx.FindSession(); session != nil && session.MatchHook(hookName) { logger.Infof("executing %s via debug-hooks", hookName) err = session.RunHook(hookName, runner.paths.GetCharmDir(), env) } else { err = runner.runCharmHook(hookName, env, charmLocation) } return runner.context.Flush(hookName, err) There are two scripts: - a client script executed via an ssh client when you run juju debug-hooks - a server script which is executed in the `RunHook` function by an agent and creates a new window for an existing tmux session. client side: https://github.com/juju/juju/blob/develop/cmd/juju/commands/debughooks.go#L137 script := base64.StdEncoding.EncodeToString([]byte(unitdebug.ClientScript(debugctx, c.hooks))) innercmd := fmt.Sprintf(`F=$(mktemp); echo %s | base64 -d > $F; . $F`, script) args := []string{fmt.Sprintf("sudo /bin/bash -c '%s'", innercmd)} c.Args = args return c.sshCommand.Run(ctx) Server script: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/server.go#L90 Client script: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/client.go#L49 A worker waits until a client exists by monitoring a file lock at ClientExitFileLock: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/server.go#L34 https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/server.go#L53 A path of a lock itself for a particular session: https://github.com/juju/juju/blob/develop/worker/uniter/runner/debug/common.go#L24 --- If this approach with killing a tmux session is fine then I could create a PR for the doc repo and for the description in the debugHooksServerScript to explicitly mention it. I doubt it deserves a helper command but rather a more verbose explanation. Have anybody else encountered the need to do the same? Best Regards, Dmitrii Shcherbakov Field Software Engineer IRC (freenode): Dmitrii-Sh -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev