To: bug-ddrescue@gnu.org Subject: Command mode deadlock when stdout is fully buffered
The following short Python program forks a ddrescue command mode process, sends it a command, and tries to read the result. However, it ends up waiting forever. ddrescue never sends the response, because stdout is fully buffered and ddrescue doesn't flush it before trying to read the next command. (This uses the files in the testsuite directory of a ddrescue source tarball.) #!/usr/bin/env python3 import subprocess with subprocess.Popen(['ddrescue', '--command-mode', 'test1.txt', '/tmp/dest.txt', 'mapfile1'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, encoding='utf-8') as p: p.stdin.write('s 0 512\n') p.stdin.flush() response = p.stdout.readline() print(response) To confirm buffering is the problem, run this script under strace, following forks: strace -ff --decode-fds=all -o /tmp/trace.txt ./script.py When the deadlock occurs, the tail of the trace of the ddrescue process will look like: newfstatat(0<pipe:[475022]>, "", {st_mode=S_IFIFO|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0 read(0<pipe:[475022]>, "s 0 512\n", 4096) = 8 newfstatat(1<pipe:[475023]>, "", {st_mode=S_IFIFO|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0 read(0<pipe:[475022]>, ddrescue (actually the C library on ddrescue's behalf) stat()s stdout, but then reads stdin again without actually writing to stdout. If you modify the script to ...Popen(['stdbuf', '-o', 'L', 'ddrescue'... the deadlock does not occur. stdbuf sets stdout to line-buffered even though it is non-interactive (a pipe). To do this from within ddrescue, I think you would call setvbuf(stdout, NULL, _IOLBF, BUFSIZ). (But cppreference[1] warns that line buffering on non-terminals is not portable.) Of course, inserting calls to std::fflush(stdout) at appropriate places will also work. (I think in command_mode.cc Rescuebook::do_commands?) Note that programs that send ddrescue commands and then close ddrescue's stdin currently work properly even when ddrescue's stdout is fully buffered. ddrescue will read until it gets EOF on stdin, then exit, and the C library will flush stdout before the process terminates. (Try it by adding "p.stdin.close()" to the Python script above. It doesn't deadlock, and strace shows the write to stdout happening just before the exit_group syscall.) Thank you for your time and effort on ddrescue. --Jeffrey [1]: https://en.cppreference.com/w/c/io/setvbuf