Bug#814089: Please call fdatasync on the target file before removing the source file
On 2016-02-08 13:42:47 [+0100], Enrico Zini wrote: > Hello, Hi, > I was archiving and xz-compressing mail from last year when my laptop > tripped thermal protection and switched off. The resulting filesystem > situation is this: … > The source files are gone, and the target files are empty. > > I have not checked xz's sources, but this looks like the result of a > missing fdatasync on the target file before the source has been removed. First all, I'm sorry for your loss. I've been looking at this a few years ago and then somehow forgot about it. Now I kinda remembered. I *think* the problem (was/is) that write() close() unlink() ended up with commited metadata but not data so you see the file but data did not hit the disk (completely). That probably was the time before ext4 changed its default behaviour with a few hacks once user reported that KDE lost all its configurations files because there were rewritten at boot time. A fdatasync() would probably avoid that[0]. Should we really try to do that? If so, we should also go after gzip, bzip2, … and make the change there, too. They don't do that (behave like xz does) but I think if we make such a change we should behave consistenly across those tools. In theory we might have worse performance due to that sync but this shouldn't hit most users. I think most ppl use xz via a pipe (say `tar J') so they won't be affected by that sync. We probably could also avoid that sync together with the `--keep' option. And then we would be able to catch errors before close() and unlink of the original. [0] I would check if linux's fs ppl if we really consider it. > Enrico Sebastian
Bug#814089: Please call fdatasync on the target file before removing the source file
Package: xz-utils Version: 5.1.1alpha+20120614-2.1 Severity: normal Hello, I was archiving and xz-compressing mail from last year when my laptop tripped thermal protection and switched off. The resulting filesystem situation is this: $ ls -la total 588256 drwxr-xr-x 2 enrico enrico 16384 Feb 8 13:29 . drwxr-xr-x 15 enrico enrico 4096 Feb 8 13:07 .. -rw-r--r-- 1 enrico enrico 80788 Feb 8 13:21 debian-curiosa.xz -rw-r--r-- 1 enrico enrico 77312 Feb 8 13:21 debian-custom.xz [..] -rw-r--r-- 1 enrico enrico 6741700 Feb 8 13:21 osm-talk-it.xz -rw-r--r-- 1 enrico enrico 0 Feb 8 13:21 sent-mail.xz -rw-r--r-- 1 enrico enrico 0 Feb 8 13:21 smartphone.xz -rw-r--r-- 1 enrico enrico 0 Feb 8 13:21 sourceforge.xz -rw-r--r-- 1 enrico enrico 0 Feb 8 13:21 tham.xz The source files are gone, and the target files are empty. I have not checked xz's sources, but this looks like the result of a missing fdatasync on the target file before the source has been removed. I'll now go and cry for the loss of my whole sent-mail archive from last year. Enrico -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (500, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.3.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages xz-utils depends on: ii libc6 2.21-7 ii liblzma5 5.1.1alpha+20120614-2.1 xz-utils recommends no packages. xz-utils suggests no packages. -- no debconf information
