Re: broken repo after power cut
Am 22.06.2015 um 02:35 schrieb Theodore Ts'o: On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote: I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) s/ext4/all modern file systems/ POSIX makes **no guarantees** about what happens after a power failure unless you use fsync() --- which git does not do by default (see below). Thanks for pointing this out. The bottome lins is that if you care about files being written, you need to use fsync(). Should git use fsync() by default? Well, if you are willing to accept that if your system crashes within a second or so of your last git operation, you might need to run git fsck and potentially recover from a busted repo, maybe speed is more important for you (and git is known for its speed/performance, after all. :-) The actual state of the source tree would have been written using a text editor which tends to be paranoid about using fsync (at least, if you use a real editor like Emacs or Vi, as opposed to the toy notepad editors shipped with GNOME or KDE :-). So as long as you know what you're doing, it's unlikely that you will actually lose any work. Personally, I have core.fsyncobjectfiles set to yes in my .gitconfig. Part of this is because I have an SSD, so the speed hit really doesn't bother me, and needing to recover a corrupted git repository is a pain (although I have certainly done it in the past). I think core.fsyncObjectFiles documentation really needs an update. What about this one? diff --git a/Documentation/config.txt b/Documentation/config.txt index 43bb53c..b08fa11 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -693,10 +693,16 @@ core.whitespace:: core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with data=writeback). +For performance reasons git does not call 'fsync()' after writing object +files. This means that after a power cut your git repository can get +corrupted as not all data hit the storage media. Especially on modern +filesystems like ext4, xfs or btrfs this can happen very easily. +If you have to face power cuts and care about your data it is strongly +recommended to enable this setting. +Please note that git's behavior used to be safe on ext3 with data=ordered, +for any other filesystems or mount settings this is not the case as +POSIX clearly states that you have to call 'fsync()' to make sure that +all data is written. core.preloadIndex:: Enable parallel index preload for operations like 'git diff' -- Thanks, //richard -- To unsubscribe from this list: send the line unsubscribe git in
Re: broken repo after power cut
On Mon, Jun 22, 2015 at 01:19:59PM +0200, Richard Weinberger wrote: The bottome lins is that if you care about files being written, you need to use fsync(). Should git use fsync() by default? Well, if you are willing to accept that if your system crashes within a second or so of your last git operation, you might need to run git fsck and potentially recover from a busted repo, maybe speed is more important for you (and git is known for its speed/performance, after all. :-) I made a typo in the above. s/second/minute/. (Linux's writeback timer is 30 seconds, but if the disk is busy it might take a bit longer to get all of the data blocks written out to disk and committed.) I think core.fsyncObjectFiles documentation really needs an update. What about this one? diff --git a/Documentation/config.txt b/Documentation/config.txt index 43bb53c..b08fa11 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -693,10 +693,16 @@ core.whitespace:: core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with data=writeback). +For performance reasons git does not call 'fsync()' after writing object +files. This means that after a power cut your git repository can get +corrupted as not all data hit the storage media. Especially on modern +filesystems like ext4, xfs or btrfs this can happen very easily. +If you have to face power cuts and care about your data it is strongly +recommended to enable this setting. +Please note that git's behavior used to be safe on ext3 with data=ordered, +for any other filesystems or mount settings this is not the case as +POSIX clearly states that you have to call 'fsync()' to make sure that +all data is written. My main complaint about this is that it's a bit Linux-centric. For example, the fact that fsync(2) is needed to push data out of the cache is also true for MacOS (and indeed all other Unix systems going back three decades) as well as Windows. In fact, it's not a matter of POSIX says, but POSIX documented, but since standards are held in high esteem, it's sometimes a bit more convenient to use them as an appeal to authority. :-) (Ext3's data=ordered behaviour is an outlier, and in fact, the reason why it mostly safe to skip fsync(2) calls when using ext3 data=ordered was an accidental side effect of another problem which was trying to solve based on the relatively primitive way it handled block allocation.) Cheers, - Ted -- To unsubscribe from this list: send the line unsubscribe git in
Re: broken repo after power cut
Hi Johannes, [CC'ing linux-fsdevel and tytso] Am 21.06.2015 um 14:28 schrieb Johannes Schindelin: Hi Richard, On 2015-06-20 21:40, Richard Weinberger wrote: Yesterday our git server faced a power cut and a git repository broke. The server is running a ext4 filesystem on top of Linux 3.16 (stable from openSUSE) and git 2.1.4. We had a backup, so no data was lost but I really would like to figure out what happened. This is the output of git fsck: Checking object directories: 100% (256/256), done. error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt To me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it. As the object was referenced before syncing the data to disk the repo broke. Could this have happened? Also, is git designed to survive power cuts? Then referencing an object before synching it do disk would make no sense. I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully. I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled. You mean the ext4 delayed block allocation feature/issue? IIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE). But to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems. Ted, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered. Maybe this leads you into the direction of a work-around in your setup? I'm still not sure who to blame. ;-) Thanks, //richard -- To unsubscribe from this list: send the line unsubscribe git in
Re: broken repo after power cut
Am 21.06.2015 um 15:59 schrieb Christoph Hellwig: On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote: To me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it. As the object was referenced before syncing the data to disk the repo broke. Git doesn't fsync by default, and because of that I've seen similar data losses on ext4/xfs/btrfs. You can set the core.fsyncobjectfiles to mitigate it, but even with that I've seen corrupted index files. Yeah, after inspecting git's source I've found that config option too. Now it's also crystal clear that git is not power cut safe at all by default. ;-\ So, anyone that cares about his repos has to enable core.fsyncobjectfiles, which is IMHO kind of sad. Thanks, //richard -- To unsubscribe from this list: send the line unsubscribe git in
Re: broken repo after power cut
On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote: To me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it. As the object was referenced before syncing the data to disk the repo broke. Git doesn't fsync by default, and because of that I've seen similar data losses on ext4/xfs/btrfs. You can set the core.fsyncobjectfiles to mitigate it, but even with that I've seen corrupted index files. Note that I've been mostly on old git versions from various distros, so in case this was fixed recently I'll take everything I said back. -- To unsubscribe from this list: send the line unsubscribe git in
Re: broken repo after power cut
Hi Richard, On 2015-06-20 21:40, Richard Weinberger wrote: Yesterday our git server faced a power cut and a git repository broke. The server is running a ext4 filesystem on top of Linux 3.16 (stable from openSUSE) and git 2.1.4. We had a backup, so no data was lost but I really would like to figure out what happened. This is the output of git fsck: Checking object directories: 100% (256/256), done. error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt To me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it. As the object was referenced before syncing the data to disk the repo broke. Could this have happened? Also, is git designed to survive power cuts? Then referencing an object before synching it do disk would make no sense. I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully. I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled. Maybe this leads you into the direction of a work-around in your setup? Ciao, Johannes -- To unsubscribe from this list: send the line unsubscribe git in
Re: broken repo after power cut
On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote: I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) s/ext4/all modern file systems/ POSIX makes **no guarantees** about what happens after a power failure unless you use fsync() --- which git does not do by default (see below). You mean the ext4 delayed block allocation feature/issue? IIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE). But to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems. Ted, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered. The hacks (which were agreed upon by all of the major file system developers --- ext4, btfs, xfs --- at the Linux File Systems and Storage summit a couple of years ago --- protects against the default text editors of GNOME and KDE which were saving file without using fsync(), and in one particularly egregious example (although I don't remember which program was doing this), updated files by opening the file with O_TRUNC and then rewritng the new contents of the file. So if you crashed just after the open(2), and before the file data was written, you were guaranteed to lose data. The hack protects against data loss when programs updated a file incompetently. What we agreed to do was that upon renaming a fileA on top of another fileB, there is an implicit writeback initiated of fileA. If the program properly called fsync(2) before closing the file descriptor for fileA and doing the rename, this implicit writeback would be no-op. Simiarly, if a file descriptor was opened with O_TRUNC, when the file descriptor is closed, we start an implicit writeback at that point. Note that this is not the same as a full fsync; it merely closes the race window from 30 seconds down to a second or so (depending on how busy the disk is). But this hack does not protect against freshly written files, which is the case of git object files or git pack files. The basic idea here is that you could have just as easily crashed before the commit as after the commit, and doing an implicit writeback after all file closes would have destroyed performance and penalized progams that didn't really care so much about the file hitting disk. (For example, if you do a compile, and you crash, it's not such a big deal.) The bottome lins is that if you care about files being written, you need to use fsync(). Should git use fsync() by default? Well, if you are willing to accept that if your system crashes within a second or so of your last git operation, you might need to run git fsck and potentially recover from a busted repo, maybe speed is more important for you (and git is known for its speed/performance, after all. :-) The actual state of the source tree would have been written using a text editor which tends to be paranoid about using fsync (at least, if you use a real editor like Emacs or Vi, as opposed to the toy notepad editors shipped with GNOME or KDE :-). So as long as you know what you're doing, it's unlikely that you will actually lose any work. Personally, I have core.fsyncobjectfiles set to yes in my .gitconfig. Part of this is because I have an SSD, so the speed hit really doesn't bother me, and needing to recover a corrupted git repository is a pain (although I have certainly done it in the past). - Ted -- To unsubscribe from this list: send the line unsubscribe git in
broken repo after power cut
Hi! Yesterday our git server faced a power cut and a git repository broke. The server is running a ext4 filesystem on top of Linux 3.16 (stable from openSUSE) and git 2.1.4. We had a backup, so no data was lost but I really would like to figure out what happened. This is the output of git fsck: Checking object directories: 100% (256/256), done. error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt To me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it. As the object was referenced before syncing the data to disk the repo broke. Could this have happened? Also, is git designed to survive power cuts? Then referencing an object before synching it do disk would make no sense. Thanks, //richard -- To unsubscribe from this list: send the line unsubscribe git in
