Re: btrfs check --repair now runs in minutes instead of hours? aborting
On Sun, Sep 10, 2017 at 02:01:58PM +0800, Qu Wenruo wrote: > > > On 2017年09月10日 01:44, Marc MERLIN wrote: > > So, should I assume that btrfs progs git has some issue since there is > > no plausible way that a check --repair should be faster than a regular > > check? > > Yes, the assumption that repair should be no faster than RO check is > correct. > Especially for clean fs, repair should just behave the same as RO check. > > And I'll first submit a patch (or patches) to output the consumed time for > each tree, so we could have a clue what is going wrong. > (Digging the code is just a little too boring for me) Cool. Let me know when I should sync and re-try. In the meantime, though, my check --repair went back to 170mn after triggering an FS bug for Josef, so it seems back to normal. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
On 2017年09月10日 01:44, Marc MERLIN wrote:
So, should I assume that btrfs progs git has some issue since there is
no plausible way that a check --repair should be faster than a regular
check?
Yes, the assumption that repair should be no faster than RO check is
correct.
Especially for clean fs, repair should just behave the same as RO check.
And I'll first submit a patch (or patches) to output the consumed time
for each tree, so we could have a clue what is going wrong.
(Digging the code is just a little too boring for me)
Thanks,
Qu
Thanks,
Marc
On Tue, Sep 05, 2017 at 07:45:25AM -0700, Marc MERLIN wrote:
On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote:
gargamel:~# btrfs fi df /mnt/btrfs_pool1
Data, single: total.60TiB, used.54TiB
System, DUP: total2.00MiB, used=1.19MiB
Metadata, DUP: totalX.00GiB, used.69GiB
Wait for a minute.
Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format
(again)?
This output format must be changed, at least to 0.69 GiB, or 706 MiB.
Email client problem. I see control characters in what you quoted.
Let's try again
gargamel:~# btrfs fi df /mnt/btrfs_pool1
Data, single: total=10.66TiB, used=10.60TiB => 10TB
System, DUP: total=64.00MiB, used=1.20MiB=> 1.2MB
Metadata, DUP: total=57.50GiB, used=12.76GiB => 13GB
GlobalReserve, single: total=512.00MiB, used=0.00B => 0
You mean lowmem is actually FASTER than original mode?
That's very surprising.
Correct, unless I add --repair and then original mode is 2x faster than
lowmem.
Is there any special operation done for that btrfs?
Like offline dedupe or tons of reflinks?
In this case, no.
Note that btrfs check used to take many hours overnight until I did a
git pull of btrfs progs and built the latest from TOT.
BTW, how many subvolumes do you have in the fs?
gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | wc -l
91
If I remove snapshots for btrfs send and historical 'backups':
gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | grep -Ev
'(hourly|daily|weekly|rw|ro)' | wc -l
5
This looks like a bug. My first guess is related to number of
subvolumes/reflinks, but I'm not sure since I don't have many real-world
btrfs.
I'll take sometime to look into it.
Thanks for the very interesting report,
Thanks for having a look :)
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems
what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
So, should I assume that btrfs progs git has some issue since there is no plausible way that a check --repair should be faster than a regular check? Thanks, Marc On Tue, Sep 05, 2017 at 07:45:25AM -0700, Marc MERLIN wrote: > On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote: > > > gargamel:~# btrfs fi df /mnt/btrfs_pool1 > > > Data, single: total.60TiB, used.54TiB > > > System, DUP: total2.00MiB, used=1.19MiB > > > Metadata, DUP: totalX.00GiB, used.69GiB > > > > Wait for a minute. > > > > Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format > > (again)? > > This output format must be changed, at least to 0.69 GiB, or 706 MiB. > > Email client problem. I see control characters in what you quoted. > > Let's try again > gargamel:~# btrfs fi df /mnt/btrfs_pool1 > Data, single: total=10.66TiB, used=10.60TiB => 10TB > System, DUP: total=64.00MiB, used=1.20MiB=> 1.2MB > Metadata, DUP: total=57.50GiB, used=12.76GiB => 13GB > GlobalReserve, single: total=512.00MiB, used=0.00B => 0 > > > You mean lowmem is actually FASTER than original mode? > > That's very surprising. > > Correct, unless I add --repair and then original mode is 2x faster than > lowmem. > > > Is there any special operation done for that btrfs? > > Like offline dedupe or tons of reflinks? > > In this case, no. > Note that btrfs check used to take many hours overnight until I did a > git pull of btrfs progs and built the latest from TOT. > > > BTW, how many subvolumes do you have in the fs? > > gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | wc -l > 91 > > If I remove snapshots for btrfs send and historical 'backups': > gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | grep -Ev > '(hourly|daily|weekly|rw|ro)' | wc -l > 5 > > > This looks like a bug. My first guess is related to number of > > subvolumes/reflinks, but I'm not sure since I don't have many real-world > > btrfs. > > > > I'll take sometime to look into it. > > > > Thanks for the very interesting report, > > Thanks for having a look :) > > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems > what McDonalds is to gourmet > cooking > Home page: http://marc.merlins.org/ | PGP > 1024R/763BE901 > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote: > > gargamel:~# btrfs fi df /mnt/btrfs_pool1 > > Data, single: total.60TiB, used.54TiB > > System, DUP: total2.00MiB, used=1.19MiB > > Metadata, DUP: totalX.00GiB, used.69GiB > > Wait for a minute. > > Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format > (again)? > This output format must be changed, at least to 0.69 GiB, or 706 MiB. Email client problem. I see control characters in what you quoted. Let's try again gargamel:~# btrfs fi df /mnt/btrfs_pool1 Data, single: total=10.66TiB, used=10.60TiB => 10TB System, DUP: total=64.00MiB, used=1.20MiB=> 1.2MB Metadata, DUP: total=57.50GiB, used=12.76GiB => 13GB GlobalReserve, single: total=512.00MiB, used=0.00B => 0 > You mean lowmem is actually FASTER than original mode? > That's very surprising. Correct, unless I add --repair and then original mode is 2x faster than lowmem. > Is there any special operation done for that btrfs? > Like offline dedupe or tons of reflinks? In this case, no. Note that btrfs check used to take many hours overnight until I did a git pull of btrfs progs and built the latest from TOT. > BTW, how many subvolumes do you have in the fs? gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | wc -l 91 If I remove snapshots for btrfs send and historical 'backups': gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | grep -Ev '(hourly|daily|weekly|rw|ro)' | wc -l 5 > This looks like a bug. My first guess is related to number of > subvolumes/reflinks, but I'm not sure since I don't have many real-world > btrfs. > > I'll take sometime to look into it. > > Thanks for the very interesting report, Thanks for having a look :) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
Qu Wenruo posted on Tue, 05 Sep 2017 17:06:35 +0800 as excerpted: >> See if these numbers, copied and reformatted from his post with spaces >> inserted either side of the numbers and the equals signs deleted, >> arrive any less garbled: >> >> Data, single: total 10.60 TiB, used 10.54 TiB System, DUP: total 32.00 >> MiB, used 1.19 MiB Metadata, DUP: total 58.00 GiB, used 12.69 GiB >> GlobalReserve, single: total 512.00 MiB, used 0.00 B >> > Thanks a lot for this. It worked. =:^) (But thinking about it now, that smiley with an equals sign probably won't!) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
On 2017年09月05日 16:54, Duncan wrote: Qu Wenruo posted on Tue, 05 Sep 2017 16:05:04 +0800 as excerpted: On 2017年09月05日 10:55, Marc MERLIN wrote: gargamel:~# btrfs fi df /mnt/btrfs_pool1 Data, single: total.60TiB, used.54TiB System, DUP: total2.00MiB, used=1.19MiB Metadata, DUP: totalX.00GiB, used.69GiB Wait for a minute. Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format (again)? It appears to be your end. Based on the fact that I'm seeing a a bunch of weird characters in your quote of the message that I didn't see in the original, I'm guessing it's charset related, very possibly due to the "equal" sign being an escape character for mime/quoted-printable (tho his post was text/plain; charset equals utf-8, full 8-bit, so not quoted- printable encoded at all) and I believe various i18n escapes as well, with the latter being an issue if the client assumes local charset despite the utf8 specified in the header. See if these numbers, copied and reformatted from his post with spaces inserted either side of the numbers and the equals signs deleted, arrive any less garbled: Data, single: total 10.60 TiB, used 10.54 TiB System, DUP: total 32.00 MiB, used 1.19 MiB Metadata, DUP: total 58.00 GiB, used 12.69 GiB GlobalReserve, single: total 512.00 MiB, used 0.00 B Thanks a lot for this. I'd better double check my client setup to avoid such embrassment. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
Qu Wenruo posted on Tue, 05 Sep 2017 16:05:04 +0800 as excerpted: > On 2017年09月05日 10:55, Marc MERLIN wrote: >> >> gargamel:~# btrfs fi df /mnt/btrfs_pool1 >> Data, single: total.60TiB, used.54TiB >> System, DUP: total2.00MiB, used=1.19MiB >> Metadata, DUP: totalX.00GiB, used.69GiB > > Wait for a minute. > > Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the > format (again)? It appears to be your end. Based on the fact that I'm seeing a a bunch of weird characters in your quote of the message that I didn't see in the original, I'm guessing it's charset related, very possibly due to the "equal" sign being an escape character for mime/quoted-printable (tho his post was text/plain; charset equals utf-8, full 8-bit, so not quoted- printable encoded at all) and I believe various i18n escapes as well, with the latter being an issue if the client assumes local charset despite the utf8 specified in the header. See if these numbers, copied and reformatted from his post with spaces inserted either side of the numbers and the equals signs deleted, arrive any less garbled: Data, single: total 10.60 TiB, used 10.54 TiB System, DUP: total 32.00 MiB, used 1.19 MiB Metadata, DUP: total 58.00 GiB, used 12.69 GiB GlobalReserve, single: total 512.00 MiB, used 0.00 B -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
On 2017年09月05日 10:55, Marc MERLIN wrote: On Tue, Sep 05, 2017 at 09:21:55AM +0800, Qu Wenruo wrote: On 2017年09月05日 09:05, Marc MERLIN wrote: Ok, I don't want to sound like I'm complaining :) but I updated btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB filesystem that used to take 12H or so to check. How much space allocated for that 8T fs? If metadata is not that large, 10min is valid. Here fi df output could help. gargamel:~# btrfs fi df /mnt/btrfs_pool1 Data, single: total.60TiB, used.54TiB System, DUP: total2.00MiB, used=1.19MiB Metadata, DUP: totalX.00GiB, used.69GiB Wait for a minute. Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format (again)? This output format must be changed, at least to 0.69 GiB, or 706 MiB. I'll fix this first. GlobalReserve, single: totalQ2.00MiB, used=0.00B And, without --repair, how much time it takes to run? Well, funny that you ask, it's now been running for hours, still waiting... Just before, I ran lowmem, and it was pretty quick too (didn't time it, but less than 1h): You mean lowmem is actually FASTER than original mode? That's very surprising. Is there any special operation done for that btrfs? Like offline dedupe or tons of reflinks? IIRC original mode did a quite slow check for tons of reflink, which may be related. BTW, how many subvolumes do you have in the fs? gargamel:/var/local/src/btrfs-progs# btrfs check --mode=wmem /dev/mapper/dshelf1 Checking filesystem on /dev/mapper/dshelf1 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d checking extents checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 11674263330816 bytes used, no error found total csum bytes: 11384482936 total tree bytes: 13738737664 total fs tree bytes: 758988800 total extent tree bytes: 482623488 btree space waste bytes: 1171475737 file data blocks allocated: 12888981110784 referenced 12930453286912 Now, this is good news for my filesystem being probably clean (previous versions of lowmem before my git update found issues that were unclear, but apparently errors in the code, and this version finds nothing) But I'm not sure why --repair would be fast, and not --repair would be slow? This looks like a bug. My first guess is related to number of subvolumes/reflinks, but I'm not sure since I don't have many real-world btrfs. I'll take sometime to look into it. Thanks for the very interesting report, Qu Marc -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
Ok, not quite hours, but check takes 88mn, check --repair takes 11mn gargamel:/var/local/src/btrfs-progs# time btrfs check /dev/mapper/dshelf1 Checking filesystem on /dev/mapper/dshelf1 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d checking extents checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 11674263330816 bytes used, no error found total csum bytes: 11384482936 total tree bytes: 13704478720 total fs tree bytes: 724729856 total extent tree bytes: 482623488 btree space waste bytes: 1167009013 file data blocks allocated: 12041456693248 referenced 12063146434560 real88m56.597s user2m13.985s sys 2m7.880s gargamel:/var/local/src/btrfs-progs# time btrfs check --repair /dev/mapper/dshelf1 enabling repair mode Checking filesystem on /dev/mapper/dshelf1 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d checking extents Fixed 0 roots. checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 11674263330816 bytes used, no error found total csum bytes: 11384482936 total tree bytes: 13704478720 total fs tree bytes: 724729856 total extent tree bytes: 482623488 btree space waste bytes: 1167009013 file data blocks allocated: 12041456693248 referenced 12063146434560 real11m10.499s user1m55.067s sys 1m31.666s And lowmem is 24mn: gargamel:/var/local/src/btrfs-progs# time btrfs check --mode=lowmem /dev/mapper/dshelf1 Checking filesystem on /dev/mapper/dshelf1 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d checking extents checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 11674263363584 bytes used, no error found total csum bytes: 11384482936 total tree bytes: 13738770432 total fs tree bytes: 758988800 total extent tree bytes: 482656256 btree space waste bytes: 1171508121 file data blocks allocated: 12888981110784 referenced 12930453286912 real24m20.493s user5m45.749s sys 1m40.204s Does this make any sense at all that check without --repair is so much slower than with --repair or lowmem? Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
On Tue, Sep 05, 2017 at 09:21:55AM +0800, Qu Wenruo wrote: > > > On 2017年09月05日 09:05, Marc MERLIN wrote: > >Ok, I don't want to sound like I'm complaining :) but I updated > >btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB > >filesystem that used to take 12H or so to check. > > How much space allocated for that 8T fs? > If metadata is not that large, 10min is valid. > > Here fi df output could help. gargamel:~# btrfs fi df /mnt/btrfs_pool1 Data, single: total=10.60TiB, used=10.54TiB System, DUP: total=32.00MiB, used=1.19MiB Metadata, DUP: total=58.00GiB, used=12.69GiB GlobalReserve, single: total=512.00MiB, used=0.00B > And, without --repair, how much time it takes to run? Well, funny that you ask, it's now been running for hours, still waiting... Just before, I ran lowmem, and it was pretty quick too (didn't time it, but less than 1h): gargamel:/var/local/src/btrfs-progs# btrfs check --mode=lowmem /dev/mapper/dshelf1 Checking filesystem on /dev/mapper/dshelf1 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d checking extents checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 11674263330816 bytes used, no error found total csum bytes: 11384482936 total tree bytes: 13738737664 total fs tree bytes: 758988800 total extent tree bytes: 482623488 btree space waste bytes: 1171475737 file data blocks allocated: 12888981110784 referenced 12930453286912 Now, this is good news for my filesystem being probably clean (previous versions of lowmem before my git update found issues that were unclear, but apparently errors in the code, and this version finds nothing) But I'm not sure why --repair would be fast, and not --repair would be slow? Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check --repair now runs in minutes instead of hours? aborting
On 2017年09月05日 09:05, Marc MERLIN wrote: Ok, I don't want to sound like I'm complaining :) but I updated btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB filesystem that used to take 12H or so to check. How much space allocated for that 8T fs? If metadata is not that large, 10min is valid. Here fi df output could help. And, without --repair, how much time it takes to run? It finished in maybe 10mn, just 10mn! :) gargamel:/var/local/src/btrfs-progs# btrfs check --repair /dev/mapper/dshelf1 enabling repair mode Checking filesystem on /dev/mapper/dshelf1 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d checking extents Fixed 0 roots. checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 11674263347200 bytes used, no error found total csum bytes: 11384482936 total tree bytes: 13704495104 total fs tree bytes: 724729856 total extent tree bytes: 482639872 btree space waste bytes: 1167025205 file data blocks allocated: 12041456693248 referenced 12063146434560 This is great news, but can I trust that the program worked properly and indeed that my filesystem is fully clean? Normally we rely on btrfs-progs selftest to make sure fsck works. But maybe we could cross check original mode and lowmem mode. Or at this point if I'm not running --mode=lowmem, the regular mode is really doesn't check much and only lowmem can do a proper check? (even though it can't fix problems once it finds them) No, original mode is still the core (although code is not that elegant), please consider lowmem mode as a corner case for guys with small memory (without swap) but want to check super large fs. So if it turns out to be a bug, we must fix it. Thanks, Qu Thanks Marc -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
