Re: balance hangs and starts again on reboot

[Duncan]

Holger Hoffstätte posted on Sat, 05 Mar 2016 16:38:57 +0100 as excerpted:

> On 03/05/16 15:17, Marc Haber wrote:
>>> Then try to balance in small increments.
>> 
>> -dusage=5 and incrementing? Or what do you mean with "in small
>> increments"?
> 
> Exactly, yes. Sorry for not being more clear.
> 
> FWIW I've been balancing a lot recently (both for stress testing and
> cleaning up a few filesystems) and have never run into this particular
> stall, but only ever do filtered balances. Also I wouldn't be surprised
> at all if this is yet another problem where md does something in a way
> that btrfs doesn' expect, and things go wrong.

What I thought you meant was the drange= or vrange= filters, changing the 
range to eventually cover the entire filesystem.

That should work too, tho I've never actually used it myself and I 
suspect I'd have to play around with the ranges a bit to figure out what 
numbers I should actually be supplying, as the filter descriptions in the 
manpage are somewhat vague on this point.

(Anyone who knows where to actually find those numbers to plug in and/or 
has useful examples, feel free to consider this an invitation to 
elucidate. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Holger Hoffstätte]

On 03/05/16 18:25, Marc Haber wrote:
> On Sat, Mar 05, 2016 at 04:38:57PM +0100, Holger Hoffstätte wrote:
>> On 03/05/16 15:17, Marc Haber wrote:
 Then try to balance in small increments.
>>>
>>> -dusage=5 and incrementing? Or what do you mean with "in small
>>> increments"?
>>
>> Exactly, yes. Sorry for not being more clear.
> 
> So you would recommend something along
> 
> for nr in $(seq 5 5 100); do
>   btrfs balance start -dusage=$nr $FS
> done
> 
> right?

Except for the 100 part, which seems pointless. Maybe more like
10,20..80 max. If that doesn't help you are probably out of space
anyway.

> Won't this take ages longer than a straight unfiltered balance?

Touching less stuff conditionally is pretty much guaranteed to be
faster than unconditionally rewriting everything, and less likely to
end up out of space since you garbage-collect the smallest chunks
first, freeing up a larger one and so on.

> md as in the Linux Software RAID? That's not in the game here, it's a
> single SATA hard disk.

I thought your df output contained md or something. If not, all the
better.

-h

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Marc Haber]

On Sat, Mar 05, 2016 at 04:38:57PM +0100, Holger Hoffstätte wrote:
> On 03/05/16 15:17, Marc Haber wrote:
> >> Then try to balance in small increments.
> > 
> > -dusage=5 and incrementing? Or what do you mean with "in small
> > increments"?
> 
> Exactly, yes. Sorry for not being more clear.

So you would recommend something along

for nr in $(seq 5 5 100); do
  btrfs balance start -dusage=$nr $FS
done

right?

Won't this take ages longer than a straight unfiltered balance?

> FWIW I've been balancing a lot recently (both for stress testing and
> cleaning up a few filesystems) and have never run into this particular
> stall, but only ever do filtered balances. Also I wouldn't be surprised
> at all if this is yet another problem where md does something in a way
> that btrfs doesn' expect, and things go wrong.

md as in the Linux Software RAID? That's not in the game here, it's a
single SATA hard disk.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Chris Murphy]

On Sat, Mar 5, 2016 at 7:12 AM, Marc Haber  wrote:

> What is the most helpful way to include logs?

Attach as a text file. If they're too big and get rejected, then it
depends. If I'm pretty sure it's a bug, I open a bug report on
bugzilla.kernel.org and attach there, then URL in list. If I'm not
sure, I put the text file on google drive or dropbox and post public
URL here.

> including them
> causes "annoying MUA wrapping".

It can be either sending or receiving client that does this. I have no
doubt gmail does it sending and receiving because I always have this
problem.

> I do only have 24 years of e-mail
> experience, so I'm a clueless newbie, maybe one can give advice how to
> do that properly.

I've given up with pasting kernel messages inline.



-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Holger Hoffstätte]

On 03/05/16 15:17, Marc Haber wrote:
>> Then try to balance in small increments.
> 
> -dusage=5 and incrementing? Or what do you mean with "in small
> increments"?

Exactly, yes. Sorry for not being more clear.

FWIW I've been balancing a lot recently (both for stress testing and
cleaning up a few filesystems) and have never run into this particular
stall, but only ever do filtered balances. Also I wouldn't be surprised
at all if this is yet another problem where md does something in a way
that btrfs doesn' expect, and things go wrong.

-h

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Marc Haber]

On Fri, Mar 04, 2016 at 07:09:39PM +0100, Holger Hoffstätte wrote:
> On 03/04/16 18:31, Marc Haber wrote:
> > I have another btrfs on the same host that has no the no space left on
> > device balance issue, but on another disk. On this btrfs, it seems
> > like a balance process is stuck, with a lot of hanging kernel
> > threads. After a reboot, when I mount the filesystem, the balance
> > immediately starts again. btrfs balance cancel just hangs around with
> > no visible reaction for hours.
> > 
> > Log appended. Is there rescue?
> 
> Can't offer much help other than to recommend to *always* mount with
> -o skip_balance, which IMHO should have been the default behaviour
> from the beginning.

That's an important hint. The btrfs balance cancel has worked over
night though.

> Then try to balance in small increments.

-dusage=5 and incrementing? Or what do you mean with "in small
increments"?

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Marc Haber]

Hi Chris,

I apologize for not being able to deliver logs in the way you might
find them more helpful.

On Fri, Mar 04, 2016 at 12:08:10PM -0700, Chris Murphy wrote:
> On Fri, Mar 4, 2016 at 10:31 AM, Marc Haber  
> wrote:
> > I have another btrfs on the same host that has no the no space left on
> > device balance issue, but on another disk. On this btrfs, it seems
> > like a balance process is stuck, with a lot of hanging kernel
> > threads. After a reboot, when I mount the filesystem, the balance
> > immediately starts again. btrfs balance cancel just hangs around with
> > no visible reaction for hours.
> >
> > Log appended. Is there rescue?
> 
> The log is made much more useful if you can sysrq+w while the blocked
> task is happening; and then dmesg or journalctl -k to get the results
> into a file for attachment to avoid the annoying MUA wrapping.

This list has repeatedly eaten log attachments without giving any
indication why. I had assumed that attachments are disallowed here,
and am taking careful attention that inserted logs are not wrapped on
my side. The list archives
(http://www.spinics.net/lists/linux-btrfs/msg52663.html) show that my
efforts not to cause wrapping on my side were actually successful.

What is the most helpful way to include logs? Pastebinning them would
probably reduce the list archives' usefulness due to pastebin
expiring, attaching doesn't work (see above), and including them
causes "annoying MUA wrapping".  I do only have 24 years of e-mail
experience, so I'm a clueless newbie, maybe one can give advice how to
do that properly.

I'm going to try the sysrq+w thing next time things happen.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Chris Murphy]

On Fri, Mar 4, 2016 at 10:31 AM, Marc Haber  wrote:
> Hi,
>
> I have another btrfs on the same host that has no the no space left on
> device balance issue, but on another disk. On this btrfs, it seems
> like a balance process is stuck, with a lot of hanging kernel
> threads. After a reboot, when I mount the filesystem, the balance
> immediately starts again. btrfs balance cancel just hangs around with
> no visible reaction for hours.
>
> Log appended. Is there rescue?

The log is made much more useful if you can sysrq+w while the blocked
task is happening; and then dmesg or journalctl -k to get the results
into a file for attachment to avoid the annoying MUA wrapping.




-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: balance hangs and starts again on reboot

[Holger Hoffstätte]

On 03/04/16 18:31, Marc Haber wrote:
> I have another btrfs on the same host that has no the no space left on
> device balance issue, but on another disk. On this btrfs, it seems
> like a balance process is stuck, with a lot of hanging kernel
> threads. After a reboot, when I mount the filesystem, the balance
> immediately starts again. btrfs balance cancel just hangs around with
> no visible reaction for hours.
> 
> Log appended. Is there rescue?

Can't offer much help other than to recommend to *always* mount with
-o skip_balance, which IMHO should have been the default behaviour
from the beginning. Then try to balance in small increments.

-h

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html