Re: Pre-made live image with actual functioning persistent storage?

[Josef Bacik]

On 12/11/20 1:48 PM, Matthew Miller wrote:

We're going to spend some of our remaining budget on a swag refresh, and as
part of that, we're getting some shiny new USB sticks, and partly
considering the recent discussion about making it easy for new users to try
without committing, we're getting ones with decent speed and quality and
room for a persistent overlay.

It has been approximately a decade since I made such an image. Does anyone
have a recipe for creating one with persistent storage (and possibly a
separate home filesystem, or however that should be set up on btrfs?)



I'm going to be less than helpful with specifics, but btrfs has this feature 
called "seed devices" that allows you to generate a btrfs device that is read 
only.  The mechanical steps are here


https://btrfs.wiki.kernel.org/index.php/Seed-device

This is actually what we use for all of our network switches inside FB, we 
generate a seed image that is dd'ed onto the devices.  On bootup they find the 
other partition on the device and add it to the boot up device (which is the 
seed device) before the remount rw step, which gives you the scratch space. 
This scratch space is cleared on reboot because the seed device is the boot 
target and since it's RO it can't be modified to know anything about the scratch 
device.


This provides a kind neat/clean way to make the live environment permanent. 
Simply btrfs device add /dev/whatever /, btrfs device delete /dev/seed /, btrfs 
device delete /dev/scratch-partition /.  This copies all of the data we need 
from the seed device to the local drive and then you're good to go.  Obviously 
the boot loader needs to do its boot loader thing and such, but the copy is nice 
and efficient.


There's no tooling for this, but if one wanted to build tooling around it then 
this would be the way to do it generally.  Not super helpful for you right now, 
but in case anybody is bored and looking for something to do, this is how I 
would do it.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: btrfs and default page sizes (4k vs 64k)

[Josef Bacik]

On 9/16/20 3:18 PM, Eugene Syromiatnikov wrote:

On Wed, Sep 16, 2020 at 03:04:45PM -0400, Josef Bacik wrote:

At the time we tied the fs blocksize to the
page size, because it was unlikely that a user would mkfs a fs on one arch
and move it over to another arch.


But one doesn't need "another arch" for page size to change; many
architectures (arm, mips, powerpc, sparc, to name a few) support multiple
page sizes.


Sure, but again you are not likely to change page size for an existing system. 
The decision early on was to forgo this particular ability for simplicity, and 
then we would revisit the decision later on.  It's been a while and there's 
still not been enough demand to justify the work until recently.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: btrfs and default page sizes (4k vs 64k)

[Josef Bacik]

On 9/14/20 3:31 AM, Daniel Pocock wrote:



Given the plans to make btrfs the default, I'll share some of my own
recent experiences, hopefully this can make it easier for the next person

One issue I've come across is that a btrfs filesystem can only be used
on hosts with the same page size as the host that created the filesystem

E.g. x86-64 kernels have a 4k default page size but powerpc64le kernels
have been compiled with the optional 64k page size.  This impacts
various distributions.

If somebody creates some filesystems with the 4k parameter and then they
migrate them to the powerpc64le host, they won't mount

If they try to go the other way, the filesystems won't mount either

There are other non-btrfs issues related to the 64k page size, for
example, nouveau driver won't work either

To make things easier for btrfs, could it be worthwhile changing the
default page size from 64k back to 4k on default kernels for most
ordinary users?



Yeah subpage blocksize support isn't something that we've prioritized.  When 
btrfs was originally written the only option for that was to use buffer heads, 
which removed a lot of the flexibility we needed to support things like multi 
device file systems.  At the time we tied the fs blocksize to the page size, 
because it was unlikely that a user would mkfs a fs on one arch and move it over 
to another arch.


There is work ongoing from Suse to bring this support in, there was a patchset 
last week posted to add read-only support for sub-page blocksizes.  Write 
support will be harder but is coming along.  However these are obviously not 
going to be ready for F33 timeline, nor probably F34.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/9/20 9:30 PM, Eric Sandeen wrote:

On 7/9/20 8:22 PM, Josef Bacik wrote:

On 7/9/20 7:23 PM, Eric Sandeen wrote:

On 7/9/20 4:27 PM, Eric Sandeen wrote:

On 7/9/20 3:32 PM, Davide Cavalca via devel wrote:


...


As someone on one of the teams at FB that has to deal with that, I can
assure you all the scenarios you listed can and do happen, and they
happen a lot. While we don't have the "laptop's out of battery" issue
on the production side, we have plenty of power events and unplanned
maintenances that can and will hit live machines and cut power off.
Force reboots (triggered by either humans or automation) are also not
at all uncommon. Rebuilding machines from scratch isn't free, even with
all the automation and stuff we have, so if power loss or reboot events
on machines using btrfs caused widespread corruption or other issues
I'm confident we'd have found that out pretty early on.


It is a bare minimum expectation that filesystems like btrfs, ext4, and xfs
do not suffer filesystem corruptions and inconsistencies due to reboots
and power losses.

So for the record I am in no way insinuating that btrfs is less crash-safe
than other filesystems (though I have not tested that, so if I have time
I'll throw that into the mix as well.)


So, we already have those tests in xfstests, and I put btrfs through a few
loops.  This is generic/475:

# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
#
# FS QA Test No. 475
#
# Test log recovery with repeated (simulated) disk failures.  We kick
# off fsstress on the scratch fs, then switch out the underlying device
# with dm-error to see what happens when the disk goes down.  Having
# taken down the fs in this manner, remount it and repeat.  This test
# is a Good Enough (tm) simulation of our internal multipath failure
# testing efforts.

It fails within 2 loops.  Is it a critical failure? I don't know; the
test looks for unexpected things in dmesg, and perhaps the filter is
wrong.  But I see stack traces during the run, and message like:

[689284.484258] BTRFS: error (device dm-3) in btrfs_sync_log:3084: errno=-117 
Filesystem corrupted


You might want to change that message, then.  If it's not corrupted, I'd suggest not doing 
printk("corrupted!") because that will make people think that it's corrupted, because it 
says "Filesystem corrupted..." ;)



Yeah probably not the best, but again not something a user will generally see.



Yeah, because dm-error throws EIO, and thus we abort the transaction, which 
results in an EUCLEAN if you run fsync.  This is a scary sounding message, but 
its _exactly_ what's expected from generic/475.  I've been running this in a 
loop for an hour and the thing hasn't failed yet.  There's all sorts of scary 
messages


That's weird.  The test fails very quickly for me - again, AFAICT it fails due 
to things in dmesg that aren't recognized as safe by the test harness, but a 
variety of things - not just stack dumps - seem to trigger the failure.


Do you know what's tripping it?  Because my loop is still running happily along.




[17929.939871] BTRFS warning (device dm-13): direct IO failed ino 261 rw 
1,34817 sector 0xb8ce0 len 24576 err no 10
[17929.943099] BTRFS: error (device dm-13) in btrfs_commit_transaction:2323: 
errno=-5 IO failure (Error while writing out transaction)

again, totally expected because we're forcing EIO's at random times.


Right, of course it will get IO errors, that's why I didn't highlight those in 
my email.


so I can't say for sure.

Are btrfs devs using these tests to assess crash/powerloss resiliency
on a regular basis?  TBH I honestly did not expect to see any test
failures here, whether or not they are test artifacts; any filesystem
using xfstests as a benchmark needs to be keeping things up to date.



It depends on the config options.  Some of our transaction abort sites dump 
stack, and that trips the dmesg filter, and thus it fails.  Generally when I 
run this test I turn those options off.


It would be good, in general, to fix up the test for btrfs so that it does not 
yield false positives, if that's what this is.  Otherwise it trains people to 
ignore it or not run it


Except it doesn't, it's not failing for me now.  Like I said we pay particularly 
close attention to this test because it has a habit of finding memory leaks or 
reference accounting bugs.





This test is run constantly by us, specifically because it's the error cases 
that get you.  But not for crash consistency reasons, because we're solid 
there.  I run them to make sure I don't have stupid things like reference leaks 
or whatever in the error path.  Thanks,


or "corrupted!" printk()s that terrify the hapless user? ;)


I'd love to know what hapless user is running xfstests.  Thanks,

Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedorapr

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/9/20 7:23 PM, Eric Sandeen wrote:

On 7/9/20 4:27 PM, Eric Sandeen wrote:

On 7/9/20 3:32 PM, Davide Cavalca via devel wrote:


...


As someone on one of the teams at FB that has to deal with that, I can
assure you all the scenarios you listed can and do happen, and they
happen a lot. While we don't have the "laptop's out of battery" issue
on the production side, we have plenty of power events and unplanned
maintenances that can and will hit live machines and cut power off.
Force reboots (triggered by either humans or automation) are also not
at all uncommon. Rebuilding machines from scratch isn't free, even with
all the automation and stuff we have, so if power loss or reboot events
on machines using btrfs caused widespread corruption or other issues
I'm confident we'd have found that out pretty early on.


It is a bare minimum expectation that filesystems like btrfs, ext4, and xfs
do not suffer filesystem corruptions and inconsistencies due to reboots
and power losses.

So for the record I am in no way insinuating that btrfs is less crash-safe
than other filesystems (though I have not tested that, so if I have time
I'll throw that into the mix as well.)


So, we already have those tests in xfstests, and I put btrfs through a few
loops.  This is generic/475:

# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
#
# FS QA Test No. 475
#
# Test log recovery with repeated (simulated) disk failures.  We kick
# off fsstress on the scratch fs, then switch out the underlying device
# with dm-error to see what happens when the disk goes down.  Having
# taken down the fs in this manner, remount it and repeat.  This test
# is a Good Enough (tm) simulation of our internal multipath failure
# testing efforts.

It fails within 2 loops.  Is it a critical failure? I don't know; the
test looks for unexpected things in dmesg, and perhaps the filter is
wrong.  But I see stack traces during the run, and message like:

[689284.484258] BTRFS: error (device dm-3) in btrfs_sync_log:3084: errno=-117 
Filesystem corrupted



Yeah, because dm-error throws EIO, and thus we abort the transaction, which 
results in an EUCLEAN if you run fsync.  This is a scary sounding message, but 
its _exactly_ what's expected from generic/475.  I've been running this in a 
loop for an hour and the thing hasn't failed yet.  There's all sorts of scary 
messages


[17929.939871] BTRFS warning (device dm-13): direct IO failed ino 261 rw 1,34817 
sector 0xb8ce0 len 24576 err no 10
[17929.943099] BTRFS: error (device dm-13) in btrfs_commit_transaction:2323: 
errno=-5 IO failure (Error while writing out transaction)


again, totally expected because we're forcing EIO's at random times.


so I can't say for sure.

Are btrfs devs using these tests to assess crash/powerloss resiliency
on a regular basis?  TBH I honestly did not expect to see any test
failures here, whether or not they are test artifacts; any filesystem
using xfstests as a benchmark needs to be keeping things up to date.



It depends on the config options.  Some of our transaction abort sites dump 
stack, and that trips the dmesg filter, and thus it fails.  Generally when I run 
this test I turn those options off.


This test is run constantly by us, specifically because it's the error cases 
that get you.  But not for crash consistency reasons, because we're solid there. 
 I run them to make sure I don't have stupid things like reference leaks or 
whatever in the error path.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/9/20 1:51 PM, Eric Sandeen wrote:

On 7/6/20 12:07 AM, Chris Murphy wrote:

On Fri, Jul 3, 2020 at 8:40 PM Eric Sandeen 
wrote:


On 7/3/20 1:41 PM, Chris Murphy wrote:

SSDs can fail in weird ways. Some spew garbage as they're
failing, some go read-only. I've seen both. I don't have stats on
how common it is for an SSD to go read-only as it fails, but once
it happens you cannot fsck it. It won't accept writes. If it
won't mount, your only chance to recover data is some kind of
offline scrape tool. And Btrfs does have a very very good scrape
tool, in terms of its success rate - UX is scary. But that can
and will improve.


Ok, you and Josef have both recommended the btrfs restore
("scrape") tool as a next recovery step after fsck fails, and I
figured we should check that out, to see if that alleviates the
concerns about recoverability of user data in the face of
corruption.

I also realized that mkfs of an image isn't representative of an
SSD system typical of Fedora laptops, so I added "-m single" to
mkfs, because this will be the mkfs.btrfs default on SSDs (right?).
Based on Josef's description of fsck's algorithm of throwing away
any block with a bad CRC this seemed worth testing.

I also turned fuzzing /down/ to hitting 2048 bytes out of the 1G
image, or a bit less than 1% of the filesystem blocks, at random.
This is 1/4 the fuzzing rate from the original test.

So: -m single, fuzz 2048 bytes of 1G image, run btrfsck --repair,
mount, mount w/ recovery, and then restore ("scrape") if all that
fails, see what we get.


What's the probability of this kind of corruption occurring in the
real world? If the probability is so low it can't practically be
computed, how do we assess the risk? And if we can't assess risk,
what's the basis of concern?


 From 20 years of filesystem development experience, I know that people
run filesystem repair tools.  It's just a fact.  For a wide variety of
reasons - from bugs, to hardware errors, to admin errors, you name it,
filesystems experience corruption and inconsistencies.  At that point
the administrator needs a path forward.

"people won't need to repair btrfs" is, IMHO, the position that needs
to be supported, not "filesystem repair tools should be robust."


I ran 50 loops, and got:

46 btrfsck failures 20 mount failures

So it ran btrfs restore 20 times; of those, 11 runs lost all or
substantially all of the files; 17 runs lost at least 1/3 of the
files.


Josef states reliability of ext4, xfs, and Btrfs are in the same
ballpark. He also reports one case in 10 years in which he failed to
recover anything. How do you square that with 11 complete failures,
trivially produced? Is there even a reason to suspect there's
residual risk?


Extrapolating from Facebook's usecases to the fedora desktop should be
approached with caution, IMHO.

I've provided evidence that if/when damage happens for whatever reason,
btrfs is unable to recover in place far more often than other filesytems.


When metadata is single profile, Btrfs is basically an early warning
system.> The available research on uncorrectable errors, errors that drive ECC
does not catch, suggests that users are decently likely to experience
at least one block of corruption in the life of the drive. And that
it tends to get worse up until drive failure. But there is much less
chance to detect this, if the file system isn't also checksumming the
vastly larger payload on a drive: the data.


One of the problems in this whole discussion is the assumption that filesystem
inconsistencies only arise from disk bitflips etc; that's just not the case.

Look, I'm just providing evidence of what I've found when re-evaluating the
btrfs administration/repair tools.  I've found them to be quite weak.

 From what I've gathered from these responses, btrfs is unique in that it is
/expected/ that if anything goes wrong, the administrator should be prepared
to scrape out remaining data, re-mkfs, and start over.  If that's acceptable
for the Fedora desktop, that's fine, but I consider it a risk that should not
be ignored when evaluating this proposal.



Agreed, it's the very first thing I said when I was asked what are the 
downsides.  There's clearly more work to be done in the recovery arena.  How 
often do disks fail for Fedora?  Do we have that data?  Is this a real risk? 
Nobody can say because Fedora doesn't have data.


Facebook does however have that data, and it's a microscopically small 
percentage.  I agree that Facebook is vastly different from Fedora from a 
recovery standpoint, but our workloads and hardware I think extrapolate to the 
normal Fedora user quite well.  We drive the disks harder than the normal Fedora 
user does of course, but in the end we're updating packages, taking snapshots, 
and building code.  We're just doing it at 1000x what a normal Fedora user does. 
 Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an 

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/3/20 10:39 PM, Eric Sandeen wrote:

On 7/3/20 1:41 PM, Chris Murphy wrote:

SSDs can fail in weird ways. Some spew garbage as they're failing,
some go read-only. I've seen both. I don't have stats on how common it
is for an SSD to go read-only as it fails, but once it happens you
cannot fsck it. It won't accept writes. If it won't mount, your only
chance to recover data is some kind of offline scrape tool. And Btrfs
does have a very very good scrape tool, in terms of its success rate -
UX is scary. But that can and will improve.


Ok, you and Josef have both recommended the btrfs restore ("scrape")
tool as a next recovery step after fsck fails, and I figured we should
check that out, to see if that alleviates the concerns about
recoverability of user data in the face of corruption.

I also realized that mkfs of an image isn't representative of an SSD
system typical of Fedora laptops, so I added "-m single" to mkfs,
because this will be the mkfs.btrfs default on SSDs (right?).  Based
on Josef's description of fsck's algorithm of throwing away any
block with a bad CRC this seemed worth testing.

I also turned fuzzing /down/ to hitting 2048 bytes out of the 1G
image, or a bit less than 1% of the filesystem blocks, at random.
This is 1/4 the fuzzing rate from the original test.

So: -m single, fuzz 2048 bytes of 1G image, run btrfsck --repair,
mount, mount w/ recovery, and then restore ("scrape") if all that
fails, see what we get.

I ran 50 loops, and got:

46 btrfsck failures
20 mount failures

So it ran btrfs restore 20 times; of those, 11 runs lost all or
substantially all of the files; 17 runs lost at least 1/3 of the
files.


Hmm I wonder if some of my "ignore X failures" stuff got lost over the years, we 
should be able to recover far more than that.  I'll add it to the list of things 
to dig into this week.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/3/20 9:37 AM, Eric Sandeen wrote:

On 7/1/20 2:50 PM, Josef Bacik wrote:

On 7/1/20 2:24 PM, Matthew Miller wrote:

On Wed, Jul 01, 2020 at 06:54:02AM +, Zbigniew Jędrzejewski-Szmek wrote:

Making btrfs opt-in for F33 and (assuming the result go well) opt-out for F34
could be good option. I know technically it is already opt-in, but it's not
very visible or popular. We could make the btrfs option more prominent and
ask people to pick it if they are ready to handle potential fallout.


I'm leaning towards recommending this as well. I feel like we don't have
good data to make a decision on -- the work that Red Hat did previously when
making a decision was 1) years ago and 2) server-focused, and the Facebook
production usage is encouraging but also not the same use case. I'm
particularly concerned about metadata corruption fragility as noted in the
Usenix paper. (It'd be nice if we could do something about that!)



There's only so much we can do about this.  I've sent up patches to ignore 
failed global trees to allow users to more easily recover data in case of 
corruption in the case of global trees, but as they say if only 1 bit is off in 
a node, we throw the whole node away.  And throwing a node away means you lose 
access to any of its children, which could be a large chunk of the file system.

This sounds like a "wtf, why are you doing this btrfs?" sort of thing, but this 
is just the reality of using checksums.  It's a checksum, not ECC.  We don't know _which_ 
bits are fucked, we just know somethings fucked, so we throw it all away.  If you have 
RAID or DUP then we go read the other copy, and fix the broken copy if we find a good 
copy.  If we don't, well then there's nothing really we can do.


There is often a path forward when a bad metadata checksum is detected.
i.e. e2fsck:

scan_extent_node() {
...

 /* Failed csum but passes checks?  Ask to fix checksum. */
 if (failed_csum &&
 fix_problem(ctx, PR_1_EXTENT_ONLY_CSUM_INVALID, pctx)) {
 pb->inode_modified = 1;
 pctx->errcode = ext2fs_extent_replace(ehandle, 0, );
 if (pctx->errcode)
 return;
 }

it does similarly for many types of metadata:

/* inode passes checks, but checksum does not match inode */
#define PR_1_INODE_ONLY_CSUM_INVALID0x010068
--
/* Inode extent block passes checks, but checksum does not match extent */
#define PR_1_EXTENT_ONLY_CSUM_INVALID   0x01006A
--
/* Inode extended attribute block passes checks, but checksum does not
  * match block. */
#define PR_1_EA_BLOCK_ONLY_CSUM_INVALID 0x01006C
--
/* dir leaf node passes checks, but fails checksum */
#define PR_2_LEAF_NODE_ONLY_CSUM_INVALID0x02004D

Does btrfsck really never attempt to salvage a metadata block with a bad CRC by
validating its fields?


No, I suppose we could, I'll add it to the list.  Generally speaking if there's 
a bad checksum detected we just attempt to recover based on what we couldn't get 
access to.  However that's difficult if it's a node.  If it's a leaf then 
usually you just lose some metadata that can be inferred from other data.  For 
example if you lose a leaf in the extent tree, well we can add all that 
information back once we've scanned the rest of the file system and know what 
extents are missing in the extent tree.


Same goes for directory items, we detect that we are missing directory items, 
but we have references for them and so we add the missing directory items that 
were lost from that corrupt block.


But again, if you lose a node you lose access to many leaves, which makes it 
more likely we'll lose somehting because we'll lose the other information we can 
use to recover what was lost.  The extent tree and checksum trees are exceptions 
to this, since they can be rebuilt from scratch, provided everything else is fine.


And then if we did decide to validate nodes, we _might_ be ok, but we might end 
up with old versions of leaves because it happens to point at something that 
appears to be correct, but isn't really.  Our metadata changes all the time, so 
it's not outside the realm of possiblities that the corruption points at a 
seemlingly valid piece of metadata, but isn't and thus makes us do something 
_really_ wrong.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

Yeah I mean the general discussion, not you specifically.  Thanks,

Josef

On Thu, Jul 2, 2020 at 8:38 PM Eric Sandeen  wrote:

> On 7/2/20 4:44 PM, Josef Bacik wrote:
> > We're talking about this issue like it's reasonable that xfs and ext4
> are going to allow the user to get back a bunch of data they don't know is
> ok or not. We're also talking about it like the user should be able to
> carry on his happy merry way.  In these cases the drive is dying and needs
> to be shredded, and a new install needs to happen and a restore from
> backups needs to happen.  Is the btrfs failure much less user friendly?  No
> doubt about it.  Is it any comfort at all when a user shows up and we say
> "where are your backups" and they say "what backups?", no.  But if we're
> going to talk about this like ext4 and xfs are much better because they
> give you the _appearance_ that your data is fine, that's a bit disingenuous.
>
> If I had talked about it like that, it would have been disingenuous.
>
> But I didn't; this was an investigation of resiliency to metadata
> corruption, not data error detection, and to what degree metadata
> corruption can render files or even entire filesystems unreachable after
> normal administrative recovery efforts.
>
> -Eric
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/2/20 4:38 PM, Eric Sandeen wrote:

On 7/1/20 12:50 PM, Chris Murphy wrote:

...


Integrity checking is highly valued by some and less by others.
Considering that we know hardware isn't 100% reliable, and doesn't
always report its own failures as expected, and hence why most file
systems now at least checksum metadata, it's not persuasive to me that
the data should be left unchecked, and corruption ought to be handled
by user space somehow.


There's a flip side to this coin - in my experience, if the right btrfs
metadata blocks experience this disk corruption, there can be
a complete inability to recover the btrfs filesystem from that error -
i.e. it won't mount, and btrfsck --repair won't get it to a mountable
state.

So if we're saying disk corruption happens often enough that data
checksumming is critical, then it happens often enough that metadata
recovery is at least as critical.

I've been trying to quantify this and have not come up with a particularly
compelling test scenario, because it involves purposefully (though at random)
corrupting enough blocks on a filesystem image that a critical block gets
hit, so it looks synthetic.  But the net result is frequently a filesystem
where btrfsck and/or mount fails, and at first blush this type of failure
happens much more often than on other filesystems.[1]

I think Josef has alluded to this situation as well.  To me, that's a big
concern.  Not trying to be a wet blanket here but I think this needs to be
carefully investigated and evaluated to understand what impact it may have
on Fedora btrfs users and their ability to recover their data in the face
of metadata corruption, because it looks to me like a definite btrfs weak
spot.


Yeah this is what I've said many times over the last 3 weeks.  Btrfs is more 
vulnerable to metadata corruption.


Now there's things that we can do to mitigate this.  I have one patch up to 
handle one of the main cases (a corrupt global tree).  The next patch set will 
be to keep entire metadata tree's around for longer as long as we have space to 
handle it.  These two things will drastically improve the situation, but of 
course if I'm being evil we can still end up in a bad spot.  These patches are 
not hard or controversial, they'll likely land in 5.9 which will be what F33 
ships with (if I'm doing my math right).


And this sort of ignores the other side of the coin.  fsfuzzer isn't just 
corrupting metadata, it's corrupting data.  Btrfs is the only file system that's 
going to notice that and let the user know.


Checksumming is great because it lets the user know things are going wrong 
before they go catastrophically wrong.  However just because we know something 
went wrong doesn't mean we can do anything about it, it just means that the user 
knows now that they need to restore from backups and find a new drive.  These 
features do not mean you are absolved of good practices.  If you care about 
data, you need to have it in multiple places.  End of story.  Btrfs is just 
going to let you know in advance that things are going wrong.


We're talking about this issue like it's reasonable that xfs and ext4 are going 
to allow the user to get back a bunch of data they don't know is ok or not. 
We're also talking about it like the user should be able to carry on his happy 
merry way.  In these cases the drive is dying and needs to be shredded, and a 
new install needs to happen and a restore from backups needs to happen.  Is the 
btrfs failure much less user friendly?  No doubt about it.  Is it any comfort at 
all when a user shows up and we say "where are your backups" and they say "what 
backups?", no.  But if we're going to talk about this like ext4 and xfs are much 
better because they give you the _appearance_ that your data is fine, that's a 
bit disingenuous.


"Well what if it was just /usr."  Sure, then you got lucky and you could copy 
things off.  But what if it wasn't?  That's the measure that's being applied to 
btrfs here.  Is it likely that random corruption is going to be so bad that you 
end up with an unmountable file system?  It's about as likely that the random 
corruption is on your dissertation or your family photographs.  The difference 
is that btrfs will tell you that your dissertation or your family photographs 
are now bad, whereas ext4 and xfs will not.


These are tradeoffs no doubt.  Every file system choice is a series of trade 
offs.  We're arguing/optimizing for the narrowest usecase.  Arguments can be 
made either way, but in the end is it important enough to not move ahead with 
btrfs?  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/1/20 9:49 PM, Chris Adams wrote:

Once upon a time, Josef Bacik  said:

This sounds like a "wtf, why are you doing this btrfs?" sort of
thing, but this is just the reality of using checksums.  It's a
checksum, not ECC.  We don't know _which_ bits are fucked, we just
know somethings fucked, so we throw it all away.  If you have RAID
or DUP then we go read the other copy, and fix the broken copy if we
find a good copy.  If we don't, well then there's nothing really we
can do.


That's where an fsck and a lost+found type directory should come into
play.  Maybe punt to user space, but still try to see what you can make
sense of to try to salvage.  If you are saying a single bit error in the
wrong place can basically lop off a good chunk of a filesystem, then I'm
going to say that's not an improvement in reliability.



We do, the recovery tools allow you to just ignore checksums.  This is 
specifically separate from everything else because there's the expectation of 
results.  The user is acknowledging that things are bad and the tools are going 
to do their very best.  If you know you only have a single bit off then hooray, 
you got everything back (probably), but if not then you don't.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/1/20 2:24 PM, Matthew Miller wrote:

On Wed, Jul 01, 2020 at 06:54:02AM +, Zbigniew Jędrzejewski-Szmek wrote:

Making btrfs opt-in for F33 and (assuming the result go well) opt-out for F34
could be good option. I know technically it is already opt-in, but it's not
very visible or popular. We could make the btrfs option more prominent and
ask people to pick it if they are ready to handle potential fallout.


I'm leaning towards recommending this as well. I feel like we don't have
good data to make a decision on -- the work that Red Hat did previously when
making a decision was 1) years ago and 2) server-focused, and the Facebook
production usage is encouraging but also not the same use case. I'm
particularly concerned about metadata corruption fragility as noted in the
Usenix paper. (It'd be nice if we could do something about that!)



There's only so much we can do about this.  I've sent up patches to ignore 
failed global trees to allow users to more easily recover data in case of 
corruption in the case of global trees, but as they say if only 1 bit is off in 
a node, we throw the whole node away.  And throwing a node away means you lose 
access to any of its children, which could be a large chunk of the file system.


This sounds like a "wtf, why are you doing this btrfs?" sort of thing, but this 
is just the reality of using checksums.  It's a checksum, not ECC.  We don't 
know _which_ bits are fucked, we just know somethings fucked, so we throw it all 
away.  If you have RAID or DUP then we go read the other copy, and fix the 
broken copy if we find a good copy.  If we don't, well then there's nothing 
really we can do.


As for their complaint about DIR_INDEX vs DIR_ITEM recovery, that's been around 
for a while now.  A lot of these things have been added over the last year.


Another thing to keep in mind is that fsck is _very_ conservative for a reason. 
It's only job is to get the fs back to the point that it can be mounted, it has 
no knowledge of what data is important and which is not.  So by default it 
doesn't do much, because we want the user to be able to use the rescue tools to 
pull off any data they can before they run repair.  Because it's possible that 
fsck decides to delete problematic entries, and maybe those entries are to data 
you cared about.


I've stated this many times before, btrfs is more vulnerable to things going 
wrong.  It's also more likely to notice things going wrong.  There's things we 
can do to make it easier in the face of these issues, they're patches I've 
written and submitted in the last few days.  There's bigger, more complex things 
that I can do to make us more resilient in the face of these corruptions.  But 
even with all of the things I have in my head, I could still go do one or two 
things and render the file system unusable.  Would these things happen in 
practice?  Unlikely.  Is it impossible?  Unfortunately no.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 7/1/20 7:49 AM, Steven Whitehouse wrote:

Hi,

On 01/07/2020 12:09, Zbigniew Jędrzejewski-Szmek wrote:

On Wed, Jul 01, 2020 at 11:28:10AM +0100, Steven Whitehouse wrote:

Hi,

On 01/07/2020 07:54, Zbigniew Jędrzejewski-Szmek wrote:

On Mon, Jun 29, 2020 at 03:15:23PM -0400, Solomon Peachy wrote:

So yes, I think an explicit "let's all test btrfs (as anaconda
configures it) before we make it default" period is warranted.

Perhaps one can argue that Fedora has already been doing that for the
past two years (since 2018-or-later-btrfs is what everyone with positive
results appears to be talking about), but it's still not clear that
those deployments utilize the same feature set as Fedora's defaults, and
how broad the hardware sample is.

Making btrfs opt-in for F33 and (assuming the result go well) opt-out for F34
could be good option. I know technically it is already opt-in, but it's not
very visible or popular. We could make the btrfs option more prominent and
ask people to pick it if they are ready to handle potential fallout.

Normally we just switch the default or we don't, without half measures. But
the fs is important enough and complicated enough to be extra careful about
any transitions.

Zbyszek

Indeed, it is an important point, and taking care is very important
when dealing with other people's data, which is in effect what we
are discussing here.

When we looked at btrfs support in RHEL, we took quite a long time
over it. In fact I'm not quite sure how long, since the process had
started before I was involved, but it was not a decision that was
made quickly, and a great deal of thought went into it. It was
difficult to get concrete information about the stability aspects at
the time. Just like the discussions that have taken place on this
thread, there was a lot of anecdotal evidence, but that is not
always a good indicator. Since time has passed since then, and there
is now more evidence, this part of the process should be easier.
That said to get a meaningful comparison then ideally one would want
to compare on the basis of user populations of similar size and
technical skill level, and look not just at the overall number of
bugs reported, but at the rate those bugs are being reported too.

Yeah. I have no doubt that the decision was made carefully back then.
That said, time has passed, and btrfs has evolved and our use cases
have evolved too, so a fresh look is good.

We have https://fedoraproject.org/wiki/Changes/DNF_Better_Counting,
maybe this could be used to collect some statistics about the fs type
too.


Yes, and also the questions that Fedora is trying to answer are different too. 
So I don't think that our analysis for RHEL is applicable here in general. The 
method that we went through, in general terms, may potentially be helpful.




It is often tricky to be sure of the root cause of bugs - just
because a filesystem reports an error doesn't mean that it is at
fault, it might be a hardware problem, or an issue with volume
management. Figuring out where the real problem lies is often very
time consuming work. Without that work though, the raw numbers of
bugs reported can be very misleading.
It would be worth taking that step here, and
asking each of the spins what are the features that they would most
like to see from the storage/fs stack. Comparing filesystems in the
abstract is a difficult task, and it is much easier against a
context. I know that some of the issues have already been discussed
in this thread, but maybe if someone was to gather up a list of
requirements from those messages then that would help to direct
further discussion,

Actually that part has been answered pretty comprehensively. The split
between / and /home is hurting users and we completely sidestep it
with this change. The change page lists a bunch of other benefits,
incl. better integration with the new resource allocation mechanisms
we have with cgroups2. So in a way this is a follow-up to the
cgroupsv2-by-default change in F31. Snapshots and subvolumes also give
additional powers to systemd-nspawn and other tools. I'd say that the
huge potential of btrfs is clear. It's the possibility of the loss of
stability that is my (and others') worry and the thing which is hard
to gauge.

Zbyszek


If the / and /home split is the main issue, then dm-thin might be an alternative 
solution, and we should check to see if some of the issues listed on the change 
page have been addressed. I'm copying in Jon for additional comment on that. Are 
those btrfs benefits which are listed on the change page in priority order?


File system resize is mentioned there, but pretty much all local filesystems 
support grow. Also, no use cases are listed for that benefit. Shrink is more 
tricky, and can easily result in poor file layouts, particularly if there are 
repeated grow/shrink operations, not to mention potential complications with NFS 
if the fs is exported. So is there some specific use case there that cannot be 

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/29/20 2:23 PM, Eric Sandeen wrote:

On 6/29/20 8:39 AM, Josef Bacik wrote:

On 6/29/20 5:33 AM, Florian Weimer wrote:

* Josef Bacik:


That being said I can make btrfs look really stupid on some workloads.
There's going to be cases where Btrfs isn't awesome.  We still use xfs
for all our storage related tiers (think databases).  Performance is
always going to be workload dependent, and Btrfs has built in overhead
out the gate because of checksumming and the fact that we generate far
more metadata.


Just to be clear here, the choice of XFS here is purely based on
performance, not on the reliability of the file systems, right?
(So it's not “all the really important data is stored in XFS”.)



Yes that's correct.  At our scale everything falls over, including XFS, and as 
I've stated elsewhere in this thread we actually see a higher rate of failure 
(relative to the install size) with XFS.  The databases we use already do all 
of the fancy things that btrfs does in the application.  If we could get away 
with it we'd just use raw disks for those applications. and in fact may do that 
in the future.  Thanks,


Josef, with my XFS hat on, are these recent failures?  Have they
all been reported to the XFS list?

It makes sense to look at reliability in the context of this thread, but
offering "btrfs fails less often than XFS for us" without any context
(what kind of failure, what kernel, when, etc) doesn't help much, it's
just more anecdotes.



Yup this is why I try to avoid talking about other file systems.  This shouldn't 
be interpreted as "XFS drools, btrfs rules!", just that in our own environment, 
btrfs does not fail at any significant rate higher than xfs.


Xfs is used in completely different workloads, and with completely different 
(much better) hardware.


And the reason they haven't been brought up to the list is because it fails at 
such a low rate that I didn't even realize we were having xfs reprovisions until 
I went and looked at the data.  So far of the 15 machines that fell over, 10 of 
them appear to be hardware related.  The other 5 have logs that are in a 
different database that take longer to pull out.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/29/20 12:23 PM, J. Bruce Fields wrote:

Maybe not a desktop question, but do you know btrfs's change
attribute/i_version status?  Does it default to bumping i_version on
each change, or does that still need to be opted in?  And has anyone
measured the performance delta (i_version vs. noi_version) recently?



Yeah it defaults to bumping it all the time, we just use the normal inode 
changing infrastructure so it gets updated the same way everybody else does. 
AFAIK there's no way to opt out of it, unless there's a -o noiversion that 
exists?  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/29/20 5:33 AM, Florian Weimer wrote:

* Josef Bacik:


That being said I can make btrfs look really stupid on some workloads.
There's going to be cases where Btrfs isn't awesome.  We still use xfs
for all our storage related tiers (think databases).  Performance is
always going to be workload dependent, and Btrfs has built in overhead
out the gate because of checksumming and the fact that we generate far
more metadata.


Just to be clear here, the choice of XFS here is purely based on
performance, not on the reliability of the file systems, right?
(So it's not “all the really important data is stored in XFS”.)



Yes that's correct.  At our scale everything falls over, including XFS, and as 
I've stated elsewhere in this thread we actually see a higher rate of failure 
(relative to the install size) with XFS.  The databases we use already do all of 
the fancy things that btrfs does in the application.  If we could get away with 
it we'd just use raw disks for those applications. and in fact may do that in 
the future.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/27/20 4:53 PM, Gerald B. Cox wrote:



On Sat, Jun 27, 2020 at 1:23 PM Chris Murphy > wrote:



The proposal has nothing to do with raid56, let alone by default. The
installer doesn't offer it as an option. And it's not relevant to the
desktop. We're talking about single device btrfs file systems.

Isn't the proposal talking about BTRFS as a default for workstations?  Are you 
saying that Anaconda is just going to check to see if a PC has only one hard 
drive and then install BTRFS there, but if it has two devices use something 
else?  Why would we be installing something by default that has widely known 
broken functionality?  I would think it would be more appropriate to have people 
who specifically want to use BTRFS functionality and are aware and knowledgeable 
of the risks to seek it out rather than have it be some sort of selective 
default.  The target audience you're aiming at by making it the default doesn't 
know FAT from NTFS from EXT4 from XFS from BTRFS or do they care.   Neither are 
they aware or even care about purported benefits.



 > So, BTRFS is great, ready for prime time... many people are using it,
etc. etc. etc. until something goes wrong and then you get... well, it's
experimental and not intended for production.  Sucks to be you.

The raid56 criticism is relevant to raid56 use cases. What you're
doing above is an association fallacy.


Actually no, this doesn't apply to just raid56.  I haven't seen where BTRFS has 
been declared production ready.  It's always been this or that feature is OK, 
another feature is mostly OK, or don't use this feature.  Then when something 
goes wrong, the response is it's still experimental and not intended for 
production workloads.  People then mention Facebook uses it... but my 
understanding is that Facebook is "testing it in production".  They don't RELY 
on it in production.  Why do we want to push something out like that as a default?


Huh?  We absolutely rely on it in production, our whole container system is 
built around it.  It's literally the only thing that works with IO isolation 
with cgroups.  IDK where you're getting your information, but its wrong.  It's 
deployed on millions of machines, most of our infrastructure is built around 
using multiple core features.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/27/20 9:57 AM, Peter Robinson wrote:

I've been very clear from the outset that Facebook's fault tolerance is much
higher than the average Fedora user.  The only reason I've agreed to assist in
answering questions and support this proposal is because I have multi-year data
that shows our failure rates are the same that we see on every other file
system, which is basically the failure rate of the disks themselves.

And I specifically point out the hardware that we use that most closely reflects
the drives that an average Fedora user is going to have.  We of course have a
very wide variety of hardware.  In fact the very first thing we deployed on were
these expensive hardware RAID setups.  Btrfs found bugs in that firmware that
was silently corrupting data.  These corruptions had been corrupting AI test
data for years under XFS, and Btrfs found it in a matter of days because of our
checksumming.

We use all sorts of hardware, and have all sorts of similar stories like this.
I agree that the hardware is going to be much more varied with Fedora users,
and that Facebook has mch higher fault tolerance.  But higher production
failures inside FB means more engineering time spent dealing with those
failures, which translates to lost productivity.  If btrfs was causing us to run
around fixing it all the time then we wouldn't deploy it.  The fact is that it's
not, it's perfectly stable from our perspective.  Thanks,


Thanks for the details, you have any data/information/opinions on non
x86 architectures such as aarch64/armv7/ppc64le all of which have
supported desktops too?



I can't speak to ppc* at all, and I'm not sure how much I can talk about our arm 
stuff, but it was tested and used in production on arm a few years ago.  But 
obviously the bulk of our workload is x86.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/27/20 2:57 AM, Nicolas Mailhot via devel wrote:

Le vendredi 26 juin 2020 à 12:30 -0400, Josef Bacik a écrit :

On 6/26/20 11:15 AM, Matthew Miller wrote:

On Fri, Jun 26, 2020 at 11:13:39AM -0400, Josef Bacik wrote:

Not Fedora land, but Facebook installs it on all of our root
devices, so millions of machines.  We've done this for 5 years.
It's worked out very well. Thanks,


Josef, I'd love to hear your comments on any differences between
that
situation and the typical laptop-user case for Fedora desktop
systems.
Anything we should consider?



We buy worse hardware than a typical laptop user uses, at least for
our hard drives.


The difference between an operation like Facebook and the Fedora user
base, it that Facebook will have a huge fleet of crap hardware, with
the support teams to baby-sit the crap hardware, and attention to
reducing the variety of crap hardware to limit the support matrix
breadth, while Fedora has to deal with a huge support matrix breadth,
without the support teams and the support team tooling to baby-sit
hardware. (Besides Facebook designs the levels of crapiness they allow
in their hardware, meaning they know exactly where they are pushing
limits to lower hardware costs).

And, it’s not always the crap hardware that hits bugs. Sometimes
expensive gamer hardware will fail first because its manufacturer has
pushed the limits to eke some performance points over the competition.

Therefore, using btrfs in Fedora, is inherently more ambitious, than
using it at Facebook.



I've been very clear from the outset that Facebook's fault tolerance is much 
higher than the average Fedora user.  The only reason I've agreed to assist in 
answering questions and support this proposal is because I have multi-year data 
that shows our failure rates are the same that we see on every other file 
system, which is basically the failure rate of the disks themselves.


And I specifically point out the hardware that we use that most closely reflects 
the drives that an average Fedora user is going to have.  We of course have a 
very wide variety of hardware.  In fact the very first thing we deployed on were 
these expensive hardware RAID setups.  Btrfs found bugs in that firmware that 
was silently corrupting data.  These corruptions had been corrupting AI test 
data for years under XFS, and Btrfs found it in a matter of days because of our 
checksumming.


We use all sorts of hardware, and have all sorts of similar stories like this. 
I agree that the hardware is going to be much more varied with Fedora users, 
and that Facebook has mch higher fault tolerance.  But higher production 
failures inside FB means more engineering time spent dealing with those 
failures, which translates to lost productivity.  If btrfs was causing us to run 
around fixing it all the time then we wouldn't deploy it.  The fact is that it's 
not, it's perfectly stable from our perspective.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/27/20 4:35 AM, Roberto Ragusa wrote:

On 2020-06-26 22:13, Justin Forbes wrote:


Saying production on millions of systems is a bit misleading here,
when you are talking about millions of systems at a single company.


...in a redundant configuration where losing a disk is tolerated by design
and managing data that have very low vale (mostly pictures of cats
and random chats).



Huh?  I can assure you that we care very much about our users data, and do not 
lose "cat pictures" randomly and call it a day.  If you are going to make 
technical arguments then I'm happy to talk about actual issues, but insulting 
the hard work we put into maintaining a very high quality production environment 
is not helpful or relevant.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/26/20 5:44 PM, Matthew Miller wrote:

On Fri, Jun 26, 2020 at 03:22:07PM -0400, Josef Bacik wrote:

I described this case to the working group last week, because it hit
us in production this winter.  Somebody screwed up and suddenly
pushed 2 extra copies of the whole website to everybody's VM.  The
website is mostly metadata, because of the inline extents, so it
exhausted everybody's metadata space.  Tens of thousands of machines
affected.  Of those machines I had to hand boot and run balance on
~20 of them to get them back.  The rest could run balance from the
automation and recover cleanly.


Is there a way to mitigate this by reserving space or setting quotas? Users
running out of space on their laptops because:

* they downloaded a lot of media
* they created huge vms
* some sort of horrible log thing gone awry

are pretty common in both a) my anecdotal experience helping people
professionally and personally and b) um, me.



There's a difference between data ENOSPC and metadata ENOSPC.  And again, this 
is a pretty specific failure case.  Obviously it's not impossible to hit, but 
it's not something that's going to be a common occurrence.  The two times we've 
hit these issues in production was the thing that I mentioned, which had 750gib 
fs's completely full with 20gib of metadata completely filled up.


The second was a bad service that was spewing empty files onto the disk slowly 
filling up the metadata chunks, coupled with a bug in how we allocated data and 
metadata chunks.  The chunk allocation thing has been fixed for a year or two 
now.  This isn't something a normal user is going to hit most of the time.  It 
obviously does happen, I'm aware of it, and I've made progress on making it less 
likely to get you into a "call Josef" situation.  I'm sure there's still more 
work to be done, but there's continual progress on this particular edgecase. 
Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/26/20 2:58 PM, James Szinger wrote:

On Fri, 26 Jun 2020 12:30:02 -0500
Chris Adams  wrote:

So... I freely admit I have not looked closely at btrfs in some time,
so I could be out of date (and my apologies if so).  One issue that I
have seen mentioned as an issue within the last week is still the
problem of running out of space when it still looks like there's
space free.  I didn't read the responses, so not sure of the
resolution, but I remember that being a "thing" with btrfs.  Is that
still the case?  What are the causes, and if so, how can we keep from
getting a lot of the same question on mailing lists/forums/etc.?


Yes, it happened to me last week.  The workstation has been upgraded
since F25 and is now at F31.  A yum update last week ran a restorecon
-r / which filled up the filesystem and RAM and swap.  The 460 GB
filesystem had about 140GB of real data, 100 GB of data bloat from
underfull blocks, and the rest (200GB) was metadata.  I had to boot
from a live USB and run btrfs balance to free up the bloat.  I expect
to reformat it to ext4 when the quarantine is over.

This is my last BTRFS filesystem.  One was on a laptop hard disk that
was painfully slow, especially when compared with it's ext4 twin
sitting next to it.  It was reformatted to ext4.  I also had a BTRFS
RAID 0 hard disk array.  It was also slow and also ended up needing
rescue.  I converted it over to xfs on MD raid and it's been faster
and perfectly reliable ever since.

While I like subvolumes and snapshots, I find the maintenance,
reliability, and performance overhead to be not worth it.

Not recommended.



Generally speaking btrfs performance has been the same if not better for our 
workloads.  This is millions of boxes with thousands of different workloads and 
performance requirements.


That being said I can make btrfs look really stupid on some workloads.  There's 
going to be cases where Btrfs isn't awesome.  We still use xfs for all our 
storage related tiers (think databases).  Performance is always going to be 
workload dependent, and Btrfs has built in overhead out the gate because of 
checksumming and the fact that we generate far more metadata.


As for your ENOSPC issue, I've made improvements on that area.  I see this in 
production as well, I have monitoring in place to deal with the machine before 
it gets to this point.  That being said if you run the box out of metadata space 
things get tricky to fix.  I've been working my way down the list of issues in 
this area for years, this last go around of patches I sent were in these corner 
cases.


I described this case to the working group last week, because it hit us in 
production this winter.  Somebody screwed up and suddenly pushed 2 extra copies 
of the whole website to everybody's VM.  The website is mostly metadata, because 
of the inline extents, so it exhausted everybody's metadata space.  Tens of 
thousands of machines affected.  Of those machines I had to hand boot and run 
balance on ~20 of them to get them back.  The rest could run balance from the 
automation and recover cleanly.


It's a shit user experience, and its a shitty corner case that still needs work. 
 It's a top priority of mine.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/26/20 12:43 PM, Matthew Miller wrote:

On Fri, Jun 26, 2020 at 12:30:35PM -0400, Josef Bacik wrote:

Obviously the Facebook scale, recoverability, and workload is going
to be drastically different from a random Fedora user.  But hardware
wise we are pretty close, at least on the disk side.  Thanks,


Thanks. I guess it's really recoverability I'm most concerned with. I expect
that if one of these nodes has a metadata corruption that results in an
unbootable system, that's really no big deal in the big scheme of things.
It's a bigger deal to home users. :)



Sure, I've answered this a few different times with various members of the 
working group committee (or whatever they're called nowadays).  I'll copy and 
paste what I said to them.  The context is "what do we do with bad drives that 
blow up at the wrong time".


Now as for what does the average Fedora user do?  I've also addressed that a 
bunch over the last few weeks, but instead of pasting like 9 emails I'll just 
summarize.


The UX of a completely fucked fs sucks, irregardless of the file system. 
Systemd currently (but will soon apparently) does not handle booting with a read 
only file system, which is essentially what you get when you have critical 
metadata corrupted.  You are dumped to a emergency shell, and then you have to 
know what to do from there.


With ext4/xfs, you mount read only or you run fsck.  With Btrfs you can do that 
too, but then there's like a whole level of other options depending on how bad 
the disk is.  I've written a lot of tools over the years (which are in 
btrfs-progs) to recover various levels of broken file systems.  To the point 
that you can pretty drastically mess up a FS and I'll still be able to pull data 
from the disk.


But, again, the UX for this _sucks_.  You have to know first of all that you 
should try mounting read only, and then you have to get something plugged into 
the box and copy it over.  And then assume the worst, you can't mount read only. 
 Now with ext4/xfs that's it, you are done.  With btrfs you are just getting 
started.  You have several built in mount options for recovering different 
failures, all read only.  But you have to know that they are there and how to 
use them.


These things are easily addressed with documentation, but that's only so good. 
This sort of scenario needs to be baked into Fedora itself, because it's the 
same problem no matter which file system you use.  Thanks,


Josef


Email elaborating my comments about btrfs's sensitivity to bad hardware and how 
we test.


---

The fact is I can make any file system unmountable with the right corruption.
The only difference with btrfs is that our metadata is completely dynamic, while
xfs and ext4 are less so.  So they're overwriting the same blocks over and over
again, and there is simply less of "important" metadata for the file system to
function.

The "problem" that btrfs has is it's main strength, it does COW.  That means our
important metadata is constantly being re-written to different segments of the
disk.  So if you have a bad disk, you are much more likely to get unlucky and
end up with some core piece of metadata getting corrupted, and thus resulting in
a file system that cannot be mounted read/write.

Now you are much more likely to hit this in a data segment, because generally
speaking there's more data writes than metadata writes.  The thing I brought up
in the meeting last week was a potential downside for sure, but not something
that will be a common occurrence.  I just checked the fleet for this week and
we've had to reprovision 20 machines out of 138 machines that threw crc errors,
out of N total machines with btrfs fs'es, which is in the millions.  In the same
time period I have 15 xfs boxes that needed to be reprovisioned because of
metadata corruption, out of <100k machines that have xfs.  I don't have data on
ext4 because it doesn't exist in our fleet anymore.

As for testing, there are 8 tests in xfstests that utilize my dm-log-writes
target.  These tests mount the file system, do a random workload, and then
replay the workload one write at a time to validate the file system isn't left
in some intermediate broken state.  This simulates the case of weird things
happening but in a much more concrete and repeatable manner.

There's 65 tests that utilize dm-flakey, which randomly corrupts or drops
writes, and again these are to test different scenarios that have given us
issues in the past.  There's more of these because up until a few years ago this
was our only mechanism for testing this class of failures.  I wrote
dm-log-writes to bring some determinism to our testing.

All of our file systems in linux are extremely thoroughly tested for a variety
of power fail cases.  The only area that btrfs is more likely to screw up is in
the case of bad hardware, and again we're not talking like huge percentage
points difference.  It's a trade off.  Y

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/26/20 11:15 AM, Matthew Miller wrote:

On Fri, Jun 26, 2020 at 11:13:39AM -0400, Josef Bacik wrote:

Not Fedora land, but Facebook installs it on all of our root
devices, so millions of machines.  We've done this for 5 years.
It's worked out very well. Thanks,


Josef, I'd love to hear your comments on any differences between that
situation and the typical laptop-user case for Fedora desktop systems.
Anything we should consider?



We buy worse hardware than a typical laptop user uses, at least for our hard 
drives.  Also we hit our disks harder than most typical Fedora users.  Consider 
the web tier for example, we push the entire website to every box in the web 
tier (measured in hundreds of thousands of machines) probably 6-10 times a day. 
This is roughly 40 gib of data, getting written to these truly terrible consumer 
grade flash drives (along with some spinning rust), 6-10 times a day.  In 
addition to the normal sort of logging, package updates, etc that happen.


Also keep in mind we pay really close attention to burn rates for our drives, 
because obviously at our scale it translates to millions of dollars.  Btrfs has 
improved our burn rates with the compression, as the write amplification goes 
drastically down, thus extending the life of the drives.


Obviously the Facebook scale, recoverability, and workload is going to be 
drastically different from a random Fedora user.  But hardware wise we are 
pretty close, at least on the disk side.  Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/26/20 11:04 AM, Solomon Peachy wrote:

On Fri, Jun 26, 2020 at 10:42:25AM -0400, Ben Cotton wrote:

For laptop and workstation installs of Fedora, we want to provide file
system features to users in a transparent fashion. We want to add new
features, while reducing the amount of expertise needed to deal with
situations like [https://pagure.io/fedora-workstation/issue/152
running out of disk space.] Btrfs is well adapted to this role by
design philosophy, let's make it the default.


So... can btrfs now be trusted to not crap itself?


The change is based on the installer's custom partitioning Btrfs
preset. It's been well tested for 7 years.


What does "Well tested" mean, in this context?  Do we have data that
shows roughly how many installs were done in Fedora-land, and how long
they lasted?


Not Fedora land, but Facebook installs it on all of our root devices, so 
millions of machines.  We've done this for 5 years.  It's worked out very well. 
Thanks,


Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: PSA: Do not run 'dnf update' inside GNOME, KDE or any other graphical desktop on Fedora 24

[Josef Bacik]

On Wed, Oct 12, 2016 at 8:35 PM, Gerald B. Cox  wrote:
>
> On Wed, Oct 12, 2016 at 3:29 PM, Chris Murphy 
> wrote:
>>
>> On Wed, Oct 12, 2016 at 6:29 AM, Gerald B. Cox  wrote:
>> > On Tue, Oct 11, 2016 at 8:52 PM, Chris Murphy 
>> > wrote:
>> >>
>> >>
>> >> About the rewrite comment: that did not come from a developer, and is
>> >> definitely overstated. In any case, rewrites are not inherently bad
>> >> news, there's a bunch of OpenZFS videos from last yearss summit in
>> >> which the developers talk about various things being completely
>> >> rewritten from scratch, some things more than twice. So kinda par for
>> >> the course, and given enough time things get rewritten anyway. XFS has
>> >> had substantial changes over its history including numerous on disk
>> >> format changes even before it found its way onto Linux.
>> >
>> >
>> > Could be, should be, may be... that's fine - but it all says the same
>> > thing... they
>> > don't know how much time it is going to take to fix - and who knows what
>> > their
>> > priority is to get around to it.  The advantages over what already is
>> > available
>> > don't appear to be that compelling, especially when weighed with the
>> > risks.
>>
>> So you are saying that you started using raid56 when it was brand new,
>> before it had *any* kind of persistent repairing or device replacement
>> and only now, due to a bug that manifests remarkably less bad than the
>> normal behavior of everything else (minus ZFS), are you now
>> complaining? So basically this is, "I want it now!" complaint. Because
>> all the available information has been saying don't use raid56 for
>> production, but you did so anyway. This is a subjective change in your
>> evaluation. It has nothing to do with the state of Btrfs so you really
>> shouldn't blame it when your requirements have clearly changed.
>
>
> Well no, what I said was that I thought that they might have a somewhat
> stable product
> after three years... instead I get that a rewrite may be required with no
> idea of when it would ever
> be production ready.  If you think that is acceptable, more power to you.  I
> do not and I would
> venture to guess that many reasonable people would think three years would
> be ample time.
> My requirements haven't changed... I just think the time to fish or cut bait
> has come.  I've cut
> bait.
>
>>
>>
>> >
>> > When all this started I did some searches and found Kent Overstreet's
>> > page
>> > on
>
>
>>
>> > bcachefs:  https://goo.gl/U0UFfN
>>
>>
>> >
>> > He had some words about the different filesystems - and had this to say
>> > about btrfs:
>> >
>> > btrfs, which was supposed to be Linux's next generation COW filesystem -
>> > Linux's answer to zfs. Unfortunately, too much code was written too
>> > quickly
>> > without focusing on getting the core design correct first, and now it
>> > has
>> > too many design mistakes baked into the on disk format and an enormous,
>> > messy codebase - bigger that xfs. It's taken far too long to stabilize
>> > as
>> > well - poisoning the well for future filesystems because too many people
>> > were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs
>> > multiple times and had to switch at the last minute, and server vendors
>> > who
>> > years ago hoped to one day roll out btrfs are now quietly migrating to
>> > xfs
>> > instead).
>>
>> I have heard from a couple developers that it was a victim of its own
>> hype/success and too many feature additions without equivalent effort
>> on error reporting, debugging, and fault injection tools. I have yet
>> to hear a Btrfs developer say the core design or on disk format has
>> anything to do with the problems, but to the contrary. The comment
>> it's bigger than XFS is kinda funny, seeing as it does more than XFS,
>> md, and LVM combined, so a proper comparison would be comparing all of
>> them to Btrfs minus their user space tools (for sure Btrfs tools do
>> not come anywhere near the metric ass tonne of switches or
>> documentation in LVM or mdadm).
>
>
> I can't speak for him, but don't believe that was his point.  I read it as
> that the
> code was bloated.
>>
>>
>> The Fedora report is simply nonsense. Fedora made no meaningful
>> attempt to switch to Btrfs once, let alone multiple times. FESCo
>> considered and approved it, with conditions attached.
>
>
> Well, we're getting into semantics here.  I would call that a serious
> attempt - and there
> are many articles that are available that talk about Fedora discussing
> making BTRFS
> a default.  If you don't consider any of those attempts "meaningful" I
> suppose that's
> a good thing.
>
>>
>> Josef kept
>> pushing it off because he thought it wasn't ready.
>
>
> Smart man, he was right.
>
>>
>> And then Josef
>> moved on from Red Hat, and wasn't replaced. Characterizing this as
>> "tried to switch" and "had to switch at the last minute" is at best
>> 

Re: Fedora IPv6 testing and improvements - request for ideas

[Josef Bacik]

So one thing I would suggest is testing ipv6 only environments.  At
Facebook we are running into and fixing a whole host of problems with
NetworkManager, Anaconda, Dracut, etc. because they don't handle ipv6
only very well.  It seems that having ipv4 enabled allows things to
work well enough that nobody notices problems with ipv6.  Thanks,

Josef

On Tue, Nov 3, 2015 at 12:50 PM, Moez Roy  wrote:
> Hi Pavel Simerda,
>
> The IPv6 updates are breaking stuff (and probably increasing the
> attack surface):
>
> Bug 1231946 - unbound-anchor ignores net.ipv6.conf.all.disable_ipv6=1
> in /etc/sysctl.conf
> https://bugzilla.redhat.com/show_bug.cgi?id=1231946
>
> Bug 1251762 - dnssec-triggerd ignores net.ipv6.conf.all.disable_ipv6=1
> in /etc/sysctl.conf
> https://bugzilla.redhat.com/show_bug.cgi?id=1251762
>
> (maybe other software like avahi also don't remember right now)
>
> You can reproduce this by putting "ipv6.disable=1" in the kernel command line.
>
> Doing 'setsebool -P domain_kernel_load_modules 1' would reduce the
> security provided by SELinux so it is not an option.
>
> Would appreciate fixes please. Thanks.
> --
> devel mailing list
> devel@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: dnf is completly broken

[Josef Bacik]

On Sun, Sep 27, 2015 at 11:43 AM, Reindl Harald  wrote:
>
>
> Am 27.09.2015 um 13:57 schrieb Neal Gompa:
>>
>> On Sun, Sep 27, 2015 at 7:38 AM, Reindl Harald wrote:
>>
>> Am 27.09.2015 um 11:27 schrieb Richard W.M. Jones:
>>
>> This is quite tiresome.  dnf clearly isn't "completely broken".
>> It
>> may have a bug, but the correct place to put that is in Bugzilla.
>>
>>
>> a package manager which pretends "nothing to do" after rm -rf
>> /var/cache/dnf/* while there are two fresh builds is by definition
>> broken
>>
>> My question to you is... why are you doing "rm -rf /var/cache/dnf/*"?
>> Why not just do "sudo dnf  --refresh"? That forces DNF to
>> actually look at everything again. If your goal is to clean everything
>> out, then "sudo dnf clean all" would do the trick too (which also worked
>> in the yum days)
>
>
> how often do you ask that question again?
>
> a empty "/var/cache/yum|dnf" is a by definition and unconditional empty
> cache - why should i trust a software obviously not working with the basic
> commands right in case of other ones?
>
> and BTW we are not a Ubuntu - what's up with all that "sudo" stuff - if i am
> root then i am root, that's it
>

This is unnecessarily combative and derogatory to a fellow
distribution, why are you going out of your way to be mean spirited?
Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: dnf is completly broken

[Josef Bacik]

There is a bugzilla, ranting on a mailing list every time you hit a bug
makes you an unlikeable person who doesn't get taken seriously.  Thanks,

Josef
On Sep 26, 2015 10:21 PM, "Reindl Harald"  wrote:

> besides https://bugzilla.redhat.com/show_bug.cgi?id=1263888 and the
> ridicolous size of /var/cache/dnf/ in genereal it's unacceptable that "dnf
> -v update dbmail-3.1.17-3.fc22.20150927.rh.3.1.17.x86_64.rpm
> dbmail-manpages-3.1.17-3.fc22.20150927.rh.3.1.17.x86_64.rpm
> dbmail-tools-3.1.17-3.fc22.20150927.rh.3.1.17.x86_64.rpm
> gmime-2.6.20-7.fc22.20150927.rh.x86_64.rpm" *directly* in the repofolder
> works (after repeat the command because the bug above) while DNF pretends
> "nothing to do"
>
> and YES, "createrepo" was called corerctly by the script filling it for
> many years now and a "rm -rf /var/cache/dnf/*" as well as the fact that
> it's a own repo without mirrors leaves not much questions
>
> so how many update do we randomly miss now?
>
> Sep 22 14:53:55 INFO Upgraded: python-dnf-plugins-core-0.1.12-1.fc22.noarch
> Sep 22 14:53:55 INFO Upgraded: python-dnf-plugins-core-0.1.12-1.fc22.noarch
> Sep 22 14:53:55 INFO Upgraded: dnf-plugins-core-0.1.12-1.fc22.noarch
> Sep 22 14:53:55 INFO Upgraded: dnf-plugins-core-0.1.12-1.fc22.noarch
> Sep 22 14:54:00 INFO Cleanup: dnf-plugins-core-0.1.11-1.fc22.noarch
> Sep 22 14:54:00 INFO Cleanup: dnf-plugins-core-0.1.11-1.fc22.noarch
> Sep 22 14:54:00 INFO Cleanup: python-dnf-plugins-core-0.1.11-1.fc22.noarch
> Sep 22 14:54:00 INFO Cleanup: python-dnf-plugins-core-0.1.11-1.fc22.noarch
>
>
> rhsoft-fedora: using metadata from Sun Sep 27 04:08:01 2015.
> rpmfusion-free-updates-testing: using metadata from Fri Apr 10 19:47:11
> 2015.
> rpmfusion-nonfree: using metadata from Wed Aug  5 09:07:52 2015.
> Last metadata expiration check performed 0:02:36 ago on Sun Sep 27
> 04:12:39 2015.
> --> Starting dependency resolution
> --> Finished dependency resolution
> Dependencies resolved.
> Nothing to do.
> Completion plugin: Generating completion cache...
> Complete!
>
>
>
> --
> devel mailing list
> devel@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: dnf is completly broken

[Josef Bacik]

Being tired doesn't justify acting like my 5 year old. Thanks,

Josef

On September 27, 2015, at 1:15 PM, Reindl Harald <h.rei...@thelounge.net> wrote:



Am 27.09.2015 um 19:08 schrieb Josef Bacik:
> This is unnecessarily combative and derogatory to a fellow
> distribution, why are you going out of your way to be mean spirited?

because i am tired of regressions over regressinos from replacements of 
replacements tyring to sell me improvements which aren't ones?

that may work for new users facing "new" things in the next releases as 
new and imporved but that won't work for people using Fedora for many 
years and just say "no, that's not a improvement, it's just a iteration 
and bringing back things which worked perfectly fine before it got 
replaced and improved"

and yes: i know it's not only Fedora specific, it's the result of 
throwing away working things left and right in the hope making all from 
scratch and get rid of the dirty parts of the past makes things better 
until realizing that dirty parts where the one making a difference and 
catching all the bordercases

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Btrfs as default filesystem for Fedora 23?

[Josef Bacik]

On Tue, Jun 23, 2015 at 12:24 PM, Neal Gompa ngomp...@gmail.com wrote:
 Hey all,

 Over the last few months (since Fedora 22 beta's release), I've been using
 Btrfs as my daily driver filesystem across a multitude of machines. After
 Fedora 22 released, I tried it with RAID 5 and RAID 6 enabled on a few
 machines with fantastic success (there aren't even any scary warnings about
 being experimental anymore!).

 Admittedly, my tests have been specific to my needs (media center storage,
 workstations, laptops with SSDs, etc.), but it appears to work really well
 now.

 Also, with kernel 4.1 imported into rawhide, we've now got performance
 improvements for large (20TB) filesystems (though it's been plenty fast for
 my 48TB array).

 As I recall, Josef Bacik mentioned that he'd be pushing for Btrfs becoming
 the default in Fedora 23. At this point, I'm personally convinced that it is
 certainly ready and doable for F23.

 Perhaps other guys with more experience on this stuff could chime in with
 feedback/information/etc, but it feels like we should start the process to
 get everything ready for Btrfs being default in Fedora 23.

 The question now is: as a distribution, where are we on this? The tools seem
 to work, the filesystem appears stable, and I've not been able to cause the
 filesystem to corrupt itself with any kind of user error or cause it to keel
 over. So, what's left?


Sorry I completely missed this conversation.  I'm not interested in
pushing btrfs into Fedora now.  There is nobody to support it if
things go wrong.  If you want to use btrfs you can, or you can use
Suse.  We're finding and fixing things in our internal testing at
Facebook, and the power fail testing stuff I added early this year has
given me a lot of confidence in our ability to not lose all of your
data due to some weird bug.  In a few months we'll have switched over
lots of our boxes onto btrfs so that will give us a pretty good way to
keep track of stability in a production environment.  After that I
imagine it'll be good to go for Fedora, but that'll be somebody else's
decision, I'm no longer super interested in driving anything in
Fedora.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: /boot on Btrfs still not supported, main problem is anaconda and grubby

[Josef Bacik]

On Mar 17, 2015 3:30 PM, Chris Murphy li...@colorremedies.com wrote:

 What's it going to take to fix this? Ubuntu supports it, openSUSE
 supports it, GRUB 2 has supported it for many years now.

 This is a 2.5 year old bug, with patches to fix the problem for ~9
 months, which have been tested and work
 https://bugzilla.redhat.com/show_bug.cgi?id=864198

 The short version of that bug (and the next one) is that on live
 installs, Anaconda copies the kernel with rsync, excluding the install
 media's initramfs, then calls grub2-mkconfig. The resulting grub.cfg
 lacks an initrd line for the primary entry, because there's no
 initramfs yet. This happens on all file systems.

 Next, the installer execute new-kernel-pkg, which creates the
 initramfs, and executes grubby which updates the grub.cfg to include
 the initrd line. Except on Btrfs.

 Every now and then the installer starts permitting /boot on Btrfs,
 which then causes blockers like this:
 https://bugzilla.redhat.com/show_bug.cgi?id=1200539

 But due to the dependency on grubby, anaconda folks have fixed such
 blockers by disallowing /boot on Btrfs.

 This email basically constitutes the harassment approach to getting a
 really old bug fixed. But it's like, everything that should be done
 has been done, and yet we're stuck in the mud on this while other
 distros have totally bypassed this because they don't depend on
 grubby, and instead use upstream tools for generating grub.cfg.

 Now what?


Why are we even using grubby? I was under the impression that the grub2
tools did all of the same work, so why keep maintaining it? Seems silly
something this easy to fix still hasn't been fixed. Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: /boot on Btrfs still not supported, main problem is anaconda and grubby

[Josef Bacik]

On Mar 20, 2015 8:15 AM, Peter Robinson pbrobin...@gmail.com wrote:

 On Fri, Mar 20, 2015 at 11:47 AM, Josef Bacik jo...@toxicpanda.com
wrote:
  On Mar 17, 2015 3:30 PM, Chris Murphy li...@colorremedies.com wrote:
 
  What's it going to take to fix this? Ubuntu supports it, openSUSE
  supports it, GRUB 2 has supported it for many years now.
 
  This is a 2.5 year old bug, with patches to fix the problem for ~9
  months, which have been tested and work
  https://bugzilla.redhat.com/show_bug.cgi?id=864198
 
  The short version of that bug (and the next one) is that on live
  installs, Anaconda copies the kernel with rsync, excluding the install
  media's initramfs, then calls grub2-mkconfig. The resulting grub.cfg
  lacks an initrd line for the primary entry, because there's no
  initramfs yet. This happens on all file systems.
 
  Next, the installer execute new-kernel-pkg, which creates the
  initramfs, and executes grubby which updates the grub.cfg to include
  the initrd line. Except on Btrfs.
 
  Every now and then the installer starts permitting /boot on Btrfs,
  which then causes blockers like this:
  https://bugzilla.redhat.com/show_bug.cgi?id=1200539
 
  But due to the dependency on grubby, anaconda folks have fixed such
  blockers by disallowing /boot on Btrfs.
 
  This email basically constitutes the harassment approach to getting a
  really old bug fixed. But it's like, everything that should be done
  has been done, and yet we're stuck in the mud on this while other
  distros have totally bypassed this because they don't depend on
  grubby, and instead use upstream tools for generating grub.cfg.
 
  Now what?
 
 
  Why are we even using grubby? I was under the impression that the grub2
  tools did all of the same work, so why keep maintaining it? Seems silly
  something this easy to fix still hasn't been fixed. Thanks,

 Because there's still usecases and architectures that don't use grub2?

What usescases?  And what architectures don't have boot tools that handle
their own configuration? Specifics would be helpful so I have a scope of
what I need to fix.  AFAIK yaboot can handle its own config stuff, not sure
what ARM uses. If they really need this help then by all means let's
continue using grubby for those arch's and fix the scripts where we use
grub2 to simply use grub2's tooling.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: /boot on Btrfs still not supported, main problem is anaconda and grubby

[Josef Bacik]

On Fri, Mar 20, 2015 at 9:08 AM, Peter Robinson pbrobin...@gmail.com wrote:
  What's it going to take to fix this? Ubuntu supports it, openSUSE
  supports it, GRUB 2 has supported it for many years now.
 
  This is a 2.5 year old bug, with patches to fix the problem for ~9
  months, which have been tested and work
  https://bugzilla.redhat.com/show_bug.cgi?id=864198
 
  The short version of that bug (and the next one) is that on live
  installs, Anaconda copies the kernel with rsync, excluding the install
  media's initramfs, then calls grub2-mkconfig. The resulting grub.cfg
  lacks an initrd line for the primary entry, because there's no
  initramfs yet. This happens on all file systems.
 
  Next, the installer execute new-kernel-pkg, which creates the
  initramfs, and executes grubby which updates the grub.cfg to include
  the initrd line. Except on Btrfs.
 
  Every now and then the installer starts permitting /boot on Btrfs,
  which then causes blockers like this:
  https://bugzilla.redhat.com/show_bug.cgi?id=1200539
 
  But due to the dependency on grubby, anaconda folks have fixed such
  blockers by disallowing /boot on Btrfs.
 
  This email basically constitutes the harassment approach to getting a
  really old bug fixed. But it's like, everything that should be done
  has been done, and yet we're stuck in the mud on this while other
  distros have totally bypassed this because they don't depend on
  grubby, and instead use upstream tools for generating grub.cfg.
 
  Now what?
 
 
  Why are we even using grubby? I was under the impression that the grub2
  tools did all of the same work, so why keep maintaining it? Seems silly
  something this easy to fix still hasn't been fixed. Thanks,

 Because there's still usecases and architectures that don't use grub2?

 What usescases?  And what architectures don't have boot tools that handle
 their own configuration? Specifics would be helpful so I have a scope of
 what I need to fix.  AFAIK yaboot can handle its own config stuff, not sure
 what ARM uses. If they really need this help then by all means let's
 continue using grubby for those arch's and fix the scripts where we use
 grub2 to simply use grub2's tooling.  Thanks,

 Well there's u-boot on ARMv7 and probably on aarch64 (uEFI is for the
 server spec, not necessarily other form factors), what about those
 users interested in using gummiboot instead of grub2?

 How would you handle non grub tools with grub2-tools, ultimately there
 will need to be some glue between the two, although it's possible that
 it might be a tool other than grubby or a refactored grubby (I think
 pjones has plans here).


Cool so then we use grubby for these other cases and use the grub2
stuff for the grub2 case which covers the majority of installs and
lets us use btrfs for /boot.  Then as new features are added to grub2
for btrfs we don't have to worry about some other package being
updated, we just automatically get them when we update the package.
I'll take a look at what needs to be done when I get back from
vacation.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: /boot on Btrfs still not supported, main problem is anaconda and grubby

[Josef Bacik]

On Fri, Mar 20, 2015 at 4:27 PM, drago01 drag...@gmail.com wrote:
 On Fri, Mar 20, 2015 at 9:18 PM, Chris Murphy li...@colorremedies.com wrote:
 On Fri, Mar 20, 2015 at 1:51 PM, drago01 drag...@gmail.com wrote:
 On Fri, Mar 20, 2015 at 8:36 PM, Chris Murphy li...@colorremedies.com 
 wrote:
 On Fri, Mar 20, 2015 at 11:58 AM, drago01 drag...@gmail.com wrote:
 That is not the only issue.
 If grub itself gets updated and you run grub2-mkconfig the generated
 config file might cause issues with the grub that is actually
 installed on disk (we do not reinstall grub on updates).

 On UEFI it is.

 OK, so what we do with old BIOS machines or ones for whatever reason using 
 CSM?

 Like you said. The grub package might get updated, but the installed
 bootloader (the core.img) remains the same. On UEFI, a new grub2-efi
 package means the grubx64.efi in EFI/fedora is replaced.

 Just use grubby for those?

 I'm not quite following the question. Grubby always gets used in any
 case. And the same Btrfs subvolume bug happens on either UEFI or BIOS.

 You (and Josef) seem to be proposing to switch to grub2-mkconfig
 instead of grubby.
 I am just pointed one one case where this doesn't work hence the
 question how this case is supposed to be handled in your new world
 order.

This is what I wanted to hear, reasons why we use grubby.  So it seems
like we can't just use the grub utilities and need something else, so
perhaps the something else should be something we can actually get
patches into?  Like whatever debian and suse use?  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: /boot on Btrfs still not supported, main problem is anaconda and grubby

[Josef Bacik]

On Fri, Mar 20, 2015 at 1:22 PM, Chris Murphy li...@colorremedies.com wrote:
 On Fri, Mar 20, 2015 at 7:20 AM, Josef Bacik jo...@toxicpanda.com wrote:
 Cool so then we use grubby for these other cases and use the grub2
 stuff for the grub2 case which covers the majority of installs and
 lets us use btrfs for /boot.  Then as new features are added to grub2
 for btrfs we don't have to worry about some other package being
 updated, we just automatically get them when we update the package.
 I'll take a look at what needs to be done when I get back from
 vacation.  Thanks,

 Using grub2-mkconfig instead of grubby on x86 was suggested and
 rejected primarily on the basis that grub-mkconfig discards the
 existing grub.cfg, which can contain user add/modified entries, and
 generates a completely new one. Ergo, grubby and GRUB upstreams
 fundamentally disagree on the fairly basic principle of whether the
 grub.cfg should be modifiable outside of GRUB tools or regenerated
 from scratch each time a kernel version or boot parameter change
 happens.

 And that's why Gene Czarcinski wrote patches for grubby, and AdamW did
 a lot of testing of it.
 https://bugzilla.redhat.com/show_bug.cgi?id=1094489

 This is the most recent grubby patch
 https://bugzilla.redhat.com/attachment.cgi?id=907052


 I don't know if the grubby failure to fix missing initrd lines
 https://bugzilla.redhat.com/show_bug.cgi?id=1200539

 is related to the overall grubby failure to understand Btrfs
 subvolumes (that Gene's patch addresses)
 https://bugzilla.redhat.com/show_bug.cgi?id=864198


Seems like it's easier to just ditch grubby than patch it, if it takes
years to merge patches to provide simple functionality that other
distro's already have then we just need to drop it and go with some
other solution.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Josef Bacik]

On Tue, Oct 7, 2014 at 2:05 PM, Matthias Clasen mcla...@redhat.com wrote:
 On Tue, 2014-10-07 at 13:24 -0400, Josh Boyer wrote:
 On Tue, Oct 7, 2014 at 1:19 PM, Gerald B. Cox gb...@bzb.us wrote:
  Thanks James... I am aware of all the warnings.  They might as well put up 
  a
  skull  crossbones.  I have all my data backed up twice.  But this is my
  point... you don't say toxic and then simultaneously talk about proposing 
  it
  as the default file system on Fedora.

 Right... no single person is saying both things.  We don't have
 split-personality disorder here.  Someone started discussing it as
 default, and a bunch of other people chimed in that it wasn't ready.
 Until those concerns are dealt with, it's not really even a candidate
 for default consideration.

 I think the point is somewhat valid though. To just keep repeating the
 mantra 'its not ready' is not going to make it any more ready. If suse
 can identify a stable subset of btrfs features and use it as their
 default file system with those restrictions, why can't we do the same ?
 The approach makes sense to me, at least...


Because they still have the support staff for when users don't listen,
Fedora doesn't.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Josef Bacik]

On Oct 6, 2014 8:29 AM, Gene Czarcinski gczarcin...@gmail.com wrote:

 On 10/05/2014 08:25 PM, Josh Boyer wrote:

 On Sun, Oct 5, 2014 at 3:03 PM, Gene Czarcinski gczarcin...@gmail.com
wrote:

 On 10/05/2014 09:50 AM, Josef Bacik wrote:

 On Oct 2, 2014 11:32 PM, Andre Robatino robat...@fedoraproject.org
 wrote:

 openSUSE 13.2, scheduled for release in November, will have btrfs as
the
 default filesystem. What are the chances that F22 will follow suit,
 assuming
 openSUSE has no major problems with it?

 https://news.opensuse.org/2014/09/22/

 My plan is to push for F23, I'm still wrapping up some balance bugs and
some
 other issues we've found at work and then this will be my next priority.
 Suse benefits from having a narrow supported criteria, like only use
it
 with lots of space and don't use any of the RAID stuff, plus they have
two
 kernel guys on it and Dave Sterba who is now in charge of btrfs-progs.
 Fedora unfortunately has me who has Facebook work to do and Eric who is
a
 professional fs juggler. We will get there, and when we do it will be
less
 painful than its going to be for Suse since they will have fixed it all
for
 us ;).  Thanks,


 F23 sounds about right to me.  I am very much a fan of BTRFS and
currently
 use it on all of my systems with few problems but I think that F22 is a
bit
 early to make it the default.

 However, I do believe that there a couple of things that need to happen
to
 make thjings easier/better:

 1.  The Fedora developers/maintainers need to take BTRFS more seriously
and
 address btrfs related problems seriously and quickly.  Yes, I know that
 BTRFS swap draining is a little bit difficult to deal with when you are
up
 to you rear end in alligators but, a little more attension please.

 Nope.  See, we deal with what we think we can deal with and what is
 impacting the most people.  There are two of us.  A non-default
 filesystem with a lot of known issues isn't high on the priority list.
 If you would like to see more attention on btrfs, find some people
 that share your interest to triage and work on the bugs (which are all
 upstream bugs) or chip in yourself.

 Nope, I understand.  However, what I said is with the understanding to
change to using BTRFS as the default ... whenever it takes place and I
stand by my statement of needing more attention as a requirement.


 2.  Currently anaconda supports supports installation of /boot on BTRFS
 either as a simple directory on a BTRFS volume (yes, I don't understand
why
 someone would do this but ...), as a simple directory on a rootfs (/)
 subvolume, or as a spepate subvolume.  Grub2.02 also supports this and
 grubby will support if real soon now when pjones can get enough time
to
 examine and integrate my grubby patches adding BTRFS support.

 This isn't a requirement for btrfs by default.  It's a nice to have.

 Using btrfs by default needs more users to want to use it.  Making it
easier for more users to install into btrfs will see increased use.  It
will also demonstrate that Fedora development/maintenance is paying
attention to BTRFS.

 Now, there is another question which has not been voiced: what is the
plan for filessystems in Fedora (and by implication RHEL)?  Is it BTRFS?
Or, perhaps is it LVM with XFS?  IIRC, some time ago it was stated that the
plan was to move to BTRFS.  It is not clear to me that everyone is onboard
with that decision.  Or, perhaps that decision is being reconsidered.


 Anyway, we are getting close.

 If it's getting close, it's entirely because of Josef's and the SuSE
 team's efforts.  They should be applauded.

 Yes, they should!  But Fedora moving to increased use of BTRFS seems to
be receding at the speed of time: it is always a couple of releases from
now; go away I am busy with other stuff.  Sorry is this bothers some people
but that is the impression I get.

Well that's exactly what it is, go away I'm busy with other stuff :).  The
fact is I'm the only one who can drive btrfs as the default filesystem
feature in Fedora, and since I've left Red Hat that has become much less of
an priority for me.  But my other stuff is still mostly related to btrfs,
so its not like this has just been abandoned, the focus has just shifted
and I no longer feel like we need to be using btrfs as the default fs in
Fedora to have a successful project, so it got moved down the priority
list.  It will happen, and when it happens it will be relatively painless
because Suse will have worked out a lot of the distro esque kinks and us at
Facebook will have worked out a lot of the at scale kinks.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Josef Bacik]

On Mon, Oct 6, 2014 at 9:50 AM, Ric Wheeler rwhee...@redhat.com wrote:
 On 10/06/2014 08:54 AM, Josef Bacik wrote:


 Well that's exactly what it is, go away I'm busy with other stuff :).  The
 fact is I'm the only one who can drive btrfs as the default filesystem
 feature in Fedora, and since I've left Red Hat that has become much less of
 an priority for me.  But my other stuff is still mostly related to btrfs,
 so its not like this has just been abandoned, the focus has just shifted and
 I no longer feel like we need to be using btrfs as the default fs in Fedora
 to have a successful project, so it got moved down the priority list.  It
 will happen, and when it happens it will be relatively painless because Suse
 will have worked out a lot of the distro esque kinks and us at Facebook will
 have worked out a lot of the at scale kinks.  Thanks,

 Josef


 I think that the out of space issues are not that different from any file
 system on write enabled snapshots - it certainly can be mysterious and
 confuse users, but that is something that we have to deal with in order to
 get this kind of sophistication into end users hands (documentation? better
 tooling like the snapper tool, etc?).

 One of the harder challenges I think for btrfs is still getting the repair
 tools rock solid - how is our track record these days with repairing btrfs
 after bad things happen :) ?


Funny you should ask, I just added a bunch of functionality last week!
 So right now our fsck fixes anything wrong in the extent tree, which
is where a majority of the problems happen.  The other side of course
is the fs tree which is infinitely easier to deal with, but I usually
wait for a user with a broken fs to fix before adding the ability to
fix it into fsck so I'm sure we have a good testcase to work against.
Every fix I add to fsck gets a test image added to btrfs-progs to make
sure we're never regressing.

Obviously we aren't in xfs/e2fsprogs territory, but it'll fix 90% of
the problems and then the other 10% are just a matter of having an
example to work off of.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Josef Bacik]

On Mon, Oct 6, 2014 at 11:52 AM, Eric Sandeen sand...@redhat.com wrote:
 On 10/6/14 9:26 AM, Josef Bacik wrote:
 Obviously we aren't in xfs/e2fsprogs territory, but it'll fix 90% of
 the problems and then the other 10% are just a matter of having an
 example to work off of.  Thanks,

 Josef

 Josef, just as a datapoint: after corrupting 32k random bytes on a 2G
 image lightly populated and made with default mkfs options, then running
 fsck with all of your recent fixes, I got 9 mount failures out of 20
 attempts, 55% success.

 Running the same test, but w/ 2 devices, each randomly damaged to
 the same extent, and created with -m raid1 -d raid0, I get
 10 failures out of 20, 50% success.

 (Note that this is just a low-bar will it mount test, I'm
 not looking at what's in the repaired filesystem at all).

 What sort of testing yielded your 90% success rate?


The 90% is just off the top of my head, I used to be doing lots of
fsck work for users with corrupted fs, I do that a lot less now, so it
seems like we're making progress.  I have never done any fsfuzzer
testing, I'll put that on the list.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Josef Bacik]

On Mon, Oct 6, 2014 at 2:19 PM, Ric Wheeler rwhee...@redhat.com wrote:
 On 10/06/2014 10:26 AM, Josef Bacik wrote:

 On Mon, Oct 6, 2014 at 9:50 AM, Ric Wheeler rwhee...@redhat.com wrote:

 On 10/06/2014 08:54 AM, Josef Bacik wrote:


 Well that's exactly what it is, go away I'm busy with other stuff :).
 The
 fact is I'm the only one who can drive btrfs as the default filesystem
 feature in Fedora, and since I've left Red Hat that has become much less
 of
 an priority for me.  But my other stuff is still mostly related to
 btrfs,
 so its not like this has just been abandoned, the focus has just shifted
 and
 I no longer feel like we need to be using btrfs as the default fs in
 Fedora
 to have a successful project, so it got moved down the priority list.
 It
 will happen, and when it happens it will be relatively painless because
 Suse
 will have worked out a lot of the distro esque kinks and us at Facebook
 will
 have worked out a lot of the at scale kinks.  Thanks,

 Josef

 I think that the out of space issues are not that different from any file
 system on write enabled snapshots - it certainly can be mysterious and
 confuse users, but that is something that we have to deal with in order
 to
 get this kind of sophistication into end users hands (documentation?
 better
 tooling like the snapper tool, etc?).

 One of the harder challenges I think for btrfs is still getting the
 repair
 tools rock solid - how is our track record these days with repairing
 btrfs
 after bad things happen :) ?

 Funny you should ask, I just added a bunch of functionality last week!
   So right now our fsck fixes anything wrong in the extent tree, which
 is where a majority of the problems happen.  The other side of course
 is the fs tree which is infinitely easier to deal with, but I usually
 wait for a user with a broken fs to fix before adding the ability to
 fix it into fsck so I'm sure we have a good testcase to work against.
 Every fix I add to fsck gets a test image added to btrfs-progs to make
 sure we're never regressing.

 Obviously we aren't in xfs/e2fsprogs territory, but it'll fix 90% of
 the problems and then the other 10% are just a matter of having an
 example to work off of.  Thanks,

 Josef


 One of the things that the GFS2 people have done really well in helping
 their repair tools is to keep an anonymized (file names randomized, data
 blocks skipped) set of corrupt file system metadata layouts around to use to
 verify the tools.  Every time they hit a really nasty file system, they try
 to get this kind of dump so they can validate the tools...

 Adding in Bob who does most of this :)


Yup we have something similar, btrfs-image will pull all the metadata
off of the fs and compress it and then I can replay it to have
something to reproduce on.  We have a sanitize option that will
sanitize filenames and such if the user cares about that.  I'm pretty
happy with fsck at this point and it is really easy for me to fix it
to fix whatever new corruption we've found, its everything else that's
making me go prematurely bald (well ok maybe Liam is to blame for most
of that too.)  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Josef Bacik]

On Oct 2, 2014 11:32 PM, Andre Robatino robat...@fedoraproject.org
wrote:

 openSUSE 13.2, scheduled for release in November, will have btrfs as the
 default filesystem. What are the chances that F22 will follow suit,
assuming
 openSUSE has no major problems with it?

 https://news.opensuse.org/2014/09/22/


My plan is to push for F23, I'm still wrapping up some balance bugs and
some other issues we've found at work and then this will be my next
priority.  Suse benefits from having a narrow supported criteria, like
only use it with lots of space and don't use any of the RAID stuff, plus
they have two kernel guys on it and Dave Sterba who is now in charge of
btrfs-progs.  Fedora unfortunately has me who has Facebook work to do and
Eric who is a professional fs juggler. We will get there, and when we do it
will be less painful than its going to be for Suse since they will have
fixed it all for us ;).  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: blivet-gui announcement

[Josef Bacik]

On Sep 5, 2014 5:05 AM, Vratislav Podzimek vpodz...@redhat.com wrote:

 Good news, everyone! We (me and CC'd Vojtech Trefny) would like to
 introduce you the next generation tool for storage management -- the
 **blivet-gui** tool [1]_. It is a GUI tool based on the blivet python
 library (originally Anaconda's storage management and configuration
 tool) inspired by GParted and other storage management tools. Why not
 use GParted you ask? The reason we came up with blivet-gui is that
 none of the existing storage management tools supports all storage
 technologies supported in modern Linux distributions. Anaconda does
 support them all so it's only logical to take Anaconda's storage
 backend, combine it with a nice, intuitive and in general
 user-friendly frontend and build a standalone application for storage
 management.

 The GUI of blivet-gui is heavily based on GParted's UI to minimize the
 surprise which is very important in such a critical task as storage
 management is. If you know how to work with GParted, you'll almost
 instantly know how to work with blivet-gui. All requested changes are
 just enqueued first and then processed/committed to disk by the
 Apply button just like in GParted. However, with blivet-gui those
 actions may be something like the following sequence:

 create 10GB partition sda1, create a LUKS device on top of it and use
 the LUKS device as a PV for a VG called ``dataVG`` with a single LV
 ``data`` occupying half of the VG space and with XFS on top of the LV,

 not only partitioning and file system operations like GParted does.

 Having troubles writing partitioning part of a kickstart file for
 automated installation? Run blivet-gui with the ``--kickstart`` option
 and export the partitioning portion of a kickstart file instead of
 committing changes to disk.

 On top of the features described above, the blivet-gui is embedable so
 any application using any toolkit with the XEmbed protocol support
 (Gtk, Qt,...) may use blivet-gui as a part of its GUI.

 The application only started its history few months ago, is under
 heavy development now and will get new features in the next months
 (RAID, BTRFS), but even now it is a very nice and useful tool.

 Suggestions, feature requests, bug reports and of course PATCHES ARE
 WELCOME! It is quite a simple Gtk application written in Python which
 makes it an easy target for everybody who misses something in the
 other storage management tools. Don't like Gtk? Text mode would be
 really, really useful too! Don't feel like adding new features and
 diving deep in blivet to implement them? The code always needs
 refactorization and cleanup!

 I think everybody can submit patches for blivet-gui and I'm really
 looking forward to see the hundreds of patches from all those people
 that hate every storage management GUIs and tools that have every
 existed. Let's spend some time on pushing the Linux storage management
 further instead of just complaining!

 .. [1] http://blog.vojtechtrefny.cz/blivet-gui

 P.S. Do you know about any other mailing lists or individuals that may
 be interested in this announcement? Feel free to forward the message and
 spread the word!


This looks excellent, thank you.  I look forward to the BTRFS integration,
let me know if you have questions on that front.  Super excited to see
something non-gnome specific.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Transferring ownership of Ceph

[Josef Bacik]

Hello,

Red Hat is going to take over maintainership of Ceph so I'm orphaning
it and Kaleb Keithley is taking over it.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Josef Bacik]

On Mar 3, 2014 7:34 AM, Stephen Gallagher sgall...@redhat.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 03/01/2014 06:38 PM, Chris Murphy wrote:
 
  On Mar 1, 2014, at 3:58 PM, Reindl Harald h.rei...@thelounge.net
  wrote:
 
 
 
  Am 01.03.2014 22:55, schrieb poma:
  On 27.02.2014 01:33, Josef Bacik wrote:
 
  Just popping in here to say that btrfs is not ready to be
  default in Fedora yet.  Optional is fine but not default.
  Thanks,
 
  This is actually a good news. Thanks.
 
  Now all we need is fair support in the installer. BTRFS as
  alternative scheme: +1 F-Server +1 F-Workstation
 
  one of the BTRFS maintainers explains is is *not* ready and you
  start we need in context of BTRFS? strange logic
 
  Josef said it's not ready to be default. Poma suggested making it
  available as an alternate to whatever the default is, which is
  consistent with how Fedora has been for three releases. His
  suggestion is still fewer permutations than the partition scheme
  outcomes in Fedora 20; and is about the same or on par with Fedora
  18/19, but still one more than oldui.
 


 One of the things that we have been seriously discussing here is that
 non-default options (particularly those known not to be ready) do
 not need or deserve to be presented with the same prominence as other
 options.

 In my opinion, only the default layout should be provided prominently.
 Other choices (such as btrfs) should be available as part of the
 custom layout options. Users should be permitted to install it (and
 without annoying hoops), but they are not entitled to us developing a
 best effort default of a technology we aren't sure they should be
 using, which is essentially what the btrfs drop-down in Fedora 20
 meant.


I'm not saying it isn't ready at all, just not the default.  I and others
still need a way to install on to btrfs if they need to, and frankly it is
good enough for most people to use.  I hope we aren't talking about taking
that option away completely right?  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Josef Bacik]

On Mon, Mar 3, 2014 at 8:59 AM, Stephen Gallagher sgall...@redhat.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 03/03/2014 08:51 AM, Ric Wheeler wrote:
 On 03/03/2014 03:43 PM, Stephen Gallagher wrote:
 So if you were asking me Are we removing btrfs from the install
 options completely?, the answer is a resounding NO. However,
 if you're asking Are we removing btrfs from the drop-down of
 simple-install layouts?, my personal recommendation is yes.

 I disagree - why would we remove the drop down option?

 That would make it exceedingly hard and rare for casual users to
 install and test.  Basically, our Fedora btrfs user base would drop
 to nothing.

 Making it easy to test is a critical part of taking btrfs up to the
 next level of stability!



 It's a matter of user experience, here. By presenting something in the
 guided drop-down, we are effectively asserting that they are of equal
 utility and support. This is *not* the reality.

 Now, if you want to talk about having some sort of click-through for
 I want to try out some experimental options without going all the way
 to customizing my layout manually, that (to me) needs to be a
 different, third path. But listing it directly alongside the default
 gives a false expectation.

 Now, this might be as simple as changing the modern drop-down from
  * EXT4
  * BTRFS
  * XFS
  [] Use LVM

 To something like:
  * XFS-LVM (Recommended)
  * XFS
  * EXT4-LVM
  * EXT4
  * BTRFS (Experimental)

 But even still, Fedora QA is at least ostensibly supposed to test all
 guided paths and best-effort of custom paths. This is more paths than
 are strictly necessary, especially considering that we don't expect
 many people to actually USE the guided paths (in favor of custom
 and/or kickstart).

Ok I was just confused as I haven't done a normal install in a few
releases.  So you can still get to btrfs going through some new
custom layout option but you want to remove the install onto btrfs
easy button in the normal guided option?  I'm ok with this, I just
want to make sure that I/users don't have to jump through
icantbelieveitsnotbtr hoops to install onto btrfs.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Josef Bacik]

On Feb 26, 2014 10:18 AM, Jaroslav Reznik jrez...@redhat.com wrote:

 - Original Message -
  On Wed, Feb 26, 2014 at 9:25 AM, Josh Boyer jwbo...@fedoraproject.org
  wrote:
 
 
 
  Yeah, agreed here. Everyone wants the latest shiniest thing, even if
that
  thing isn't ready. I really don't want to wade through tons of bug
reports
  for btrfs just because it has a lot of hype.
 
  Also, right now cloud is plain old ext4. Let's see if we can ship *all*
of
  the filesystems! It'll be fun!

 Yep, a lot of fun - three different file systems for free different
products.
 And we are back to the question how much these products could differ -
with
 limited resources we have right now - at least short term. Who can answer
it
 - filesystem/kernel guys, if they are able and willing to support all
 potential filesystem, as David stated, it's possible in Anaconda but again
 the same question if the team would be able to maintain more filesystems
 support with high bar in terms of quality (even for example brtfs limited
 to bare minimum), QA... And it could be pretty confusing for users but
that's
 up to us/marketing to explain that products aim specific goal and it's for
 good (if we would be able to support it - then it's for good, if not...).

 Adding devel list to CC - I expect another topic Base should be involved
 too.

 And no, no elections for file system. It's really up to WGs and
coordination
 with the rest teams.


Just popping in here to say that btrfs is not ready to be default in Fedora
yet.  Optional is fine but not default.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Default boot/root filesystem

[Josef Bacik]

On Sat, Sep 7, 2013 at 6:40 PM, Jóhann B. Guðmundsson
johan...@gmail.com wrote:
 On 09/07/2013 10:35 PM, Eric Sandeen wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 9/6/13 5:48 PM, Sam Varshavchik wrote:

 According to this:


 http://www.serverwatch.com/server-news/where-is-red-hat-enterprise-linux-7.html

   RHEL7 will use XFS for the default boot/root.

 I could certainly have been out of town, for a while, and missed
 this. But, to the best of my knowledge, Fedora uses ext4 as the
 default boot/root. Just sounds a bit strange to me, that this is
 getting dumped into RHEL without tossing it into Fedora first, to
 rattle around, and shake any bugs out, beforehand. Don't really have
 an opinion either way, myself, I would just expect that something
 like this would go into Fedora first.

 If people want to switch the Fedora default to XFS, I'll gladly file
 the feature.  :)


 Given that we could not kill lvm as an default and switch to ext4 until
 btrfs was in ready enough shape to improve the desktop experience I doubt
 that feature would pass the acks of the shadow of the storage and filesystem
 monks that appeared from nowhere at that time...

 In a 21 century distro the sub community would decide which default
 filesystem would be their preference to deliverer the best out of the box
 experience for their product but hey rings to rule them all and hobbits too.


I'd like to see this become a reality, that way I don't have to deal
with the bureaucracy that is the fedora feature process to make btrfs
the default, I can just make my own spin for people who want to try it
out and when it becomes stable enough the other spins can integrate it
if they so choose.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: XFS and trim

[Josef Bacik]

On Sat, Mar 30, 2013 at 9:38 PM, Steven Haigh net...@crc.id.au wrote:
 Hi all,

 Firstly, Please CC me into replies as I'm not subscribed to this list.

 I'm trying to confirm that Fedora 18 has enabled trim for XFS filesystems. I
 have added discard to the mount options in /etc/fstab - however I do not see
 it when looking at the output of 'mount':

 UUID=b95a7b45-d197-45c0-a7d6-c13f930a38b3 /   xfs
 defaults,discard 0 1

 /dev/sda2 on / type xfs (rw,relatime,attr2,inode64,noquota)

 fstrim shows that trim is at least enabled, but I am unable to confirm if it
 is done in realtime, or I am required to run fstrim manually.

 # fstrim -v /
 /: 48919015424 bytes were trimmed

 Can anyone share some insight into this?


IIRC you have to rebuild your initrd so it picks up the new fstab
mount options for /, the other options is you should be able to do
mount -o remount,discard /.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Status to make btsfs to the standard filesystem of Fedora

[Josef Bacik]

On Wed, Jan 16, 2013 at 12:00 PM, Rahul Sundaram methe...@gmail.com wrote:




 On Wed, Jan 16, 2013 at 11:41 AM, Jóhann B. Guðmundsson  wrote:


 Afik Josef just left Red Hat not Fedora...


 I haven't seen any recent activity in Fedora from him.  Have you?


Did you see consistent activity in Fedora while I was at Red Hat?  Because
that would be news to me ;)

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Status to make btsfs to the standard filesystem of Fedora

[Josef Bacik]

On Wed, Jan 16, 2013 at 3:53 AM, Jochen Schmitt joc...@herr-schmitt.dewrote:

 Hallo,

 for Fedora 17 we had a feature to make btrfs to the
 standard filesystem of Fedora. This feature was defered
 because the fsck utitlities for btrfs was not available
 on the stable state for Fedora 17.

 So, I would like to ask, if there any plans to make this to
 a feautre for F-19. I think it may be sense to integreate
 this feature to the rewrite of the part of anaconde which
 is responsible for disc partitioning. In a article of the
 c't magazin (a german computer magazin) I could read, that
 this part of the new release of anaconda may be get any
 more love.

 Additionally, because I have read about an issue relating
 btrfs with LVM2 on this mailing list and lost the thread, I
 woould like to ask about the starte of this issue.


I'm waiting until Anaconda settles down before I pursue btrfs in Fedora
again.  Things change too much and Btrfs is too reliant on the anaconda
part working properly to even bother trying to push it through at this
point.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Status to make btsfs to the standard filesystem of Fedora

[Josef Bacik]

On Wed, Jan 16, 2013 at 1:50 PM, Richard W.M. Jones rjo...@redhat.comwrote:

 On Wed, Jan 16, 2013 at 10:12:34AM -0800, Zach Brown wrote:
   So there are a couple of issues with btrfs which I believe absolutely
   must be fixed before it can become the default
 
  I'd agree, though I'd have a different list of pet bugs.
 
  But that's a subjective judgement.  I'd be the first to admit that I'm
  pretty risk averse, especially when it comes to losing data and
  rendering machines unbootable.

 I think both of us are making a subjective judgement.  For myself, I
 want to believe in btrfs, having championed immutable
 state/wandering trees, and real databases for many years.

 BUT I'm deeply unhappy about data corrupting bugs being effectively
 ignored by upstream for months.  That's not good.


I see no data corruption bugs that have been reported that are being
ignored, link to the email?  The invalidate stuff was causing problems (not
a btrfs problem, we just got hurt by it the most), and it looks like those
were cleared up.  I'm working on the only data corruption problem I know of
at the moment and it's not super clear its a data corruption problem.
 Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Anaconda is totally trashing the F18 schedule (was Re: f18: how to install into a LVM partitions (or RAID))

[Josef Bacik]

On Wed, Oct 31, 2012 at 7:54 AM, Jóhann B. Guðmundsson
johan...@gmail.com wrote:
 On 10/31/2012 11:42 AM, Peter Robinson wrote:

 It's already been pushed back once, the first iteration of newui was
 attempted to land in F-17 and was pushed back to F-18 if my memory
 serves me correctly.


 Dont think it did


   So I think we need to land it now and deal with
 the fall out then move on. The one thing that concerns me is the lack
 of communications about LVM with the storage team as it makes me
 wonder what else has been missed/assumed.


 Lack of communication lol those RH storage developers could have.

 A) subscribed to the Anaconda developers list to monitor changes relevant to
 their setup as anyone else affected by any upstream changes ( this got
 mentioned in August )
 B) bothered to do a simple test install of alpha they would have noticed
 that the installer did not default to LVM partition layout by default and
 had that discussion then and there...

 So the Storage team within Red Hat they themselves expecting the Anaconda
 team to be running around notify them or FESCO for that matter is just utter
 and total bullocks and their little lvm no being turned on by default pails
 in comparison with what we ( QA Community )  discovered where missing in
 the installer early on...


You know what the storage team does right?  I can only speak for
myself really, but 26 hours out of the day my head is buried in btrfs.
 Sure I'm subscribed to anaconda devel and fedora devel, which means I
search btrfs in my fedora-devel and anaconda folders once a week to
see if somebody is complaining.  We just had a big get together in
August with the storage developers and anaconda people and I don't
remember hearing anything about this.  The anaconda guys are the same
way, they focus on the installer and don't look up unless they have
to.  So when I need something btrfs-y done in Anaconda I go find Dave
or somebody and tell them what I need and we get it worked out
together.  The same thing should be done from the anaconda side when
it comes to changing the basic behavior of storage in Fedora.  Red Hat
employs the top storage developers in the world, why would they not
take advantage of that expertise and experience?  So no it's not
utter and total bullocks to expect some sort of heads up when it
comes to storage related changes in anaconda, we're all on the same
team and why would you not talk to each other?  We should be working
to create a well integrated solution for our users that provides the
best possible experience, and the only way we get that done is if we
all work together.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 TC1 DVD still no btrfs as install option?

[Josef Bacik]

On Wed, Apr 25, 2012 at 11:07 AM, Tomasz Torcz to...@pipebreaker.pl wrote:
 On Wed, Apr 25, 2012 at 01:34:07PM +0200, Matthias Runge wrote:
 On 25/04/12 09:48, Vít Ondruch wrote:
  +1. I'd like to see this fixed before final.
 Yes, me too. Afaik, SuSE supports btrfs even for their enterprise distro.

  For added laughs, SLES does not support read-write ext4. Ext4 read-only 
 support
 is provided for migrating to btrfs.


That's because Btrfs is way more stable than ext4,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 TC1 DVD still no btrfs as install option?

[Josef Bacik]

On Wed, Apr 25, 2012 at 12:25 PM, Colin Walters walt...@verbum.org wrote:
 On Wed, 2012-04-25 at 12:03 -0400, Josef Bacik wrote:
 On Wed, Apr 25, 2012 at 11:07 AM, Tomasz Torcz to...@pipebreaker.pl wrote:
  On Wed, Apr 25, 2012 at 01:34:07PM +0200, Matthias Runge wrote:
  On 25/04/12 09:48, Vít Ondruch wrote:
   +1. I'd like to see this fixed before final.
  Yes, me too. Afaik, SuSE supports btrfs even for their enterprise distro.
 
   For added laughs, SLES does not support read-write ext4. Ext4 read-only 
  support
  is provided for migrating to btrfs.
 

 That's because Btrfs is way more stable than ext4,

 [citation needed]


https://twitter.com/#!/josefbacik/status/195190540529184768
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 TC1 DVD still no btrfs as install option?

[Josef Bacik]

On Wed, Apr 25, 2012 at 1:22 PM, Chris xchris...@googlemail.com wrote:
 2012/4/25 Josef Bacik jo...@toxicpanda.com:
 On Wed, Apr 25, 2012 at 12:25 PM, Colin Walters walt...@verbum.org wrote:
 On Wed, 2012-04-25 at 12:03 -0400, Josef Bacik wrote:
 On Wed, Apr 25, 2012 at 11:07 AM, Tomasz Torcz to...@pipebreaker.pl 
 wrote:
  On Wed, Apr 25, 2012 at 01:34:07PM +0200, Matthias Runge wrote:
  On 25/04/12 09:48, Vít Ondruch wrote:
   +1. I'd like to see this fixed before final.
  Yes, me too. Afaik, SuSE supports btrfs even for their enterprise 
  distro.
 
   For added laughs, SLES does not support read-write ext4. Ext4 read-only 
  support
  is provided for migrating to btrfs.
 

 That's because Btrfs is way more stable than ext4,

 [citation needed]


 https://twitter.com/#!/josefbacik/status/195190540529184768

 Is this a joke? Btrfs more stable than ext4??? Not really???


According to the stab-o-matic stability testing framework[1] btrfs
scores a 63 whereas ext4 scores a 37, so yes, I'm completely serious,

Josef

[1] credit goes to Sandeen for that one.
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 TC1 DVD still no btrfs as install option?

[Josef Bacik]

On Wed, Apr 25, 2012 at 1:35 PM, Reindl Harald h.rei...@thelounge.net wrote:


 Am 25.04.2012 18:03, schrieb Josef Bacik:
 On Wed, Apr 25, 2012 at 11:07 AM, Tomasz Torcz to...@pipebreaker.pl wrote:
 On Wed, Apr 25, 2012 at 01:34:07PM +0200, Matthias Runge wrote:
 On 25/04/12 09:48, Vít Ondruch wrote:
 +1. I'd like to see this fixed before final.
 Yes, me too. Afaik, SuSE supports btrfs even for their enterprise distro.

  For added laughs, SLES does not support read-write ext4. Ext4 read-only 
 support
 is provided for migrating to btrfs.


 That's because Btrfs is way more stable than ext4,

 oh no, please do not make such jokes

 as long there exists even no fsck and as long there are
 zero production worksloads generic tests are useless

 and what about workloads like virtual machines where AFAIK
 BTRFS is still completly unuseable

 BTRFS has many nice things and will be in the future a good
 FS but call it more stable than ext4 while ext4 is since
 years useable for production is a past-1st-april joke


Wow ok I would have thought after the stab-o-matic thing it would have
been obvious I was kidding, Btrfs still has a little ways to go before
it is as stable as Ext4, though we are getting closer.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Guaranteed way to break btrfs

[Josef Bacik]

There was a problem with 3.3 kernels that we fixed in 3.4, please try
Linus's most recent git tree and see if you have the same problems.
Thanks,

Josef

On Thu, Apr 19, 2012 at 10:23 AM, valent.turko...@gmail.com
valent.turko...@gmail.com wrote:
 Hi,
 I found a perfect way how you can test small file issue in btrfs
 because every time I tried it it breaks btrfs.

 I tried on several partition with over 200+GB of free space, and after
 starting openwrt compile process that creates few hundred thousand
 small files I get out of space error.

 On ext4 partition openwrt doesn't fail and it uses around 4GB of space.
 All test were done on Fedora 16 with Linux 3.3.0 kernel.

 And here are the instructions if you like to try out for yourself:
 http://wiki.openwrt.org/doc/howto/buildroot.exigence
 http://wiki.openwrt.org/doc/howto/build

 After finding out this issue I reformatted my /home and /storage
 partitions to ext4.

 I read on few mailing lists that this is an old and fixed btrfs issue,
 but it looks that (at least in Fedora) it still affect some extreme
 use cases, like building openwrt firmware.

 Cheers,
 Valent.

 --
 follow me - www.twitter.com/valentt  http://kernelreloaded.blog385.com
 linux, anime, spirituality, wireless, scuba, linuxmce smart home, zwave
 ICQ: 2125241, Skype: valent.turkovic, MSN: valent.turko...@hotmail.com
 --
 devel mailing list
 devel@lists.fedoraproject.org
 https://admin.fedoraproject.org/mailman/listinfo/devel
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 btrfsck crash

[Josef Bacik]

Sorry I need to include a patch I sent recently to the list.  Those
errors are bogus, btrfsck just needs to be taught about our special
space cache inodes.  I'll pull that into fedora.  You are right, you
don't need to run btrfsck on boot, just if you run into issues.
Thanks,

Josef

On Thu, Apr 12, 2012 at 6:13 PM, Chris Murphy li...@colorremedies.com wrote:


 On Apr 11, 2012, at 12:00 PM, Josef Bacik wrote:

 Got it all updated, it's in koji, 0.19-18 is what you want.  Thanks,

 Thanks Josef. This is now available with a yum update and has fixed the 
 problem. However, this is what I get as a result.

 I'm under the impression that a btrfsck isn't needed if there are no mount 
 errors. Yet in this case there are no mount errors in dmesg, yet btrfsck says 
 there are errors. Expected, or not? And is this better taken to the btrfs 
 list?


 [root@f17v chris]# btrfsck /dev/sdb4
 checking extents
 checking fs roots
 root 5 inode 18446744073709551604 errors 2000
 root 5 inode 18446744073709551605 errors 1
 found 6472544256 bytes used err is 1
 total csum bytes: 5872152
 total tree bytes: 459329536
 total fs tree bytes: 426516480
 btree space waste bytes: 134110780
 file data blocks allocated: 15564447744
  referenced 625522
 Btrfs Btrfs v0.19


 Chris Murphy
 --
 devel mailing list
 devel@lists.fedoraproject.org
 https://admin.fedoraproject.org/mailman/listinfo/devel
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 btrfsck crash

[Josef Bacik]

Got it all updated, it's in koji, 0.19-18 is what you want.  Thanks,

Josef

On Tue, Apr 10, 2012 at 2:28 PM, Chris Murphy li...@colorremedies.com wrote:
 If it's only updated in rawhide, what will be the state of btrfs-progs in F17?

 On Apr 10, 2012, at 7:10 AM, Josef Bacik wrote:

 Hrm sorry, I need to update btrfs-progs in rawhide, I will try and get
 to that today.  Thanks,

 Josef

 On Mon, Apr 9, 2012 at 2:24 AM, Chris Murphy li...@colorremedies.com wrote:
 I have an inconsequential F16 VM that uses btrfs. The VM went psycho for 
 unknown reasons and I had to force quit. Realizing btrfsck can't fix a 
 problem, I was curious if it would see any problems. So I loaded the 
 F16.vdi as a 2nd SATA disk in an F17 VM. I get the following results:

 [root@f17v chris]# btrfsck /dev/sdb4
 couldn't open because of unsupported option features (8).
 btrfsck: disk-io.c:684: open_ctree_fd: Assertion `!(1)' failed.
 Aborted (core dumped)

 [root@f17v chris]# btrfsck /dev/sdb5
 couldn't open because of unsupported option features (8).
 btrfsck: disk-io.c:684: open_ctree_fd: Assertion `!(1)' failed.
 Aborted (core dumped)

 If this is expected, fine. If it's unexpected, I can provide more 
 information. The F16 VM appears to be working fine, all data is intact, and 
 I get no mount errors in dmesg. But btrfsck crashing is, to me, unexpected.

 --
 devel mailing list
 devel@lists.fedoraproject.org
 https://admin.fedoraproject.org/mailman/listinfo/devel
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: F17 btrfsck crash

[Josef Bacik]

Hrm sorry, I need to update btrfs-progs in rawhide, I will try and get
to that today.  Thanks,

Josef

On Mon, Apr 9, 2012 at 2:24 AM, Chris Murphy li...@colorremedies.com wrote:
 I have an inconsequential F16 VM that uses btrfs. The VM went psycho for 
 unknown reasons and I had to force quit. Realizing btrfsck can't fix a 
 problem, I was curious if it would see any problems. So I loaded the F16.vdi 
 as a 2nd SATA disk in an F17 VM. I get the following results:

 [root@f17v chris]# btrfsck /dev/sdb4
 couldn't open because of unsupported option features (8).
 btrfsck: disk-io.c:684: open_ctree_fd: Assertion `!(1)' failed.
 Aborted (core dumped)

 [root@f17v chris]# btrfsck /dev/sdb5
 couldn't open because of unsupported option features (8).
 btrfsck: disk-io.c:684: open_ctree_fd: Assertion `!(1)' failed.
 Aborted (core dumped)

 If this is expected, fine. If it's unexpected, I can provide more 
 information. The F16 VM appears to be working fine, all data is intact, and I 
 get no mount errors in dmesg. But btrfsck crashing is, to me, unexpected.


 Chris Murphy
 --
 devel mailing list
 devel@lists.fedoraproject.org
 https://admin.fedoraproject.org/mailman/listinfo/devel
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: phoronix benchmarks ext4 vs. btrfs

[Josef Bacik]

On Fri, Mar 9, 2012 at 10:11 AM, David Quigley seli...@davequigley.com wrote:
 On 03/09/2012 08:42, Przemek Klosowski wrote:

 On 03/09/2012 01:43 AM, Adam Williamson wrote:

 On Thu, 2012-03-08 at 22:19 -0700, Chris Murphy wrote:

 I'm not sure how useful 'time' is as a benchmark for file copies.


 Don't file transfers get cached and return to a console as 'complete'
 long before the data is ever written, sometimes?

 I'm pretty sure you sometimes hit the case where you copy 200MB to a USB
 stick, it returns to the console pretty fast, but the light on the stick
 is still flashing, and if you run 'sync', it sits there for quite a
 while before returning to the console, indicating the transfer wasn't
 really complete. So I'm not sure 'time'ing a 'cp' is an accurate test of
 actual final-write-to-device.


 That is true---but in that case, we could flush the disks. and then
 time the operation followed by another flush, i.e.:

 sync; time (cp ...; sync)

 I assume that the old-time Unix superstition of calling sync three
 times no longer applies :)

 Perhaps a dedicated disk benchmark like bonnie++ would be a better
 test, though.



 If you want to look seriously into file-system benchmarking I would suggest
 looking into the work done by the fsbench people at Stony Brook University's
 Filesystem and Storage Lab (FSL). There is a survey paper there for the last
 decade of FS benchmarks and their short commings and what should be
 addressed.


 http://www.fsl.cs.sunysb.edu/project-fsbench.html


fsbench is amazing, I also use fio and fs_mark to test various things.
 But these are artificial workloads!  These numbers don't mean a
damned thing to anybody, the only way you know if a fs is going to
work for you is if you run your application on a couple of fses and
figure out which one is faster for you!  For example if you mostly
compile kernels, btrfs is fastest.  However if you mostly use a fs for
your virt images, don't use btrfs!  It's all a matter of workloads and
no amount of benchmarking is going to be able to tell you if your pet
workload is going to work well at all.

The work that we file system developers do with benchmarking is to
stress particular areas of our respective filesystems.  For example,
with Dave's tests he was testing our ability to scale as the amount of
metadata gets ridiculously huge.  He has exposed real problems that we
are working on fixing.  However these real problems are things that I
imagine 99% of you will never run into, and therefore should not be
what you use to base your decisions on.

So let's try to remember that benchmarks mean next to nothing to real
users, unless watching iozone output happens to be what you use your
computer for.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: btrfs scrub not included in F16?

[Josef Bacik]

On Tue, Feb 7, 2012 at 12:46 PM, Richard Hughes hughsi...@gmail.com wrote:
 On 7 February 2012 17:40, Josh Boyer jwbo...@gmail.com wrote:
 The scrub commands weren't added in btrfs-progs until October of last year.
 The version F16 and rawhide has is just too old to contain that support.

 That worries me a little, seeing how we're pressing on with btrfs by
 default for F17... Surely shipping new snapshots of the btrfs-progs
 package might be a good idea in rawhide?


We're not pressing on with btrfs by default for F17.  At this point
I'm waiting for fsck so I can bring it all up to date at once, which
should be shortly (like today or tomorrow).  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Draft schedule for today's FESCO meeting (6th February 2012)

[Josef Bacik]

On Mon, Feb 6, 2012 at 9:18 AM, Matthew Garrett mj...@srcf.ucam.org wrote:
 Following is the list of topics that will be discussed in the FESCo
 meeting tomorrow at 18:00UTC (1:00pm EST) in #fedora-meeting on
 irc.freenode.net.

 Links to all tickets below can be found at:
 https://fedorahosted.org/fesco/report/9

 = Followups =

 #topic #690     F17 Feature: move all to /usr - 
 https://fedoraproject.org/wiki/Features/UsrMove
 .fesco 690

 #topic #756     F17 Feature: English Typing Booster - 
 https://fedoraproject.org/wiki/Features/english-typing-booster
 .fesco 756

 #topic #796     F17 Feature: Network Zones - 
 https://fedoraproject.org/wiki/Features/network-zones
 .fesco 796

 #topic #797     F17 Feature: firewall-d - default firewall solution -- 
 https://fedoraproject.org/wiki/Features/firewalld-default
 .fesco 797

 = Fedora Engineering Services tickets =

 https://fedorahosted.org/fedora-engineering-services/report/6

 = Open Floor =

 For more complete details, please visit each individual ticket.  The
 report of the agenda items can be found at
 https://fedorahosted.org/fesco/report/9

 If you would like to add something to this agenda, you can reply to
 this e-mail, file a new ticket at https://fedorahosted.org/fesco,
 e-mail me directly, or bring it up at the end of the meeting, during
 the open floor topic. Note that added topics may be deferred until
 the following meeting.


We're running close to the wire on this but it looks like Chris will
have fsck out for btrfs tomorrow, so I'd like to get fesco's opinion
on the viability of moving forward with the Btrfs as default feature,
whether we actually want to make a push for it or leave it for F18.
Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Draft schedule for today's FESCO meeting (6th February 2012)

[Josef Bacik]

On Mon, Feb 6, 2012 at 10:55 AM, Matthew Garrett mj...@srcf.ucam.org wrote:
 On Mon, Feb 06, 2012 at 10:51:37AM -0500, Josef Bacik wrote:

 We're running close to the wire on this but it looks like Chris will
 have fsck out for btrfs tomorrow, so I'd like to get fesco's opinion
 on the viability of moving forward with the Btrfs as default feature,
 whether we actually want to make a push for it or leave it for F18.

 There's been a couple of concerns raised about btrfs's resiliance
 against local attacks (forced hash collisions and the like) - is local
 hardening something that's been on the test plan so far?


We've fixed the recent one and audited everything else, it seems like
we're in pretty good shape.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS on LVM causes long fedora-storage-init run?

[Josef Bacik]

On Sat, Oct 8, 2011 at 5:13 PM, Richard Shaw hobbes1...@gmail.com wrote:
 On Sat, Oct 8, 2011 at 4:01 PM, Richard Shaw hobbes1...@gmail.com wrote:
 On Fri, Oct 7, 2011 at 11:56 AM, Josef Bacik jo...@toxicpanda.com wrote:
 What Tomasz said, and if that doesn't help use bootchart and upload
 the chart somewhere so I can see what's going on.  Thanks,

 Here's my bootchart link before making any modifications:

 http://dl.dropbox.com/u/34775202/boot.png

 Here's after adding the suggested mount options:

 http://dl.dropbox.com/u/34775202/reboot.png


This one still shows a btrfs-cache-# thread, which means it was doing
the old slow method.  This is expected since we'll do the old method
for every block group we don't have our space cache setup for.  So
reboot a few times, let it get all of it's cache setup, and then do
another bootchart so I can see if it's still doing the old caching for
some reason.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS on LVM causes long fedora-storage-init run?

[Josef Bacik]

On Fri, Oct 7, 2011 at 7:20 PM, Michael Cronenworth m...@cchtml.com wrote:
 On 10/07/2011 12:19 PM, Josef Bacik wrote:
 I don't doubt it's btrfs, but bootchart will tell me which one of our
 kernel threads is running so I can tell_what_  in btrfs is taking it's
 sweet time.  Thanks,

 Here you go:
 http://i284.photobucket.com/albums/ll20/daumas/bootchart.png

 It looks like btrfs-endio-met is the time waster.


The picture won't zoom all the way in for me so I can't read it at
all, can you re-upload the full sized image?  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS on LVM causes long fedora-storage-init run?

[Josef Bacik]

On Fri, Oct 7, 2011 at 12:40 PM, Tomasz Torcz to...@pipebreaker.pl wrote:
 On Fri, Oct 07, 2011 at 11:29:45AM -0500, Michael Cronenworth wrote:
 Richard Shaw wrote:
  I rebooted a few times just to make sure it's not a scheduled fsck
  (not that there is a full fsck for BTRFS yet) and the hard drive light
  is on pretty solid the whole time...

 The fsck tool does nothing and will not be the cause of your delay. If
 your fs gets even one bit corrupted you'll lose everything (just FYI).

 I've had long mount times on a btrfs drive (no LVM) for a while now. I'm
 seeing mount take about 7 seconds to complete. I posted this delay to
 the btrfs list and my message didn't receive a single reply.

  Do you have caches enabled and fully built?  Remounting with
 -o space_cache,inode_cache will enable them.  Then wait few minutes
 for caches to be built (I/O will stop when they're ready).  Subsequent
 mounts should be faster.


What Tomasz said, and if that doesn't help use bootchart and upload
the chart somewhere so I can see what's going on.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS on LVM causes long fedora-storage-init run?

[Josef Bacik]

On Fri, Oct 7, 2011 at 1:16 PM, Michael Cronenworth m...@cchtml.com wrote:
 Josef Bacik wrote:
 What Tomasz said, and if that doesn't help use bootchart and upload
 the chart somewhere so I can see what's going on.  Thanks,

 I posted[1] my systemd-analyze results to the list to show as proof that
 it is btrfs.

 Plus my system is silent (root is an SSD, btrfs drive is a HDD) and the
 only thing making noise at boot is the HDD for 7 seconds.


I don't doubt it's btrfs, but bootchart will tell me which one of our
kernel threads is running so I can tell _what_ in btrfs is taking it's
sweet time.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Btrfs status for F16

[Josef Bacik]

Hello,

In order to hopefully (and I understand this is a unrealistically big
hope) stem the amount of hostile comments and random remarks about
Btrfs not being ready for F16 that I get with _every_ bz that get's
filed against it, let me announce this as clearly as possible

BTRFS WILL NOT BE THE DEFAULT FILE SYSTEM FOR F16

Fesco outlined basic requirements that needed to be met by Alpha for
the switch to be allowed to happen and we have not met those
requirements so it won't be happening for F16.

I appreciate those who will continue to use it and report bugs, we are
working very hard on trying to get everything more stable and it is a
slow going process.  With your help we will be in a better situation
for F17.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Btrfs status for F16

[Josef Bacik]

On Mon, Aug 8, 2011 at 9:22 AM, Genes MailLists li...@sapience.com wrote:
 On 08/08/2011 08:55 AM, Josef Bacik wrote:
 On Mon, Aug 8, 2011 at 8:53 AM, Matej Cepl mc...@redhat.com wrote:
 On 8.8.2011 14:44, Josef Bacik wrote:
 I appreciate those who will continue to use it and report bugs, we are
 working very hard on trying to get everything more stable and it is a
 slow going process.  With your help we will be in a better situation
 for F17.  Thanks,


  Sounds good ... can you give us an update and ballpark timeline of
 RAID-5 on btrfs  as well if you don't mind?

It requires the larger than page size blocksize work which is slated
for 3.2, I'm not sure what Chris has in mind specifically but I
imagine that stuff will land in 3.2 to get plenty of testing, and then
the RAID5 stuff will follow.  Or both could land in 3.2, I'm not
entirely sure.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Thu, Jul 14, 2011 at 5:22 AM, Richard W.M. Jones rjo...@redhat.com wrote:
 Good timing!  I have a related BTRFS / Fedora 16 question.

 I have used the btrfs Anaconda option, and I get btrfs appearing as
 a choice in the menus.  However if I just change the root filesystem
 to btrfs, then I would get:

  /boot          /dev/sda2
                 PV
                 VG
                 LV
                 btrfs     swap

 Is this how F16 will be set up?  The Feature page[1] suggests that LVM
 will be turned off by default, in which case it should look more like:

  /boot          /dev/sda2
                 btrfs

Well it should be more like

/boot/dev/sda1
swap/dev/sda2
btrfs

Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Thu, Jul 14, 2011 at 6:08 AM, JB jb.1234a...@gmail.com wrote:
 Ric Wheeler rwheeler at redhat.com writes:

 ...
 I think that it would be really rare to see pristine, academic algorithms
 implemented exactly as a non-coding mathematician designed them in code :)
 ...

 Well, not convinced ... :-)

 The algorithm  has to be taken holisticly - it has been designed, tested,
 fine tuned, optimized, tested again, and then submitted to internal rigor,
 and then to external rigor (e.g. of academia, professional community, etc).

 When an implementer picks it up, she can not interpret it second-handedly.
 She has to take it as a whole. No games !

 The algorithm *must* be coded as designed and *not* have programming coding
 bugs.

 Next, after coding it (even presumably as originally intended), you have
 to submit it to that same rigor of testing as done by the original algorithm
 inventor.
 And make no mistake, you have to do it repeatedly, at every stage of
 development, iteratively.

 Your BTRFS fs's integrity relies on that !

 So, no wobbling, strictly as the doctor prescribed :-)


The only thing we do not strictly adhere to is the static sized
elements in the tree because we inline data in the tree if it is small
enough, which works out awesomely if you have small files because you
only do the read once.  Is this not in keeping with the original
algorithm?  Maybe.  Is it a problem?  More than likely not.  Can you
turn it off?  Yes.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Thu, Jul 14, 2011 at 9:08 AM, Genes MailLists li...@sapience.com wrote:



  I think RAID-5 support would be reasonably important to have too ... I
 dont think we want to have raid on top of btrfs ... right?

  Ric - what is the current status of RAID-5 ?

This requires some other big changes that are disk format changes.  It
seems like the big block support will go into 3.1, so after that it
should be relatively shortly after that, possibly 3.2 or 3.3.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Thu, Jul 14, 2011 at 10:33 AM, Genes MailLists li...@sapience.com wrote:
 On 07/14/2011 10:17 AM, Ric Wheeler wrote:
 On 07/14/2011 02:54 PM, Josef Bacik wrote:
 On Thu, Jul 14, 2011 at 9:08 AM, Genes MailListsli...@sapience.com  wrote:


   I think RAID-5 support would be reasonably important to have too ... I
 dont think we want to have raid on top of btrfs ... right?

   Ric - what is the current status of RAID-5 ?
 This requires some other big changes that are disk format changes.  It
 seems like the big block support will go into 3.1, so after that it
 should be relatively shortly after that, possibly 3.2 or 3.3.  Thanks,

 Josef

 And just to add in here, you can run btrfs on top of MD devices although 
 that is
 not the most popular thing to do currently.

 Ric


  Thanks Josef/Ric -

  Another (Q) - once the format changes, will there be tools to change
 the online format of existing filesystems - or will we need to delete
 and start fresh ?


All format changes happen automatically (usually with a mount option
so as not to surprise users) online so there's nothing that needs to
be done beyond giving the special mount option.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Wed, Jul 13, 2011 at 10:06 PM, Reindl Harald h.rei...@thelounge.net wrote:


 Am 14.07.2011 03:57, schrieb Eric Sandeen:
 bleeeding edge / modern technology is not the same as dangerous defaults
 unstable / unfinsihed packages should never be default in GA nor replace
 existing and over a long time well working things - never!

 You might have said the same thing about ext4 in Fwhatever it was
 and yet, here we are, shipping it as default for many releases now, with
 little trouble.

 ext4 is not written from scratch and it is a hughe difference
 if we are speaking from a improvement of ext2/ext3 with many
 years history on all sorts of setups (desktop, server, notebook)

 the troubles of lost kde settings was enough on crashes and yes
 i had the luck of a system-disk hangig some times exactly as
 this was discussed in a way POSIX do not say what exactly
 happens after that, it says only the FS must be clean and
 not what data are there or not

 virtual machines this time are really a normal thing and if
 there are hughe troubles now a FS should NOT be DEFAULT in a
 few months - why everytime this hurry? jesus one of the biggest
 benefits of open source is that there ar eno marketing idiots
 with release plans which forcing a release without enough testing

 so why do we not need this benefits and fire out permanently
 new technologies in a not understandable hurry?


Blah blah blah blah, that's all I hear out of you.  I've already said
that if it's not in good shape by Alpha the switch won't even be made,
so quit your bitching.

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Thu, Jul 14, 2011 at 11:21 AM, JB jb.1234a...@gmail.com wrote:
 Josef Bacik josef at toxicpanda.com writes:

 ...
 I've already said
 that if it's not in good shape by Alpha the switch won't even be made,
 so quit your bitching.

 Josef

 Josef,
 would it be possible, BEFORE (in case that) you decide to switch on before
 Alpha, to present some test suite results (a la Phoronix, etc) that would
 assure yourselves and Fedora testers/users of BTRFS fitness ?

 If it came from you it would have a special weight and a sign that you
 do no want to sell cats in a bag :-)

 I hope, that the community at large will parallel it with their own tests.


The problem is we test it regularly with lot's of artificial tests.
We all run xfstests for each patch, I have a bunch of ENOSPC tests
that I run and Chris does large stress tests before he sends pull
requests to Linus.  This thing is tested to death.  We are at the
point that it is pretty stable, which is why we're having this
conversation.  We've reached the point where we really need wider user
testing, because no amount of testing we do will ever be able to match
up to the crazy things users do.  For example I don't see these random
high latency issues that have been discussed in this thread with my
testing, so I'm going to have to figure out a way to make that happen
so I can try and fix it.  I would have never seen that in my own
testing.

So far the wider testing I've gotten has been good, basically we just
have a couple of key performance issues, VMs and random latency
spikes.  That's not too bad all things considered.  The VM thing I
think I can have fixed by 3.1, but the latency spikes thing is going
to take some looking into, and if I can't get that nailed down that
will be a big reason not to go with Btrfs as default.

But you have to remember a huge requirement is going to be having a
working fsck by Alpha, and that seems less and less likely every day,
so chances are we won't even make Alpha and I'll have to fill out one
of these annoying feature pages for F17.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Wed, Jul 13, 2011 at 4:53 PM, Manuel Escudero jmlev...@gmail.com wrote:
 Today I'll be switching from BTRFS to Ext4 again because of the troubles
 I've been having with
 the New Linux Filesystem. As BTRFS is going to be the Default in F16 I
 wanted the developers to
 know what kind of troubles I've been experiencing with this FS in F15 so
 they can take a look
 at them in order to have a better F16 release:
 The Good:
 Since BTRFS arrived into my computer (Everything in the HDD is formated with
 BTRFS excluding /boot)
 I've seen a performance improvement in the data transfer part from and to
 the computer (copying files seem to
 be faster than before) But that's all about the good things I noticed...
 The Bad:
 BTRFS has reduced system's overall performance, at this point, sometimes it
 is OK, sometimes it is
 VERY BAD, I've noticed Performance Peaks in F15 with BTRFS and the Boot
 times are not nice: I mean,
 they are not the slowest ones, but they're not as good as Before in F14 with
 Ext4 instead of BTRFS.
 The performance Running/Launching apps has been afected too and now the PC
 freezes sometimes (that never
 happened in F14 unless I forced it a lot with 4 VM's to suck the 4GB of RAM
 I have); And Now it freezes
 very often when it wants without a lot of effort.
 The Ugly:
 Running VM's when having their virtual HDD's stored in a BTRFS partition is
 DEATH!
 They're very slow, sometimes they open, sometimes they not, usually they
 freeze, You can't
 work with them. Same thing about Gnome Shell working over a BTRFS partition:
 it is really slow,
 sometimes it reacts but most of the time is pretty unresponsive.
 Reading in the Web, I found that some users think that the BTRFS poor
 performance is caused by some
 special kind of fragmentation it suffers, others think it's because of it's
 CopyonWrite attributes and some
 others blame other stuff, God Knows! the only thing I know is that BTRFS is
 not ready for being
 used in normal production machines (as I tought) and it needs to be fixed
 before the release of F16, because it's
 performance is really far from good...
 Other Stuff I noticed is that with Kernel 2.6.38.8-35 the system seems to
 work better that with the previous one,
 just a little, but is some kind of improvement.
 Here you have all the info I found on the net about BTRFS Performance
 issues noticed by users:
 https://bugzilla.redhat.com/show_bug.cgi?id=689127
 http://arosenfeld.wordpress.com/2010/12/27/back-to-ext4-from-btrfs/
 http://www.vyatta4people.org/btrfs-is-a-bad-choice-when-running-kvm/
 http://lkml.org/lkml/2010/7/13/475
 http://blog.patshead.com/2011/03/btrfs---six-months-later.html
 I only have a question:
 Why Any Kind of VM is Sooo Slow when being stored on a BTRFS
 partition? Any Way to Solve this? or at least have a BTRFS performance
 improvement?

Yeah VMs are a particular problem with Btrfs.  There are a ton of
reasons for this, for example by default we use fsync.  Fsync _sucks_
for btrfs currently, and it has historically not been a well optimized
piece of code.  I'm working on fixing this, but it requires VFS level
changes that are currently sitting in Al's queue.  I suspect they will
go into 3.1 and so we can move ahead with our work, but for now, it
sucks.  You can use cache=none you get better performance, but still
not that great.  And this is all because of one major thing

Btrfs has threads for _everything_.  This works out fantastically when
you have big chunks of reads or writes you want done.  This _sucks_
when you are doing little piddly io's.  The reason for all of this is
because we don't want you to get bottlenecked on us
calculating/verifying checksums, so we farm all IO and endio out to
different threads, which as I said works out great if you are trying
to cram gigs of data down your drives throat.

But with VMs you are doing small scattered IO's, so the IO comes down,
we prepare it, and farm it off to a thread and wait for that thread to
wake up and submit the io.  Then the io is completed and that is
farmed off to another thread and we wait on that.  This switching
around and waiting for things to wake up is hugely painful when all
you want to do is write a few bytes.  If you were to do

dd if=/dev/zero of=/mnt/btrfs/file bs=4k count=100 oflag=direct

on a btrfs fs and then do it on an ext4 fs, you would see about a 20%
difference between the 2.  But if you do say bs=20M, the gap closes
quite a bit.

I fixed part of this problem for O_DIRECT (which is cache=none with
qemu), if the IO's are small we don't send it off to a thread but
submit it within our threads context, which is what got us with 20% of
ext4 as opposed to 50%.  The other half is doing the completion in the
submitters context, which is going to take some extra work.  I'm
fixing this in the fsync case as well, but as I said we need a VFS
patch to do it properly so that will be a little later coming.  After
that I can do the endio part of it and hopefully get us within
spitting 

Re: BTRFS: The Good, The Bad and The Ugly

[Josef Bacik]

On Wed, Jul 13, 2011 at 5:59 PM, Reindl Harald h.rei...@thelounge.net wrote:


 Am 13.07.2011 23:54, schrieb Michael Cronenworth:
 Farkas Levente wrote:
 if you said that this's the current state of btrfs than it's not ready
 as a default fs for f16. so please postpone it at least to f17.

 If f16 gets kernel 3.1 (or backported stuff into 3.0), IMHO there is no
 reason to slip it one release

 there are many reasons!

 replacing an essential part of the OS as filesystems are with
 finally not well tested piece of new software is simply a
 dangerous game with no benefit

 hopefully stable at release is my definition of untested


That's not the case at all, I'm not sure where you are getting that.
If we don't have a released offline fsck by Alpha, which IIRC is the
beginning of August we're not even going to make the switch.  We
aren't aiming for hopefully stable, we're aiming for actually stable
and reasonably safe.  If we don't meet certain basic requirements no
switch will be made and everything will carry on as normal.

I'm not trying to shove Btrfs down peoples throats.  The last thing I
want is to switch over to Btrfs before it's fully ready for everybody
to be using it, which is why there are a bunch of requirements that
need to be met before the switch is actually met.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plan for tomorrow's FESCo meeting (2011-06-08)

[Josef Bacik]

On Tue, Jun 7, 2011 at 7:59 PM, Kevin Fenzi ke...@scrye.com wrote:
 Following is the list of topics that will be discussed in the FESCo
 meeting tomorrow at 17:30UTC (1:30pm EDT) in #fedora-meeting on
 irc.freenode.net.

 Links to all tickets below can be found at:
 https://fedorahosted.org/fesco/report/9

 = Followups =

 #topic #563 suggested policy: all daemons must set RELRO and PIE flags
 .fesco 563

 #topic #594 F16Feature: F16 BTRFS default file system - 
 http://fedoraproject.org/wiki/Features/F16BtrfsDefaultFs
 .fesco 594


Unfortunately I had something come up this afternoon so I can't make
the meeting for this.  I'll check the logs and such when I get back
this afternoon and respond to any questions and such.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS concerns (was: Re: Plan for tomorrow's FESCo meeting (2011-06-01))

[Josef Bacik]

On Thu, Jun 2, 2011 at 11:46 AM, Richard W.M. Jones rjo...@redhat.com wrote:
 On Wed, Jun 01, 2011 at 04:15:59PM +0100, Peter Robinson wrote:
 I will be unable to attend tomorrow but I have concerns of making btrfs
 default without a well tested fsck. I'm aware one is due soon but I don't
 believe 3-4 months is enough time to test it well enough. On 2.6.38.x I
 still get regular kernel abrt crashes on resume. Is it even marked stable in
 the upstream kernel yet?

 Another concern is whether btrfs is going to work well to store
 virtual machine disk images (ie. to replace LVM for that purpose,
 where LVM is known to work very efficiently).

 Last time I looked -- which I admit was a really long time ago -- it
 behaved fairly pathologically with these huge monolithic files that
 are rewritten in-place.

 (Edit: just noticed this bug:
 https://bugzilla.redhat.com/show_bug.cgi?id=689127 )


These sort of issues are my priority and I've spent the last 2 months
specifically working on the kvm performance differences between ext4
and btrfs.  Now we're not on par with ext4 yet, but we aren't 2-3
times slower any more, maybe at the most we're 20% slower.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plan for tomorrow's FESCo meeting (2011-06-01)

[Josef Bacik]

On Wed, Jun 1, 2011 at 11:15 AM, Peter Robinson pbrobin...@gmail.com wrote:
 I will be unable to attend tomorrow but I have concerns of making btrfs
 default without a well tested fsck. I'm aware one is due soon but I don't
 believe 3-4 months is enough time to test it well enough. On 2.6.38.x I
 still get regular kernel abrt crashes on resume. Is it even marked stable in
 the upstream kernel yet?


Have you filed bugs for these crashes?

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Status of btrfs in rawhide

[Josef Bacik]

On Tue, May 10, 2011 at 5:37 PM, Clyde E. Kunkel
clydekunkel7...@cox.net wrote:
 Hi,

 Is rawhide following current btrfs development releases with kernel
 patches and userland programs?  Or, should we role our own?  I am
 interested in testing only.


btrfs-progs hasn't changed much so you should be fine with whats in
fedora.  If you want to be testing the latest and greatest it's
probably better to either follow Chris's btrfs-unstable.git tree or my
btrfs-work.git tree on kernel.org.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Status of btrfs in rawhide

[Josef Bacik]

On Tue, May 10, 2011 at 6:09 PM, Matej Cepl mc...@redhat.com wrote:
 Dne 10.5.2011 23:49, Josef Bacik napsal(a):
 btrfs-progs hasn't changed much so you should be fine with whats in
 fedora.  If you want to be testing the latest and greatest it's
 probably better to either follow Chris's btrfs-unstable.git tree or my
 btrfs-work.git tree on kernel.org.  Thanks,

 Mandatory question: how is btrfsck doing? Is it working and robust to be
 trusted?


Should be out by the end of May, we'll see how well it's working then.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS vs LVM for VM storage (was: Re: Plans for BTRFS in Fedora)

[Josef Bacik]

On Wed, Mar 2, 2011 at 9:23 AM, Richard W.M. Jones rjo...@redhat.com wrote:
 On Tue, Feb 22, 2011 at 02:51:50PM -0500, Josef Bacik wrote:
 2) Fedora 16 ships without LVM as the volume manager and instead use
 BTRFS's built in volume management, again just for the default.

 Sorry I'm a bit late on this gentle discussion, but I have one
 question about this:

 I use LVM to store virtual machines, one VM per LV, and it's very good
 for that.

 How is BTRFS's performance when used to store VMs (presumably they are
 stored as files)?


Good, but the problem is the default behavior of virt manager is to
use fsync for everything, you have to manually go in and set the
Cache to None so it will use O_DIRECT, and then it's just as fast
as anything else.  Not a big deal if you create everything via the
command line, kind of annoying if you do it via the GUI, tho all you
have to do is say let me edit the options before starting this vm
when you first create it, set the cache type and then do the install
and you are good to go.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Mon, Feb 28, 2011 at 1:10 PM, Jon Masters jonat...@jonmasters.org wrote:
 On Sat, 2011-02-26 at 17:33 -0500, Lyos Gemini Norezel wrote:
 On 02/23/2011 04:37 PM, Kevin Kofler wrote:
 
  And I'd like to counter-counter-propose that we just stop using ANY kind of
  subvolumes or volume management by default and just default to plain old
  partitions. IMHO, LVM causes more problems than it fixes. Sure, you can
  easily add storage from another disk, but in exchange there's no
  straightforward way to resize your partitions, at least none of the common
  partition editors can do it. There's also a performance penalty.

 +1

 This subvolume nonsense has no real place on any home computer/consumer
 device.

 With all due respect, that's the path chosen by certain other Linux
 distributions (ones where if I install them I have to jump through all
 kinds of hoops to turn on LVM). That is not the way we should be going.
 I've made my objections known, added a comment on the wiki discussion
 for the feature, and will raise an objection at the appropriate time
 that it is proposed to drop LVM use by default. Until then, I'm done :)


I don't see any comments on the discussion page.  And I'm not talking
about adding extra hoops, you just have to select the custom partition
scheme.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Thu, Feb 24, 2011 at 8:44 AM, Matej Cepl mc...@redhat.com wrote:
 Dne 23.2.2011 20:49, Matthew Garrett napsal(a):
 btrfs does the former without anywhere near as much of the latter.

 BTRFS so far only makes my kernel panicking as it did anytime I have
 been trying it since Fedora 9 (yes, I am crazy). This is absolutely not
 meant as anything personal against Josef (I know very well how
 incredibly small group are BTRFS developers), but just a bit of
 suspicion, whether we have fsck now (or we will have fsck soon) really
 leads so quickly let's make it default.

 I am quite OK with having crashing and unstable systemd or Gnome 3 (and
 again, nothing against their developers, this is Rawhide and Fedora, so
 when my kids are alive despite me using it I am pretty happy), but
 unstable file system is quite a different matter.

 Could we slow down a bit, please?

Sure but where's the fun in that? :).

Seriously though, when I'm not on RHEL duty most of my time has been
spent on stability vs. adding features, and honestly there aren't a
lot of OMFG this shit is broken complaints, it's more of hey when I
do this thing that is specific to my work load, BTRFS does X wrong.
At this point what is left is the normal day to day users using this
thing and breaking it in ways that we as developers have not been able
to break it using it the way we use it and test it the way we test it.
 I think that if I could get a large base to test for F15 that we
could squash most/all of the problems that crop up from that to be in
great shape for default in F16.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Tue, Feb 22, 2011 at 10:25 PM, Jon Masters jonat...@jonmasters.org wrote:
 On Tue, 2011-02-22 at 14:51 -0500, Josef Bacik wrote:

 2) Fedora 16 ships without LVM as the volume manager and instead use
 BTRFS's built in volume management, again just for the default.

 In my personal opinion, this is a poor design decision. Yes, BTRFS can
 do a lot of volume-y things, and these are growing by the day, but I
 don't want my filesystem replacing a full volume manager and I am
 concerned that this will lead to less testing and exposure to full LVM
 use within the Fedora community. Instead, I'd like to counter-propose
 that everything stay exactly as it is, with users being able to elect to
 switch to BTRFS (sub)volumes if they are interested in doing so.

 Should the switch to BTRFS by default happen, this will be one more
 thing I will have to fix immediately during installation. The list grows
 longer and longer over time - please don't make this change.


Theres no sense in using LVM if BTRFS can do the same job better.  I'm
not suggesting removing LVM altogether, just not using it for the
default pick the layout for me option, you can always do your own
thing with a custom layout.

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Tue, Feb 22, 2011 at 11:57 PM, Bruno Wolff III br...@wolff.to wrote:
 On Tue, Feb 22, 2011 at 14:51:50 -0500,
  Josef Bacik jo...@toxicpanda.com wrote:

 3) All the various little tools that we have for putting together
 LiveCD's that are very ext* centered.  I've not even looked at this
 yet, but I assume it's going to be kind of a pain.

 I like to see live CDs just use squashfs directly and not a squashed copy
 of an ext3/4 image. However this will break using dd to copy the image
 for installs which will slow things down. Further however, there is at least
 some interest in having live CDs and ananconda share the same install method,
 so this might be happening in any case.


This would be a great thing in general since the default ext* image is
shrunk down to be installed which creates a bad fs layout which has
performance implications.  Sure it may be faster for the install, but
overall it hurts our users.  So moving away from this dd an image
thing would be good for everybody.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Wed, Feb 23, 2011 at 8:29 AM, Camilo Mesias cam...@mesias.co.uk wrote:
 Hi

 I wanted to second these questions...

 2011/2/22 Jóhann B. johan...@gmail.com:
 Will there be any performance penalties making this move?
 [...]
 What benefit will this switch bring to the novice desktop end users?

 Will the novice desktop end user ever take advantages of any of the features 
 that btrfs brings?

 Since upgrading (downgrading?) my netbook to use an SSD I went with
 the Anaconda defaults (using LVM etc) and that probably wasn't in my
 best interest - judging from benchmarks at the time of read
 performance in the LVM compared to the underlying device, also from
 the point of not being able to add complex disk arrays to the netbook
 any time soon.

 I think Fedora could do more to support lower end devices *well*, in
 addition to allowing people to use the very latest technology on
 larger (ie. desktop and server) platforms. My impression right now is
 I'd be interested to try BTRFS for the server and maybe larger
 desktops, but would probably want to avoid it for anything smaller.


Your impression is wrong, there has been quite a bit of work done to
make BTRFS work well on small devices, it is the default filesystem
for meego which goes on phones, which is much smaller than anything
you are going to have on your netbook.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Wed, Feb 23, 2011 at 9:18 AM, John Reiser jrei...@bitwagon.com wrote:
 On 02/23/2011 05:07 AM, drago01 wrote:
 Defaults should be chooses on the metric what provides the best
 experience for the users not based on what we have been doing in the
 past (i.e stagnation).

 *One* data corruption constitutes EPIC FAIL.  Btrfs is too young,
 and will be for yet a while longer.


Well if data corruption is the test then we shouldn't be using Ext4
either, there was one fixed as recently as the beginning of this
month.  File systems are software like anything else, there will be
bugs.  Off the top of my head I can think of 3 data corrupters we've
had in 4 years of working on BTRFS, and they've all been hard to hit
and have not to my knowledge been seen by users, only us developers in
testing.  BTRFS is young, but we have to start somewhere.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Wed, Feb 23, 2011 at 10:19 AM, Dennis Jacobfeuerborn
denni...@conversis.de wrote:
 On 02/23/2011 03:27 PM, Josef Bacik wrote:
 On Wed, Feb 23, 2011 at 9:18 AM, John Reiserjrei...@bitwagon.com  wrote:
 On 02/23/2011 05:07 AM, drago01 wrote:
 Defaults should be chooses on the metric what provides the best
 experience for the users not based on what we have been doing in the
 past (i.e stagnation).

 *One* data corruption constitutes EPIC FAIL.  Btrfs is too young,
 and will be for yet a while longer.


 Well if data corruption is the test then we shouldn't be using Ext4
 either, there was one fixed as recently as the beginning of this
 month.  File systems are software like anything else, there will be
 bugs.  Off the top of my head I can think of 3 data corrupters we've
 had in 4 years of working on BTRFS, and they've all been hard to hit
 and have not to my knowledge been seen by users, only us developers in
 testing.  BTRFS is young, but we have to start somewhere.  Thanks,

 I'm actually not that worried about corruption as that is something that
 can be fixed once discovered. What creeps me out about btrfs at the moment
 is this:

 https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21__Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21

 The fact that the FS needs manual rebalance operations and that these can
 take a while (even tough this can be done online) doesn't exactly make
 btrfs the ideal candidate for an end-user desktop system that should pretty
 much be able to look after itself.
 I'm actually quite interested in btrfs especially for servers because of
 it's features but this problem really worries me.


Yes this is one of the more complicated areas of BTRFS and tends to
blow up in our faces a lot.  That being said it's only a big deal if
you tend to run your filesystem close to full a lot, which most people
do not.  It is an area that we work very hard to make sure it's not a
problem, hopefully we have eliminated all of the big problems and you
should really only see ENOSPC when you actually fill up the disk.
Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS on servers (was Re: Plans for BTRFS in Fedora)

[Josef Bacik]

On Wed, Feb 23, 2011 at 2:00 PM, Jonathan Dieter jdie...@lesbg.com wrote:
 On Wed, 2011-02-23 at 16:19 +0100, Dennis Jacobfeuerborn wrote:
 I'm actually quite interested in btrfs especially for servers because
 of it's features

 For what it's worth, we've been running btrfs on our school fileservers
 since September.  After a few teething problems (fixed by
 increasing /proc/sys/vm/min_free_kbytes), we've had pretty much zero
 trouble.


Wait what?  I know we use lots o ram, but you shouldn't have to bump
min_free_kbytes.  What were you seeing?  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: BTRFS on servers (was Re: Plans for BTRFS in Fedora)

[Josef Bacik]

On Wed, Feb 23, 2011 at 2:42 PM, Jonathan Dieter jdie...@lesbg.com wrote:
 On Wed, 2011-02-23 at 14:18 -0500, Josef Bacik wrote:
 On Wed, Feb 23, 2011 at 2:00 PM, Jonathan Dieter jdie...@lesbg.com wrote:
  On Wed, 2011-02-23 at 16:19 +0100, Dennis Jacobfeuerborn wrote:
  I'm actually quite interested in btrfs especially for servers because
  of it's features
 
  For what it's worth, we've been running btrfs on our school fileservers
  since September.  After a few teething problems (fixed by
  increasing /proc/sys/vm/min_free_kbytes), we've had pretty much zero
  trouble.
 

 Wait what?  I know we use lots o ram, but you shouldn't have to bump
 min_free_kbytes.  What were you seeing?  Thanks,

 Sorry, I didn't mean to imply it was btrfs that was causing the problem.
 We got new servers when we did our upgrade and ran into problems with
 the e1000e kernel module being unable to allocate memory even though
 there was loads free.  Bumping up min_free_kbytes fixed it.

 See https://bugzilla.redhat.com/show_bug.cgi?id=626851 for more detail.


Ah ok, good, nevermind then :),

Joesf
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Wed, Feb 23, 2011 at 4:37 PM, Kevin Kofler kevin.kof...@chello.at wrote:
 Jon Masters wrote:
 In my personal opinion, this is a poor design decision. Yes, BTRFS can
 do a lot of volume-y things, and these are growing by the day, but I
 don't want my filesystem replacing a full volume manager and I am
 concerned that this will lead to less testing and exposure to full LVM
 use within the Fedora community. Instead, I'd like to counter-propose
 that everything stay exactly as it is, with users being able to elect to
 switch to BTRFS (sub)volumes if they are interested in doing so.

 And I'd like to counter-counter-propose that we just stop using ANY kind of
 subvolumes or volume management by default and just default to plain old
 partitions. IMHO, LVM causes more problems than it fixes. Sure, you can
 easily add storage from another disk, but in exchange there's no
 straightforward way to resize your partitions, at least none of the common
 partition editors can do it. There's also a performance penalty.


Sorry I should clarify, when I say use Btrfs's volume management stuff
I mean just doing normal partitions and then creating a Btrfs
filesystem and then add disks to the fs as required.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Plans for BTRFS in Fedora

[Josef Bacik]

Hello,

So we're getting close to having a working fsck tool so I wanted to
take the opportunity to talk about the future of BTRFS in Fedora.
Coming up in F15 we're going to have the first release of Fedora where
we don't need the special boot option to have the ability to format
you filesystem as BTRFS.  This is in hopes that we can open it up for
wider testing before possibly making it the default filesystem.  I
realize we're in the early stages of F15, but since filesystems are
big and important I'd like to get an idea of the amount of work that
needs to still be done to get BTRFS in shape for being Fedora's
default filesystem.  So here are my goals

1) Fedora 16 ships with BTRFS as the default root filesystem.
2) Fedora 16 ships without LVM as the volume manager and instead use
BTRFS's built in volume management, again just for the default.

Fedora 16 is a very aggressive target, which is also why I'm bringing
it up now.  I think we will be ready by then.  We have had the ability
to install Fedora onto BTRFS since F11, and I have been testing it
since then without too many issues.  Some things that I know are going
to be gotcha's at this point

1) GRUB support.  Edward Shishkin did GRUB1 patches for BTRFS a while
ago, but they were obviously never merged upstream and were also not
included into fedora.  These would either need to be cleaned up and
put into our grub package, or we'd need to put /boot on a different
filesystem.  I personally hate the idea of having a non-btrfs /boot
partition but I'm not the one in charge of GRUB.

2) Anaconda support.  I've already talked with Will Woods about this
some.  Really anaconda will format a normal disk with BTRFS with no
problem today, the biggest issue here is adding the volume management
stuff and allowing users to create subvolumes via anaconda.

3) All the various little tools that we have for putting together
LiveCD's that are very ext* centered.  I've not even looked at this
yet, but I assume it's going to be kind of a pain.

4) FSCK does actually need to be finished.  I don't see this as a
problem as Chris is 90% done with it at this point so it should be
ready even before F15 ships, but it's worth mentioning.

I would really like to see a lot of testing on BTRFS in the F15 cycle
just so we know how well it works in the non-developer's use case.
It's easy for me to use it because I work on it and I understand the
limitations, so it would be nice to have much broader testing.

So what are your thoughts?  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Tue, Feb 22, 2011 at 3:00 PM, Kevin Fenzi ke...@scrye.com wrote:
 On Tue, 22 Feb 2011 14:51:50 -0500
 Josef Bacik jo...@toxicpanda.com wrote:

 Hello,

 So we're getting close to having a working fsck tool so I wanted to
 take the opportunity to talk about the future of BTRFS in Fedora.
 ...snip...

 1) GRUB support.  Edward Shishkin did GRUB1 patches for BTRFS a while
 ago, but they were obviously never merged upstream and were also not
 included into fedora.  These would either need to be cleaned up and
 put into our grub package, or we'd need to put /boot on a different
 filesystem.  I personally hate the idea of having a non-btrfs /boot
 partition but I'm not the one in charge of GRUB.

 Perhaps if we are going to the pain of getting btrfs patches working
 and stable, we could just look at moving to grub2?

Well I don't think cleaning up the existing patches will be that big
of a deal, it's mostly a matter of testing.  The problem with GRUB2 is
it's GPLv3, explicitly to be a giant pain in the ass for porting any
new fs to GRUB since we're all GPLv2 only.  Aren't open source
licenses grand?


 ...snip...

 So what are your thoughts?  Thanks,

 Sounds good. I'd suggest filing a feature page already and getting it
 setup with all the status, etc. That can be a central point for
 coordinating that effort.


Sounds good, I'll do that, thank you,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

On Tue, Feb 22, 2011 at 5:07 PM, Lennart Poettering
mzerq...@0pointer.de wrote:
 On Tue, 22.02.11 14:51, Josef Bacik (jo...@toxicpanda.com) wrote:

 Hello,

 So we're getting close to having a working fsck tool so I wanted to
 take the opportunity to talk about the future of BTRFS in Fedora.
 Coming up in F15 we're going to have the first release of Fedora where
 we don't need the special boot option to have the ability to format
 you filesystem as BTRFS.  This is in hopes that we can open it up for
 wider testing before possibly making it the default filesystem.  I
 realize we're in the early stages of F15, but since filesystems are
 big and important I'd like to get an idea of the amount of work that
 needs to still be done to get BTRFS in shape for being Fedora's
 default filesystem.  So here are my goals

 1) Fedora 16 ships with BTRFS as the default root filesystem.

 Hmm, what are your plans regarding hierarchy layout for this?

 My hope is that one day we can ship a read-only root dir by default, or
 more specifically a btrfs file system with three subvolumes in it: one
 read-only one mounted to /, and two writable ones mounted to /home and
 /var, with /tmp mounted from tmpfs.


Yeah the hope is to separate out various things into different
subvolumes so we can have things that can be independently
snapshottable things.

 We came a long way with supporting read-only root dirs and it should
 mostly work now. In F15 for example /etc/mtab is a symlink, and even
 smaller stuff like /etc/nologin got moved to /var/run, to make write
 accesses to /etc unnecessary during normal operation. /etc/resolv.conf
 is the only thing often updated that's left, but with NM using dnsmasq
 it's static too.

 By using btrfs subvolumes doing this kind of seperation into writable
 and non-writable subtrees we don't have to think anymore about the sizes
 for those file systems at install time, since they all live in the same
 fs.

 If this is adopted package managers and system configuration UIs would
 need to invoke mount / -o rw,remount before doing their work.

 So, I'd like to see this implemented one day, maybe the adoption of
 btrfs is the right time to push this through too?

 I have not filed a feature page for this, as I am not sure I want to
 push this on F16, and I don't even know if people in general are onboard
 with this idea. The benefits of this are mostly security and robustness
 since we know that the actual subvolume the OS is booted from is always
 in a consistent state during operation and cannot normally be
 changed. And of course, we get all kind of magic by doing this because
 we can easily use snapshots to roll back system upgrades while leaving
 /home completely untouched.

 Sorry for hijacking your thread like this,


It's cool, just hilights all the cool stuff we can do with BTRFS :).  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Plans for BTRFS in Fedora

[Josef Bacik]

2011/2/22 Jóhann B. johan...@gmail.com:
 On Tue, 2011-02-22 at 14:51 -0500, Josef Bacik wrote:
 So what are your thoughts?  Thanks,

 Will there be any performance penalties making this move?


Who knows, thats what testing is for :).  There are some things that
suck with BTRFS, but so it goes with filesystems.

 I ran it on my workstation when F13 came out with updates+snapshots and while 
 updating my desktop responsiveness was well not so good.

 Novice end users tools ( GUI ) and policy to clean those snapshots and 
 perhaps storing those snapshots in it's own directory and or partition rather 
 than directly on / need be in place before we implement this as an default.


They would be helpful, but not necessary.

 I suggest people try it during the F15 atleast those that are considering as 
 an default to catch any potential usability issues it did not eat any babes 
 from me when I ran it.

 I think we should also ask ourselves these questions..

 What benefit will this switch bring to the novice desktop end users?


Various things, better data integrity to start with, and if you
install the yum-fs-snapshot you have the ability to rollback easily.

 Will the novice desktop end user ever take advantages of any of the features 
 that btrfs brings?


At the very least you get better data integrity via our checksumming
and duplicate metadata on single spindle fs's.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Fedora 15, new and exciting plans

[Josef Bacik]

On Fri, Nov 12, 2010 at 12:03 PM,  clum...@redhat.com wrote:
 We have been spending a lot of time and thought trying to scheme
 about how to accelerate btrfs. At this point, it is actually fairly
 stable but still missing key things (most notably a fsck that can
 fix the file system!).

 Last week at plumbers, we did get a large chunk of the btrfs
 developers together for a few hours and have been working to make
 sure that we have sufficient people working to get this done...

 This is good information, and makes me feel even better about removing
 the need for a btrfs command line option.

 What I mean about we're not quite there yet is that I don't really
 want to change over to it as the default in anaconda until we (1) have a
 release or two with it as a non-hidden option to get some feedback, and
 (2) know what our strategy with it in a greater sense is.  If it's going
 to allow us to do all sorts of crazy new things, I'd like to enable the
 crazy new things at the same time as we switch it in as the default.


Agreed, my plan was to take away the option for F15 (I sent the
anaconda patch already right?) and leave it as an optional thing for
at least 2 releases until we even begin to discuss setting it as
default.

Another criteria I have for making it default is I want grub support,
specifically grub2.  So until grub2 is in Fedora and we have btrfs
patches upstream in grub2 I have no intention of making it the default
in Fedora.  Thanks,

Josef
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel