Re: Restricting automounting of uncommon filesystems?

[Eric Sandeen]

On 7/24/23 10:40 PM, Demi Marie Obenour wrote:

On 7/24/23 15:11, Eric Sandeen wrote:

On 7/23/23 7:22 PM, Steve Grubb wrote:

On Saturday, July 22, 2023 2:01:34 AM EDT Matthew Garrett wrote:

A discussion within Debian again brought up the problem that:

1) Automounting of removable media exposes the kernel to a lot of
untrusted input
2) Kernel upstream are not terribly concerned with ensuring that kernel
filesystems are resilient against deliberately malformed filesystems so
are mostly not proactively looking for bugs there
3) Uncommonly used filesystems are less likely to be tested against
adverse input in the real world and so are more likely to contain
exploitable bugs

There are various cases where people do need to make use of uncommon
filesystems, but the majority of them aren't related to removable media.
udisks2 supports setting the UDISKS_AUTO variable to 0 on devices that
shouldn't be automounted, which means something like:

SUBSYSTEM!="block", GOTO="udisks_insecure_fs_end"
ENV{ID_FS_TYPE}=="hfs", ENV{UDISKS_AUTO}="0"
# repeat as necessary for anything else that shouldn't be automounted
LABEL="udisks_insecure_fs_end"

ought to be enough. So:

a) Does this seem like a good idea?
b) If so, is dealing with it via udev rules the right approach? This way
seems desktop-agnostic
c) Where should it ship, and what should the process be for disabling it
for people who need this functionality?

Long-term I'd love to see more work put into having FUSE support for
these and leaving the in-kernel fs to stuff we know is trustworthy, but
that's rather more work.


If "a malicious input can't cause problems" is the threshold for
trustworthy, I'm not sure we have any trustworthy filesystems as this point.

https://syzkaller.appspot.com/upstream/s/ext4
https://syzkaller.appspot.com/upstream/s/xfs
https://syzkaller.appspot.com/upstream/s/btrfs
https://syzkaller.appspot.com/upstream/s/fat


A while back, I wrote the fsfuzzer specifically to find, in a repeatable way,
filesystem bugs so they can be fixed:

https://github.com/stevegrubb/fsfuzzer

It does not support all filesystems, but it is easy to add support through
adding the correct mounter to the scrips. It has found *so* *many* filesystem
bugs over time.


That was awesome, back in the day! syzbot/syzcaller is the new shiny
here though, finding filesystem flaws day after day that (with all due
respect) fsfuzzer could never have reached (think: fuzzing metadata and
then fixing up the checksum so it passes initial validation on read.)

And frankly that is some of my motivation to find an improvement here. A
small cadre of filesystem developers are near the breaking point trying
to keep up with an army of machines running syzkaller.

-Eric


How much of the problem is the C programming language itself?  I’m NOT
suggesting that you rewrite your filesystem in Rust; that would be an
extremely unreasonable request.  I’m merely trying to figure out how
much of this is a case of “filesystems are hard” and how much of this
is C providing essentially no help.


Well, this has been discussed but it's obviously not a short-term 
solution. :) (Nor would Rust be a panacea. Some of the problems may be 
mitigated by a language like Rust, but certainly not all of them.)


I think it's a little bit of both scenarios, tbh.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Restricting automounting of uncommon filesystems?

[Eric Sandeen]

On 7/23/23 7:22 PM, Steve Grubb wrote:

On Saturday, July 22, 2023 2:01:34 AM EDT Matthew Garrett wrote:

A discussion within Debian again brought up the problem that:

1) Automounting of removable media exposes the kernel to a lot of
untrusted input
2) Kernel upstream are not terribly concerned with ensuring that kernel
filesystems are resilient against deliberately malformed filesystems so
are mostly not proactively looking for bugs there
3) Uncommonly used filesystems are less likely to be tested against
adverse input in the real world and so are more likely to contain
exploitable bugs

There are various cases where people do need to make use of uncommon
filesystems, but the majority of them aren't related to removable media.
udisks2 supports setting the UDISKS_AUTO variable to 0 on devices that
shouldn't be automounted, which means something like:

SUBSYSTEM!="block", GOTO="udisks_insecure_fs_end"
ENV{ID_FS_TYPE}=="hfs", ENV{UDISKS_AUTO}="0"
# repeat as necessary for anything else that shouldn't be automounted
LABEL="udisks_insecure_fs_end"

ought to be enough. So:

a) Does this seem like a good idea?
b) If so, is dealing with it via udev rules the right approach? This way
seems desktop-agnostic
c) Where should it ship, and what should the process be for disabling it
for people who need this functionality?

Long-term I'd love to see more work put into having FUSE support for
these and leaving the in-kernel fs to stuff we know is trustworthy, but
that's rather more work.


If "a malicious input can't cause problems" is the threshold for 
trustworthy, I'm not sure we have any trustworthy filesystems as this point.


https://syzkaller.appspot.com/upstream/s/ext4
https://syzkaller.appspot.com/upstream/s/xfs
https://syzkaller.appspot.com/upstream/s/btrfs
https://syzkaller.appspot.com/upstream/s/fat


A while back, I wrote the fsfuzzer specifically to find, in a repeatable way,
filesystem bugs so they can be fixed:

https://github.com/stevegrubb/fsfuzzer

It does not support all filesystems, but it is easy to add support through
adding the correct mounter to the scrips. It has found *so* *many* filesystem
bugs over time.


That was awesome, back in the day! syzbot/syzcaller is the new shiny 
here though, finding filesystem flaws day after day that (with all due 
respect) fsfuzzer could never have reached (think: fuzzing metadata and 
then fixing up the checksum so it passes initial validation on read.)


And frankly that is some of my motivation to find an improvement here. A 
small cadre of filesystem developers are near the breaking point trying 
to keep up with an army of machines running syzkaller.


-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Restricting automounting of uncommon filesystems?

[Eric Sandeen]

On 7/24/23 10:00 AM, Daniel P. Berrangé wrote:

On Mon, Jul 24, 2023 at 10:08:50AM -0400, Demi Marie Obenour wrote:


...


I still believe that mounting should _not_ be automatic, though, because
it could have side-effects (such as replaying the FS journal) that might
not be wanted.  To prevent prompt fatigue, Fedora could offer to remember
the user’s choice.


Remember the choice per device.

If I have a USB flash stick I plug in every day, it shouldn't ask me
about that after the first time I use it.

If I acquire a new USB flash stick I've never plugged in before, I
don't want it auto-mounting before I can wipe & reformat it.


FWIW, OSX does this today.

-Eric


With regards,
Daniel

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Restricting automounting of uncommon filesystems?

[Eric Sandeen]

On 7/22/23 9:12 AM, Neal Gompa wrote:

On Sat, Jul 22, 2023 at 9:53 AM Florian Weimer  wrote:


* Matthew Garrett:


a) Does this seem like a good idea?
b) If so, is dealing with it via udev rules the right approach? This way
seems desktop-agnostic
c) Where should it ship, and what should the process be for disabling it
for people who need this functionality?


Maybe a first step would be to disable automounting while the screen is
locked.


Long-term I'd love to see more work put into having FUSE support for
these and leaving the in-kernel fs to stuff we know is trustworthy, but
that's rather more work.


Fedora moved in the opposite direction (from gvfs to unprivileged
mounting via udisks2 IIRC).



Several years ago, SUSE distributions moved to disabling the modules
by default for a number of filesystems, but making it pretty easy to
turn them back on:
https://github.com/openSUSE/suse-module-tools/pull/5


That's approaching it from the wrong angle, IMHO. It's not that you 
don't want to be able to mount these filesytems at all, it's that you 
don't want let any random unprivileged user do so.


If the system administrator wants to mount $UNCOMMONFS, they should be 
able to do so without hassle, but that doesn't mean that a normal user 
who got handed a sketchy USB stick at a conference should be able to do 
so with no restrictions at all.


-Eric


We could take a similar approach, maybe...?

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Restricting automounting of uncommon filesystems?

[Eric Sandeen]

On 7/22/23 7:57 AM, Michael Catanzaro wrote:

I've been thinking about this for a while. The status quo is really awful.

On Sat, Jul 22 2023 at 11:31:22 AM +, Zbigniew Jędrzejewski-Szmek 
 wrote:

A bigger problem I see, is that if a user plugins in a usb stick,
expecting to make use of it, and it's not automounted without any
explanation, they are most likely to just click for it to be mounted,
or to even issue an explicit 'mount', defeating the whole protection.


Yup, there's the problem. The user will almost always mount it manually, 
so disabling automount seems pointless.



A more reasonable UI would be to display a pop-up that says "Device
ASDF uses file system AmigaFS 1982 which is not well supported. See
https://some.link/ for details. Do you want to a) mount once, b) not
mount, c) mount this fs type always?". This would require some work
in DE.


The UI would have to not mention technical details like the name of the 
filesystem. Also, warning a user that the filesystem is not 
"well-supported" is also not likely to accomplish much. The user plugged 
in the USB stick in order to look at files, and will almost always 
choose to do so if presented with the option. Even if we warn that the 
device may be malicious (which we don't actually know), users will still 
just think about it and decide "probably not, I want to see my files."


The desktop security model assumes the kernel is robust to malformed 
filesystems and removing that assumption is just not workable. This 
development mindset might be prevalent among kernel developers, but it's 
not acceptable for desktop users.  Filesystems that are not designed to 
be robust to untrusted input should be disabled outright and not 
supported at all. I'm not sure how practical this actually is, though, 
because I think it would probably entail disabling common filesystems 
(ext4? xfs? btrfs?) in addition to uncommon filesystems. Starting with 
disabling uncommon filesystems is better than nothing, though.


mount is a privileged operation in the kernel; when system configuration 
works around that to allow non-trusted users to initiate a mount, that 
breaks the restrictions that the kernel intentionally put on the ability 
to ingest a stream of untrusted data from a block device.


Filesystems generally *do* care about malformed input, but they do so in 
the context of the security model that the kernel defines.


As an example, XFS goes to great lengths to validate input, and placing 
checksums on every piece of metadata is a big part of that. In the real 
world, this validation catches and handles the vast majority of 
malformed/corrupted input that XFS should expect to see.


Other filesystems have similar validation, with similar assumptions.

But when fuzzers intentionally malform metadata and then fix up the 
checksum to look valid, this is something that would never happen 
organically; it is an intentionally malicious work-around to the 
existing mechanism. But again, mount is supposed to be a privileged 
operation, so if the system administrator decides to mount the USB drive 
they found in the parking lot, that's on them, and not a security issue. 
They are root, after all.


Once you let any random user do this, working around the kernel's 
privilege checks, it's a different scenario, and it exposes the system 
to risks that were not part of the original threat model.


I wholeheartedly support any effort to restrict (by default) 
auto-mounting to a smaller set of filesystems that could reasonably be 
expected to be found on removable media (isofs, fat, exfat ...) and shut 
off all the rest to limit the attack surface here. This issue and the 
associated CVEs that get raised have been consuming an inordinate amount 
of time from a very limited set of filesystem developers.


-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: btrfs loses 32-bit application compatibility after a while

[Eric Sandeen]

On 7/14/23 4:47 PM, Fabio Valentini wrote:

On Fri, Jul 14, 2023 at 10:45 PM Eric Sandeen  wrote:


On 7/14/23 6:53 AM, Florian Weimer wrote:

* Neal Gompa:


On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini  wrote:


On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer  wrote:


Fedora lists are hostile to upstream collaboration via cross-posting, so
I can only forward this for your information.

This causes problems with the i686 builders.


I wonder how this only started to happen recently? Has something
changed in BTRFS with the 6.3 kernel?
This only started happening a few days after builders were rebooted at
the end of June to apply updates (and kernel 6.3 was among those
updates, as far as I can tell).



This was always possible. I'm curious as to why it took so long for us
to hit it, though.

The recommended solution is to create a new subvolume for these
environments, since the inode count is reset for each subvolume.


What about impact beyond the builders?

Are end users are expected to do this?  Do we have a tool for this?


FWIW, 64 bit inodes have existed in some Linux filesystems for a very
long time. On XFS, you'll get them by default - and quickly, not after
extended use - on a filesystem of sufficient size (around 1T-2T by
default, if I remember right.)

XFS does have a hack^Wmount option to force inodes into the 32-bit
range, but just FWIW we almost never see users running into problems
with 32-bit applications (but maybe because they know about the mount
option...)


But that still raises the question - why does it look like this
started to happen pretty suddenly around June 30?
The list of updates that were applied to builders in that timeframe
doesn't raise any alarm bells (except maybe the 6.3 kernel):
(see https://pagure.io/releng/issue/11531#comment-864471)
I read the release notes for the 6.3 kernel but didn't see any
mentions of BTRFS changes that could explain this. :(


Sure - I can't speak to what might have changed in btrfs, sorry - just 
saying that binaries that choke on 64 bit inodes are going to get less 
and less usable over time, I think.


One thing you could consider if you're ever really in a pinch is an 
ld_preload that fakes up a 32-bit inode number for i.e. stat. The vast 
majority of applications don't care about the inode number at all, and 
it's really kind of unfortunate to fail the whole stat call just for that.


-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: btrfs loses 32-bit application compatibility after a while

[Eric Sandeen]

On 7/14/23 6:53 AM, Florian Weimer wrote:

* Neal Gompa:


On Thu, Jul 13, 2023 at 8:29 AM Fabio Valentini  wrote:


On Thu, Jul 13, 2023 at 10:33 AM Florian Weimer  wrote:


Fedora lists are hostile to upstream collaboration via cross-posting, so
I can only forward this for your information.

This causes problems with the i686 builders.


I wonder how this only started to happen recently? Has something
changed in BTRFS with the 6.3 kernel?
This only started happening a few days after builders were rebooted at
the end of June to apply updates (and kernel 6.3 was among those
updates, as far as I can tell).



This was always possible. I'm curious as to why it took so long for us
to hit it, though.

The recommended solution is to create a new subvolume for these
environments, since the inode count is reset for each subvolume.


What about impact beyond the builders?

Are end users are expected to do this?  Do we have a tool for this?


FWIW, 64 bit inodes have existed in some Linux filesystems for a very 
long time. On XFS, you'll get them by default - and quickly, not after 
extended use - on a filesystem of sufficient size (around 1T-2T by 
default, if I remember right.)


XFS does have a hack^Wmount option to force inodes into the 32-bit 
range, but just FWIW we almost never see users running into problems 
with 32-bit applications (but maybe because they know about the mount 
option...)


-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: filesystems and year 2038

[Eric Sandeen]

On 4/4/22 2:51 PM, Justin Forbes wrote:
> On Mon, Apr 4, 2022 at 11:47 AM Colin Walters  wrote:
>>
>> Hi, creating a thread on this from:
>> https://github.com/coreos/fedora-coreos-config/pull/1650
>>
>> Basically I'd propose that not just our default images have y2038-compatible 
>> filesystem setups, we ensure that if e.g. XFS is explicitly chosen for a 
>> Workstation installation then it is set up with bigtime=1.
>>
>> (Note btrfs uses 64 bit time today, so I think this is mostly about ext4 and 
>> xfs, but perhaps we also need to look at the longer tail too, e.g. squashfs. 
>>  OTOH, because squashfs is read-only we can just worry about that closer to 
>> 10 years from now...)
>>
>> If no one objects I guess I can look at re-learning Mediawiki syntax again 
>> and writing a Change.
> 
> Or, you could ignore it and it will happen anyway:
> 
> xfsprogs-5.15.0-rc1 (11 Mar 2022)
> - mkfs: enable inobtcount and bigtime by default (Darrick J. Wong)

*nod*

I just pushed xfsprogs-5.15.0 out today and it'll be built in rawhide shortly.
If we need this in released Fedoras I'm happy to push 5.15.0 to older
releases as well.

As for ext4, it might be wise to raise the small-filesystem issue on the
ext4 list.

OTOH, the ext4 "small" threshold is 512MB so another option might be to
just be sure to size /boot to at least 512mb to get Y2038-happy inodes.

else if (fs_blocks_count < 512 * meg)
size_type = "small";

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Meaning of Size Directories

[Eric Sandeen]

On 3/16/21 11:51 AM, John Reiser wrote:
> On 3/16/21, David Howells wrote:
>> John Reiser  wrote:
>>
>>> See the manual page "man 2 getdents".
>>
>> Um, which bit?  I don't see anything obvious to that end.
> 
> On that manual page:
> =
> The system call getdents() reads several linux_dirent structures from the 
> directory
> referred to by the open file descriptor fd into the buffer pointed to by dirp.
>    [snip]]
> On  success, the number of bytes read is returned.

But the original question was about the st_size returned stat, which is not
calling getdents.

Two different numbers, which mean 2 different things.

> =
> 
> So the return value is related to the size of the directory; the sum of the 
> values
> returned before End-Of-File should be quite close to the .st_size of the 
> directory.

Again, that's not at all correct. Counter-example on ext4:

# stat -c %s dir
2547712
# ls -a1 dir
.
..
file
# strace -v -egetdents ls dir
getdents(3, [{d_ino=524289, d_off=4294967296, d_reclen=24, d_name=".", 
d_type=DT_DIR}, {d_ino=2, d_off=3358761300848251151, d_reclen=24, d_name="..", 
d_type=DT_DIR}, {d_ino=534290, d_off=9223372036854775807, d_reclen=24, 
d_name="file", d_type=DT_REG}], 32768) = 72

72 is not anywhere close to 2547712

> If a program is walking through the directory, reading all the entries via 
> getdents64(),
> then .st_size of the directory is the only thing known in advance about the 
> total size.

But it tells you nothing about how much is likely to be returned by getdents.

You should not use st_size to infer anything about the amount of data which 
will be
returned by getdents. POSIX does not define the meaning of st_size for 
directories,
and different filesystems can do wildly different things.

As Steve mentioned, st_blksize is your best hint for this purpose.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Meaning of Size Directories

[Eric Sandeen]

On 2/25/21 6:56 PM, John Reiser wrote:
>> Tools such as ls or stat report the size of a directory. Of course it is not 
>> the content size.
>> stat -c %s  /home/sergio/.config
>> 6550
>>
>> What does 6550 mean in btrfs context?
> 
> Regardless of filesystem type, the size of a directory is the sum of the sizes
> of the struct linux_dirent (or linux_dirent64) for the filenames of the 
> contained files.
> See the manual page "man 2 getdents".

That's not correct; dirents are in-memory structures, unrelated
to what the [fl]stat(2) interface used by ls returns for a directory.

The size returned by stat(2) (aka ls) of a directory inode is
filesystem implementation dependent, and AFAIK has no well-defined
meaning. stat(2) refers to st_size only for files and symlinks,
not for directories.  Same w/ POSIX:

off_t st_sizeFor regular files, the file size in bytes. 
 For symbolic links, the length in bytes of the 
 pathname contained in the symbolic link. 


So, as one example on ext4 - directories never shrink.

# mkdir dir
# ls -ld dir
drwxr-xr-x. 2 root root 4096 Feb 26 00:48 dir

# touch dir/123456789
# ls -ld dir
drwxr-xr-x. 2 root root 4096 Feb 26 01:00 dir

# for I in `seq 1 2000`; do touch dir/longfilename$I; done
# ls -ld dir
drwxr-xr-x. 2 root root 69632 Feb 26 00:49 dir

# rm -f dir/*
# ls -ld dir
drwxr-xr-x. 2 root root 69632 Feb 26 00:49 dir

69632 byte directory with no files in it, wh.

xfs is different:

# mkdir dir
# ls -ld dir
drwxr-xr-x. 2 root root 6 Feb 26 00:58 dir

# touch dir/123456789
# ls -ld dir
drwxr-xr-x. 2 root root 23 Feb 26 00:59 dir

# for I in `seq 1 2000`; do touch dir/longfilename$I; done
# ls -ld dir
drwxr-xr-x. 2 root root 65536 Feb 26 00:59 dir

# rm -f dir/*
# ls -ld dir
drwxr-xr-x. 2 root root 6 Feb 26 00:59 dir

btrfs is still different:

# mkdir dir
# ls -ld dir
drwxr-xr-x. 1 root root 0 Feb 26 01:05 dir

# touch dir/123456789
# ls -ld dir
drwxr-xr-x. 1 root root 18 Feb 26 01:06 dir

# for I in `seq 1 2000`; do touch dir/longfilename$I; done
# ls -ld dir
drwxr-xr-x. 1 root root 61804 Feb 26 01:06 dir

# rm -f dir/*
# ls -ld dir
drwxr-xr-x. 1 root root 0 Feb 26 01:06 dir



In short, "size" of a dir doesn't tell you much.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: btrfs and default page sizes (4k vs 64k)

[Eric Sandeen]

On 9/16/20 10:22 AM, Benjamin Block wrote:
> On Wed, Sep 16, 2020 at 09:31:50AM -0500, Eric Sandeen wrote:

...

>> Sub-page block support in filesystems is not a wild, esoteric, unexpected
>> feature.
>>
> 
> These kinds of problems are not really that rare across different
> Filesystems.
> 
> Try creating a XFS fs on a system with 64k PAGE_SIZE and a blocksize of
> 64k, then try mounting that fs on a x86_64 machine. It won't work:
> https://elixir.bootlin.com/linux/v5.8/source/fs/xfs/xfs_mount.c#L165
> And IIRC xfs is the default for RHEL, no?

It is.  mkfs.xfs defaults to 4k blocks, so XFS filesystem are, by default,
compatible across all supported architectures.  RHEL would not choose a
default fs with this sort of incompatibility across arches.

Block > page size is a different problem vs what is described in this
thread.

If you /manually/ create a large block size fs, overriding the defaults,
then yes you will have a compatibility problem on smaller page systems.

That's not the same as "you cannot create any btrfs filesystem that is
usable on both 4k and 64k page systems"

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: btrfs and default page sizes (4k vs 64k)

[Eric Sandeen]

On 9/15/20 7:29 PM, Neal Gompa wrote:
> On Tue, Sep 15, 2020 at 7:57 PM Kevin Kofler  wrote:
>>
>> Daniel Pocock wrote:
>>> One issue I've come across is that a btrfs filesystem can only be used
>>> on hosts with the same page size as the host that created the filesystem
>>
>> Ewww! That alone should disqualify btrfs as a default file system!
>>
>> Why does a file system depend on the kernel page size? The kernel page size
>> is an internal implementation detail of the kernel, whereas a file system
>> ought to be a stable interchange format that is compatible across all
>> machines.
>>
>> It is unfortunate that this showstopper was not mentioned when the switch to
>> btrfs by default was proposed.

I'm not sure that it would have been deemed any more important than other
concerns which were raised at the time, TBH.

> I hate to break it to you, but this problem is not just in
> filesystems, it's in basically everything in the kernel. And we've had
> variations of problems like this for years (endianness, page size,
> pointer size, single bit vs multi-bit booleans, etc.). I've personally
> been bitten by all of these issues in some way. This comes from the
> fact that there's no such thing as "internal implementation detail of
> the kernel" by design. This is the "joy" of the monorepo "design"
> where everything leaks into everything else.

That's simply not accurate.  Handling 32/64 bit interfaces, endianness, etc
are long-solved problems.  Longstanding lack of design or support for
sub-page block support in a filesystem is not /at all/ the same thing.

Are there occasional endianness bugs, pointer size bugs, etc?  Sure.
But that's different from "We did not design this."

> This didn't become a serious problem until Red Hat made the
> unfortunate (though not realized at the time) mistake of switching to
> 64k pages for ARM and POWER. We got that change in Fedora for POWER
> but not ARM. It has led to all kinds of unfortunate problems that are
> gradually being worked on and fixed upstream.

Sub-page block support in filesystems is not a wild, esoteric, unexpected
feature.

It's something that is generally available in nearly every other widely used
Linux filesystem.  It's not accurate to suggest that this is some unexpected
side effect of page size choice, or that 64k pages were somehow a "mistake"
now that this btrfs compatibility issue has been made more obvious.

btw, Fedora has shipped kernels with 64k pages for almost a decade:

commit 737c9c7da818f1da0bdf3f6a0dda5c38a3cba769
Author: Josh Boyer 
Date:   Fri Sep 9 11:21:22 2011 -0400

Change to 64K page size for ppc64 kernels (rhbz 736751)

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Very strange compiler/linker related build failures in rawhide

[Eric Sandeen]

On 7/24/20 1:31 AM, Fabio Valentini wrote:
> Hi all,
> 
> I'm starting to see various very strange kinds of build failures in
> rawhide, that seem to have started with either of these updates (or a
> combination of them):
> 
> - annobin 9.21-1.fc33 → 9.22-1.fc33
> - binutils 2.34.0-6.fc33 → 2.34.0-7.fc33
> - elfutils 0.179-2.fc33 → 0.180-2.fc33
> - glibc 2.31.9000-13.fc33 → 2.31.9000-14.fc33
> 
> These rawhide updates all happened at roughly the same time, so it's
> difficult to say which one of them is to blame (if any of them).

Hm, is this related?  (libtool segfault building xfsprogs)

https://kojipkgs.fedoraproject.org//work/tasks/9149/47779149/build.log

/bin/sh ../libtool --quiet --tag=CC --mode=link gcc -Wl,-z,relro 
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld 
-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now 
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -rpath /lib64 -version-info 0:0:0 
-static -o libfrog.la avl64.lo bitmap.lo bulkstat.lo convert.lo crc32.lo 
fsgeom.lo list_sort.lo linux.lo logging.lo paths.lo projects.lo ptvar.lo 
radix-tree.lo scrub.lo topology.lo util.lo workqueue.lo 
../libtool: line 1085: 137776 Segmentation fault  (core dumped) ar cru 
.libs/libfrog.a avl64.o bitmap.o bulkstat.o convert.o crc32.o fsgeom.o 
list_sort.o linux.o logging.o paths.o projects.o ptvar.o radix-tree.o scrub.o 
topology.o util.o workqueue.o
gmake[2]: *** [../include/buildrules:71: libfrog.la] Error 139
gmake[1]: *** [include/buildrules:36: libfrog] Error 2
make: *** [Makefile:92: default] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.fcTM51 (%build)
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.fcTM51 (%build)
Child return code was: 1
EXCEPTION: [Error()]
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/9/20 9:15 PM, Josef Bacik wrote:
> On 7/9/20 9:30 PM, Eric Sandeen wrote:

...

>>> This test is run constantly by us, specifically because it's the error 
>>> cases that get you.  But not for crash consistency reasons, because we're 
>>> solid there.  I run them to make sure I don't have stupid things like 
>>> reference leaks or whatever in the error path.  Thanks,
>>
>> or "corrupted!" printk()s that terrify the hapless user? ;)
> 
> I'd love to know what hapless user is running xfstests.  Thanks,

*sigh*

the point is, telling the user "your filesystem is corrupted" if it's not 
actually corrupted is bad news.  Discovering that communication problem via 
xfstests does not make the concern less valid.  I was trying to gently tease 
you that the test not only discovers leaks, but also discovers terrifyingly 
inaccurate messages in response to IO errors, but I guess that didn't come 
through.

Thanks.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/9/20 8:22 PM, Josef Bacik wrote:
> On 7/9/20 7:23 PM, Eric Sandeen wrote:
>> On 7/9/20 4:27 PM, Eric Sandeen wrote:
>>> On 7/9/20 3:32 PM, Davide Cavalca via devel wrote:
>>
>> ...
>>
>>>> As someone on one of the teams at FB that has to deal with that, I can
>>>> assure you all the scenarios you listed can and do happen, and they
>>>> happen a lot. While we don't have the "laptop's out of battery" issue
>>>> on the production side, we have plenty of power events and unplanned
>>>> maintenances that can and will hit live machines and cut power off.
>>>> Force reboots (triggered by either humans or automation) are also not
>>>> at all uncommon. Rebuilding machines from scratch isn't free, even with
>>>> all the automation and stuff we have, so if power loss or reboot events
>>>> on machines using btrfs caused widespread corruption or other issues
>>>> I'm confident we'd have found that out pretty early on.
>>>
>>> It is a bare minimum expectation that filesystems like btrfs, ext4, and xfs
>>> do not suffer filesystem corruptions and inconsistencies due to reboots
>>> and power losses.
>>>
>>> So for the record I am in no way insinuating that btrfs is less crash-safe
>>> than other filesystems (though I have not tested that, so if I have time
>>> I'll throw that into the mix as well.)
>>
>> So, we already have those tests in xfstests, and I put btrfs through a few
>> loops.  This is generic/475:
>>
>> # Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
>> #
>> # FS QA Test No. 475
>> #
>> # Test log recovery with repeated (simulated) disk failures.  We kick
>> # off fsstress on the scratch fs, then switch out the underlying device
>> # with dm-error to see what happens when the disk goes down.  Having
>> # taken down the fs in this manner, remount it and repeat.  This test
>> # is a Good Enough (tm) simulation of our internal multipath failure
>> # testing efforts.
>>
>> It fails within 2 loops.  Is it a critical failure? I don't know; the
>> test looks for unexpected things in dmesg, and perhaps the filter is
>> wrong.  But I see stack traces during the run, and message like:
>>
>> [689284.484258] BTRFS: error (device dm-3) in btrfs_sync_log:3084: 
>> errno=-117 Filesystem corrupted

You might want to change that message, then.  If it's not corrupted, I'd 
suggest not doing printk("corrupted!") because that will make people think that 
it's corrupted, because it says "Filesystem corrupted..." ;)

> 
> Yeah, because dm-error throws EIO, and thus we abort the transaction, which 
> results in an EUCLEAN if you run fsync.  This is a scary sounding message, 
> but its _exactly_ what's expected from generic/475.  I've been running this 
> in a loop for an hour and the thing hasn't failed yet.  There's all sorts of 
> scary messages

That's weird.  The test fails very quickly for me - again, AFAICT it fails due 
to things in dmesg that aren't recognized as safe by the test harness, but a 
variety of things - not just stack dumps - seem to trigger the failure.

> [17929.939871] BTRFS warning (device dm-13): direct IO failed ino 261 rw 
> 1,34817 sector 0xb8ce0 len 24576 err no 10
> [17929.943099] BTRFS: error (device dm-13) in btrfs_commit_transaction:2323: 
> errno=-5 IO failure (Error while writing out transaction)
> 
> again, totally expected because we're forcing EIO's at random times.

Right, of course it will get IO errors, that's why I didn't highlight those in 
my email.

>> so I can't say for sure.
>>
>> Are btrfs devs using these tests to assess crash/powerloss resiliency
>> on a regular basis?  TBH I honestly did not expect to see any test
>> failures here, whether or not they are test artifacts; any filesystem
>> using xfstests as a benchmark needs to be keeping things up to date.
>>
> 
> It depends on the config options.  Some of our transaction abort sites dump 
> stack, and that trips the dmesg filter, and thus it fails.  Generally when I 
> run this test I turn those options off.

It would be good, in general, to fix up the test for btrfs so that it does not 
yield false positives, if that's what this is.  Otherwise it trains people to 
ignore it or not run it

> This test is run constantly by us, specifically because it's the error cases 
> that get you.  But not for crash consistency reasons, because we're solid 
> there.  I run them to make sure I don't have stupid things like reference 
> leaks or whatever in the error path.  Thanks,

or "corrupted!" printk()s that terrify the hapless user? ;)

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/9/20 4:27 PM, Eric Sandeen wrote:
> On 7/9/20 3:32 PM, Davide Cavalca via devel wrote:

...

>> As someone on one of the teams at FB that has to deal with that, I can
>> assure you all the scenarios you listed can and do happen, and they
>> happen a lot. While we don't have the "laptop's out of battery" issue
>> on the production side, we have plenty of power events and unplanned
>> maintenances that can and will hit live machines and cut power off.
>> Force reboots (triggered by either humans or automation) are also not
>> at all uncommon. Rebuilding machines from scratch isn't free, even with
>> all the automation and stuff we have, so if power loss or reboot events
>> on machines using btrfs caused widespread corruption or other issues
>> I'm confident we'd have found that out pretty early on.
> 
> It is a bare minimum expectation that filesystems like btrfs, ext4, and xfs
> do not suffer filesystem corruptions and inconsistencies due to reboots
> and power losses.
> 
> So for the record I am in no way insinuating that btrfs is less crash-safe
> than other filesystems (though I have not tested that, so if I have time
> I'll throw that into the mix as well.)

So, we already have those tests in xfstests, and I put btrfs through a few
loops.  This is generic/475:

# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
#
# FS QA Test No. 475
#
# Test log recovery with repeated (simulated) disk failures.  We kick
# off fsstress on the scratch fs, then switch out the underlying device
# with dm-error to see what happens when the disk goes down.  Having
# taken down the fs in this manner, remount it and repeat.  This test
# is a Good Enough (tm) simulation of our internal multipath failure
# testing efforts.

It fails within 2 loops.  Is it a critical failure? I don't know; the
test looks for unexpected things in dmesg, and perhaps the filter is
wrong.  But I see stack traces during the run, and message like:

[689284.484258] BTRFS: error (device dm-3) in btrfs_sync_log:3084: errno=-117 
Filesystem corrupted

so I can't say for sure.

Are btrfs devs using these tests to assess crash/powerloss resiliency
on a regular basis?  TBH I honestly did not expect to see any test
failures here, whether or not they are test artifacts; any filesystem
using xfstests as a benchmark needs to be keeping things up to date.

As a further test, I skipped the dmesg check, which may or may not be
finding false positives, and replaced it with a mount/umount/check cycle.
That seems to pass, so if fsck validation is complete and correct, perhaps
all is well in this regard.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/9/20 3:38 PM, Chris Murphy wrote:
> On Thu, Jul 9, 2020 at 1:57 PM Eric Sandeen  wrote:
>> On 7/9/20 2:11 PM, Josef Bacik wrote:
>>>>  From what I've gathered from these responses, btrfs is unique in that it 
>>>> is
>>>> /expected/ that if anything goes wrong, the administrator should be 
>>>> prepared
>>>> to scrape out remaining data, re-mkfs, and start over.  If that's 
>>>> acceptable
>>>> for the Fedora desktop, that's fine, but I consider it a risk that should 
>>>> not
>>>> be ignored when evaluating this proposal.
>>>>
>>> Agreed, it's the very first thing I said when I was asked what are the 
>>> downsides.  There's clearly more work to be done in the recovery arena.  
>>> How often do disks fail for Fedora?  Do we have that data?  Is this a real 
>>> risk? Nobody can say because Fedora doesn't have data.
>> But again, let me reiterate that disk failures are far from the only
>> reason that admins need capable filesystem repair tools, in general.
>>
>> We see users running fsck all the time, for various reasons.  I can't
>> back it up, but my hunch is that bugs and misconfigurations (i.e. write
>> cache) are more often the root cause for filesystem inconsistencies.
>>
>> IMHO, focusing on physical disk failure rates is focusing too narrowly,
>> but I suppose I'm just joining the chorus of hunches and anecdotes now.
> Actually there's quite a lot of evidence of this, even though there's
> no precise estimate - not least of which these populations are
> constantly dying and reemerging, and can be batch (firmware version)
> specific. This is only the most recent such story on linux-btrfs@ (and
> warning, this reads like an alien autopsy):
> 
> https://lore.kernel.org/linux-btrfs/20200708034407.ge10...@hungrycats.org/
> 
> fsck.btrfs is a no op, same as fsck.xfs. And recently the actual
> repair utility dissuades users from running it casually.


Honestly, that's not relevant. They are no-ops because they do not need to
be run at boot time after an unclean shutdown, because the filesystems are
explicitly designed to handle that.  This is clearly stated in the man page,
the script itself, and the commit log.  In fact fsck.btrfs was copied from
fsck.xfs.

(Honestly fsck.ext[34] could be a no-op too, but for $REASONS it chooses to do
journal replay in userspace instead, via fsck.)

They are no-ops for this reason, and /not/ because fsck isn't /ever/ expected
to be needed.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/9/20 3:32 PM, Davide Cavalca via devel wrote:
> On Thu, 2020-07-09 at 16:15 -0400, Simo Sorce wrote:
>> However I have had bad kernels, power outages, loss of battery power
>> (laptops on too long suspend) and other random reasons to force
>> reboot
>> a system. That has been the primary case of file system checks
>> through
>> my Fedora usage. And luckily so far I never had a loss of filesystem
>> or
>> data that way, fsck always ended up solving most of the issues, and
>> whenever I lost file they ended up being temporary files I did not
>> care
>> for.
>>
>> I do not think those failures are common in Facebook fleets, so I am
>> quite skeptical FB data and failure modes are representative of
>> Fedora
>> usage as a desktop/laptop OS and therefore of the behavior of btrfs
>> in
>> those cases.
> 
> As someone on one of the teams at FB that has to deal with that, I can
> assure you all the scenarios you listed can and do happen, and they
> happen a lot. While we don't have the "laptop's out of battery" issue
> on the production side, we have plenty of power events and unplanned
> maintenances that can and will hit live machines and cut power off.
> Force reboots (triggered by either humans or automation) are also not
> at all uncommon. Rebuilding machines from scratch isn't free, even with
> all the automation and stuff we have, so if power loss or reboot events
> on machines using btrfs caused widespread corruption or other issues
> I'm confident we'd have found that out pretty early on.

It is a bare minimum expectation that filesystems like btrfs, ext4, and xfs
do not suffer filesystem corruptions and inconsistencies due to reboots
and power losses.

So for the record I am in no way insinuating that btrfs is less crash-safe
than other filesystems (though I have not tested that, so if I have time
I'll throw that into the mix as well.)

We do at times see corrupted filesystems when something has a writeback
cache w/o a battery backup, though, because then the hardware violates
its guarantees to the filesystem this is the sort of thing I'd put
in the "misconfiguration" bucket.  Which happens from time to time, and
from which it is nice to be able to recover w/o heroics.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/9/20 2:11 PM, Josef Bacik wrote:
>>
>>  From what I've gathered from these responses, btrfs is unique in that it is
>> /expected/ that if anything goes wrong, the administrator should be prepared
>> to scrape out remaining data, re-mkfs, and start over.  If that's acceptable
>> for the Fedora desktop, that's fine, but I consider it a risk that should not
>> be ignored when evaluating this proposal.
>>
> 
> Agreed, it's the very first thing I said when I was asked what are the 
> downsides.  There's clearly more work to be done in the recovery arena.  How 
> often do disks fail for Fedora?  Do we have that data?  Is this a real risk? 
> Nobody can say because Fedora doesn't have data.

But again, let me reiterate that disk failures are far from the only
reason that admins need capable filesystem repair tools, in general.

We see users running fsck all the time, for various reasons.  I can't
back it up, but my hunch is that bugs and misconfigurations (i.e. write
cache) are more often the root cause for filesystem inconsistencies.

IMHO, focusing on physical disk failure rates is focusing too narrowly,
but I suppose I'm just joining the chorus of hunches and anecdotes now.

-Eric

> Facebook does however have that data, and it's a microscopically small 
> percentage.  I agree that Facebook is vastly different from Fedora from a 
> recovery standpoint, but our workloads and hardware I think extrapolate to 
> the normal Fedora user quite well.  We drive the disks harder than the normal 
> Fedora user does of course, but in the end we're updating packages, taking 
> snapshots, and building code.  We're just doing it at 1000x what a normal 
> Fedora user does.  Thanks,
> 
> Josef
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/6/20 8:21 PM, Chris Murphy wrote:

...

> Yes. Also in fuzzing there is the concept of "when to stop fuzzing"
> because it's a rabbit hole, you have to come up for air at some point,
> and work on other things. But you raise a good and subtle point which
> is also that ext4 has a very good fsck built up over decades, they
> succeed today from past failures. It's no different with Btrfs.
> 
> But also there is a bias. ext4 needs fsck to succeed in the worst
> cases in order to mount the file system.

Really?

> Btrfs doesn't need that.
> Often it can tolerate a read-only mount without any other mount
> option; 

Well, this assertion can be tested, so let's do that as well;
I'll do 50 runs of:

* mkfs w/ -m single as would happen on SSD
* fuzz 2048 byte of that 1G image at random
* mount -o ro, tally mount failures
* count missing/unreachable files if mount -o ro succeeds

<50 runs later on btrfs>

16 readonly mounts failed (32% failure rate)
Within the successful mounts, 1 or more files were unreachable in 30 attempts.
Across all 50 attempts, 7720 files were lost.

Is that better than ext4, and will ext4 need fsck just to be able to mount?

<50 runs later on ext4, same strategy>

zero mount failures for ext4.
Within the successful mounts, 1 or more files were unreachable in 2 attempts.
Across all 50 attempts, 48 files were lost.

It does not seem that btrfs has any unique or superior mount -o ro
recovery capabilities, either.

> and optionally can be made more tolerant to errors while still
> mounting read-only. This is a significant difference in recovery
> strategy. An fsck is something of a risk because it is writing changes
> to the file system. It is irreversible. Btrfs takes a different view,
> which is to increase the chance of recovery without needing a risky
> repair as the first step. Once your important data is out, now try the
> repair. Good chance it works, but maybe not as good as ext4's.

That's not supported by any of these test results.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/6/20 12:07 AM, Chris Murphy wrote:
> On Fri, Jul 3, 2020 at 8:40 PM Eric Sandeen 
> wrote:
>> 
>> On 7/3/20 1:41 PM, Chris Murphy wrote:
>>> SSDs can fail in weird ways. Some spew garbage as they're
>>> failing, some go read-only. I've seen both. I don't have stats on
>>> how common it is for an SSD to go read-only as it fails, but once
>>> it happens you cannot fsck it. It won't accept writes. If it
>>> won't mount, your only chance to recover data is some kind of
>>> offline scrape tool. And Btrfs does have a very very good scrape
>>> tool, in terms of its success rate - UX is scary. But that can
>>> and will improve.
>> 
>> Ok, you and Josef have both recommended the btrfs restore
>> ("scrape") tool as a next recovery step after fsck fails, and I
>> figured we should check that out, to see if that alleviates the
>> concerns about recoverability of user data in the face of
>> corruption.
>> 
>> I also realized that mkfs of an image isn't representative of an
>> SSD system typical of Fedora laptops, so I added "-m single" to
>> mkfs, because this will be the mkfs.btrfs default on SSDs (right?).
>> Based on Josef's description of fsck's algorithm of throwing away
>> any block with a bad CRC this seemed worth testing.
>> 
>> I also turned fuzzing /down/ to hitting 2048 bytes out of the 1G 
>> image, or a bit less than 1% of the filesystem blocks, at random. 
>> This is 1/4 the fuzzing rate from the original test.
>> 
>> So: -m single, fuzz 2048 bytes of 1G image, run btrfsck --repair, 
>> mount, mount w/ recovery, and then restore ("scrape") if all that 
>> fails, see what we get.
> 
> What's the probability of this kind of corruption occurring in the 
> real world? If the probability is so low it can't practically be 
> computed, how do we assess the risk? And if we can't assess risk, 
> what's the basis of concern?

From 20 years of filesystem development experience, I know that people
run filesystem repair tools.  It's just a fact.  For a wide variety of
reasons - from bugs, to hardware errors, to admin errors, you name it,
filesystems experience corruption and inconsistencies.  At that point
the administrator needs a path forward.

"people won't need to repair btrfs" is, IMHO, the position that needs
to be supported, not "filesystem repair tools should be robust."

>> I ran 50 loops, and got:
>> 
>> 46 btrfsck failures 20 mount failures
>> 
>> So it ran btrfs restore 20 times; of those, 11 runs lost all or 
>> substantially all of the files; 17 runs lost at least 1/3 of the 
>> files.
> 
> Josef states reliability of ext4, xfs, and Btrfs are in the same 
> ballpark. He also reports one case in 10 years in which he failed to 
> recover anything. How do you square that with 11 complete failures, 
> trivially produced? Is there even a reason to suspect there's
> residual risk?

Extrapolating from Facebook's usecases to the fedora desktop should be
approached with caution, IMHO.

I've provided evidence that if/when damage happens for whatever reason,
btrfs is unable to recover in place far more often than other filesytems.

> When metadata is single profile, Btrfs is basically an early warning 
> system.> The available research on uncorrectable errors, errors that drive ECC
> does not catch, suggests that users are decently likely to experience
> at least one block of corruption in the life of the drive. And that
> it tends to get worse up until drive failure. But there is much less
> chance to detect this, if the file system isn't also checksumming the
> vastly larger payload on a drive: the data.

One of the problems in this whole discussion is the assumption that filesystem
inconsistencies only arise from disk bitflips etc; that's just not the case.

Look, I'm just providing evidence of what I've found when re-evaluating the
btrfs administration/repair tools.  I've found them to be quite weak.

From what I've gathered from these responses, btrfs is unique in that it is
/expected/ that if anything goes wrong, the administrator should be prepared
to scrape out remaining data, re-mkfs, and start over.  If that's acceptable
for the Fedora desktop, that's fine, but I consider it a risk that should not
be ignored when evaluating this proposal.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/3/20 1:41 PM, Chris Murphy wrote:
> SSDs can fail in weird ways. Some spew garbage as they're failing,
> some go read-only. I've seen both. I don't have stats on how common it
> is for an SSD to go read-only as it fails, but once it happens you
> cannot fsck it. It won't accept writes. If it won't mount, your only
> chance to recover data is some kind of offline scrape tool. And Btrfs
> does have a very very good scrape tool, in terms of its success rate -
> UX is scary. But that can and will improve.

Ok, you and Josef have both recommended the btrfs restore ("scrape")
tool as a next recovery step after fsck fails, and I figured we should
check that out, to see if that alleviates the concerns about
recoverability of user data in the face of corruption.

I also realized that mkfs of an image isn't representative of an SSD
system typical of Fedora laptops, so I added "-m single" to mkfs,
because this will be the mkfs.btrfs default on SSDs (right?).  Based
on Josef's description of fsck's algorithm of throwing away any
block with a bad CRC this seemed worth testing.

I also turned fuzzing /down/ to hitting 2048 bytes out of the 1G
image, or a bit less than 1% of the filesystem blocks, at random.
This is 1/4 the fuzzing rate from the original test.

So: -m single, fuzz 2048 bytes of 1G image, run btrfsck --repair,
mount, mount w/ recovery, and then restore ("scrape") if all that
fails, see what we get.

I ran 50 loops, and got:

46 btrfsck failures
20 mount failures

So it ran btrfs restore 20 times; of those, 11 runs lost all or
substantially all of the files; 17 runs lost at least 1/3 of the 
files.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/1/20 2:50 PM, Josef Bacik wrote:
> On 7/1/20 2:24 PM, Matthew Miller wrote:
>> On Wed, Jul 01, 2020 at 06:54:02AM +, Zbigniew Jędrzejewski-Szmek wrote:
>>> Making btrfs opt-in for F33 and (assuming the result go well) opt-out for 
>>> F34
>>> could be good option. I know technically it is already opt-in, but it's not
>>> very visible or popular. We could make the btrfs option more prominent and
>>> ask people to pick it if they are ready to handle potential fallout.
>>
>> I'm leaning towards recommending this as well. I feel like we don't have
>> good data to make a decision on -- the work that Red Hat did previously when
>> making a decision was 1) years ago and 2) server-focused, and the Facebook
>> production usage is encouraging but also not the same use case. I'm
>> particularly concerned about metadata corruption fragility as noted in the
>> Usenix paper. (It'd be nice if we could do something about that!)
>>
> 
> There's only so much we can do about this.  I've sent up patches to ignore 
> failed global trees to allow users to more easily recover data in case of 
> corruption in the case of global trees, but as they say if only 1 bit is off 
> in a node, we throw the whole node away.  And throwing a node away means you 
> lose access to any of its children, which could be a large chunk of the file 
> system.
> 
> This sounds like a "wtf, why are you doing this btrfs?" sort of thing, but 
> this is just the reality of using checksums.  It's a checksum, not ECC.  We 
> don't know _which_ bits are fucked, we just know somethings fucked, so we 
> throw it all away.  If you have RAID or DUP then we go read the other copy, 
> and fix the broken copy if we find a good copy.  If we don't, well then 
> there's nothing really we can do.

There is often a path forward when a bad metadata checksum is detected.
i.e. e2fsck:

scan_extent_node() {
...

/* Failed csum but passes checks?  Ask to fix checksum. */
if (failed_csum &&
fix_problem(ctx, PR_1_EXTENT_ONLY_CSUM_INVALID, pctx)) {
pb->inode_modified = 1;
pctx->errcode = ext2fs_extent_replace(ehandle, 0, );
if (pctx->errcode)
return;
}

it does similarly for many types of metadata:

/* inode passes checks, but checksum does not match inode */
#define PR_1_INODE_ONLY_CSUM_INVALID0x010068
--
/* Inode extent block passes checks, but checksum does not match extent */
#define PR_1_EXTENT_ONLY_CSUM_INVALID   0x01006A
--
/* Inode extended attribute block passes checks, but checksum does not
 * match block. */
#define PR_1_EA_BLOCK_ONLY_CSUM_INVALID 0x01006C
--
/* dir leaf node passes checks, but fails checksum */
#define PR_2_LEAF_NODE_ONLY_CSUM_INVALID0x02004D

Does btrfsck really never attempt to salvage a metadata block with a bad CRC by
validating its fields?

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/2/20 4:44 PM, Josef Bacik wrote:
> We're talking about this issue like it's reasonable that xfs and ext4 are 
> going to allow the user to get back a bunch of data they don't know is ok or 
> not. We're also talking about it like the user should be able to carry on his 
> happy merry way.  In these cases the drive is dying and needs to be shredded, 
> and a new install needs to happen and a restore from backups needs to happen. 
>  Is the btrfs failure much less user friendly?  No doubt about it.  Is it any 
> comfort at all when a user shows up and we say "where are your backups" and 
> they say "what backups?", no.  But if we're going to talk about this like 
> ext4 and xfs are much better because they give you the _appearance_ that your 
> data is fine, that's a bit disingenuous.

If I had talked about it like that, it would have been disingenuous.

But I didn't; this was an investigation of resiliency to metadata corruption, 
not data error detection, and to what degree metadata corruption can render 
files or even entire filesystems unreachable after normal administrative 
recovery efforts.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/2/20 3:58 PM, José Abílio Matos wrote:
> On Thursday, 2 July 2020 21.38.46 WEST Eric Sandeen wrote:
>> 3 files in lost+found, -1 files gone/unreachable
> 
> This last line from the xfs test seems suspicious (the -1 file gone). :-)

It is weird, but it shows I didn't fudge the numbers ;)

directory repair may have inadvertently created a file or something, not sure.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/1/20 12:50 PM, Chris Murphy wrote:

...

> Integrity checking is highly valued by some and less by others.
> Considering that we know hardware isn't 100% reliable, and doesn't
> always report its own failures as expected, and hence why most file
> systems now at least checksum metadata, it's not persuasive to me that
> the data should be left unchecked, and corruption ought to be handled
> by user space somehow.

There's a flip side to this coin - in my experience, if the right btrfs
metadata blocks experience this disk corruption, there can be
a complete inability to recover the btrfs filesystem from that error -
i.e. it won't mount, and btrfsck --repair won't get it to a mountable
state.

So if we're saying disk corruption happens often enough that data
checksumming is critical, then it happens often enough that metadata
recovery is at least as critical.

I've been trying to quantify this and have not come up with a particularly
compelling test scenario, because it involves purposefully (though at random)
corrupting enough blocks on a filesystem image that a critical block gets
hit, so it looks synthetic.  But the net result is frequently a filesystem
where btrfsck and/or mount fails, and at first blush this type of failure
happens much more often than on other filesystems.[1]

I think Josef has alluded to this situation as well.  To me, that's a big
concern.  Not trying to be a wet blanket here but I think this needs to be
carefully investigated and evaluated to understand what impact it may have
on Fedora btrfs users and their ability to recover their data in the face
of metadata corruption, because it looks to me like a definite btrfs weak
spot.

-Eric

[1] some details - I used the mangle.c fuzzer from fsfuzzer, and modified
it so that it corrupts 8192 bytes of an image, which in fs terms
can be up to 8192 filesystem blocks.  I also avoided the first 4k so that
any filesystem signature was not damaged.

I then ran a loop where I created a 1G base image, populated it, fuzzed it
in this way, (so up to 3% of blocks were damaged) and ran the filesystem's
fsck utility  (in btrfs' case, btrfsck --repair) and then tried to mount
(in btrfs' case, with bare mount, then -o usebackuproot if mount failed). 
If it mounted, I used "find | wc" to see how many files were reachable vs
the original image.

If either fsck or mount reports an exit code that reflects failure to
complete properly, I recorded that.

It was a quick hack, and it's not beautiful, so there are probably holes
to be poked in it; if you want to look, I threw the bash script and the C
source up at https://people.redhat.com/esandeen/fsckfuzzer/

Running 10 loops on each of btrfs, ext4, and xfs I got results that look
like this (ext4 always creates empty lost+found so it will always find at
least 1 file there)

btrfs

fsck failed
0 files in lost+found, 628 files gone/unreachable
0 files in lost+found, 0 files gone/unreachable
526 files in lost+found, 9 files gone/unreachable
595 files in lost+found, 55 files gone/unreachable
53 files in lost+found, 8 files gone/unreachable
57 files in lost+found, 44 files gone/unreachable
fsck failed
7 files in lost+found, 1491 files gone/unreachable
fsck failed, mount failed
fsck failed, mount failed
88 files in lost+found, 40 files gone/unreachable
== 4 fsck failures, 2 mount failures

ext4

1 files in lost+found, 0 files gone/unreachable
1 files in lost+found, 0 files gone/unreachable
164 files in lost+found, 2 files gone/unreachable
1 files in lost+found, 0 files gone/unreachable
1 files in lost+found, 0 files gone/unreachable
1 files in lost+found, 1 files gone/unreachable
1 files in lost+found, 0 files gone/unreachable
9 files in lost+found, 1 files gone/unreachable
1 files in lost+found, 0 files gone/unreachable
1 files in lost+found, 0 files gone/unreachable
== 0 fsck failures, 0 mount failures

xfs

0 files in lost+found, 1 files gone/unreachable
0 files in lost+found, 0 files gone/unreachable
958 files in lost+found, 629 files gone/unreachable
0 files in lost+found, 0 files gone/unreachable
2 files in lost+found, 0 files gone/unreachable
0 files in lost+found, 1 files gone/unreachable
0 files in lost+found, 0 files gone/unreachable
0 files in lost+found, 0 files gone/unreachable
8 files in lost+found, 1 files gone/unreachable
3 files in lost+found, -1 files gone/unreachable
== 0 fsck failures, 0 mount failures


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/1/20 9:24 AM, Josef Bacik wrote:
> On 7/1/20 7:49 AM, Steven Whitehouse wrote:
>> Hi,
>>
>> On 01/07/2020 12:09, Zbigniew Jędrzejewski-Szmek wrote:
>>> On Wed, Jul 01, 2020 at 11:28:10AM +0100, Steven Whitehouse wrote:
 Hi,

 On 01/07/2020 07:54, Zbigniew Jędrzejewski-Szmek wrote:
> On Mon, Jun 29, 2020 at 03:15:23PM -0400, Solomon Peachy wrote:
>> So yes, I think an explicit "let's all test btrfs (as anaconda
>> configures it) before we make it default" period is warranted.
>>
>> Perhaps one can argue that Fedora has already been doing that for the
>> past two years (since 2018-or-later-btrfs is what everyone with positive
>> results appears to be talking about), but it's still not clear that
>> those deployments utilize the same feature set as Fedora's defaults, and
>> how broad the hardware sample is.
> Making btrfs opt-in for F33 and (assuming the result go well) opt-out for 
> F34
> could be good option. I know technically it is already opt-in, but it's 
> not
> very visible or popular. We could make the btrfs option more prominent and
> ask people to pick it if they are ready to handle potential fallout.
>
> Normally we just switch the default or we don't, without half measures. 
> But
> the fs is important enough and complicated enough to be extra careful 
> about
> any transitions.
>
> Zbyszek
 Indeed, it is an important point, and taking care is very important
 when dealing with other people's data, which is in effect what we
 are discussing here.

 When we looked at btrfs support in RHEL, we took quite a long time
 over it. In fact I'm not quite sure how long, since the process had
 started before I was involved, but it was not a decision that was
 made quickly, and a great deal of thought went into it. It was
 difficult to get concrete information about the stability aspects at
 the time. Just like the discussions that have taken place on this
 thread, there was a lot of anecdotal evidence, but that is not
 always a good indicator. Since time has passed since then, and there
 is now more evidence, this part of the process should be easier.
 That said to get a meaningful comparison then ideally one would want
 to compare on the basis of user populations of similar size and
 technical skill level, and look not just at the overall number of
 bugs reported, but at the rate those bugs are being reported too.
>>> Yeah. I have no doubt that the decision was made carefully back then.
>>> That said, time has passed, and btrfs has evolved and our use cases
>>> have evolved too, so a fresh look is good.
>>>
>>> We have https://fedoraproject.org/wiki/Changes/DNF_Better_Counting,
>>> maybe this could be used to collect some statistics about the fs type
>>> too.
>>
>> Yes, and also the questions that Fedora is trying to answer are different 
>> too. So I don't think that our analysis for RHEL is applicable here in 
>> general. The method that we went through, in general terms, may potentially 
>> be helpful.
>>
>>
 It is often tricky to be sure of the root cause of bugs - just
 because a filesystem reports an error doesn't mean that it is at
 fault, it might be a hardware problem, or an issue with volume
 management. Figuring out where the real problem lies is often very
 time consuming work. Without that work though, the raw numbers of
 bugs reported can be very misleading.
 It would be worth taking that step here, and
 asking each of the spins what are the features that they would most
 like to see from the storage/fs stack. Comparing filesystems in the
 abstract is a difficult task, and it is much easier against a
 context. I know that some of the issues have already been discussed
 in this thread, but maybe if someone was to gather up a list of
 requirements from those messages then that would help to direct
 further discussion,
>>> Actually that part has been answered pretty comprehensively. The split
>>> between / and /home is hurting users and we completely sidestep it
>>> with this change. The change page lists a bunch of other benefits,
>>> incl. better integration with the new resource allocation mechanisms
>>> we have with cgroups2. So in a way this is a follow-up to the
>>> cgroupsv2-by-default change in F31. Snapshots and subvolumes also give
>>> additional powers to systemd-nspawn and other tools. I'd say that the
>>> huge potential of btrfs is clear. It's the possibility of the loss of
>>> stability that is my (and others') worry and the thing which is hard
>>> to gauge.
>>>
>>> Zbyszek
>>
>> If the / and /home split is the main issue, then dm-thin might be an 
>> alternative solution, and we should check to see if some of the issues 
>> listed on the change page have been addressed. I'm copying in Jon for 
>> additional comment on that. Are those btrfs benefits 

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/1/20 4:08 PM, Neal Gompa wrote:
> On Wed, Jul 1, 2020 at 5:06 PM Eric Sandeen  wrote:
>>
>> On 7/1/20 11:53 AM, Michael Catanzaro wrote:
>>> On Wed, Jul 1, 2020 at 4:25 pm, Nicolas Mailhot via devel 
>>>  wrote:
>>>> Actually this split is a godsend because you can convince anaconda to
>>>> leave your home alone when reinstalling, while someone always seems too
>>>> invent a new Fedora change that justifies the reformatting of /.
>>>>
>>>> Good luck dealing with user data the next time workstation (or any
>>>> other group) feels the / filesystem should change, once you've put user
>>>> data on the same mount point
>>>
>>> So for the avoidance of doubt: if the btrfs change is rejected, we are 
>>> almost certain to put everything on the same mount point. We haven't 
>>> approved this yet, but odds are very high IMO. The options we are seriously 
>>> considering for our default going forward are (a) btrfs, (b) failing that, 
>>> probably ext4 all one big partition without LVM, (c) less-likely, maybe xfs 
>>> all one big partition without LVM. This is being discussed in 
>>> https://pagure.io/fedora-workstation/issue/152
>>>
>>> We have a high number of complaints from developers running out of space on 
>>> / with plenty of space left on /home (happens to me all the time). The 
>>> opposite scenario is a problem too. Separate mountpoints by default is just 
>>> not a good default, sorry. Ensuring users don't run out of space due to bad 
>>> partitioning is more important than keeping /home during reinstall IMO. But 
>>> with btrfs, then /home will just be a subvolume so we can have our cake and 
>>> eat it too.
>>
>> This can be mitigated with directory (project) quotas, btw.
>>
>> On XFS, exceeding a directory tree quota even yields ENOSPC.
>> (on ext4, it's EDQUOT right now.)
>>
>> So one big / partition including /home, with a directory quota set
>> on /home at 20G, will yield ENOSPC when home contains 20G and will now
>> allow / to get filled with user files.
>>
>> It's also trivial to adjust the directory quota on /home up or down, as
>> needed.
>>
>> It's another cake eating-and-having option which is a pretty trivial
>> thing to implement.
> 
> This does not solve the "Anaconda will blow away /home because it's
> technically part of /" problem, though. Btrfs subvolumes do.
> 
> Directory quotas only protect against space contention, and while
> Btrfs quotas do the same thing, we're deliberately not proposing
> setting those up because we want space allocation to be flexible.

I was not proposing directory quotas as any protection against mkfs of the
root device, of course.  Changing that behavior in Anaconda would be another
rather minor change as well, i.e. the equivalent of "rm -rf /usr /var/ ..."
instead of mkfs at reinstall time.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 7/1/20 11:53 AM, Michael Catanzaro wrote:
> On Wed, Jul 1, 2020 at 4:25 pm, Nicolas Mailhot via devel 
>  wrote:
>> Actually this split is a godsend because you can convince anaconda to
>> leave your home alone when reinstalling, while someone always seems too
>> invent a new Fedora change that justifies the reformatting of /.
>>
>> Good luck dealing with user data the next time workstation (or any
>> other group) feels the / filesystem should change, once you've put user
>> data on the same mount point
> 
> So for the avoidance of doubt: if the btrfs change is rejected, we are almost 
> certain to put everything on the same mount point. We haven't approved this 
> yet, but odds are very high IMO. The options we are seriously considering for 
> our default going forward are (a) btrfs, (b) failing that, probably ext4 all 
> one big partition without LVM, (c) less-likely, maybe xfs all one big 
> partition without LVM. This is being discussed in 
> https://pagure.io/fedora-workstation/issue/152
> 
> We have a high number of complaints from developers running out of space on / 
> with plenty of space left on /home (happens to me all the time). The opposite 
> scenario is a problem too. Separate mountpoints by default is just not a good 
> default, sorry. Ensuring users don't run out of space due to bad partitioning 
> is more important than keeping /home during reinstall IMO. But with btrfs, 
> then /home will just be a subvolume so we can have our cake and eat it too.

This can be mitigated with directory (project) quotas, btw.

On XFS, exceeding a directory tree quota even yields ENOSPC.
(on ext4, it's EDQUOT right now.)

So one big / partition including /home, with a directory quota set
on /home at 20G, will yield ENOSPC when home contains 20G and will now
allow / to get filled with user files.

It's also trivial to adjust the directory quota on /home up or down, as
needed.

It's another cake eating-and-having option which is a pretty trivial
thing to implement.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 6/29/20 1:47 PM, Josef Bacik wrote:
 Just to be clear here, the choice of XFS here is purely based on
 performance, not on the reliability of the file systems, right?
 (So it's not “all the really important data is stored in XFS”.)

>>>
>>> Yes that's correct.  At our scale everything falls over, including XFS, and 
>>> as I've stated elsewhere in this thread we actually see a higher rate of 
>>> failure (relative to the install size) with XFS.  The databases we use 
>>> already do all of the fancy things that btrfs does in the application.  If 
>>> we could get away with it we'd just use raw disks for those applications. 
>>> and in fact may do that in the future.  Thanks,
>>
>> Josef, with my XFS hat on, are these recent failures?  Have they
>> all been reported to the XFS list?
>>
>> It makes sense to look at reliability in the context of this thread, but
>> offering "btrfs fails less often than XFS for us" without any context
>> (what kind of failure, what kernel, when, etc) doesn't help much, it's
>> just more anecdotes.
>>
> 
> Yup this is why I try to avoid talking about other file systems.  This 
> shouldn't be interpreted as "XFS drools, btrfs rules!", just that in our own 
> environment, btrfs does not fail at any significant rate higher than xfs.
> 
> Xfs is used in completely different workloads, and with completely different 
> (much better) hardware.
> 
> And the reason they haven't been brought up to the list is because it fails 
> at such a low rate that I didn't even realize we were having xfs reprovisions 
> until I went and looked at the data.  So far of the 15 machines that fell 
> over, 10 of them appear to be hardware related.  The other 5 have logs that 
> are in a different database that take longer to pull out.  Thanks,
> 
> Josef

Thanks for the context, Josef, I appreciate it.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 6/29/20 12:44 PM, Mark Otaris wrote:
> That’s one fewer reason not to use XFS then. It seems
> Documentation/admin-guide/cgroup-v2.rst was not updated and still says
> only ext2, ext4, and btrfs have writeback implemented.

Interesting, thanks for the heads up - I'll get that fixed.

Looks like f2fs also supports it; the dangers of listing details
like that in a doc file I guess :(

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 6/29/20 8:39 AM, Josef Bacik wrote:
> On 6/29/20 5:33 AM, Florian Weimer wrote:
>> * Josef Bacik:
>>
>>> That being said I can make btrfs look really stupid on some workloads.
>>> There's going to be cases where Btrfs isn't awesome.  We still use xfs
>>> for all our storage related tiers (think databases).  Performance is
>>> always going to be workload dependent, and Btrfs has built in overhead
>>> out the gate because of checksumming and the fact that we generate far
>>> more metadata.
>>
>> Just to be clear here, the choice of XFS here is purely based on
>> performance, not on the reliability of the file systems, right?
>> (So it's not “all the really important data is stored in XFS”.)
>>
> 
> Yes that's correct.  At our scale everything falls over, including XFS, and 
> as I've stated elsewhere in this thread we actually see a higher rate of 
> failure (relative to the install size) with XFS.  The databases we use 
> already do all of the fancy things that btrfs does in the application.  If we 
> could get away with it we'd just use raw disks for those applications. and in 
> fact may do that in the future.  Thanks,

Josef, with my XFS hat on, are these recent failures?  Have they
all been reported to the XFS list?

It makes sense to look at reliability in the context of this thread, but
offering "btrfs fails less often than XFS for us" without any context
(what kind of failure, what kernel, when, etc) doesn't help much, it's
just more anecdotes.

Thanks,
-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: User experience issue on btrfs

[Eric Sandeen]

On 6/29/20 3:19 AM, John M. Harris Jr wrote:
> On Monday, June 29, 2020 1:09:16 AM MST Markus Larsson wrote:
>> On 29 June 2020 08:26:21 CEST, "John M. Harris Jr" 
>> wrote:
>>> On Sunday, June 28, 2020 5:37:08 PM MST Chris Adams wrote:
>>>
 Once upon a time, John M. Harris Jr  said:


> XFS proved to be troublesome, and still is up to the latest of RHEL7.
> It's
> not  uncommon to have to run xfs_repair on smaller XFS partitions,
> especially / boot. I'm not sure if btrfs has the same issue there?



 [citation needed]

 I haven't run xfs_repair in probably 15 years (and so never on Fedora or
 RHEL/CentOS).
>>>
>>>
>>> I haven't had time to figure out why the RHEL systems I have that are 
>>> (mistakenly I assume, though they were created before I was hired) using
>>> XFS  run into that issue, after about a month, they report 100% disk
>>> space utilization on /boot, and I've gotta run xfs_repair in order to fix
>>> that. In the unlikely event that I have the time to figure out why, before
>>> I just re- install them (which is already planned), I'd be happy to follow
>>> up with a citation. :)
>>
>>
>> That is very odd. I haven't seen it once in over a decade in an environment
>> with thousands of machines. Very interesting though, I think I will have
>> to try to replicate this. Is there anything special about them like odd
>> partition layout etc? 
> 
> I can't confirm at the moment, but I'm pretty sure /boot is a 1GiB (maybe 
> 2GiB) XFS partition. I've marked your message as "TODO" in my client, so I 
> can 
> get you more info tomorrow if you're interested.

FWIW, I've seen this occasionally on one of my development machines, but 
figured it
could be related to the fact that this box runs development/testing filesystems 
all
day long, and I hadn't dug into it yet.  It would have been very helpful if 
you'd
filed a bug for the issue, so that we knew it had been seen in the wild, and 
could
raise the priority.

One of my least favorite things, TBH, is people reporting in threads like this 
that
they've had this or that problem with a filesystem, but they never filed a bug 
with
developers to get it noticed or fixed.

(If you have filed the bug, kindly share the link here) ;)

Thanks,
-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Eric Sandeen]

On 6/29/20 1:31 AM, Mark Otaris wrote:
> The master branch for cp now defaults to copy-on-write on filesystems
> that support reflinks, which should make copies more efficient if
> Fedora starts using btrfs:
> https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=25725f9d41735d176d73a757430739fb71c7d043.
> 
> Dolphin and KIO also seem like they will start doing this:
> https://bugs.kde.org/show_bug.cgi?id=326880,
> https://invent.kde.org/frameworks/kio/commit/c2faaae697f11ee600989b67b4406981838ae628.
> 
> Beyond these recent changes, there are many other reasons to use
> btrfs, such as that Podman has a btrfs driver that might make
> containers more efficient, that ostree makes limited use of reflinks
> when they are available, that many filesystem options can be changed
> and new features and better defaults used even after the filesystem
> was initially created, that resize operations can be done online, and
> that there are uniform checksums on all metadata blocks, giving
> guarantees against corruption.
> 
> XFS also has reflinks, but lacks many features of btrfs, and switching
> from ext4 to XFS would mean losing cgroup writeback. 

That's incorrect:

commit adfb5fb46af059387eca0fce1d8cd8733f9ee3a0
Author: Christoph Hellwig 
Date:   Fri Jun 28 19:30:22 2019 -0700

xfs: implement cgroup aware writeback

Link every newly allocated writeback bio to cgroup pointed to by the
writeback control structure, and charge every byte written back to it.

Tested-by: Stefan Priebe - Profihost AG 
Signed-off-by: Christoph Hellwig 
Reviewed-by: Darrick J. Wong 
Signed-off-by: Darrick J. Wong 

-Eric

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: I would like to propose that we turn on XFS Reflink in Fedora 29 by default

[Eric Sandeen]

On 4/29/18 11:45 AM, Eric Sandeen wrote:
>> Assuming that the plan is to leave it enabled in F-29 on branching and
>> have it ship enabled in F-29 I agree, if the intention is to leave it
>> enabled in rawhide and disable it on branching then the Change
>> Proposal mechanism isn't the way to ensure wider knowledge.
> I'd be happy to have it left on for F29, it /might/ even be default upstream
> by the time F29 ships.  I'll need to re-familiarize myself w/ the 
> "System-Wide Change Proposal" process, I guess.
> 
> FWIW, turning it on has very little effect until it's actually used, so it
> really should not be destabilizing for most, even if bugs lurk.  ;)
> 
> -Eric

Getting back on topic (back from the Shrink Wars), I'd like to get this
turned on.  I'd prefer to do this via a soon-to-land patch which enables
config files for mkfs, so that distros can set their own defaults without
having to patch source.  I'm going to give it a bit of time to see if we
can get that upstream, and then will see about simply shipping that
patch along w/ a config file that enables reflink for F29.

Thanks,
-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: I would like to propose that we turn on XFS Reflink in Fedora 29 by default

[Eric Sandeen]

On 5/2/18 8:42 AM, Ian Pilcher wrote:
> On 05/02/2018 08:25 AM, Eric Sandeen wrote:
>> I've always seen the need for shrink as an indicator that someone had
>> poor planning along the way, or insufficient tools for provisioning to
>> start with.  Sure, there are exceptions, but in general who needs shrink
>> on a regular basis?
> 
> The point isn't so much that you need it on a regular basis, it's that
> when you need it, you *really* need it.

Given that it is exception activity, dump/mks/restore is also a less
convenient but more robust solution to the problem.

I mostly want to point out that shrink permanently de-optimizes your
filesystem, and carries some risk as well, especially if you have no
backups.

> I'll buy the poor planning argument on a server that does pretty much
> the same thing for the entirety of its life/deployment, but the case of
> a laptop/desktop that goes years without being reinstalled, and then
> unexpectedly needs tens of gigabytes of space to bisect a kernel bug
> is very different.

If you're putting your years-old root or home filesystem at risk to bisect
a bug I'd humbly suggest that an external or additional disk might be more
suited to the task.

I don't have any real horse in this race - if Fedora feels that shrink
capability trumps features like reflink, that's fine.  Just offering my
thoughts on the matter, and trying to point out that shrink has its
downsides.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: I would like to propose that we turn on XFS Reflink in Fedora 29 by default

[Eric Sandeen]

On 5/2/18 7:15 AM, Neal Gompa wrote:
> On Wed, May 2, 2018 at 5:36 AM Marius Vollmer 
> wrote:
> 
>> Neal Gompa  writes:
> 
>>> And there's still the fun restriction of XFS not being able to shrink.
> 
>> But note that even ext4 can't shrink while being mounted.
> 
> But it can shrink when it's not. This is incredibly important for being
> able to deal with resizing both / and /home at the same time, or even
> trying to make space for multi-booting (typically with Windows but some
> people do other OSes too).

I've always seen the need for shrink as an indicator that someone had
poor planning along the way, or insufficient tools for provisioning to
start with.  Sure, there are exceptions, but in general who needs shrink
on a regular basis?

Shrink is actually pretty damaging to the filesystem; it takes all the
locality that the allocator tried to provide, and scatters it to the
wind.  The result is a stitched-together mess. 

Not only that, but wouldn't any sane administrator with important data
to take care of make a backup before an invasive action like shrink?

And if you have a backup, you're halfway to mkfs & restore, which will
leave you in a much better place.

So yes, you can shrink ext4, but it really should be seen as a last resort
IMHO.  I know it can be expedient at times, but I'm not sure people really
consider the downsides of the action.  On the surface, "yay it's smaller
now!" but a bit more investigation shows that it's a de-optimizing,
potentially dangerous administrative action.  Just my $0.02.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: I would like to propose that we turn on XFS Reflink in Fedora 29 by default

[Eric Sandeen]

On 4/30/18 1:16 PM, Neal Gompa wrote:
> On Mon, Apr 30, 2018 at 2:14 PM Jason L Tibbitts III 
> wrote:
> 
>>> "CW" == Colin Walters  writes:
> 
>> CW> I'd say it makes sense to revisit the default here globally in
>> CW> Anaconda.
> 
>> Maybe.  Have the issues which made XFS less suitable for use on laptops
>> been resolved?  The primary one I recall was that each mounted
>> filesystem would have a corresponding kernel thread doing about 20
>> wakeups per second.  This was not really good for battery life and power
>> consumption in general.
> 
>> Last I checked (which was 2016 or so) those wakeups were slated to be
>> around for a while longer.
>> https://www.spinics.net/lists/xfs/msg40210.html

As we discussed on IRC, I think idle filesystems won't get these wakeups;
if this is still a concern, investigation of the actual effects on battery
life (if any) are recommended.

> 
> And there's still the fun restriction of XFS not being able to shrink. It's
> not particularly important in the server case, but in the desktop/laptop
> case, it happens enough in my experience that I'm not sure I'd want a
> default filesystem that can't shrink.

XFS realistically is not going to get shrink.  Fedora will need to decide if
the other capabilities & features outweigh the lack of shrink capability.

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: I would like to propose that we turn on XFS Reflink in Fedora 29 by default

[Eric Sandeen]

On 4/29/18 6:23 AM, Peter Robinson wrote:
>>>> On 28/04/18 14:55, Peter Robinson wrote:
>>>>> On Sat, Apr 28, 2018 at 11:09 AM, Daniel Walsh <dwa...@redhat.com>
>>>>> wrote:
>>>>>> We are adding some features to container projects for User Namespace
>>>>>> support
>>>>>> that can take advantage of XFS Reflink.  I have talked to some of the
>>>>>> XFS
>>>>>> Reflink kernel engineers in Red Hat and they have informed me that
>>>>>> they
>>>>>> believe it is ready to be turned on by default.
>>>>>>
>>>>>> I am not sure who in Red Hat I should talk to about this? Whether we
>>>>>> should
>>>>>> turn it on in the installer or in the mkfs.xfs command?
>>>>>>
>>>>>> Who should I be talking to?  To make this happen.
>>>>> I would speak to Eric Sandeen I believe he's the Red Hat maintainer
>>>>> (or one of them) of XFS.
>>>>>
>>>>> Peter
>>>> Indeed, and also we should look at this in the context of what is done
>>>> for upstream. Ideally Fedora would just inherit the changes there, and 
>>>> there
>>>> should not be anything special required for Fedora,
>>>>
>>>> Steve.
>>>
>>> So, for context, I am the upstream maintainer of xfsprogs as well as for
>>> Fedora xfsprogs.
>>>
>>> Historically, new features in XFS have gone from "Experimental" (i.e.
>>> under
>>> development), then dropped Experimental (development is ~done) but still
>>> optional,
>>> and eventually default.  We do this very conservatively, to give bugs a
>>> chance
>>> to shake out, which is one of the reasons XFS has a good reputation for
>>> /not/
>>> eating your data.
>>>
>>> Reflink on XFS only recently dropped "Experimental" and is not yet default
>>> upstream;
>>> it won't be default upstream for some time to come - think on the order of
>>> months.
>>>
>>> However, we do want to give reflink more exposure, and so jumping the gun
>>> a bit and
>>> turning it on for rawhide / Fedora 29 is probably a good idea.
>>>
>>> I'm mostly ok with patching it on by default in mkfs.xfs; it does confuse
>>> things a bit
>>> when "our" version behaves fundamentally differently from upstream, but
>>> it's probably
>>> the right thing to do here.  I'll make sure that none of the other xfs
>>> developers have
>>> strong objections, and if not, I'll patch it in for fedora 29.
>>>
>>> Unless this should be a full blown Feature?  If so, I'm ok with following
>>> that path
>>> as well.
>>
>>
>>
>> XFS is the default filesystem on Fedora Server Edition, so yes: I think we
>> should really have a System-Wide Change Proposal submitted for this,
>> primarily to ensure that the information is spread widely (Change Proposals
>> like this are picked up by Fedora Marketing and tech news, so it’ll be more
>> widely dispersed than just on the fedora-devel list).
> 
> Assuming that the plan is to leave it enabled in F-29 on branching and
> have it ship enabled in F-29 I agree, if the intention is to leave it
> enabled in rawhide and disable it on branching then the Change
> Proposal mechanism isn't the way to ensure wider knowledge.

I'd be happy to have it left on for F29, it /might/ even be default upstream
by the time F29 ships.  I'll need to re-familiarize myself w/ the 
"System-Wide Change Proposal" process, I guess.

FWIW, turning it on has very little effect until it's actually used, so it
really should not be destabilizing for most, even if bugs lurk.  ;)

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: I would like to propose that we turn on XFS Reflink in Fedora 29 by default

[Eric Sandeen]

On 4/28/18 9:21 AM, Steven Whitehouse wrote:
> Hi,
> 
> 
> On 28/04/18 14:55, Peter Robinson wrote:
>> On Sat, Apr 28, 2018 at 11:09 AM, Daniel Walsh <dwa...@redhat.com> wrote:
>>> We are adding some features to container projects for User Namespace support
>>> that can take advantage of XFS Reflink.  I have talked to some of the XFS
>>> Reflink kernel engineers in Red Hat and they have informed me that they
>>> believe it is ready to be turned on by default.
>>>
>>> I am not sure who in Red Hat I should talk to about this? Whether we should
>>> turn it on in the installer or in the mkfs.xfs command?
>>>
>>> Who should I be talking to?  To make this happen.
>> I would speak to Eric Sandeen I believe he's the Red Hat maintainer
>> (or one of them) of XFS.
>>
>> Peter
> Indeed, and also we should look at this in the context of what is done for 
> upstream. Ideally Fedora would just inherit the changes there, and there 
> should not be anything special required for Fedora,
> 
> Steve.

So, for context, I am the upstream maintainer of xfsprogs as well as for
Fedora xfsprogs.

Historically, new features in XFS have gone from "Experimental" (i.e. under
development), then dropped Experimental (development is ~done) but still 
optional,
and eventually default.  We do this very conservatively, to give bugs a chance
to shake out, which is one of the reasons XFS has a good reputation for /not/
eating your data.

Reflink on XFS only recently dropped "Experimental" and is not yet default 
upstream;
it won't be default upstream for some time to come - think on the order of 
months.

However, we do want to give reflink more exposure, and so jumping the gun a bit 
and
turning it on for rawhide / Fedora 29 is probably a good idea.

I'm mostly ok with patching it on by default in mkfs.xfs; it does confuse 
things a bit
when "our" version behaves fundamentally differently from upstream, but it's 
probably
the right thing to do here.  I'll make sure that none of the other xfs 
developers have
strong objections, and if not, I'll patch it in for fedora 29.

Unless this should be a full blown Feature?  If so, I'm ok with following that 
path
as well.

thanks,
-Eric

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: getting kernel-devel added

[Eric Sandeen]

On 9/12/17 10:49 AM, Matthew Miller wrote:
> On Tue, Sep 12, 2017 at 11:35:08AM -0400, Ben Williams wrote:
>> case A) Students are using Fedora on windows in a VM (Vbox in this
>> case) for a class. they are required for said class to install the
>> guest additions. they are constantly running into errors that the
>> guest addidions will not build  (there install does not have
>> kernel-devel install.  They used the F26 release so now have the
>> release kernel and so they install sudo dnf install kernel-devel and
>> still have issues (kernel and kernel-devel versions do not match).
> 
> Isn't the solution here for the guest addons to require kernel-devel?

That was my thought as well, though the addons may be circumventing
the whole rpm/dnf mechanism for all I know.
 
>> Case B) third party video or wireless driver same issue no
>> kernel-devel, no wireless no internet == fix by sneakernet
> 
> Also same?

Well, if you need it for wireless, and you only have wireless, requiring
it doesn't help much if you can't retrieve it over your non-functioning
network card.  Still, I'm not sure that's a problem for Fedora to solve?

OTOH installing kernel-devel by default isn't too heavyweight, I think.

-Eric

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: power management

[Eric Sandeen]

On 4/9/17 4:39 AM, František Zatloukal wrote:
> I had bad experience with enabling powertop' service - USB mice and
> headphones don't work very well with that. But I am using tuned
> (tuned-gtk) for few years and I didn't notice any issues
> (top-battery) profile. I see that my Haswell laptop is using both C2
> and C3 power-saving states.

I had trouble with powertop putting my USB APC UPS (Three TLA's!) into
low power mode - that made it unhappy.  Maybe a blacklisting system
is needed?  It's a pity to not tune all the things that do work
due to the few things that don't ...

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Fedora captive portal page changed output :(

[Eric Sandeen]

On 12/2/16 7:10 PM, Paul Wouters wrote:
> 
> Fedora runs a captive portal check page at:
> 
> http://fedoraproject.org/static/hotspot.txt
> 
> It used to return "OK\n".
> 
> Now it returns "OK" without the newline.

Seems like the file date is still well in the past
(2015-12-15) and does not actually contain a newline;
webserver behavior change?

-Eric
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Orphaning apcupsd

[Eric Sandeen]

On 6/22/16 2:24 PM, Jason L Tibbitts III wrote:
>> "MH" == Michal Hlavinka  writes:
> 
> MH> Hi, my APC UPS died and as I won't be buying new APC UPS, I can no
> MH> longer test and investigate bugs. So apcupsd is free for taking if
> MH> anyone wants it.
> 
> Well, I need it but I don't really _want_ it.  So it would be nice if
> someone else using it would join in.
> 
> I've taken the Fedora branches, and EPEL7.  I might be convinced to take
> the older EPEL branches but only if I can build it without a lot of
> cruft in the spec and only if someone is there to test it.
> 
> I'm cleaning up the spec now.

Eh, I could help.  I don't do anything very fancy with it, and my small
simple UPS is on my mailserver, which I don't really want to do a lot of
powerfail testing on, but at least I have it in use.

So I could help with EPEL if you want; generally though, packages like
this don't get rebased or updated much in EPEL so I doubt it's much
work.

If anyone has a more interesting apcupsd setup, it might make more
sense for them to step up but in a pinch, I'll help.

-Eric
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Imaginary single quotes in ls ?

[Eric Sandeen]

On 6/6/16 11:53 AM, Chris Adams wrote:
> Once upon a time, Eric Sandeen <sand...@redhat.com> said:
>> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commit;h=109b9220cead6e979d22d16327c4d9f8350431cc
>>
>> +  ls now quotes file names unambiguously and appropriate for use in a shell,
>> +  when outputting to a terminal.
> 
> So, when output goes to a TTY, make the output for a shell (but when the
> output is used by a shell, as in a redirect, don't do that).  Brilliant!
> 
> Did they add (yet another) option to disable the dumb behavior?

# export QUOTING_STYLE=literal

I believe. Could go in .bashrc

-Eric
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Imaginary single quotes in ls ?

[Eric Sandeen]

On 6/6/16 11:34 AM, Tom Hughes wrote:
> On 06/06/16 17:32, Tom Hughes wrote:
>> On 06/06/16 17:28, Adam Williamson wrote:
>>
>>> It showed up in F24 a few weeks back, at least in gnome-terminal. I
>>> dunno if it's a g-t thing.
>>
>> No, it's ls. See --quote-name and --quoting-style in ls(1).
> 
> Specifically I think the default for --quoting-style has changed from literal 
> to shell or something.

http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commit;h=109b9220cead6e979d22d16327c4d9f8350431cc

+  ls now quotes file names unambiguously and appropriate for use in a shell,
+  when outputting to a terminal.

-Eric
 
> Tom
> 
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: Imaginary single quotes in ls ?

[Eric Sandeen]

On 6/6/16 11:20 AM, Bernardo Sulzbach wrote:
> On 06/06/2016 12:53 PM, Paul Wouters wrote:
>>
>> paul@thinkpad:/tmp/test$ touch foo bar baz
>> paul@thinkpad:/tmp/test$ touch "touch and go"
>> paul@thinkpad:/tmp/test$ ls -l
>> total 0
>> -rw-rw-r--. 1 paul paul 0 Jun  6 11:48 bar
>> -rw-rw-r--. 1 paul paul 0 Jun  6 11:48 baz
>> -rw-rw-r--. 1 paul paul 0 Jun  6 11:48 foo
>> -rw-rw-r--. 1 paul paul 0 Jun  6 11:49 'touch and go'
>>
>>
> 
> I failed to replicate without flags or with -l, -a, or both
> 
> on
> 
> kernel 4.4.9-300.fc23.x86_64
> bash-4.3.42-3.fc23.x86_64
> gnome-terminal-3.18.3-1.fc23.x86_64

And which coreutils?

http://coreutils.gnu.narkive.com/zXX0Oazc/broken-ls-in-coreutils-8-25

-Eric
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: [Bug 1201978] dracut assumes BIOS time is UTC closed without fixing again

[Eric Sandeen]

On 5/1/15 2:18 AM, Till Maas wrote:
 On Thu, Apr 30, 2015 at 05:11:20PM -0700, Adam Williamson wrote:
 
 What doesn't work is rtc-in-local in early-boot, that's all. And that
 doesn't matter really, except if you are crazy enough to manually
 enable time-based fsck in ext234, which has been turned off by default
 in fedora since time begain, and even has been turned off upstream
 now, because it's simply a crazy feature. - Lennart, yesterday: 
 https://lists.fedoraproject.org/pipermail/devel/2015-April/210282.html
 
 Why is this a crazy feature? Since Fedora 21 tends to corrupt the root
 filesystem during resume from hibernation, regular full fsck runs make a
 lot of sense to me.

Is there a bug open for that?

Thanks,
-Eric

 Regards
 Till
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: [Bug 1201978] dracut assumes BIOS time is UTC closed without fixing again

[Eric Sandeen]

On 4/30/15 7:58 PM, Matthew Miller wrote:
 On Thu, Apr 30, 2015 at 05:22:20PM -0700, Andrew Lutomirski wrote:
 Maximum mount count:  21
 Last checked: Mon Apr  6 10:42:37 2015
 Check interval:   15552000 (6 months)
 
 The default is definely now 0, but you're right, if you've upgraded for
 a long time, it won't be changed. (You can change it now with tune2fs,
 though, if you want.)
 
 Also, the e2fsck.conf man page suggests that the program has some
 heuristics that assume that the system clock is correct — I'm not sure
 without checking the code if the check interval is really the only one.
 But you might consider setting the `broken_system_clock` option in
 /etc/e2fsck.conf.
 
 Hmmm, in fact, maybe we should consider this the _default_ (at least
 for workstation and desktop spins).

It was/is set as default, and then time-based checks never fired.
So we un-defaulted it, and then hit this stuff.

Such a mess.  :(

I need to get 1202024 resolved, ted checked in a version of a patch
I proposed to try to resolve some of this.

http://git.kernel.org/cgit/fs/ext2/e2fsprogs.git/commit/?id=f096708126412c0569e40cfbd5740729976bf12a

I'll get that pushed to testing tonight, sorry for being slow with it.

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: [Bug 1201978] dracut assumes BIOS time is UTC closed without fixing again

[Eric Sandeen]

On 4/30/15 7:35 PM, Felix Miata wrote:
 Adam Williamson composed on 2015-04-30 17:11 (UTC-0700):
 
 He never mentioned what time-based fsck is that I saw,
 
 What doesn't work is rtc-in-local in early-boot, that's all. And that
 doesn't matter really, except if you are crazy enough to manually
 enable time-based fsck in ext234, which has been turned off by default
 in fedora since time begain, and even has been turned off upstream
 now, because it's simply a crazy feature. - Lennart, yesterday: 
 https://lists.fedoraproject.org/pipermail/devel/2015-April/210282.html
 
 I saw that. It didn't tell me anything I understood, since I don't know what
 time-based fsck is, or more particularly, what enables or disables it, and so
 unlikely myself made any changes in recent weeks or months to cause the
 current boot delays these fscks consistently impose.

well, not since time began; it was 1.42-ish:

commit 3daf592646b668133079e2200c1e776085f2ffaf
Author: Eric Sandeen sand...@redhat.com
Date:   Thu Feb 17 15:55:15 2011 -0600

e2fsprogs: turn off enforced fsck intervals by default

The forced fsck often comes at unexpected and inopportune moments,
and even enterprise customers are often caught by surprise when
this happens.  Because a filesystem with an error condition will
be marked as requiring fsck anyway, I submit that the time-based
and mount-based checks are not particularly useful, and that
administrators can schedule fscks on their own time, or tune2fs
the enforced intervals if they so choose.  This patch disables the
intervals by default, and I've added a new mkfs.conf option to
turn on the old behavior of random, unexpected, time-consuming
fscks at boot time.  ;)

Signed-off-by: Eric Sandeen sand...@redhat.com
Signed-off-by: Theodore Ts'o ty...@mit.edu

(some of us are old enough that our time began before 2011) ;)

Time-based check is ext234's (old default) behavior of forcing a full
fsck just because X days or Y mounts have gone by.

Any filesystem created prior to the above would have these time-based
or mount-based checks enabled by default.

You can see if it's set on your fs by looking at dumpe2fs, i.e.:

# dumpe2fs -h /dev/md0 | grep -i mount count\|Check interval\|check after
dumpe2fs 1.41.12 (17-May-2010)
Mount count:  24
Maximum mount count:  21
Check interval:   15552000 (6 months)
Next check after: Fri Aug  2 15:55:32 2013

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Orphaning NCID (Caller ID server/client)

[Eric Sandeen]

I've not been keeping up with NCID (a caller id server/client system;
pretty neat, being able to send caller ID to mythtv, desktop notifications,
etc).

It needs some systemd love, and has newer upstream releases.  I was using
it on a CentOS 6 server, and have no good way to test it on newer releases.

There aren't a lot of bugs open; it just needs a little help to get it up
to snuff for current Fedora.

So if anyone wants it, have at it.  I'll go push the buttons on the pkgdb.
I don't mind hanging on to EPEL, because, well, it's been 0 work.  ;)
(and I have the system to test those).

Thanks,
-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Why isn't F2FS support in the Kernel?

[Eric Sandeen]

On 12/22/14 8:16 AM, Josh Boyer wrote:
 On Sun, Dec 21, 2014 at 11:09 PM, Gerald B. Cox gb...@bzb.us wrote:
 Yes, I looked at that bug report and the somewhat terse response.  I thought
 I'd post here first before I went the bugzilla route.

 Based upon the information I discovered tonight it seems a bit puzzling it
 isn't included.  Seriously, Ubuntu includes it and we don't?
 Google is using it for the Nexus 9?  The experimental rationale just
 doesn't hold weight - especially since we are allowing for
 BTRFS Raid5/6; which is made out to be toxic.  If it's good enough for
 Google and ahem:  Ubuntu - it's beyond ridiculous we don't have it.
 
 So you looked at a bug that is a year and a half old, around the time
 when F2FS was very new and under a lot of work, and assumed that
 nothing could have possibly changed?  Maybe instead of getting angry
 and incredulous, you could actually leave a comment in the bug or open
 a new RFE bug to have it enabled.  If you do, highlighting your
 findings without the snarky and aggressive tone would probably help
 your case.
 
 josh

FWIW, I see that f2fs-tools is in fedora, but it's a bit old, at v1.2.0
while upstream is at 1.4.0.

(http://git.kernel.org/cgit/linux/kernel/git/jaegeuk/f2fs-tools.git)

If you're agitating for movement in the kernel, might want to give the
userspace pkg maintainer ( echevemaster ) a heads up, too.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Why isn't F2FS support in the Kernel?

[Eric Sandeen]

On 12/22/14 12:12 PM, Gerald B. Cox wrote:
 
 On Mon, Dec 22, 2014 at 9:32 AM, Josh Boyer jwbo...@fedoraproject.org 
 mailto:jwbo...@fedoraproject.org wrote:
 
  Is there any hardware out there that uses it?
 
 Aside from the hardware already mentioned in this thread, which Fedora
 doesn't run on, there might be some generic ARM boards that could use
 it.
 
 
 One use that quickly comes to mind is USB Flash drives... check out:  
 http://en.wikipedia.org/wiki/F2FS

I'm not an F2FS expert by any means, but I think that's correct.

from Neil Brown's LWN writeup a while ago:

 f2fs is not targeted at raw flash devices, but rather at the specific
 hardware that is commonly available to consumers — SSDs, eMMC, SD
 cards, and other flash storage with an FTL (flash translation layer)
 already built in.

I'd encourage those advocating for enabling this to put it through its paces,
and see where it's at, rather than keying off phoronix  ubuntu.

In particular, getting [x]fstests up  running on f2fs shouldn't be too hard,
and it would be good to see if it uncovers any problems.

There was a patch to enable it on the list:

http://oss.sgi.com/archives/xfs/2014-06/msg00030.html

but it's not merged... I'll ask Dave about that.

Fedora succeeds when committed people share their time  effort .. can you
give f2fs a go w/ [x]fstests and report back?  Scratch that itch.  ;)
I can give you a hand running xfstests if you need it.

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Eric Sandeen]

On 10/8/14 8:39 AM, Thorsten Leemhuis wrote:
 On 08.10.2014 14:50, Josh Boyer wrote:
 On Wed, Oct 8, 2014 at 1:52 AM, Thorsten Leemhuis fed...@leemhuis.info 
 wrote:
 Josh Boyer wrote on 07.10.2014 21:15:
 On Tue, Oct 7, 2014 at 2:24 PM, Josef Bacik jo...@toxicpanda.com wrote:
 [...]
 http://kernel.opensuse.org/cgit/kernel-source/tree/patches.suse/btrfs--add-allow_unsupported-module-parameter.patch?h=SLE11-SP3
 http://kernel.opensuse.org/cgit/kernel-source/tree/patches.suse/btrfs--add-allow_unsupported-module-parameter.patch?h=SLE12
 [...] 
 I'm not thrilled with adding that patch to Fedora at all. 
 
 Fully agreed.
 
 […] They get away with this in SLE12 because it's roughly the first time
 btrfs is available in a supported fashion. […]
 
 Well, it's supported since SLE11SP2 already, which is more than two
 years old, but the point in the end is the same, yes. But FWIW, it seems
 that simply how they work afaics, as they do something similar to ext4, too:
 
 http://kernel.opensuse.org/cgit/kernel-source/tree/patches.suse/ext4-unsupported-features.patch?h=SLE12
 
 CU
 knurd
 

Interesting, bigalloc and checksums - yeah, I probably would have chosen those,
too, for now.

I think there's a little difference, though, in that bigalloc in particular,
and checksums to some degree, are really kind of niche / corner case features of
ext4.  (Who here even knew ext4 had a bigalloc feature, raise your hand!)

The list of unsupported btrfs features seems like a lot of core advertised
functionality - no compression, no raid5, no device replace, no btrfs receive...

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Eric Sandeen]

On 10/6/14 8:50 AM, Ric Wheeler wrote:
 On 10/06/2014 08:54 AM, Josef Bacik wrote:
 
 Well that's exactly what it is, go away I'm busy with other stuff
 :).  The fact is I'm the only one who can drive btrfs as the
 default filesystem feature in Fedora, and since I've left Red Hat
 that has become much less of an priority for me.  But my other
 stuff is still mostly related to btrfs, so its not like this has
 just been abandoned, the focus has just shifted and I no longer
 feel like we need to be using btrfs as the default fs in Fedora to
 have a successful project, so it got moved down the priority list.
 It will happen, and when it happens it will be relatively painless
 because Suse will have worked out a lot of the distro esque kinks
 and us at Facebook will have worked out a lot of the at scale
 kinks.  Thanks,
 
 Josef
 
 
 I think that the out of space issues are not that different from any
 file system on write enabled snapshots - it certainly can be
 mysterious and confuse users, but that is something that we have to
 deal with in order to get this kind of sophistication into end users
 hands (documentation? better tooling like the snapper tool, etc?).
 
 One of the harder challenges I think for btrfs is still getting the
 repair tools rock solid - how is our track record these days with
 repairing btrfs after bad things happen :) ?

In my recent (past couple weeks) testing, it's dismal, although a bunch
of fsck patches have shown up on the list which I haven't tried yet.

Doing what I considered to be light fuzzing of a filesystem, and attempting
a btrfsck  mount led to segfaults, aborts, and mount failures the vast
majority of the time - more or less often, depending on the particular metadata
layout.  Other filesystems fared much better in this testing.

And the best-practice repair strategy with btrfs is still not clear to me;
there is btrfs check (aka btrfsck) of course, with various suboptions the
admin should know about: init-csum-tree, init-extent-tree, backup, 
subvol-extents;
several of these are not documented ... and also mount options:
-o degraded, -o recovery, -o skip_balance.  And other tools; btrfs
rescue, with various suboptions - super-recovery, chunk-recover.
There's btrfs scrub, which will work online and Errors are corrected along the
way if possible. and btrfs-rescue, a last-ditch effort to extract files
from the smoking remains of an otherwise unrescuable filesystem.

It may be that I'm just not up to speed on btrfs - I've realized that I can't
be an in-depth expert on 3 different filesystems - but to me, the filesystem
repair and recovery plan for btrfs is not at all reliable or obvious or clear.

(I need to send this same post to the btrfs-list, I guess, but I did want to
point out that it is, to me, a big concern for Fedora decision-making.  btrfs
was first proposed as default in F17, 3 years ago, and ISTR reliable fsck being 
a
gating item back then.  And now here we are...)

Thanks,
-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Eric Sandeen]

On 10/6/14 7:45 AM, Ralf Corsepius wrote:
 On 10/06/2014 02:29 PM, Gene Czarcinski wrote:
 
 Now, there is another question which has not been voiced: what is
 the plan for filessystems in Fedora (and by implication RHEL)?
 Is it BTRFS?  Or, perhaps is it LVM with XFS?  IIRC, some time ago
 it was stated that the plan was to move to BTRFS.  It is not clear
 to me that everyone is onboard with that decision.  Or, perhaps
 that decision is being reconsidered.
 
 Let me answer from the position of a mere user. It's not clear to me
 why and when users should switch to BTRFS or xfs or else, nor am I
 not interested in using anything which would potentially endanger
 existing installations (So far, reports I am reading from openSUSE
 users don't necessarily sound convincing).
 
 In other words, you'd have to do a lot of marketing and convincing
 work to persuade users to use BTRFS/xfs etc.
 
 Ralf

I think this is an important point.  To make a fundamental change like
this, the rationale and benefits need to be very clearly spelled out,
and not just chase the new hotness (although 6-7 years in, I'm not sure
btrfs can be called new?  XFS certainly can't!) ;)

IOWs, I'd like to see much more than because it can do snapshots and
checksums as the rationale; there are most definitely interesting things
that btrfs can do (or is working on doing), but as btrfs has evolved, so has
the rest of the Linux storage ecosystem: DM thin provisioning, xfs and ext4
metadata checksums, System Storage Manager (SSM) aiming for administration
ease, etc.

It's up to those proposing a new default to clearly spell out the compelling
advantage to the change.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: btrfs as default filesystem for F22?

[Eric Sandeen]

On 10/6/14 9:26 AM, Josef Bacik wrote:
 Obviously we aren't in xfs/e2fsprogs territory, but it'll fix 90% of
 the problems and then the other 10% are just a matter of having an
 example to work off of.  Thanks,
 
 Josef

Josef, just as a datapoint: after corrupting 32k random bytes on a 2G
image lightly populated and made with default mkfs options, then running
fsck with all of your recent fixes, I got 9 mount failures out of 20
attempts, 55% success.

Running the same test, but w/ 2 devices, each randomly damaged to
the same extent, and created with -m raid1 -d raid0, I get 
10 failures out of 20, 50% success.

(Note that this is just a low-bar will it mount test, I'm
not looking at what's in the repaired filesystem at all).

What sort of testing yielded your 90% success rate?

Thanks,
-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Non-responsive maintainer Russell Cattelan

[Eric Sandeen]

I know Russell - I'll reach out to him.  I also like xxdiff ;)

He's probably quite willing to turn this over to you.

-Eric

On 8/1/14, 4:39 AM, Christopher Meng wrote:
 Hi all,
 
 I'm unable to contact Russell Cattelan (cattelan{at}xfs[dot]org) for
 almost one year[1]. I vetted his login records and found that since
 2011-11-15 he hasn't logined to Fedora any more:
 
 Last login in FAS:
cattelan 2011-11-15
 
 As the package xxdiff needs some love(it still even depends on Qt3
 whereas the latest version has ported it to Qt4), as a result I'd like
 to take it over and fix it.
 
 Thanks.
 
 [1]---https://bugzilla.redhat.com/show_bug.cgi?id=998969
 
 Yours sincerely,
 Christopher Meng
 
 http://cicku.me
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: unsigned char vs. signed char

[Eric Sandeen]

On 7/15/14, 11:40 AM, Orion Poplawski wrote:
 Did you know that char defaults to signed char on x86 but unsigned char 
 on ppc and arm?  I didn't.
 
 Just a heads up.
 

I did, because due to XFS's history, it maked an assumption that's not true on 
x86.

We had -funsigned-char in the Makefile up until around 2010, when we finally 
decided that the code was clean without it.  :)

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Eric Sandeen]

On 3/4/14, 3:43 PM, Ric Wheeler wrote:
 On 03/04/2014 11:26 PM, Przemek Klosowski wrote:
 On 02/28/2014 03:45 PM, Adam Williamson wrote:
 As a server WG member I voted +1 on XFS as I have no particular objection 
 to XFS as a filesystem, but I do think it seems a bit sub-optimal for us to 
 wind up with server and desktop having defaults that are very similar but 
 slightly different, for no apparently great reason.
 This may be a historical bias. XFS is a large code base (*), which means two 
 things: a larger bug surface, and a larger memory footprint that used to be 
 a problem for personal desktop-type machines but less so for better endowed 
 servers.

 I understand that by now XFS got so much exercise that its
 robustness is unimpeachable. As to the size, I see that while the
 latest XFS kernel module is one of the larger kernel modules
 around, it probably is no longer significant on today's multi-GB
 systems---the extra megabyte at current memory prices is just a one
 cent increase in the system cost, after all.
 
 Having said that, I don't use XFS nowadays so I don't know how much
 more memory it allocates in typical use---can anyone comment on the
 actual memory footprint of running XFS?
 
 I am pretty sure that ext4 is a built-in module in Fedora kernels,
 as well as in the boot environment; making XFS the default will
 require also building it in, pretty much forever, while we still
 need extXX, and whatever comes next (btrfs?). I am OK with that,
 though.

 (*) 2.9MB of XFS source code vs 1.3MB in ext4 dirs
 (**) xfs.ko is 1.3MB


 
 You need to count the jbd2 code for ext4 as well,
 
 Ric
 

FWIW, I we looked at lines of code vs. megabytes of source, back in 3.4, and 
did a blog post about the trend lines:

http://sandeen.net/wordpress/computers/linux-filesystems-loc-update/

I did count jbd2 in that, as well as the common mbcache code which in truth 
is only used for the extN filesystems.

I suppose I need to do another update :)

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Eric Sandeen]

On 3/3/14, 3:16 PM, Matthias Clasen wrote:
 On Mon, 2014-03-03 at 16:16 +0200, Ric Wheeler wrote:
 
 
 I am fine with something like what is proposed by Steve above - let users 
 have 
 the GUI present an option that gives preference to the default without 
 totally 
 hiding other options.
 
 You and Josef are sending mixed messages here: btrfs is fine, but just
 not ready to be the default and don't make it the default, but we
 still need users to install it so it can get better, so don't take away
 the option. I don't think that is reasonable - either it is ready to be
 widely used, ie the default, or it isn't in which case it shouldn't
 appear in the UI.

The combination of those requirements would seem to mean that only
the default filesystem may appear in the UI...

 I'm pretty firmly against filesystem choice in the workstation
 installer.

Ah!  I guess that is how you feel.  :)

We do need _some_ mechanism for willing users to test more cutting-edge
code on a real install, or everything besides the default may wither and
die, as nothing else can get any real-world exposure...

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Eric Sandeen]

On 3/3/14, 5:57 PM, Jon wrote:
 On Sat, Mar 1, 2014 at 3:18 PM, Chris Murphy li...@colorremedies.com wrote:

 On Mar 1, 2014, at 1:19 PM, Jon jdisn...@gmail.com wrote:

 The inability to shrink or reduce XFS is rather disappointing. I've
 seen a few sarcastic remarks along the lines of (paraphrased): why
 would anyone ever want to shrink a volume?

 In the context of server, and default installs, why is a valid question.


 People do shrink volumes, and this lack of flexibility is an important
 consideration I feel was ignored in the Server WG decision.

 What is the use case for volume shrinking in a server context? Dual boot is 
 a total edge case for servers.
 
 In the context of Fedora-ARM, people would regularly shrink ext4
 filesystem images to fit on different size storage.
 We would release an 8GiB image, but the ARM board might only have 4GiB
 eMMC or somebody has a smaller sdcard (stuff like that).

For what it's worth, this sort of shrinking  growing can lead to
some pretty pessimal fs layouts.

 We no longer release Fedora ARM rootfs tarballs, too hard to educate
 people to do the right thing with ACL's, xattrs, selinux, etc...
 Anyhow, it's actually a great way to ship a Fedora rootfs... just
 shrink the filesystem down to the smallest size, and allow the user to
 simply resize2fs the image into their storage.
 This would not be possible with XFS for the server variant of
 Fedora-ARM, and I feel represents a significant challenge to the ARM
 team.

I was never sure why shrinking was particularly required; maybe it's a
little harder or slower, but: do an install, see how much space it used,
make a new filesystem with 105% of that space (or so) and install onto that.
Done and done?

OTOH, the shrinkfs dance was a great way to find myriad bugs in resize2fs. ;)

Making a tiny xfs FS and then growing it significantly will also lead
to pessimal layouts there - we'll end up with many, many very small
allocation groups.

The shrink/grow thing was clever, but also a bit abusive from a filesystem
design point of view.

 
 As for the real world examples of shrinking, well drawing upon my
 experiences in the past at a managed hosting company:
 
 * VG consisted of ten 100 GiB san luns, customer asks one be removed,
 and provisioned to another server. We shrunk the filesystem, and
 removed the lun per request.
 
 Shrinking support significantly reduced the time needed for that
 maintenance window!
 Or put another way, shrinking is much faster than formatting and
 restoring data from tape to achieve smaller sized volumes.
 
 
 * customer over estimated the requirements for one mount (lets say
 /opt for example), and underestimated the requirements of another (say
 /var/log for example).
 This classic example of where customers fail to properly anticipate
 the storage requirements of their work-flow at the time of install,
 and they shrink one to grow another.
 (this might be solved by the thin provisioning idea)
 
 
 * customer wanted to shrink the SAN luns to a smaller size, but was
 averse to significant down time to restore from tape.
 ext4 shrinking to the rescue!

It's handy at times, but shrinking really scrambles data layouts.
If you don't care about performance, it's probably ok.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Eric Sandeen]

On 3/3/14, 7:34 PM, Chris Adams wrote:
 Once upon a time, Eric Sandeen sand...@redhat.com said:
 The shrink/grow thing was clever, but also a bit abusive from a filesystem
 design point of view.
 
 How does it compare to the suggested alternative, LVM thin provisioning?
 How well does thinp handle fragmentation; is there a defrag for thinp
 available (or planned)?

Well, I meant that it was clever for the minimal root fs distribution
business.

But I think you're right, fragmentation on thinp is something that is
still under investigation.  

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: lvresize and XFS, was: default file system

[Eric Sandeen]

On 2/28/14, 7:54 AM, Zdenek Kabelac wrote:
 Dne 28.2.2014 14:37, Chris Murphy napsal(a):

 On Feb 28, 2014, at 1:33 AM, Zdenek Kabelac zkabe...@redhat.com wrote:


  fsadm failed: 3


 man fsadm

 DIAGNOSTICS
On  successful completion, the status code is 0.  A status code of 2 
 indicates the operation was interrupted by the user.  A
status code of 3 indicates the requested check operation could not 
 be performed because the filesystem is mounted  and  does
not support an online fsck(8).  A status code of 1 is used for other 
 failures.

Ok, granted, I should have read the fsadm manpage.  :)

 Yeah but did fsadm fail? No, as a whole its operation succeeded.
 Can we say fsadm failed to run fsck? I guess that's one way to look
 at it, but then it failed to understand it shouldn't request a
 check operation on XFS in the first place.

 Chris Murphy

 
 Current logic of lvm is to call fsadm  to check blockdevice  (lvm2 knows 
 nothing about about filesystems).
 
 fsadm translate this to check xfs - which is not supported by xfs

Well, it failed to check it because it is *mounted*, right?  It said this:

 fsadm: Skipping filesystem check for device /dev/mapper/VG-LV as the 
 filesystem is mounted on /mnt
fsadm failed: 3

extN does the same thing:

# e2fsck /dev/sdb3 
e2fsck 1.41.12 (17-May-2010)
/dev/sdb3 is mounted.
e2fsck: Cannot continue, aborting.

# xfs_repair /dev/sdc4
xfs_repair: /dev/sdc4 contains a mounted filesystem

fatal error -- couldn't initialize XFS library

 (I've no idea if there ever be support for this operation) - so fsadm
 reports it has failed to do any check.

xfs certainly does have check and repair tools - man xfs_repair.
You can run it with -n if you want check-only.

However, I see that (at leat my copy of) fsadm reqiures xfs_check,
which has been deprecated upstream in favor of xfs_repair -n.
xfs_check doesn't scale, and xfs_repair -n performs the same
tasks.

 XFS_CHECK=xfs_check

so I guess I should file a bug on that.

 lvm2 detects through error code 3 that requested operation is not
 supported, but considered safe to be ignore and continues.
 
 Of course fsadm could return 0 - but then it wouldn't be able to
 recognize if check really was made or just skipped.
 
 This message is only shown in verbose mode - and it's mainly for
 developer to know what has happened (or actually not happened)

Ok, that's fair enough.

Thanks,
-Eric

 
 Zdenek
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: lvresize and XFS, was: default file system

[Eric Sandeen]

On 2/28/14, 8:12 AM, Eric Sandeen wrote:

 However, I see that (at least my copy of) fsadm requires xfs_check,
 which has been deprecated upstream in favor of xfs_repair -n.
 xfs_check doesn't scale, and xfs_repair -n performs the same
 tasks.
 
 XFS_CHECK=xfs_check
 
 so I guess I should file a bug on that.

Sorry, I should have checked upstream, I see that upstream handles
absence of xfs_check.

   xfs) if which $XFS_CHECK $NULL 21 ; then
 dry $XFS_CHECK $VOLUME
  else
 # Replacement for outdated xfs_check
 # FIXME: for small devices we need to force_geometry,
 # since we run in '-n' mode, it shouldn't be problem.
 # Think about better way
 dry $XFS_REPAIR -n -o force_geometry $VOLUME
  fi ;;

TBH, I would just switch 100% to xfs_repair, rather than using
it as a fallback.   We can talk about this offline though.  :)

Thanks,
-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Eric Sandeen]

On 2/27/14, 4:08 PM, James Wilson Harshaw IV wrote:
 Interesting. If someone could confirm that this remains true with Fedora 20, 
 it would be extremely beneficial.

With details, please, as Chris requested.  doesn't cooperate is not enough to 
go on.  ;)

Thanks,
-Eric

 On 02/27/2014 05:02 PM, Jochen Schmitt wrote:
 On Thu, Feb 27, 2014 at 04:08:46PM -0500, James Wilson Harshaw IV wrote:
 A question I have is XFS worth it?
 I have done some testing with RHEL 7 Beta which use XFS as a default file 
 system.

 I have to recorgnize, that the -r switch of the lvresize command doesn't 
 cooperate
 with xfs in oppoiste of ext4.

 Best Regards:

 Jochen Schmitt

 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: lvresize and XFS, was: default file system

[Eric Sandeen]

On 2/27/14, 4:40 PM, Chris Murphy wrote:
 
 On Feb 27, 2014, at 3:32 PM, Chris Murphy li...@colorremedies.com wrote:
 

 On Feb 27, 2014, at 3:02 PM, Jochen Schmitt joc...@herr-schmitt.de wrote:

 On Thu, Feb 27, 2014 at 04:08:46PM -0500, James Wilson Harshaw IV wrote:
 A question I have is XFS worth it?

 I have done some testing with RHEL 7 Beta which use XFS as a default file 
 system.

 I have to recorgnize, that the -r switch of the lvresize command doesn't 
 cooperate
 with xfs in oppoiste of ext4.

 Where you growing or shrinking the fs, and was it mounted at the time, and 
 what error did you get? XFS doesn't support shrink, and only can be grown 
 online. I'm pretty sure lvresize -r supports xfs_growfs via fsadm.
 
 worksforme
 
 Starting with a 10TB XFS volume, 5TB x 5 disk VG.
 
 
 # lvresize -r -v --size 15T VG/LV
 Finding volume group VG
 Executing: fsadm --verbose check /dev/VG/LV
 fsadm: xfs filesystem found on /dev/mapper/VG-LV
 fsadm: Skipping filesystem check for device /dev/mapper/VG-LV as the 
 filesystem is mounted on /mnt
 fsadm failed: 3

snip

 However, I don't know what fsadm failed: 3 means.

fsadm.sh:

if detect_mounted ; then
verbose Skipping filesystem check for device \$VOLUME\ as 
the filesystem is mounted on $MOUNTED;
cleanup 3
fi

...
cleanup() {
...
exit ${1:-1}
}

the script exits with error 3 meaning, well, 3, I guess, when the fs is 
mounted.  Not the nicest error reporting IMHO :)

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: default file system, was: Comparison to Workstation Technical Specification

[Eric Sandeen]

On 2/27/14, 10:53 PM, James Wilson Harshaw IV wrote:
 
 On 02/27/2014 11:20 PM, Chris Murphy wrote:
 On Feb 26, 2014, at 12:53 PM, Michael Cronenworth m...@cchtml.com wrote:

 Chris Murphy wrote:
 by default we put ext4 on LVM
 The tool works in this use-case unless something has broken it recently.
 It can be done, the convert tool should work, and Btrfs should work on any 
 device mapper instance. However…

 In the context of the default ext4+LVM layout the conversion still means 
 separate /boot, /, and /home file systems. A major benefit of the Btrfs 
 layout is these are subvolumes, which instead draw space from one volume 
 pool. And that's lost with a conversion strategy. It also means going from a 
 Fedora standard layout to a distinctly non-standard one because our Btrfs 
 layout isn't like the result you'd get from what you're talking about.


 Chris Murphy
 
 A question I have, forgive me if it seems stupid, is why not just
 change the standard? From what I have seen is that btrfs is superior
 to ext4 as even Theodore T'so (creator of ext4) said btrfs offers
 improvements in scalability, reliability, and ease of management.

Yes, that is a thing on Wikipedia isn't it.  ;)  It's referring
to a discussion held 5 years ago.

 I
 feel that eventually other filesystems may show to be better than
 ext4 and we probably will need to change the standards for the sake
 of quality. My fear is that we will be sacrificing quality for not
 having too change much.

I'll see your 5-year-old quote from the ext4 maintainer, and raise
you a 1-day-old quote from a btrfs maintainer:

Just popping in here to say that btrfs is not ready to be default in
Fedora yet. Optional is fine but not default. Thanks,
Josef.

If and when btrfs becomes a superior solution, I'm sure Fedora
will be eager to ship it.

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: May I file 1000 bugs aka upstream test suite tracking

[Eric Sandeen]

On 2/21/14, 9:25 AM, Daniel P. Berrange wrote:
 On Fri, Feb 21, 2014 at 04:19:29PM +0100, Miroslav Lichvar wrote:
 On Fri, Feb 21, 2014 at 02:50:12PM +, Daniel P. Berrange wrote:
 On Fri, Feb 21, 2014 at 09:38:56AM -0500, Josh Boyer wrote:
 Personally, I don't think %check is a good idea at all.

 I think the benefit depends on the level of patching the Fedora maintainer
 is doing. If they are shipping just vanilla upstream tar.gz then they can
 have a moderate level of confidence in the functionality of their package
 without tests, since you can assume upstream ran their test before release.

 Upstream may not have run the test on all archs supported by Fedora.
 ARM is probably still not something we can assume everyone has. I was
 dealing with a test which failed only on 32-bit systems just
 yesterday, I didn't notice it until I tried building the package in
 koji.
 
 Yes indeed, the quality of libvirt upstream on non-x86 in general has
 directly benefited from the fact that the our Fedora RPMs are running
 %check on all arches, so generating nice bugs reports for us.

Ditto for e2fsprogs, we found lots of endian bugs that way.

(But:  OMG, please, no, don't file 1000 bugs demanding new test suites
from packagers!)

-Eric

 Regards,
 Daniel
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Drawing lessons from fatal SELinux bug #1054350

[Eric Sandeen]

On 1/23/14, 5:55 PM, Kevin Kofler wrote:
 * We are enabling SELinux enabled (enforcing) by default, a tool designed to 
 prevent anything it does not like from happening. (Reread this carefully: 
 The ONLY thing that tool is designed to do at all is PREVENT things. It does 
 not have a SINGLE feature other than being a roadblock and an annoyance.)

In the same way that the lock on your front door is an annoyance, I guess.

 * SELinux works by shipping a policy that effectively tries to specify in 
 one single place (read: single point of failure!) everything any program in 
 Fedora (scalability disaster!) ever wants to do (second-guessing its actual 
 code, i.e., duplication of all logic!). (Note the 3 (!) major antipatterns 
 in a single-sentence (!) description of how SELinux works!)

If you think SELinux is duplicating all logic in application code,
I do not think you quite grasp how SELinux works.

If the solution to every serious bug that slips through the cracks of a release
is to disable the package, over time we may not have much left in Fedora.

I know that pretty much all filesystems would be out by now. ;)

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Drawing lessons from fatal SELinux bug #1054350

[Eric Sandeen]

On 1/23/14, 6:37 PM, Kevin Kofler wrote:
 Eric Sandeen wrote:
 If the solution to every serious bug that slips through the cracks of a
 release is to disable the package, over time we may not have much left in
 Fedora.
 
 But SELinux is the one package (OK, one of the few, along with, e.g., 
 firewall stuff) whose removal would actually INCREASE Fedora's 
 functionality!

Sure, removing firewalls  selinux would be a serious enhancement
of functionality.

For malware botnets  spam hosts, especially...

-Eric

 Kevin Kofler
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Packages have proxy word.

[Eric Sandeen]

On 11/2/13 10:50 AM, Kevin Kofler wrote:
 مصعب الزعبي wrote:
 Then do new rule to write proxies instead of proxy, at Fedora packaging
 guidlines.
 
 Renaming packages, censoring descriptions etc. in Fedora at the whims of 
 governments (of countries Fedora isn't supposed to be exported to in the 
 first place) is not going to happen. Sorry.
 
 Kevin Kofler
 

I think your answer conflates two issues.

If Syria is on the export control list, there's really no further discussion
to be had.

But I wonder how many i.e. school nanny-filters ban URLs with the word
proxy, and how that impacts the ability of users in any country to obtain
Fedora.

The idea of an alternate URL using package hashes seemed reasonable to me,
to route around that sort of internet damage.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Packages have proxy word.

[Eric Sandeen]

On 11/2/13 11:48 AM, Basil Mohamed Gohar wrote:
 On 11/02/2013 12:46 PM, Eric Sandeen wrote:
 On 11/2/13 10:50 AM, Kevin Kofler wrote:
 مصعب الزعبي wrote:
 Then do new rule to write proxies instead of proxy, at Fedora packaging
 guidlines.

 Renaming packages, censoring descriptions etc. in Fedora at the whims of 
 governments (of countries Fedora isn't supposed to be exported to in the 
 first place) is not going to happen. Sorry.

 Kevin Kofler


 I think your answer conflates two issues.

 If Syria is on the export control list, there's really no further discussion
 to be had.

 But I wonder how many i.e. school nanny-filters ban URLs with the word
 proxy, and how that impacts the ability of users in any country to obtain
 Fedora.

 The idea of an alternate URL using package hashes seemed reasonable to me,
 to route around that sort of internet damage.

 -Eric

 
 I think the solution in this case, that would require zero change to
 package names, and likely little else, would be to offer a way to choose
 only those repos that offer an https interface.

If that works, sure.  (I didn't read the whole thread, didn't realize that
it solved the problem.)

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: custom kernel, how to revert patch

[Eric Sandeen]

On 10/9/13 10:25 AM, Chris Murphy wrote:
 This http://fedoraproject.org/wiki/Building_a_custom_kernel has
 ApplyPatch examples in the spec file, which works fine. Is there an
 inverse command? Or do I need to do it manually with patch -p0 -R 
 ?

You could generate the reverse patch with:

$ git show 1234abcd  patch_to_revert.patch
$ interdiff patch_to_revert.patch /dev/null  revert-patch.patch

and then ApplyPatch that one.

(though you'll have to work out whether it applies cleanly, 10 months
is a long time in btrfs-time...)

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Default boot/root filesystem

[Eric Sandeen]

On 9/9/13 10:56 AM, Till Maas wrote:
 On Sat, Sep 07, 2013 at 05:35:05PM -0500, Eric Sandeen wrote:
 
 If people want to switch the Fedora default to XFS, I'll gladly file
 the feature.  :)
 
 Why is XFS better than ext4. When I checked a few months ago, XFS did
 not even support shrinking, but only growing. Even now there seems to be
 only xfs_growfs on F19, but no tool to shrink it.

There is no universal better so I won't argue that point.  TBH,
ext4 is probably more performant for the most common types of
machines, storage, and workloads under typical Fedora /handwave.

XFS really shines as things get bigger - bigger storage, more cpus,
more memory, more spindles.

As for shrink, you are right - there is no xfs shrink.

(OTOH, I have seen a fair bit of damage done by ext4 shrink, so I'm
not so sure lack of shrink should count *against* xfs) ;)

-Eric
 
 Regards
 Till
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Default boot/root filesystem

[Eric Sandeen]

On 9/7/13 3:47 AM, Chris Murphy wrote:
 
 On Sep 7, 2013, at 12:48 AM, Sam Varshavchik mr...@courier-mta.com wrote:
 
 According to this:

 http://www.serverwatch.com/server-news/where-is-red-hat-enterprise-linux-7.html

 RHEL7 will use XFS for the default boot/root.
 
 The article doesn't make it clear if this is XFS for both /boot and /
 or just /. But if it's the former, it ends the debate about
 installing the bootloader on a partition. XFS has no bootloader pad
 at all so it isn't possible. On EFI systems, it doesn't matter.

Well, that only stops you from installing the bootloader on a partition;
it doesn't stop you from putting it in the MBR.

-Eric

 Chris Murphy
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Default boot/root filesystem

[Eric Sandeen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 9/6/13 5:48 PM, Sam Varshavchik wrote:
 According to this:
 
 http://www.serverwatch.com/server-news/where-is-red-hat-enterprise-linux-7.html

  RHEL7 will use XFS for the default boot/root.
 
 I could certainly have been out of town, for a while, and missed
 this. But, to the best of my knowledge, Fedora uses ext4 as the
 default boot/root. Just sounds a bit strange to me, that this is
 getting dumped into RHEL without tossing it into Fedora first, to
 rattle around, and shake any bugs out, beforehand. Don't really have
 an opinion either way, myself, I would just expect that something
 like this would go into Fedora first.

If people want to switch the Fedora default to XFS, I'll gladly file
the feature.  :)

- -Eric
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSK6oYAAoJECCuFpLhPd7gt68P/RR8hE3aNMnnscr7LF2I5dW5
X23dYGXI93t+DcqIAYgTVY/tXP54+6xKJflOpU8k8/RxbRd6vGDeomBA81iTMZAS
rgVA6vnBC4nnyOo7NXnsgQNy/nFr0o1xQy3jcQajAZOWO9Mx9NJnC1kofBv6gtfX
s6wg18TuSl5CpUC/zRCJSk+lnJznbXZpSc1VAKekYe++VlfOEIVYfistrlwQm17X
wEC8R/ijIJl9nZqXleJIm9QdnZ7yePvKTksQ/4aGjHFJcEyxV3rxkgknhbMySvcc
F1CF+EX2ZUcEOviqAageZCs+/Vw6y333PpbTthEyVlS9ofi2B05kVGlp5kU1Sp4b
hIzyyRWt0nJ/aLmEv5ZetjsM6rOIcaXjQ0cifVviVYTnDYpyPr235wGPkLOp3BM0
WHnayAPIHTq7RmgKo0g8LRO4peLR2JNBGCQlXyJ4jy+B4jEmZ924kjCzNRDUASmz
5X+9ohSMbfJzrYwD6zraBIifETnM9pAqBcj8JiXKwokY44dzAiXve7tDthAcjcA5
luPu7HW20hxjCY5eLJk5sFpu11nMSXIFYz0GVFB5SC2vVXQ2dc7mwk4Sw/mCDL7q
grJ1trUIFvNlXQon8rwQfJmf83aYFFvG3+NHRGSicPoNpeWmsqg6WuMm0M+AmxYt
gCKIuan0Rk8tTmPvqKVK
=cT2W
-END PGP SIGNATURE-
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Suggestion: bmap files and bmaptool

[Eric Sandeen]

On 8/16/13 3:46 AM, Artem Bityutskiy wrote:
 Another approach which might (?) be more robust, is to somehow encode
  that sparseness in a single file format that can be
  transported/compressed/copied w/o losing the sparseness information,
  and another tool to operate efficiently on that format at the
  destination, either by unpacking it to a normal sparse file or piping
  it to some other process.
 Err, not sure I fully understand, but it sounds like what bmap-tools
 project actually does.

by single file I meant a _single_ file, not the original file and a mapping
file.  :)

I realize that you have a fully-fledged set of tools, and you're not looking
for new directions, but I was thinking about encoding mapping info  file data
into a single file, with a tool to extract it again.  That way there's nothing
to get out of sync.

Actually, now that I think about it qemu-img can do that already:
(sorry, I'm getting a little off topic here, bear with me)

# truncate --size=1g fsfile
# mkfs.ext4 fsfile
# cp fsfile --sparse=never fsfile.copy

// fsfile is sparse; the copy is not.

# du -hc fsfile*
49M fsfile
1.0Gfsfile.copy

# qemu-img convert -f raw -O qcow fsfile.copy fsfile.qcow

// the qcow image now contains only data+mapping info,
// no zero ranges:
# du -h fsfile.qcow
832Kfsfile.qcow

// and can be re-extracted into a sparse file

# qemu-img convert -O raw fsfile.qcow fsfile.copy2
# du -hc fsfile.copy2 
352Kfsfile.copy2

Ok, sorry for that diversion, but that's cool - the tool I want
already exists, and I hadn't realized it.  :)

So that's a decent option for encoding sparse files for efficient
transfer, too.

 Piping is not implemented, because sparseness cannot be easily passed
 though a pipe.

Err, right ;)

-Eric

  Just some thoughts...
  
 Thanks a lot for the feed-back!

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Suggestion: bmap files and bmaptool

[Eric Sandeen]

On 8/13/13 8:58 AM, Artem Bityutskiy wrote:
 # Make the image to be sparse
 $ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw 
 Fedora-x86_64-19-20130627-sda.raw.sparse
 
 # Generate the bmap file
 $ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o 
 Fedora-x86_64-19-20130627-sda.raw.sparse.bmap

So this is the part that interests me . . . 

There seem to be two issues here; how do we efficiently (compress and) 
transport sparse files while retaining sparseness, and how do we efficiently 
operate on files which are already sparse.

For the latter, you're using your bmap tool to map what is hopefully a static 
file (via fibmap or fiemap, I guess?).

I haven't looked at how you've done it, but you do need to be very careful that 
the file is stable  quiesced on disk.  Mapping it this way can be fraught with 
errors if the file is changing, or has delalloc blocks, etc.  And of course 
getting the mapping wrong means data corruption.  If the file is known to be 
sparse, then going forward, using SEEK_HOLE / SEEK_DATA is probably the best 
approach.

But then there's the issue of transporting these sparse files around.  We have 
had the same problem in the past with large e2image metadata image files, which 
may be terabytes in length, with only gigabytes or megabytes of real data.  
e2image _itself_ creates a sparse file, but bzipping it or rsyncing it still 
processes terabytes of zeros, and loses all notion of sparseness.

xfs_metadump worked around this by creating its own compact format describing a 
sparse file's data  sparseness, which is unpacked into a normal sparse file 
by xfs_mdrestore.

More recently e2image gained something slightly similar, but used the existing 
qcow format to encode the sparseness.  qemu-image convert to raw type turns 
it back into a normal sparse file readable by e2fsprogs tools.

So I guess your solution requires 2 pieces of information; the existing file, 
and the mapping file.  Are there mechanisms to ensure that they are in sync?

Another approach which might (?) be more robust, is to somehow encode that 
sparseness in a single file format that can be transported/compressed/copied 
w/o losing the sparseness information, and another tool to operate efficiently 
on that format at the destination, either by unpacking it to a normal sparse 
file or piping it to some other process.

Just some thoughts...

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Does your application depend on, or report, free disk space? Re: F20 Self Contained Change: OS Installer Support for LVM Thin Provisioning

[Eric Sandeen]

On 7/31/13 12:08 PM, Chris Murphy wrote:
 
 On Jul 31, 2013, at 8:32 AM, Mike Snitzer snit...@redhat.com
 wrote:
 
 But on the desktop the fedora developers need to provide sane
 policy/defaults.
 
 Right. And the concern I have (other than a blatant bug), is the F20
 feature for the installer to create thinp LVs; and to do that the
 installer needs to know what sane default parameters are. I think
 perhaps determining those defaults is non-obvious because of my
 experience in this bug: 
 https://bugzilla.redhat.com/show_bug.cgi?id=984236
 
 If I'm going to use thinP mostly for snapshots, then that suggests a
 smaller chunk size at thin pool creation time; whereas if I have no
 need for snapshots but a greater need for provisioning then a larger
 chunk size is better. And asking usage context in the installer, I
 think is a problem.

Quite some time ago I had asked whether we could get the allocation-tracking
snapshot niceties from dm-thinp, without actually needing it to be thin.

i.e. if you only want the efficient snapshots, a way to fully-provision
a thinp device.  I'm still not sure if this is possible...?

I guess I'm pretty nervous about offering actual thin provisioned
storage to average Fedora users.  I'm having nightmares about the bug
reports already, just based on the likelihood of most users misunderstanding
the  feature and it's requirements  expected behavior...

-Eric

 Chris Murphy
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Does your application depend on, or report, free disk space? Re: F20 Self Contained Change: OS Installer Support for LVM Thin Provisioning

[Eric Sandeen]

On 7/27/13 11:56 AM, Lennart Poettering wrote:
 On Fri, 26.07.13 22:13, Miloslav Trmač (m...@volny.cz) wrote:
 
 Hello all,
 with thin provisioning available, the total and free space values
 reported by a filesystem do not necessarily mean that that much space
 is _actually_ available (the actual backing storage may be smaller, or
 shared with other filesystems).

 If your package reports disk space usage to users, and bases this on
 filesystem free space, please consider whether it might need to take
 LVM thin provisioning into account.

 The same applies if your package automatically allocates a certain
 proportion of the total or available space.

 A quick way to check whether your package is likely to be affected, is
 to look for statfs() or statvfs() calls in C, or the equivalent in
 your higher-level library / programming language.
 
 Well, I am pretty sure the burden must be on the file systems to report
 a useful estimate free blocks value in statfs()/statvfs(). Exporting that
 problem to userspace and expecting userspace to work around this is just
 wrong. In fact, this would be quite an API breakage if applications
 cannot rely that the value returned is at least a rough estimate on how
 much data can be stored on disk.
 
 journald will scale how much disk usage it will use of /var/log/journal
 based on the file system size and free level. It will also module the
 per-service rate limit levels based on the amount of free disk space. If
 you break the API of statfs()/statvfs(), then you will end up break this
 and all programs like it.

Any program needs to be prepared for ENOSPC; as Ric mentioned elsewhere,
until you successfully write to it, it's not yours! :)  (Ok, thinp
running out of space won't generate ENOSPC today, either, but you see
my general point...)

And how much space are we really talking about here?  If you're running
thin-provisioning on thin margins, especially w/o some way to automatically 
hot-add storage, you're probably doing it wrong.

(And if journald sees 100T free and decides it can use 50T of that,
it's doing it wrong, too) ;)

The truth is somewhere in the middle, but quibbling over whether this
app or that can claim a bit of space behind a thin-provisioned volume
probably isn't useful.

The admin definitely needs tools to see the state of thinly provisioned
storage, but that's the admin's job to worry about, not the app's, IMHO.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Does your application depend on, or report, free disk space? Re: F20 Self Contained Change: OS Installer Support for LVM Thin Provisioning

[Eric Sandeen]

On 7/26/13 3:13 PM, Miloslav Trmač wrote:
 Hello all,
 with thin provisioning available, the total and free space values
 reported by a filesystem do not necessarily mean that that much space
 is _actually_ available (the actual backing storage may be smaller, or
 shared with other filesystems).
 
 If your package reports disk space usage to users, and bases this on
 filesystem free space, please consider whether it might need to take
 LVM thin provisioning into account.

Short answer: it doesn't (it can't).

Just like an application doesn't know if it's got a 2.5 or 3.5 drive
behind it, or cloud behind it, or a usb stick behind it, it doesn't
know if it's got thinly provisioned storage behind it.

It's up to the administrator to make sure that the thinly provisioned
device doesn't fill up  run out of space, but if it does, there's
nothing an app can do about that.

There's also no standard interface to query how full your thinly
provisioned storage is; it depends on what's implementing the thin
provisioning.  So again, nothing an app can do/query/handle/change.

 The same applies if your package automatically allocates a certain
 proportion of the total or available space.

I can't imagine that anything actually does that, does it?
Good lord I hope not.  ;)
 
 A quick way to check whether your package is likely to be affected, is
 to look for statfs() or statvfs() calls in C, or the equivalent in
 your higher-level library / programming language.

statfs will still tell you how much space the filesystem has allocated,
as well as how much space it thinks it has left, based on the total
space the disk has *said* it has available, just like it always ever
did.

The difference, of course, is that you might actually run out of blocks
before you fill the fs.  But I can't think offhand what apps would care.
And again, it's something the admin shouldn't let happen.

For now, consider it completely transparent to the user (unless the
admin doesn't keep up, in which case it will be anything *but*
transparent).

TBH, when the backing store runs out of space, things do get pretty
ugly at this point.  It's work that needs to be done to make it more
robust  graceful.

-Eric

  Mirek
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

Re: Bad file access on the rise

[Eric Sandeen]

On 6/7/13 3:06 PM, Steve Grubb wrote:
 On Friday, June 07, 2013 08:42:09 PM Matthew Garrett wrote:
 On Fri, Jun 07, 2013 at 03:35:28PM -0400, Steve Grubb wrote:
 So far, the discussion has focused on pulseaudio. But what about the
 O_NOATIME issue?

 Without further analysis, it doesn't tell us much. Does the code attempt
 to open a file O_NOATIME and then fall back to trying it without?
 
 It would appear so:
 
 open(/usr/share/icons/gnome/48x48/status/dialog-password.png, O_RDONLY|
 O_NOATIME) = -1 EPERM (Operation not permitted)
 open(/usr/share/icons/gnome/48x48/status/dialog-password.png, O_RDONLY) = 12
 read(12, 
 \211PNG\r\n\32\n\0\0\0\rIHDR\0\0\\0\0\\10\6\0\0\0W\2\371..., 4096) 
 = 
 2083
 close(12)   = 0
 
 Which is a bad patterm. O_NOATIME requires CAP_FOWNER and I don't think 
 graphic programs are supposed to run as root/privileged. So, there seems to 
 be 
 a misunderstanding of what O_NOATIME is for. It seems to be related to 
 loading 
 icons. Is there a common library for that?
 
 -Steve
 

This maybe?

http://permalink.gmane.org/gmane.comp.gnome.svn/621716

commit 4d2f77074802ac8b2d05f7cdfecef9bebfba2647
Author: Colin Walters walters at verbum.org
Date:   Mon Aug 27 16:07:39 2012 -0400

core: Use O_NOATIME to open metadata

We really don't need atime for metadata, it's just a speed hit.

it does try, and fall back

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Bug cloning considered harmful

[Eric Sandeen]

On 6/3/13 11:06 AM, Adam Williamson wrote:
 Hi, folks. Just wanted to throw out a random note on something that
 often irks me.
 
 Next time you're thinking about cloning a bug: think twice. I've found
 that it's rarely the case that cloning a bug is really the right thing
 to do.
 
 When you clone a bug:
 
 * The new bug gets the exact same summary and clones a lot of the other
 metadata, including Blocks, Keywords and Whiteboard
 * The new bug gets a 'description' which includes the original
 description plus every single comment from the 'parent' bug
 * The new bug has everyone from the 'parent' bug's CC list on its CC
 list
 
 This is rarely really what you want, and often can be completely
 incorrect and annoying for the people in the CC list.
 
 Unless the above accurately describes the attributes you want your new
 bug to have, you're probably better off just filing a new one with a
 link to the old one than using the 'clone' function.

Or clone, but pay attention to the content before you hit submit.  Trim
comments, re-set flags, descriptions, keywords, cc, etc.

-Eric

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: when startup delays become bugs

[Eric Sandeen]

On 5/17/13 3:38 PM, Ric Wheeler wrote:
 On 05/16/2013 02:39 PM, Lennart Poettering wrote:
 On Thu, 16.05.13 12:20, Chris Murphy (li...@colorremedies.com) wrote:

 There have been no crashes, so ext4 doesn't need fsck on every boot:

4.051s systemd-fsck-root.service
 515ms
 
 systemd-fsck@dev-disk-by\x2duuid-09c66d01\x2d8126\x2d39c2\x2db7b8\x2d25f14cbd35af.service
 Well, but only fsck itself knows that and can determine this from the
 superblock. Hence we have to start it first and it will then exit
 quickly if the fs wasn't dirty.

 Note that these times might be misleading: if fsck takes this long to
 check the superblock and exit this might be a result of something else
 which runs in parallel monopolizing CPU or IO (for example readahead),
 and might not actually be fsck's own fault.
 
 We really should not need to run fsck on boot unless the mount fails. Might 
 save some time at the cost of a bit of extra complexity?

well, ext[34]a are special little snowflakes.  ;)

Since forever, we've called fsck on boot, and e2fsck replays the log in 
userspace if needed.  If the fs isn't marked as having encountered an error 
during the previous mount (or, historically, having had too many mounts or too 
much time since last fsck), then nothing else happens.

It shouldn't take a whole ton of time, but could, depending on whether the log 
was dirty, then the size of the log and speed of the disk I suppose.

When it took 4s above, was that a from a clean reboot (i.e. was the journal 
dirty?)

-Eric

 Ric
 

 and no oops, so this seems unnecessary:

1.092s abrt-uefioops.service
 https://bugzilla.redhat.com/show_bug.cgi?id=963182

 and I'm not using LVM so these seem unnecessary:


2.783s lvm2-monitor.service
 489ms systemd-udev-settle.service
  15ms lvm2-lvmetad.service

 How do I determine what component to file a bug against? I guess I have to 
 find the package that caused these .service files to be installed?
 $ repoquery --qf=%{sourcerpm} --whatprovides 
 '*/lib/systemd/system/lvm2-monitor.service'
 lvm2-2.02.98-8.fc19.src.rpm

 Please file a bug against the lvm2 package. And make sure to add it to:

 https://bugzilla.redhat.com/show_bug.cgi?id=963210

 Hmm, on your machine, what does systemctl show -p WantedBy -p
 RequiredBy systemd-udev-settle.service show? This will tell us which
 package is actually responsible for pulling in
 systemd-udev-settle.service.

 Thanks!

 Lennart

 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: when startup delays become bugs

[Eric Sandeen]

On 5/17/13 3:58 PM, Chris Murphy wrote:
 
 On May 17, 2013, at 2:38 PM, Ric Wheeler rwhee...@redhat.com wrote:
 
 On 05/16/2013 02:39 PM, Lennart Poettering wrote:
 On Thu, 16.05.13 12:20, Chris Murphy (li...@colorremedies.com) wrote:

 There have been no crashes, so ext4 doesn't need fsck on every boot:

   4.051s systemd-fsck-root.service
515ms

 systemd-fsck@dev-disk-by\x2duuid-09c66d01\x2d8126\x2d39c2\x2db7b8\x2d25f14cbd35af.service
 Well, but only fsck itself knows that and can determine this from the
 superblock. Hence we have to start it first and it will then exit
 quickly if the fs wasn't dirty.

 Note that these times might be misleading: if fsck takes this long to
 check the superblock and exit this might be a result of something else
 which runs in parallel monopolizing CPU or IO (for example readahead),
 and might not actually be fsck's own fault.

 We really should not need to run fsck on boot unless the mount fails. Might 
 save some time at the cost of a bit of extra complexity?
 
 Seems some extra complexity is needed anyway since the way to deal
 with file system problems differs with the various fs's. XFS and
 Btrfs fsck's are noops. XFS needs xfs_repair run, and Btrfs maybe
 needs to be remounted with -o degraded, depending on the nature of
 the mount failure since most problems are autorecovered from during
 mount.

fsck.xfs is a no-op because of the xfs approach that it's a journaling
filesystem, so the mount-time recovery is simply replay the log you're
good.

If you have a corrupt filesystem (as opposed to a not-cleanly-unmounted
filesystem), xfs_repair is an administrative action,
not a boot-time auto-initiated initscript action.

-Eric


 
 Chris Murphy
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: when startup delays become bugs

[Eric Sandeen]

On 5/17/13 5:29 PM, Chris Murphy wrote:
 
 On May 17, 2013, at 3:56 PM, Eric Sandeen sand...@redhat.com wrote:
 
 On 5/17/13 3:58 PM, Chris Murphy wrote:

 Seems some extra complexity is needed anyway since the way to deal
 with file system problems differs with the various fs's. XFS and
 Btrfs fsck's are noops. XFS needs xfs_repair run, and Btrfs maybe
 needs to be remounted with -o degraded, depending on the nature of
 the mount failure since most problems are autorecovered from during
 mount.

 fsck.xfs is a no-op because of the xfs approach that it's a journaling
 filesystem, so the mount-time recovery is simply replay the log you're
 good.

 If you have a corrupt filesystem (as opposed to a not-cleanly-unmounted
 filesystem), xfs_repair is an administrative action,
 not a boot-time auto-initiated initscript action.
 
 So if the boot fails due to reasons other than an unclean mount, 

unclean mount doesn't fail boot ;)

 with
 xfs the user needs a rescue environment of some sort. At the moment,
 it's similar with Btrfs in that what to do next depends on the
 problem.

Isn't that always the case? :)

Same is true for e2fsck, TBH.  Things can go wrong...

initramfs contains tools for all these filesystems, AFAIK.

-Eric (sorry, I think I'm taking this thread off-topic)

 Chris Murphy
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: when startup delays become bugs

[Eric Sandeen]

On 5/17/13 5:39 PM, Chris Murphy wrote:
 
 On May 17, 2013, at 3:53 PM, Eric Sandeen sand...@redhat.com wrote:
 
 On Thu, 16.05.13 12:20, Chris Murphy (li...@colorremedies.com) wrote:

 There have been no crashes, so ext4 doesn't need fsck on every boot:

   4.051s systemd-fsck-root.service
515ms

 systemd-fsck@dev-disk-by\x2duuid-09c66d01\x2d8126\x2d39c2\x2db7b8\x2d25f14cbd35af.service
 
 
 When it took 4s above, was that a from a clean reboot (i.e. was the journal 
 dirty?)
 
 Clean. And it's a new file system, created within the hour of the time test. 
 I also don't understand why there are two instances. There's only one ext4 
 file system on the computer.

Can't imagine why it takes 4s then, need finer-grained tracing I guess...

If I e2fsck a clean fs here it takes about 0.1s.

Most of its time is spent in read(); several reads get about 100k of data from 
the device, and that's about it.

As a sanity check I suppose you could try e2fsck from a rescue environment and 
see if it still takes that long, or of there is other overhead / interaction 
slowing it down.

-Eric

 
 Chris Murphy
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: ext4 - scsi: storvsc: avoid usage of WRITE_SAME

[Eric Sandeen]

On 5/8/13 9:21 AM, Mario Ceresa wrote:
 Dear all,
 When accessing a VHDX disc with a f18 VM (hyperv) with kernel
 3.8.8-202 x86_64 I have tons of errors like:
 
 Add. Sense: No additional sense information
 hv_storvsc vmbus_0_12: cmd 0x41 scsi status 0x2 srb status 0x6
 
 I believe it is related to WRITE_SAME don't implemented in win2012
 server. It seems that this patch could make the trick:
 
 https://patchwork.kernel.org/patch/2172871/
 
 But how could I check if I have it in my kernel or not?

It went upstream in v3.9-rc1 so if you have that or later, you have it.

-Eric

 Of course, I can be completely wrong. If you have any other ideas,
 please share :)
 
 Thanks for any help,
 
 Mario
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: ext4 - scsi: storvsc: avoid usage of WRITE_SAME

[Eric Sandeen]

On 5/8/13 9:30 AM, Mario Ceresa wrote:
 Thanks Eric, I'll upgrade to 3.9 then.
 
 Do you know if there is an easy way to know when a kernel patch goes upstream?

You can search in the git tree:

[root@host linux-2.6]# git log --pretty=oneline | grep storvsc: avoid usage
3e8f4f4065901c8dfc51407e1984495e1748c090 [SCSI] storvsc: avoid usage of 
WRITE_SAME
[root@host linux-2.6]# git describe --contains 
3e8f4f4065901c8dfc51407e1984495e1748c090
v3.9-rc1~21^2~7

-Eric

 Mario
 
 On 8 May 2013 16:28, Eric Sandeen sand...@redhat.com wrote:
 On 5/8/13 9:21 AM, Mario Ceresa wrote:
 Dear all,
 When accessing a VHDX disc with a f18 VM (hyperv) with kernel
 3.8.8-202 x86_64 I have tons of errors like:

 Add. Sense: No additional sense information
 hv_storvsc vmbus_0_12: cmd 0x41 scsi status 0x2 srb status 0x6

 I believe it is related to WRITE_SAME don't implemented in win2012
 server. It seems that this patch could make the trick:

 https://patchwork.kernel.org/patch/2172871/

 But how could I check if I have it in my kernel or not?

 It went upstream in v3.9-rc1 so if you have that or later, you have it.

 -Eric

 Of course, I can be completely wrong. If you have any other ideas,
 please share :)

 Thanks for any help,

 Mario



-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Do you think this is a security risk and if not is it a bad UI decision?

[Eric Sandeen]

On 5/3/13 10:58 PM, Matthew Garrett wrote:
 On Fri, May 03, 2013 at 08:52:25PM -0700, Dan Mashal wrote:
 On Fri, May 3, 2013 at 8:51 PM, Matthew Garrett mj...@srcf.ucam.org wrote:
 And if the maintainers feel more than justified in closing it again?
 Bugzilla isn't a discussion forum. If disagree with a deliberate policy
 decision, discuss it on an appropriate mailing list.

 Isn't that what we're doing? That's exactly the point of this email thread.
 
 No, this isn't the most appropriate mailing list for the discussion - 
 anaconda-devel-list is a better choice if you want to interact with the 
 people who actually work on that code. In any case, I was disagreeing 
 with Rahul's assertion that he was justified in re-opening a bug merely 
 because he disagreed with a design choice.

Matthew, with all due respect the tone of the bug doesn't make me think
that there is a lot of interest in discussion from the developers.

Whether or not bugzilla was the right place to start it, the early
discussion went something like this, mostly paraphrased.

Q: This seems to be a bug.  My password is visible while I type it.  I'm 
surprised.
   Is this a bug?

A: It's not a bug.  It's intentional.  There's quite a few papers about this 
right now.

Q: Link?

A: Google it.

At this point things may have turned a bit south.  If there is active research
or new thinking on this aspect of security, it should be part of the discussion.
If there's precedent, it's worth noting specifically.   That's the transparent,
open approach.  This isn't about the placement of a widget; this is about
someone's password in clear text.  It's worth having a broader discussion
about the implications.

If this had been on the anaconda list, most impacted parties would
not have seen it.  Speaking for myself, I'm glad it was brought up here.

The principal of least surprise is a good one.
I think this change breaks it.

What is the downside to defaulting to a hidden PW, with an opt-in mechanism to
display the password as it's typed?  The downsides of defaulting to cleartext 
have
been noted, and to me are quite self-explanatory.

On the other hand, if it's the right thing to do, then it needs to be done for
GUI password change dialogs and the passwd command should be updated as well,
for consistency, no?

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: Do you think this is a security risk and if not is it a bad UI decision?

[Eric Sandeen]

On 5/3/13 11:30 PM, Matthew Garrett wrote:
 On Fri, May 03, 2013 at 11:24:01PM -0500, Eric Sandeen wrote:
 On 5/3/13 10:58 PM, Matthew Garrett wrote:
 No, this isn't the most appropriate mailing list for the discussion - 
 anaconda-devel-list is a better choice if you want to interact with the 
 people who actually work on that code. In any case, I was disagreeing 
 with Rahul's assertion that he was justified in re-opening a bug merely 
 because he disagreed with a design choice.

 Matthew, with all due respect the tone of the bug doesn't make me think
 that there is a lot of interest in discussion from the developers.
 
 Reopening bugs is generally a good way of ensuring that there's even 
 less interest in discussion from the developers, and posting to mailing 
 lists that most of the developers concerned don't read has pretty 
 obvious problems in terms of changing their minds.

Fair enough, I hadn't looked closely at the timeline on the bug
history.

 If this had been on the anaconda list, most impacted parties would
 not have seen it.  Speaking for myself, I'm glad it was brought up here.
 
 I think making people aware of it on devel was a perfectly reasonable 
 thing to do, but most of the rest of the thread has been pretty 
 pointless. We don't make technical decisions by majority vote - most get 
 left up to the maintainers of the project in question.

Well, that bears some scrutiny, IMHO.

Anaconda has a pretty special place in this project.  It is the
uber-administrator of every new Fedora install.  We would do better
as a community to hash out major changes before they're made, and
try to reach some agreement before we implement them.

For example, the choice of the default fs ultimately is implemented
in Anaconda, but I'm pretty sure the change wouldn't be made without
wider discussion.  

Maybe the question is this: How much of the Anaconda team's job is
it to set Fedora OS policies and behaviors, vs. to implement them?

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: 64-bit stat (or not) in 32-bit Fedora binaries

[Eric Sandeen]

On 4/3/13 3:49 AM, Florian Weimer wrote:
 On 02/19/2013 12:15 AM, Eric Sandeen wrote:
 On 2/18/13 5:11 PM, John Reiser wrote:
 It would be useful to have a backward compatibility LD_PRELOAD shared
 library which intercepts the caller of stat32, and sets .st_ino to 0,
 without generating EOVERFLOW, whenever the actual inode number exceeds
 32 bits.  This would allow the *option* of continued operation
 (postponing the work of portability enhancements) for the vast majority
 of packages which do use stat() but do not inspect .st_ino.

 Yep, that would make some sense as a workaround.  FWIW, here's a systemtap
 script which intercepts ext4 stat  bumps the inode nr past 2^32
 (line numbers work on F18/kernel 3.7-8-ish at least)
 
 Do you have something similar for readdir?  Do you know what the kernel does 
 in this case?
 
 It would be quite annoying if the result was a truncated or incomplete 
 directory listing.

I haven't tested readdir but changing the stap script to do it probably 
wouldn't be
too tough, I think just doing something similar to bump up fname-inode
early in ext3/4's call_filldir() would do the trick at least for hashed dirs.

Looking at filldir() in the kernel, it does look like you'd get EOVERFLOW:

static int filldir(void * __buf, const char * name, int namlen, loff_t offset,
   u64 ino, unsigned int d_type)
{
...
unsigned long d_ino;
...
d_ino = ino;
if (sizeof(d_ino)  sizeof(ino)  d_ino != ino) {
buf-error = -EOVERFLOW;
return -EOVERFLOW;
}

(note again that although we're talking about ext3/4 here, they will never(tm)
have  32-bit inodes; they're just convenient to test on since everyone is 
running
them today).

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: XFS and trim

[Eric Sandeen]

On 4/1/13 5:26 PM, Steven Haigh wrote:
 On 04/02/2013 12:19 AM, Josef Bacik wrote:
 On Sat, Mar 30, 2013 at 9:38 PM, Steven Haigh net...@crc.id.au wrote:
 Hi all,

 Firstly, Please CC me into replies as I'm not subscribed to this list.

 I'm trying to confirm that Fedora 18 has enabled trim for XFS filesystems. I
 have added discard to the mount options in /etc/fstab - however I do not see
 it when looking at the output of 'mount':

...

 Can anyone share some insight into this?


 IIRC you have to rebuild your initrd so it picks up the new fstab
 mount options for /, the other options is you should be able to do
 mount -o remount,discard /.  Thanks,
 
 Hi Josef,
 
 This was the basis of my query - even after a remount with the
 discard option, 'discard' still wasn't present in the options - as
 seen by the mount line quoted above.

Sorry I didn't see this earlier.  Josef is right; discard is not a remountable
option on xfs, unfortunately (only inode64 and barrier options are remountable
today).  So you could either put the option in the rootflags= kernel
commandline parameter, or rebuild the initrd after adding it to fstab
as Josef suggested so that it's initially mounted with the option.

I wonder if we should add something to the remount path to printk when
a non-remountable option is encountered; I might look into that, otherwise
it's a little surprising (although semi-obvious when the problem doesn't
show up in /prcoc/mounts...).

Thanks,
-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: XFS and trim

[Eric Sandeen]

On 4/2/13 9:30 AM, Ric Wheeler wrote:
 On 04/02/2013 10:26 AM, Steven Haigh wrote:
 On 03/04/13 01:08, Eric Sandeen wrote:
 On 4/1/13 5:26 PM, Steven Haigh wrote:
 On 04/02/2013 12:19 AM, Josef Bacik wrote:
 On Sat, Mar 30, 2013 at 9:38 PM, Steven Haigh net...@crc.id.au wrote:
 Hi all,

 Firstly, Please CC me into replies as I'm not subscribed to this list.

 I'm trying to confirm that Fedora 18 has enabled trim for XFS 
 filesystems. I
 have added discard to the mount options in /etc/fstab - however I do not 
 see
 it when looking at the output of 'mount':

 ...

 Can anyone share some insight into this?


 IIRC you have to rebuild your initrd so it picks up the new fstab
 mount options for /, the other options is you should be able to do
 mount -o remount,discard /.  Thanks,

 Hi Josef,

 This was the basis of my query - even after a remount with the
 discard option, 'discard' still wasn't present in the options - as
 seen by the mount line quoted above.

 Sorry I didn't see this earlier.  Josef is right; discard is not a 
 remountable
 option on xfs, unfortunately (only inode64 and barrier options are 
 remountable
 today).  So you could either put the option in the rootflags= kernel
 commandline parameter, or rebuild the initrd after adding it to fstab
 as Josef suggested so that it's initially mounted with the option.

 Thanks for the reply Eric. I believe it would be helpful if this was 
 documented. I was expecting it to work the same way as ext4 - being a simple 
 remount would enable the discard option. Once that didn't work - and 
 especially as I didn't get an error in either the syslog or returned from 
 the command, I was starting to think it was a bug in the implementation of 
 TRIM in XFS.

 I wonder if we should add something to the remount path to printk when
 a non-remountable option is encountered; I might look into that, otherwise
 it's a little surprising (although semi-obvious when the problem doesn't
 show up in /prcoc/mounts...).

 I think this is probably the best way to handle things. The first thing I 
 did was to look at the return code from mount (which returned 0 - ie 
 success), then dmesg, then /var/log/messages. As I found nothing, I had no 
 hints on where to look.

 The wiki page[1] for XFS does state that XFS supports TRIM - and it mentions 
 the changes to /etc/fstab to enable it. It doesn't however mention that it 
 needs to be on the initial mount - or that a remount will not enable it.

 The best of both worlds would be to have it output to the dmesg - or even 
 fail the mount command (if that is an option?) just like ext4 would if you 
 supply a bogus option set.

 1: http://xfs.org/index.php/FITRIM/discard

 
 Is there any reason that discard cannot be enabled on remount like it is for 
 other file systems? And disabled on remount for completeness?

Discard should be one of the easier ones to change on remount; things like 
quota are tougher.  I was talking w/ Dave about this just yesterday, and he 
pointed out that we might need to serialize access to some of the things that 
would get changed on a remount.

BTW Steven - we often recommend against runtime trim, and suggest fstrim 
instead, for performance reasons.  Just something to maybe keep in mind and 
experiment with.

-Eric

 Seems like something that we would definitely want to be consistent across 
 file systems on,
 
 Ric
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: XFS and trim

[Eric Sandeen]

On 4/2/13 9:26 AM, Steven Haigh wrote:
 On 03/04/13 01:08, Eric Sandeen wrote:

snip

 I wonder if we should add something to the remount path to printk
 when a non-remountable option is encountered; I might look into
 that, otherwise it's a little surprising (although semi-obvious
 when the problem doesn't show up in /prcoc/mounts...).
 
 I think this is probably the best way to handle things. The first
 thing I did was to look at the return code from mount (which returned
 0 - ie success), then dmesg, then /var/log/messages. As I found
 nothing, I had no hints on where to look.

FWIW, there is some history here w/ returning 0/success for these options:

/*
 * Logically we would return an error here to prevent
 * users from believing they might have changed
 * mount options using remount which can't be changed.
 *
 * But unfortunately mount(8) adds all options from
 * mtab and fstab to the mount arguments in some cases
 * so we can't blindly reject options, but have to
 * check for each specified option if it actually
 * differs from the currently set option and only
 * reject it if that's the case.
 *
 * Until that is implemented we return success for
 * every remount request, and silently ignore all
 * options that we can't actually change.
 */
#if 0
xfs_info(mp,
mount option \%s\ not supported for remount\n, p);
return -EINVAL;
#else
break;
#endif

Not great, but there it is.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: 64-bit stat (or not) in 32-bit Fedora binaries

[Eric Sandeen]

On 2/20/13 8:23 AM, Chris Adams wrote:
 Once upon a time, Bill Nottingham nott...@redhat.com said:
 Florian Weimer (fwei...@redhat.com) said: 
 If we add -DFILE_OFFSET_BITS=64 to the default CFLAGS, this comes
 pretty close to an ABI bump.  But considering the numbers, I wonder
 if it's the right thing to do if we need to cope with 64-bit inode
 numbers.

 It's kind of a shame we just did the mass rebuild, if we're
 going to have to do part of it again if we change this.
 
 I don't think you should add that to unsuspecting code.  It isn't
 default because it can create incompatible libraries, can break code
 that doesn't use the right types.
 

Agreed.

I really don't think it's safe to add -D_FILE_OFFSET_BITS=64 to the
default CFLAGS.  Code will start getting bigger numbers and will
need to cope appropriately lest they overflow.   It'd be draconian
but I'd prefer to do something like was done for open() (I think?)
a while back and fail to link if the 64 bit interfaces aren't used.
That'd require each app/maintainer to do an audit and be thoughtful
about the fixes.

-Eric
-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: 64-bit stat (or not) in 32-bit Fedora binaries

[Eric Sandeen]

On 2/19/13 6:48 AM, Petr Pisar wrote:
 On 2013-02-18, Eric Sandeen sand...@redhat.com wrote:

 Anyway, if you want to check your package(s) and maybe make them
 64-bit-stat safe, the perl script above might help.  It's more than
 just -DFILE_OFFSET_BITS=64, since you'll need to be sure not to
 overflow any large values you get back from stat64 etc.

 Is %configure spec macro the right place to define FILE_OFFSET_BITS?

I don't think so, because then rather than getting an (unpleasant
but predictable) EOVERFLOW from the syscall, you might overflow local
variables and get even more random  unpredictable behavior for large
files or inodes.  Unfortunately, I think it's up to individual app writers
to take care of, in order to do it properly and safely.

-Eric

 Because it breaks struct stat ABI, it should be defined on distrubition
 level. For the same reason I can imagine upstream could be reluctant to
 embed the defintion for everybody.
 
 -- Petr
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: 64-bit stat (or not) in 32-bit Fedora binaries

[Eric Sandeen]

On 2/19/13 4:46 AM, Richard W.M. Jones wrote:
 On Mon, Feb 18, 2013 at 03:33:33PM -0600, Eric Sandeen wrote:
 XFS recently defaulted to allowing  32 bit inode numbers, and btrfs
 can let inode numbers creep past 2^32 as well.

 While most applications don't care one bit about st_ino returned
 from a stat() call, the sad fact is that you'll get EOVERFLOW from
 stat32 if the inode number is too big to fit in 32 bits, even if you
 just wanted to get the file size.

 I have a script (http://sandeen.net/misc/summarise_stat.pl) which
 Greg Banks wrote; it can check a path or list of filenames for
 binaries which contain non-64bit-safe stat calls.  A quick look over
 my F18 install finds the situation to be only slightly in favor of
 executables using 64-bit variants:

 # ./summarize-stat.pl /usr
   270229 91.5% are scripts (shell, perl, whatever)
22633  7.7% don't use any stat() family calls at all
  913  0.3% use 32-bit stat() family interfaces only
 1335  0.5% use 64-bit stat64() family interfaces only
   73  0.0% use both 32-bit and 64-bit stat() family interfaces

 and it's not just weird obscure packages:

 # ./summarize-stat.pl `rpm -ql sendmail`
  69 78.4% are scripts (shell, perl, whatever)
   2  2.3% don't use any stat() family calls at all
  17 19.3% use 32-bit stat() family interfaces only

 Anyway, if you want to check your package(s) and maybe make them
 64-bit-stat safe, the perl script above might help.  It's more than
 just -DFILE_OFFSET_BITS=64, since you'll need to be sure not to
 overflow any large values you get back from stat64 etc.

 Might be nice to get out ahead of this before, say, btrfs comes into
 wide use.  I don't know if there could be any more of a formal
 effort in this direction?
 
 Eric, I've read this email and the summarise_stat script a couple of
 times, and I admit I'm confused.

I think Petr answered, but -

 (1) Just ensuring the code is compiled with -DFILE_OFFSET_BITS=64 is
 sufficient to ensure the 32 bit stat will never be called, right?

Yep, I think so.

 (2) If my code never mentions st_ino, it's safe?
 
   [I assume the answer to this is *no* because you seem to be saying
   that -EOVERFLOW could be returned from an innocent-looking stat
   call, even if the code never looks at st_ino.]

Right, it's stat() itself that gives EOVERFLOW, it doesn't care if
you want st_ino or not.

 (3) For my code that uses st_ino, I need to ensure this is never
 assigned to a 32 bit integer (eg. 'int', 'int32_t', 'long' on 32 bit, etc.)?

To be safe I'd use it in an u64 type, I guess.  The *internal* kernel stat
structure uses u64:

struct kstat {
u64 ino;

 (4) Is doing (1)  (3) sufficient to fix all stat32-related problems
 in my code?

Yep, I think so.

 (5) With -DFILE_OFFSET_BITS=64, is st_ino a 64 bit value?

Yes. (well, a 64-bit container).  I wish I knew where the
canonical documentation was for these interfaces.

http://opengroup.org/platform/lfs.html is certainly related, as
is http://www.gnu.org/software/libc/manual/html_node/Reading-Attributes.html
although the latter only talks about file sizes. :(

feature_test_macros(7) says:

   _FILE_OFFSET_BITS
  Defining this macro with the value  64  automatically  converts
  references  to  32-bit functions and data types related to file
  I/O and file system operations into references to their  64-bit
  counterparts.  This is useful for performing I/O on large files
  ( 2 Gigabytes) on 32-bit systems.  (Defining this  macro  per-
  mits  correctly written programs to use large files with only a
  recompilation being required.)  64-bit systems naturally permit
  file  sizes greater than 2 Gigabytes, and on those systems this
  macro has no effect.


 Also a note that the current man page for stat(2) doesn't mention this
 problem, doesn't mention that EOVERFLOW could be returned in this
 surprising situation, and also has an example that casts st_ino to a
 long which I assume would be unsafe behaviour on a 32 bit
 architecture.  These are all bugs that the man-pages maintainers would
 no doubt be interested in.

*nod* since good docs seems sparse it could be improved.  I'll file a bug
against that, thanks.

-Eric

 Rich.
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Re: 64-bit stat (or not) in 32-bit Fedora binaries

[Eric Sandeen]

On 2/19/13 9:32 AM, Jakub Jelinek wrote:
 On Tue, Feb 19, 2013 at 09:22:55AM -0600, Eric Sandeen wrote:
 (3) For my code that uses st_ino, I need to ensure this is never
 assigned to a 32 bit integer (eg. 'int', 'int32_t', 'long' on 32 bit, etc.)?

 To be safe I'd use it in an u64 type, I guess.  The *internal* kernel stat
 structure uses u64:
 
 That would be wrong.  To store st_ino values, you should be using the ino_t
 type, like for file sizes/offsets (st_size, seeking, etc.) you should be
 using off_t.  Both of these types depend on _FILE_OFFSET_BITS macro.

Oh, of course this is correct.  Sorry, and thanks for pointing it out, Jakub.

-Eric

   Jakub
 

-- 
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel