Bug#901427: Unable to enable ext4 journaled quota
On Sun, Jun 17, 2018 at 02:31:29PM -0400, Theodore Y. Ts'o wrote: > On Fri, Jun 15, 2018 at 07:37:19PM -0700, Elliott Mitchell wrote: > > > > During the process I ended up running `e2fsck -f` multiple times. I > > ended up running `e2fsck -f` after enabling each feature (`tune2fs` > > really didn't like enabling multiple features at once). > I will note that if you are starting with an ext2/ext3 file system (one > without extents and the uninit_bg or metadata_csum features), you will > get much better performance in the long run if you create a new ext4 > file system and then copy all of the data from the ext2 file system to > ext4. > > My experience from when we did an ext2 upgrade to ext4 (in no-journal > mode) some six or seven years ago, for our workload, which was for a > cluster file system, was comparing the performance of an existing file > system upgrading to ext4 in place, versus the performance of a freshly > created ext4 file system, was that the upgrade-in-place experience > resulted in roughly half the performance improvement compared to a > fresh ext4 file system. > > So if you are going to need to run tune2fs and e2fsck -f multiple > times what are you trying to do? What's the goal here? Performance isn't a critical issue for this system. As long as performance is reasonable and ideally not too much system admin time is spent, that is good enough. I'm more concerned with keeping the system up to date. Everything is moving off ext3 onto ext4 and since one crucial utility finally got updated for ext4, I'm doing upgrades. One thing which had been an issue was getting journalled quotas. Doing full quota checks by having `repquota` do what amounts to an `fsck` is kind of a bit slow. I thought I'd managed to accomplish that while still on ext3, but I wasn't absolutely certain... > Don't get me wrong; if you can get me a reproducible test case, I'm > happy to look at it. But Red Hat doesn't support upgrading file > systems via tune2fs. And that's because there are all sorts of corner > cases and it's very hard to do regression testing or bug > reproductions. We have used tune2fs to upgrade file systems at > Google, but it's something we testetd extensively, and we only did it > where it makes sense. RedHat support isn't an issue for me if you notice where this report appeared. :-) It was my understanding upgrade via `tune2fs` is supposed to work, so I used the handy tool. > > > And do you really need to enable user, group, and project quota > > > tracking? Please keep in mind there is a performance cost for > > > enabling quota tracking; it is definitely not free > > > > I definitely want user quota with journal. I'm unsure of group and > > project. > > For each quota type that you enable, you basically end up paying for > an extra random write, and potentially a random read, associated with > block or inode allocation for a given user or group id, every five > seconds. So if you don't need group or project tracking, don't turn > it on. We only track usage based on a group basis, so we only enable > group quotas. > > As far as the turning on the quota feature and that terrible error > message --- did the file system have any quota files before, and did > you check to see if the file system was full (e.g., not enough blocks > to allocate the quota file)? Yes. I could well believe `tune2fs` is unable to upgrade quota type, and that is where the problem came from. Disabling one quota type may be worthwhile. `dumpe2fs -h` with magic numbers stripped, filesystem 0: Filesystem magic number: 0xEF53 Filesystem revision #:1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit mmp sparse_super large_file metadata_csum Filesystem flags: signed_directory_hash Default mount options:user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 262144 Block count: 1047552 Reserved block count: 52377 Free blocks: 828522 Free inodes: 253868 First block: 0 Block size: 4096 Fragment size:4096 Group descriptor size:64 Reserved GDT blocks: 255 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Filesystem created: Fri May 12 18:59:53 2017 Last mount time: Tue Jun 12 23:30:50 2018 Last write time: Tue Jun 12 23:30:50 2018 Mount count: 2 Maximum mount count: 22 Last checked: Tue Jun 12 22:52:53 2018 Check interval: 0 () Lifetime writes: 350 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Default directory hash: half_md4 MMP block
Bug#901427: Unable to enable ext4 journaled quota
On Fri, Jun 15, 2018 at 07:37:19PM -0700, Elliott Mitchell wrote: > > During the process I ended up running `e2fsck -f` multiple times. I > ended up running `e2fsck -f` after enabling each feature (`tune2fs` > really didn't like enabling multiple features at once). OK, so I really need a reliable repro to be able to do anything with this report. I will note that if you are starting with an ext2/ext3 file system (one without extents and the uninit_bg or metadata_csum features), you will get much better performance in the long run if you create a new ext4 file system and then copy all of the data from the ext2 file system to ext4. My experience from when we did an ext2 upgrade to ext4 (in no-journal mode) some six or seven years ago, for our workload, which was for a cluster file system, was comparing the performance of an existing file system upgrading to ext4 in place, versus the performance of a freshly created ext4 file system, was that the upgrade-in-place experience resulted in roughly half the performance improvement compared to a fresh ext4 file system. So if you are going to need to run tune2fs and e2fsck -f multiple times what are you trying to do? What's the goal here? Don't get me wrong; if you can get me a reproducible test case, I'm happy to look at it. But Red Hat doesn't support upgrading file systems via tune2fs. And that's because there are all sorts of corner cases and it's very hard to do regression testing or bug reproductions. We have used tune2fs to upgrade file systems at Google, but it's something we testetd extensively, and we only did it where it makes sense. > > And do you really need to enable user, group, and project quota > > tracking? Please keep in mind there is a performance cost for > > enabling quota tracking; it is definitely not free > > I definitely want user quota with journal. I'm unsure of group and > project. For each quota type that you enable, you basically end up paying for an extra random write, and potentially a random read, associated with block or inode allocation for a given user or group id, every five seconds. So if you don't need group or project tracking, don't turn it on. We only track usage based on a group basis, so we only enable group quotas. As far as the turning on the quota feature and that terrible error message --- did the file system have any quota files before, and did you check to see if the file system was full (e.g., not enough blocks to allocate the quota file)? - Ted
Bug#901427: Unable to enable ext4 journaled quota
On Fri, Jun 15, 2018 at 08:39:05PM -0400, Theodore Y. Ts'o wrote: > On Tue, Jun 12, 2018 at 11:38:15PM -0700, Elliott Mitchell wrote: > > # tune2fs -O dir_index,extent,metadata_csum,mmp,quota -Q > > usrquota,grpquota,prjquota /dev/foo > > tune2fs 1.43.4 (31-Jan-2017) > > [ERROR] ../../../../lib/support/quotaio.c:275:quota_file_open:: > > qh_ops->check_file failed > > tune2fs: Unknown code 255 while updating quota limits (0) > > I apologize for lousy error messages from the quota handling code, in > particular. > > Can you tell me something about the file system you were starting > with? In particular, can you send me the output of dumpe2fs -h. > Also, have you tried running e2fsck on the file system in question > first? During the process I ended up running `e2fsck -f` multiple times. I ended up running `e2fsck -f` after enabling each feature (`tune2fs` really didn't like enabling multiple features at once). The dumpe2fs will take a bit to get to... > Finally, what are you trying to do? It looks like you're starting > with an old ext file system and trying to enable a bunch of new ext4 > features? Why do you need to enable all of these features? In > particular, unless you really know what you are doing, you probably > don't need mmp. > > And do you really need to enable user, group, and project quota > tracking? Please keep in mind there is a performance cost for > enabling quota tracking; it is definitely not free I definitely want user quota with journal. I'm unsure of group and project. > > All of the other features had been successfully enabled with some > > trouble (mostly needing several runs of `fsck -f`), but quota enabling > > failed. > > > > The error messages from `tune2fs` were less than helpful during the > > process of getting flags enabled. "mmp" was particularly troublesome. > > How so? And again, why do you need (or think you need) mmp? I'm pretty sure mmp is appropriate in this case. This is a server with multiple VMs. A few times I have found it useful to access guest filesysterms from the hypervisor, such as the upgrade `tune2fs` operation I was doing here. Adjusting the guest /etc/fstab from the hypervisor was also useful. This opens the door to accidentally mounting the guest root filesystem read-write from the hypervisor, while the guest has the filesystem mounted read-only (or worse mounting it read-write in both locations at the same time; *crunch*). So yes, I believe I have a case where MMP functionality is useful. Hopefully I won't actually need it, but it should be useful to defend against operator error (I like to think I'm perfect, but experience has demonstrated otherwise). -- (\___(\___(\__ --=> 8-) EHM <=-- __/)___/)___/) \BS (| ehem+sig...@m5p.com PGP 87145445 |) / \_CS\ | _ -O #include O- _ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
Bug#901427: Unable to enable ext4 journaled quota
On Tue, Jun 12, 2018 at 11:38:15PM -0700, Elliott Mitchell wrote: > Package: e2fsproc > Version: 1.43.4-2 > > Subject tells the story: > > # tune2fs -O dir_index,extent,metadata_csum,mmp,quota -Q > usrquota,grpquota,prjquota /dev/foo > tune2fs 1.43.4 (31-Jan-2017) > [ERROR] ../../../../lib/support/quotaio.c:275:quota_file_open:: > qh_ops->check_file failed > tune2fs: Unknown code 255 while updating quota limits (0) I apologize for lousy error messages from the quota handling code, in particular. Can you tell me something about the file system you were starting with? In particular, can you send me the output of dumpe2fs -h. Also, have you tried running e2fsck on the file system in question first? Finally, what are you trying to do? It looks like you're starting with an old ext file system and trying to enable a bunch of new ext4 features? Why do you need to enable all of these features? In particular, unless you really know what you are doing, you probably don't need mmp. And do you really need to enable user, group, and project quota tracking? Please keep in mind there is a performance cost for enabling quota tracking; it is definitely not free > All of the other features had been successfully enabled with some > trouble (mostly needing several runs of `fsck -f`), but quota enabling > failed. > > The error messages from `tune2fs` were less than helpful during the > process of getting flags enabled. "mmp" was particularly troublesome. How so? And again, why do you need (or think you need) mmp? - Ted
Bug#901427: Unable to enable ext4 journaled quota
Package: e2fsproc Version: 1.43.4-2 Subject tells the story: # tune2fs -O dir_index,extent,metadata_csum,mmp,quota -Q usrquota,grpquota,prjquota /dev/foo tune2fs 1.43.4 (31-Jan-2017) [ERROR] ../../../../lib/support/quotaio.c:275:quota_file_open:: qh_ops->check_file failed tune2fs: Unknown code 255 while updating quota limits (0) # All of the other features had been successfully enabled with some trouble (mostly needing several runs of `fsck -f`), but quota enabling failed. The error messages from `tune2fs` were less than helpful during the process of getting flags enabled. "mmp" was particularly troublesome. -- (\___(\___(\__ --=> 8-) EHM <=-- __/)___/)___/) \BS (| ehem+sig...@m5p.com PGP 87145445 |) / \_CS\ | _ -O #include O- _ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
