Re: [CentOS] 32-bit kernel+XFS+16.xTB filesystem = potential disaster (was:Re: ZFS @ centOS)
On 4/5/2011 11:21 AM, Lamar Owen wrote: Dropping to 16.37 TB on the RAID configuration by switching to RAID-6 let us put almost the entire array under a single 16 TB XFS filesystem. You really, really, really don't want to do this. Actually, it seems that you can't do it any more. I tried, just to see what would happen. (I already knew about all you're talking about.) You are still able to convince gparted to create a 16 TB partition on 32-bit CentOS 5, but when you go to format it, it gives some bogus error that doesn't tell you what actually went wrong. Repartition to get under 16 TB, and the mkfs step succeeds. I expect they added some checks for this since you last tried XFS on 32-bit. Perhaps it wasn't clear from what I wrote, but the big partition on this system is actually 15.9mumble TB, just to be sure we don't even get 1 byte over the limit. The remaining 1/3 TB is currently unused. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 32-bit kernel+XFS+16.xTB filesystem = potential disaster (was:Re: ZFS @ centOS)
On Wednesday, April 06, 2011 01:16:19 PM Warren Young wrote: I expect they added some checks for this since you last tried XFS on 32-bit. Perhaps it wasn't clear from what I wrote, but the big partition on this system is actually 15.9mumble TB, just to be sure we don't even get 1 byte over the limit. The remaining 1/3 TB is currently unused. I didn't get there in one step. Perhaps that's the difference. What you say in the last paragraph will prevent the effect I saw. Just hope you don't need to do an xfs_repair. No, it wasn't completely clear that you were keeping below 16TB from what you wrote, at least not to me. Now, I didn't do mkfs on a 16.xTB disk initially; I got there in steps with LVM, lvextend, and xfs_growfs. The starting size of the filesystem was ~4TB in two ~2TB LUNs/PV's; VMware is limited to 2TB LUNs, so I added storage, as needed, in ~2TB chunks (actually did 2,000GB chunks; pvscan reports these as 1.95TB (with some at 1.92TB for RAID group setup reasons). The 1.32TB and 1.37TB LUNs are there due to the way the RAID groups on this Clariion CX3-10c this is on are set up. So after a while of doing this, I had a hair over 14TB; xfs_growfs going from 14TB to a hair over 16TB didn't complain. But when the data hit 16TB, it quit mounting. So I migrated to a C5 x86_64 VM, and things started working again. I've added one more 1.95TB PV to the VG since then. Current setup: PV /dev/sdd1VG pachy-mirror lvm2 [1.92 TB / 0free] PV /dev/sdg1VG pachy-mirror lvm2 [1.92 TB / 0free] PV /dev/sde1VG pachy-mirror lvm2 [1.95 TB / 0free] PV /dev/sdu1VG pachy-mirror lvm2 [1.95 TB / 0free] PV /dev/sdl1VG pachy-mirror lvm2 [1.37 TB / 0free] PV /dev/sdm1VG pachy-mirror lvm2 [1.32 TB / 0free] PV /dev/sdx1VG pachy-mirror lvm2 [1.95 TB / 0free] PV /dev/sdz1VG pachy-mirror lvm2 [1.95 TB / 0free] PV /dev/sdab1 VG pachy-mirror lvm2 [1.95 TB / 0free] PV /dev/sdt1VG pachy-mirror lvm2 [1.95 TB / 0free] ACTIVE'/dev/pachy-mirror/home' [18.24 TB] inherit The growth was over a period of two years, incidentally. There are other issues with XFS and 32-bit; see: http://bugs.centos.org/view.php?id=3364 and http://www.mail-archive.com/scientific-linux-users@listserv.fnal.gov/msg05347.html and google for 'XFS 32-bit 4K stacks' for more of the gory details. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 32-bit kernel+XFS+16.xTB filesystem = potential disaster (was:Re: ZFS @ centOS)
On 4/6/2011 1:16 PM, Lamar Owen wrote: There are other issues with XFS and 32-bit; see: http://bugs.centos.org/view.php?id=3364 and http://www.mail-archive.com/scientific-linux-users@listserv.fnal.gov/msg05347.html and google for 'XFS 32-bit 4K stacks' for more of the gory details. Thanks for the info. The problem seems to be tied to LVM and high amounts of I/O, particularly writes. None of that applies to this application. The filesystem is a plain-old partition, the array is mostly going to be read-only, and due to a bottleneck elsewhere in the system, the peak read rate can't be higher than 20 Mbyte/s. (If you're wondering, then, why I bothered to benchmark the system at all, it's because we get much higher I/O rates when initially loading the array up, so that the low write speed of ZFS-FUSE would have increased that initial load from days to weeks.) That load went off without a hitch, so we've probably already done the worst thing to this server that it will ever see. (Kind of like the old advice for happiness: eat a bug every morning, and nothing worse will happen to you for the rest of the day.) We'll test it under load before shipping it anyway, however. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] 32-bit kernel+XFS+16.xTB filesystem = potential disaster (was:Re: ZFS @ centOS)
On Monday, April 04, 2011 11:09:29 PM Warren Young wrote: I did this test with Bonnie++ on a 3ware/LSI 9750-8i controller, with eight WD 3 TB disks attached. Both tests were done with XFS on CentOS 5.5, 32-bit. (Yes, 32-bit. Hard requirement for this application.) [snip] For the RAID-6 configuration, I used the 3ware card's hardware RAID, creating a single ~16 TB volume, formatted XFS. [snip] Dropping to 16.37 TB on the RAID configuration by switching to RAID-6 let us put almost the entire array under a single 16 TB XFS filesystem. You really, really, really don't want to do this. Not on 32-bit. When you roll one byte over 16TB you will lose access to your filesystem, silently, and it will not remount on a 32-bit kernel. XFS works best on a 64-bit kernel for a number of reasons; the one you're likely to hit first is the 16TB hard limit for *occupied* file space; you can mkfs an XFS filesystem on a 17TB or even larger partition or volume, but the moment the occupied data rolls over the 16TB boundary you will be in disaster recovery mode, and a 64-bit kernel will be required for rescue. The reason I know this? I had it happen. On a CentOS 32-bit backup server with a 17TB LVM logical volume on EMC storage. Worked great, until it rolled 16TB. Then it quit working. Altogether. /var/log/messages told me that the filesystem was too large to be mounted. Had to re-image the VM as a 64-bit CentOS, and then re-attached the RDM's to the LUNs holding the PV's for the LV, and it mounted instantly, and we kept on trucking. There's a reason upstream doesn't do XFS on 32-bit. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] 32-bit kernel+XFS+16.xTB filesystem = potential disaster (was:Re: ZFS @ centOS)
On Tue, Apr 5, 2011 at 10:21 AM, Lamar Owen lo...@pari.edu wrote: You really, really, really don't want to do this. Not on 32-bit. When you roll one byte over 16TB you will lose access to your filesystem, silently, and it will not remount on a 32-bit kernel. XFS works best on a 64-bit kernel for a number of reasons; the one you're likely to hit first is the 16TB hard limit for *occupied* file space; you can mkfs an XFS filesystem on a 17TB or even larger partition or volume, but the moment the occupied data rolls over the 16TB boundary you will be in disaster recovery mode, and a 64-bit kernel will be required for rescue. The reason I know this? I had it happen. On a CentOS 32-bit backup server with a 17TB LVM logical volume on EMC storage. Worked great, until it rolled 16TB. Then it quit working. Altogether. /var/log/messages told me that the filesystem was too large to be mounted. Had to re-image the VM as a 64-bit CentOS, and then re-attached the RDM's to the LUNs holding the PV's for the LV, and it mounted instantly, and we kept on trucking. There's a reason upstream doesn't do XFS on 32-bit. Afaik 32-bit binaries do run on the 64-bit build and compat libraries exist for most everything. You should evaluate if you really *really* need 32-bit. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
