Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
Joshua Baker-LePain wrote: > On Fri, 28 Aug 2009 at 8:58am, Akemi Yagi wrote > >> On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePain >> wrote: >> >>> You do not have to switch to 64bit, and your setup should be fully >>> supported. Other folks have mentioned XFS, and that's an option. >>> But if >>> you want to stay fully compatible with upstream, then ext3 is your only >>> option. >> >> Support for xfs has been added to RHEL 5.4 which will be released any >> day now. > > So it has. I recall looking in the beta release notes when they first > came out and not seeing it. So either I just plain missed it or it's > been added there since then. In any case, that's great news and > something that is *long* overdue. > Please note that it is not unheard of for them to PULL things that they have in the beta when rolling out the release. In this case, I certainly hope that they do not do this, but it is possible. Lets hope they do keep XFS in there. signature.asc Description: OpenPGP digital signature ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, Aug 28, 2009 at 9:58 AM, Joshua Baker-LePain wrote: > On Fri, 28 Aug 2009 at 8:58am, Akemi Yagi wrote >> Support for xfs has been added to RHEL 5.4 which will be released any day >> now. > > So it has. I recall looking in the beta release notes when they first came > out and not seeing it. So either I just plain missed it or it's been added > there since then. In any case, that's great news and something that is > *long* overdue. Indeed. I believe the beta release notes did not mention xfs. But it is here: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html-single/Release_Notes/ and the 5.4beta kernel does/did have xfs.ko. Akemi ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, 28 Aug 2009 at 8:58am, Akemi Yagi wrote On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePain wrote: You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option. Support for xfs has been added to RHEL 5.4 which will be released any day now. So it has. I recall looking in the beta release notes when they first came out and not seeing it. So either I just plain missed it or it's been added there since then. In any case, that's great news and something that is *long* overdue. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, Aug 28, 2009 at 9:53 AM, David Fix wrote: > And just to add to the discussion... > > We use JFS here for large filesystems. :) (We have some 24TB filesystems > in place here using JFS, with no problems like XFS has when it gets > corrupted). Because the distro kernel does not have support fort JFS, people who are looking into using JFS would need to install either the centosplus kernel or the jfs kernel module provided by ELRepo ( http://elrepo.org/tiki/kmod-jfs ). Akemi ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
And just to add to the discussion... We use JFS here for large filesystems. :) (We have some 24TB filesystems in place here using JFS, with no problems like XFS has when it gets corrupted). -- David Fix Senior Systems Administrator Mr. X Inc. 35 McCaul Street, Ste. #100 Toronto, ON M5T 1V7 T: (416) 595-6222, x 241 F: (416) 595-9122 E: dav...@mrxfx.com - Original Message - From: "Akemi Yagi" To: "CentOS mailing list" Sent: Friday, August 28, 2009 11:58:27 AM GMT -05:00 US/Canada Eastern Subject: Re: [CentOS] Setting up large (12.5 TB) filesystem howto? On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePain wrote: > You do not have to switch to 64bit, and your setup should be fully > supported. Other folks have mentioned XFS, and that's an option. But if > you want to stay fully compatible with upstream, then ext3 is your only > option. Support for xfs has been added to RHEL 5.4 which will be released any day now. Akemi ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePain wrote: > You do not have to switch to 64bit, and your setup should be fully > supported. Other folks have mentioned XFS, and that's an option. But if > you want to stay fully compatible with upstream, then ext3 is your only > option. Support for xfs has been added to RHEL 5.4 which will be released any day now. Akemi ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, 28 Aug 2009 at 1:03pm, Götz Reinicke - IT-Koordinator wrote fdisk and parted fail to create any information on the device or fail completely. You can't use fdisk on a volume that large. parted should work fine. What was the error you were getting (exactly)? For a volume that large, you must use a GPT disk label, not the default msdos one. But, I can't create a filesystem on it: mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0 mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported. As has been pointed out, you need to use "-F" to force mkfs.ext3 to make a filesystem bigger than 8TB. IMHO, this is misleading. Filesystems up to 16TB are fully supported in centos >5.1, so I don't see why the upstream vendor left the requirement for "-F" in mkfs.ext3. So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit? You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
Götz Reinicke - IT-Koordinator wrote: > Hi, > > I'm trying to set up an iscsi 12.5 TB storage for some data backup. > > Doing so, I had some difficulties to find the right tool, maybe it's > also a question of the system settings... > > The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi > connection can be establised. > > fdisk and parted fail to create any information on the device or fail > completely. > > using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly > create a logical volume: > > lvdisplay /dev/VolGroup02/lvol0 > --- Logical volume --- > LV Name/dev/VolGroup02/lvol0 > VG NameVolGroup02 > LV UUIDh7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x > LV Write Accessread/write > LV Status available > # open 0 > LV Size12,73 TB > Current LE 3337487 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:4 > > > But, I can't create a filesystem on it: > > mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 > /dev/mapper/VolGroup02-lvol0 > > > mke2fs 1.39 (29-May-2006) > mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks >(8TB using a blocksize of 4k) are currently supported. > > > The limits information provided by red hat say, that RH EL 5.1 supports > 16 TB filesystems: > > http://www.redhat.com/rhel/compare/ > > -> Maximum filesystem size (Ext3): 16TB in 5.1 > > Using a block size of 8192 gives a warning, that this size is to large > for that system. > > > So my question: What is my missunderstanding or what's wrong with my > system? Where are the real limits? Do I have to switch the OS to 64 Bit? > > Setting up large Filesystems isn't my staff of life :-) > > > Thanks for amy how to or help of any kind - Best regards, > > Götz > > Gotz, YMMV, but XFS is what I would use (and do use) as a filesystem in this instance. There are caveats, some of which are listed here: 1) Server/storage must be on stable power, backed up by a ups. I have never dealt with XFS filesystem corruption, but I have read anecdotal horror stories. 2) You need to run 64 bit, if for no other reason than memory requirements. 3) You will need to have a partition that can be used as swap space. I have not found any consistent formula, but I am running multiple servers using XFS, with 12gig of ram, 9TB filesystem, and 12 gigs was not enough memory to run xfs_check, which you should do on occasion. I wound up making a 20gig swap space on a separate partition while I was running xfs_check. That may have been excessively large, but it worked... 4) XFS is not (or has not been) part of the standard RHEL distro, so you lose the 1:1 aspects of running CentOS when you implement XFS. My experience has been that XFS is quite a bit faster than ext3, especially during file manipulation procedures. Others may not agree. It has proven rock-solid for me, both in the linux world and IRIX. Below is a link discussing the need for 64bit vs. 32bit (a bit old but still relevant): http://oss.sgi.com/archives/xfs/2005-08/msg00391.html Hope this helps. Monty ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On 28/08/2009, at 10:59 PM, Jim Perrin wrote: > On Fri, Aug 28, 2009 at 7:03 AM, Götz Reinicke - > IT-Koordinator wrote: >> Hi, >> >> I'm trying to set up an iscsi 12.5 TB storage for some data backup. >> >> Doing so, I had some difficulties to find the right tool, maybe it's >> also a question of the system settings... >> >> The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi >> connection can be establised. >> >> fdisk and parted fail to create any information on the device or fail >> completely. >> >> using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly >> create a logical volume: >> >> lvdisplay /dev/VolGroup02/lvol0 >> --- Logical volume --- >> LV Name/dev/VolGroup02/lvol0 >> VG NameVolGroup02 >> LV UUIDh7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x >> LV Write Accessread/write >> LV Status available >> # open 0 >> LV Size12,73 TB >> Current LE 3337487 >> Segments 1 >> Allocation inherit >> Read ahead sectors auto >> - currently set to 256 >> Block device 253:4 >> >> >> But, I can't create a filesystem on it: >> >> mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 >> /dev/mapper/VolGroup02-lvol0 > > You have to smack mkfs around a bit to get it to work. > The incantations are listed here, along with the various limitations. > The long and short of it is that you have to use -F to tell mkfs that > you're really, REALLY sure. > http://www.bofh-hunter.com/2008/02/11/large-filesystem-creation/ > For a 4K block size I think the maximum ext3 filesystem size is 8TB. You may need to use an 8K block size which gives a maximum filesystem size of 16TB. Neil ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, Aug 28, 2009 at 7:03 AM, Götz Reinicke - IT-Koordinator wrote: > Hi, > > I'm trying to set up an iscsi 12.5 TB storage for some data backup. > > Doing so, I had some difficulties to find the right tool, maybe it's > also a question of the system settings... > > The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi > connection can be establised. > > fdisk and parted fail to create any information on the device or fail > completely. > > using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly > create a logical volume: > > lvdisplay /dev/VolGroup02/lvol0 > --- Logical volume --- > LV Name /dev/VolGroup02/lvol0 > VG Name VolGroup02 > LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x > LV Write Access read/write > LV Status available > # open 0 > LV Size 12,73 TB > Current LE 3337487 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:4 > > > But, I can't create a filesystem on it: > > mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 > /dev/mapper/VolGroup02-lvol0 You have to smack mkfs around a bit to get it to work. The incantations are listed here, along with the various limitations. The long and short of it is that you have to use -F to tell mkfs that you're really, REALLY sure. http://www.bofh-hunter.com/2008/02/11/large-filesystem-creation/ -- During times of universal deceit, telling the truth becomes a revolutionary act. George Orwell ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, Aug 28, 2009 at 1:03 PM, Götz Reinicke - IT-Koordinator wrote: > > I'm trying to set up an iscsi 12.5 TB storage for some data backup. > > Doing so, I had some difficulties to find the right tool, maybe it's > also a question of the system settings... > > The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi > connection can be establised. > ...snip... > > The limits information provided by red hat say, that RH EL 5.1 supports > 16 TB filesystems: > > http://www.redhat.com/rhel/compare/ > > -> Maximum filesystem size (Ext3): 16TB in 5.1 > > Using a block size of 8192 gives a warning, that this size is to large > for that system. > > So my question: What is my missunderstanding or what's wrong with my > system? Where are the real limits? Do I have to switch the OS to 64 Bit? > That was discussed not so long ago. But I can't remember if it was here in the mailing list or on IRC (or somewhere else). But the conclusion was that you are better of using 64 bit and XFS for such large filesystems, and that you need a 64 bit system anyway for ext3/4 to go over 8TB. Well, that is what I remember the conclusion to be :-) Regards, Tim -- Tim Verhoeven - tim.verhoeven...@gmail.com - 0479 / 88 11 83 Hoping the problem magically goes away by ignoring it is the "microsoft approach to programming" and should never be allowed. (Linus Torvalds) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Setting up large (12.5 TB) filesystem howto?
Hi, I'm trying to set up an iscsi 12.5 TB storage for some data backup. Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings... The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised. fdisk and parted fail to create any information on the device or fail completely. using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume: lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name/dev/VolGroup02/lvol0 VG NameVolGroup02 LV UUIDh7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Accessread/write LV Status available # open 0 LV Size12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 But, I can't create a filesystem on it: mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0 mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported. The limits information provided by red hat say, that RH EL 5.1 supports 16 TB filesystems: http://www.redhat.com/rhel/compare/ -> Maximum filesystem size (Ext3): 16TB in 5.1 Using a block size of 8192 gives a warning, that this size is to large for that system. So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit? Setting up large Filesystems isn't my staff of life :-) Thanks for amy how to or help of any kind - Best regards, Götz -- Götz Reinicke IT-Koordinator Tel. +49 7141 969 420 Fax +49 7141 969 55 420 E-Mail goetz.reini...@filmakademie.de Filmakademie Baden-Württemberg GmbH Akademiehof 10 71638 Ludwigsburg www.filmakademie.de Eintragung Amtsgericht Stuttgart HRB 205016 Vorsitzende des Aufsichtsrats: Prof. Dr. Claudia Hübner Staatsrätin für Demographischen Wandel und für Senioren im Staatsministerium Geschäftsführer: Prof. Thomas Schadt ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
