Re: [Lustre-discuss] SLES 11 SP1 Client rpms built but not working
Dear Andreas, Many thanks for your response. Below are further details on this. I shall be grateful for your advice on this. Regards, Peter The system is: cmip-proc8:/etc # uname -a Linux cmip-proc8.badc.rl.ac.uk 2.6.32.29-0.3-xen #1 SMP 2011-02-25 13:36:59 +0100 x86_64 x86_64 x86_64 GNU/Linux /usr/src/linux is a symlink pointing to the source corresponding to linux-2.6.32.29-0.3-obj: cmip-proc8:/etc # ls -l /usr/src total 24 drwxr-xr-x 3 root root 4096 2011-05-09 08:31 debug lrwxrwxrwx 1 root root 19 2011-03-20 15:54 linux - linux-2.6.32.29-0.3 drwxr-xr-x 25 root root 4096 2011-05-09 08:49 linux-2.6.32.29-0.3 drwxr-xr-x 3 root root 4096 2011-03-20 15:54 linux-2.6.32.29-0.3-obj drwxr-xr-x 3 root root 4096 2011-03-20 15:54 linux-obj drwxr-xr-x 10 root root 4096 2011-05-09 08:31 lustre-1.8.5 drwxr-xr-x 7 root root 4096 2011-03-20 14:58 packages cmip-proc8:/etc # cmip-proc8:~ # ls /usr/local/kits/lustre-1.8.5 aclocal.m4 config.h.ininstall-sh Makefile autoMakefile config.log ldiskfs Makefile.in autoMakefile.am config.status libsysio missing autoMakefile.in config.sub lnet mkinstalldirs buildconfigure lustre README ChangeLogconfigure.ac lustre-1.8.5.tar.gz Rules compile COPYINGlustre-iokit snmp config.guess debian lustre.spec stamp-h1 config.h depcomplustre.spec.in tree_status cmip-proc8:~ # The build with .configure and make rpms produced rpms that are installable: cmip-proc8:/etc # ls -ls /usr/src/packages/RPMS/x86_64/*1.8.5* 4024 -rw-r--r-- 1 root root 4112883 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-1.8.52.6.32.29_0.3_xen_201105090815.x86_64.rpm 15532 -rw-r--r-- 1 root root 15881360 2011-05-09 08:54 /usr/src/packages/RPMS/x86_64/lustre-debuginfo-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 1332 -rw-r--r-- 1 root root 1358924 2011-05-09 08:54 /usr/src/packages/RPMS/x86_64/lustre-debugsource-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 1416 -rw-r--r-- 1 root root 1441937 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 3524 -rw-r--r-- 1 root root 3602163 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-source-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 2600 -rw-r--r-- 1 root root 2656393 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm cmip-proc8:/etc # rpm -e lustre-tests cmip-proc8:/etc # rpm -e lustre cmip-proc8:/etc # rpm -e lustre-modules cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre-modules ### [100%] Congratulations on finishing your Lustre installation! To register your copy of Lustre and find out more about Lustre Support, Service, and Training offerings please visit http://www.sun.com/software/products/lustre/lustre_reg.jsp cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre ### [100%] cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre-tests ### [100%] cmip-proc8:/etc # ... cmip-proc8:/etc # rpm -qa | grep lustre lustre-debuginfo-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-debugsource-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-source-1.8.5-2.6.32.29_0.3_xen_201105090815 The problem reproduces: cmip-proc8:~ # cp /var/log/messages /tmp/m0 cmip-proc8:~ # dmesg /tmp/d0 cmip-proc8:~ # lsmod | grep lustre cmip-proc8:~ # modprobe lustre Killed cmip-proc8:~ # dmesg /tmp/d1 cmip-proc8:~ # cp /var/log/messages /tmp/m1 cmip-proc8:~ # diff /tmp/d0 /tmp/d1 193a194,235 [ 84.786822] SFW2-INext-DROP-DEFLT IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:30:1e:5d:54:80:08:00 SRC=130.246.188.226 DST=224.0.0.1 LEN=28 TOS=0x00 PREC=0x00 TTL=1 ID=34816 PROTO=2 [ 104.171306] BUG: unable to handle kernel NULL pointer dereference at 0008 [ 104.171317] IP: [8002c3d2] task_rq_lock+0x42/0xa0 [ 104.171328] PGD 7d9d0067 PUD 7d94c067 PMD 0 [ 104.171333] Oops: [#1] SMP [ 104.171336] last sysfs file: /sys/module/ip_tables/initstate [ 104.171339] CPU 0 [
Re: [Lustre-discuss] SLES 11 SP1 Client rpms built but not working
The only other potential problem I see is that you are using a xen kernel and this us somehow causing problems. Cheers, Andreas On 2011-05-11, at 1:33 AM, peter.c...@stfc.ac.uk wrote: Dear Andreas, Many thanks for your response. Below are further details on this. I shall be grateful for your advice on this. Regards, Peter The system is: cmip-proc8:/etc # uname -a Linux cmip-proc8.badc.rl.ac.uk 2.6.32.29-0.3-xen #1 SMP 2011-02-25 13:36:59 +0100 x86_64 x86_64 x86_64 GNU/Linux /usr/src/linux is a symlink pointing to the source corresponding to linux-2.6.32.29-0.3-obj: cmip-proc8:/etc # ls -l /usr/src total 24 drwxr-xr-x 3 root root 4096 2011-05-09 08:31 debug lrwxrwxrwx 1 root root 19 2011-03-20 15:54 linux - linux-2.6.32.29-0.3 drwxr-xr-x 25 root root 4096 2011-05-09 08:49 linux-2.6.32.29-0.3 drwxr-xr-x 3 root root 4096 2011-03-20 15:54 linux-2.6.32.29-0.3-obj drwxr-xr-x 3 root root 4096 2011-03-20 15:54 linux-obj drwxr-xr-x 10 root root 4096 2011-05-09 08:31 lustre-1.8.5 drwxr-xr-x 7 root root 4096 2011-03-20 14:58 packages cmip-proc8:/etc # cmip-proc8:~ # ls /usr/local/kits/lustre-1.8.5 aclocal.m4 config.h.ininstall-sh Makefile autoMakefile config.log ldiskfs Makefile.in autoMakefile.am config.status libsysio missing autoMakefile.in config.sub lnet mkinstalldirs buildconfigure lustre README ChangeLogconfigure.ac lustre-1.8.5.tar.gz Rules compile COPYINGlustre-iokit snmp config.guess debian lustre.spec stamp-h1 config.h depcomplustre.spec.in tree_status cmip-proc8:~ # The build with .configure and make rpms produced rpms that are installable: cmip-proc8:/etc # ls -ls /usr/src/packages/RPMS/x86_64/*1.8.5* 4024 -rw-r--r-- 1 root root 4112883 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-1.8.52.6.32.29_0.3_xen_201105090815.x86_64.rpm 15532 -rw-r--r-- 1 root root 15881360 2011-05-09 08:54 /usr/src/packages/RPMS/x86_64/lustre-debuginfo-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 1332 -rw-r--r-- 1 root root 1358924 2011-05-09 08:54 /usr/src/packages/RPMS/x86_64/lustre-debugsource-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 1416 -rw-r--r-- 1 root root 1441937 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 3524 -rw-r--r-- 1 root root 3602163 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-source-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 2600 -rw-r--r-- 1 root root 2656393 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm cmip-proc8:/etc # rpm -e lustre-tests cmip-proc8:/etc # rpm -e lustre cmip-proc8:/etc # rpm -e lustre-modules cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre-modules ### [100%] Congratulations on finishing your Lustre installation! To register your copy of Lustre and find out more about Lustre Support, Service, and Training offerings please visit http://www.sun.com/software/products/lustre/lustre_reg.jsp cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre ### [100%] cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre-tests ### [100%] cmip-proc8:/etc # ... cmip-proc8:/etc # rpm -qa | grep lustre lustre-debuginfo-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-debugsource-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815 lustre-source-1.8.5-2.6.32.29_0.3_xen_201105090815 The problem reproduces: cmip-proc8:~ # cp /var/log/messages /tmp/m0 cmip-proc8:~ # dmesg /tmp/d0 cmip-proc8:~ # lsmod | grep lustre cmip-proc8:~ # modprobe lustre Killed cmip-proc8:~ # dmesg /tmp/d1 cmip-proc8:~ # cp /var/log/messages /tmp/m1 cmip-proc8:~ # diff /tmp/d0 /tmp/d1 193a194,235 [ 84.786822] SFW2-INext-DROP-DEFLT IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:30:1e:5d:54:80:08:00 SRC=130.246.188.226 DST=224.0.0.1 LEN=28 TOS=0x00 PREC=0x00 TTL=1 ID=34816 PROTO=2 [ 104.171306] BUG: unable to handle kernel
Re: [Lustre-discuss] SLES 11 SP1 Client rpms built but not working
Understood, Andreas, Just to supplement is that the same approach works for SLES 11 using a xen kernel (2.6.27.54-0.2-xen). The Lustre Client rpms works okay: cmip-proc1:~ # cat /etc/issue Welcome to SUSE Linux Enterprise Server 11 (x86_64) - Kernel \r (\l). cmip-proc1:~ # uname -a Linux cmip-proc1 2.6.27.54-0.2-xen #1 SMP 2010-10-19 18:40:07 +0200 x86_64 x86_64 x86_64 GNU/Linux cmip-proc1:~ # df -h /disks/ceda1 FilesystemSize Used Avail Use% Mounted on 130.246.191.64:130.246.191.65@tcp0:/ceda1 51T 130G 48T 1% /disks/ceda1 SLES 11 SP1 is a service pack update to SLES 11 (now on 2.6.32.29-0.3-xen). Is it possible to find out what the problem is? Regards, Peter -Original Message- From: Andreas Dilger [mailto:adil...@whamcloud.com] Sent: 11 May 2011 10:11 To: Chiu, Peter (STFC,RAL,RALSP) Cc: lustre-discuss@lists.lustre.org; Chiu, Peter (STFC,RAL,RALSP) Subject: Re: [Lustre-discuss] SLES 11 SP1 Client rpms built but not working The only other potential problem I see is that you are using a xen kernel and this us somehow causing problems. Cheers, Andreas On 2011-05-11, at 1:33 AM, peter.c...@stfc.ac.uk wrote: Dear Andreas, Many thanks for your response. Below are further details on this. I shall be grateful for your advice on this. Regards, Peter The system is: cmip-proc8:/etc # uname -a Linux cmip-proc8.badc.rl.ac.uk 2.6.32.29-0.3-xen #1 SMP 2011-02-25 13:36:59 +0100 x86_64 x86_64 x86_64 GNU/Linux /usr/src/linux is a symlink pointing to the source corresponding to linux-2.6.32.29-0.3-obj: cmip-proc8:/etc # ls -l /usr/src total 24 drwxr-xr-x 3 root root 4096 2011-05-09 08:31 debug lrwxrwxrwx 1 root root 19 2011-03-20 15:54 linux - linux-2.6.32.29-0.3 drwxr-xr-x 25 root root 4096 2011-05-09 08:49 linux-2.6.32.29-0.3 drwxr-xr-x 3 root root 4096 2011-03-20 15:54 linux-2.6.32.29-0.3-obj drwxr-xr-x 3 root root 4096 2011-03-20 15:54 linux-obj drwxr-xr-x 10 root root 4096 2011-05-09 08:31 lustre-1.8.5 drwxr-xr-x 7 root root 4096 2011-03-20 14:58 packages cmip-proc8:/etc # cmip-proc8:~ # ls /usr/local/kits/lustre-1.8.5 aclocal.m4 config.h.ininstall-sh Makefile autoMakefile config.log ldiskfs Makefile.in autoMakefile.am config.status libsysio missing autoMakefile.in config.sub lnet mkinstalldirs buildconfigure lustre README ChangeLogconfigure.ac lustre-1.8.5.tar.gz Rules compile COPYINGlustre-iokit snmp config.guess debian lustre.spec stamp-h1 config.h depcomplustre.spec.in tree_status cmip-proc8:~ # The build with .configure and make rpms produced rpms that are installable: cmip-proc8:/etc # ls -ls /usr/src/packages/RPMS/x86_64/*1.8.5* 4024 -rw-r--r-- 1 root root 4112883 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-1.8.52.6.32.29_0.3_xen_201105090815.x86_64.rpm 15532 -rw-r--r-- 1 root root 15881360 2011-05-09 08:54 /usr/src/packages/RPMS/x86_64/lustre-debuginfo-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 1332 -rw-r--r-- 1 root root 1358924 2011-05-09 08:54 /usr/src/packages/RPMS/x86_64/lustre-debugsource-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 1416 -rw-r--r-- 1 root root 1441937 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 3524 -rw-r--r-- 1 root root 3602163 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-source-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm 2600 -rw-r--r-- 1 root root 2656393 2011-05-09 08:53 /usr/src/packages/RPMS/x86_64/lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm cmip-proc8:/etc # rpm -e lustre-tests cmip-proc8:/etc # rpm -e lustre cmip-proc8:/etc # rpm -e lustre-modules cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-modules-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre-modules ### [100%] Congratulations on finishing your Lustre installation! To register your copy of Lustre and find out more about Lustre Support, Service, and Training offerings please visit http://www.sun.com/software/products/lustre/lustre_reg.jsp cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...### [100%] 1:lustre ### [100%] cmip-proc8:/etc # rpm -ivh /usr/src/packages/RPMS/x86_64/lustre-tests-1.8.5-2.6.32.29_0.3_xen_201105090815.x86_64.rpm Preparing...###
[Lustre-discuss] Read-ahead with blockdev --setra ?
One of our users has suggested attempting to tune our test Lustre 1.8.5 OSTs by setting a read-ahead parameter on the underlying device via blockdev --setra. I tried setting this parameter to 8MB for an OST but as soon as the OST was mounted, the parameter reverted back to 640KB. I see absolutely no mention of this method in the Lustre manual. Can someone on the development team please comment on whether this parameter is effective and can actually be tuned? -- Yemi ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Lustre 1.8.4 - Local mount of ost for backup purposes, fs type ldiskfs or ext4?
Greetings, I am doing a local mount of a 8TB ost device in a Lustre 1.8.4 installation. The ost was built with a backfstype of ldiskfs. When attempting the local mount: mount -t ldiskfs /dev/sdc /mnt/save/ost I get: mount: wrong fs type, bad option, bad superblock on /dev/sdt, missing codepage or other error I am able to mount the same block device as ext4, just not as ldiskfs. I need to be able to mount as ldiskfs to get access to the extended attributes and back them up. Is this still the case with the ext4 extensions for Lustre 1.8.4? I am able to mount read-only as ext4 but any attempt at reading the extended attributes with getfattr fails. Thanks, --Jeff -- -- Jeff Johnson Manager Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x101 f: 858-412-3845 m: 619-204-9061 4905 Morena Boulevard, Suite 1313 - San Diego, CA 92117 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre 1.8.4 - Local mount of ost for backup purposes, fs type ldiskfs or ext4?
Hi. You didn't say what your kernel version was. Check dmesg, you'll likely get a better reason for the failure there. It's possible that the force_over_8tb option is needed and not passed along appropriately. Check out Lustre bugzilla 24432 and attachment 32731 which honors the force_over_8tb mount option. It'll be included in 1.8.6. -Cory On 05/11/2011 05:23 PM, Jeff Johnson wrote: Greetings, I am doing a local mount of a 8TB ost device in a Lustre 1.8.4 installation. The ost was built with a backfstype of ldiskfs. When attempting the local mount: mount -t ldiskfs /dev/sdc /mnt/save/ost I get: mount: wrong fs type, bad option, bad superblock on /dev/sdt, missing codepage or other error I am able to mount the same block device as ext4, just not as ldiskfs. I need to be able to mount as ldiskfs to get access to the extended attributes and back them up. Is this still the case with the ext4 extensions for Lustre 1.8.4? I am able to mount read-only as ext4 but any attempt at reading the extended attributes with getfattr fails. Thanks, --Jeff ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre 1.8.4 - Local mount of ost for backup purposes, fs type ldiskfs or ext4?
Well, that's the opposite problem of Bug 24398. Are you sure you are using the ext4-based ldiskfs? Kevin On May 11, 2011, at 4:23 PM, Jeff Johnson jeff.john...@aeoncomputing.com wrote: Greetings, I am doing a local mount of a 8TB ost device in a Lustre 1.8.4 installation. The ost was built with a backfstype of ldiskfs. When attempting the local mount: mount -t ldiskfs /dev/sdc /mnt/save/ost I get: mount: wrong fs type, bad option, bad superblock on /dev/sdt, missing codepage or other error I am able to mount the same block device as ext4, just not as ldiskfs. I need to be able to mount as ldiskfs to get access to the extended attributes and back them up. Is this still the case with the ext4 extensions for Lustre 1.8.4? I am able to mount read-only as ext4 but any attempt at reading the extended attributes with getfattr fails. Thanks, --Jeff -- -- Jeff Johnson Manager Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x101 f: 858-412-3845 m: 619-204-9061 4905 Morena Boulevard, Suite 1313 - San Diego, CA 92117 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Fragmented I/O
Hi, I'm having some performance issues on my Lustre filesystem and it looks to me like it's related to I/Os getting fragmented before being written to disk, but I can't figure out why. This system is RHEL5, running Lustre 1.8.4. All of my OSTs look pretty much the same- read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 88811 38 38 | 46375 17 17 2:1497 0 38 | 7733 2 20 4:1161 0 39 | 1840 0 21 8:1168 0 39 | 7148 2 24 16:922 0 40 | 3297 1 25 32:979 0 40 | 7602 2 28 64: 1576 0 41 | 9046 3 31 128: 7063 3 44 | 16284 6 37 256:129282 55 100 | 162090 62 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 0: 51181 22 22 |0 0 0 1: 45280 19 42 | 82206 31 31 2: 16615 7 49 | 29108 11 42 3:3425 1 50 | 17392 6 49 4: 110445 48 98 | 129481 49 98 5:1661 0 99 | 2702 1 99 read | write disk I/O size ios % cum % | ios % cum % 4K: 45889 8 8 | 56240 7 7 8K: 3658 0 8 | 6416 0 8 16K: 7956 1 10 | 4703 0 9 32K: 4527 0 11 | 11951 1 10 64K:114369 20 31 | 134128 18 29 128K: 5095 0 32 | 17229 2 31 256K: 7164 1 33 | 30826 4 35 512K: 369512 66 100 | 465719 64 100 Oddly, there's no 1024K row in the I/O size table... ...and these seem small to me as well, but I can't seem to change them. Writing new values to either doesn't change anything. # cat /sys/block/sdb/queue/max_hw_sectors_kb 320 # cat /sys/block/sdb/queue/max_sectors_kb 320 Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID controllers, with MD1000 and MD1200 arrays, respectively. Any clues on where I should look next? Thanks, Kevin Kevin Hildebrand University of Maryland, College Park Office of Information Technology ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Fragmented I/O
You didn't say, but I think they are LSI-based: are you using the mptsas driver with the PERC cards? Which driver version? First, max_sectors_kb should normally be set to a power of 2 number, like 256, over an odd size like 320. This number should also match the native raid size of the device, to avoid read-modify-write cycles. (See Bug 22886 on why not to make it 1024 in general). See Bug 17086 for patches to increase the max_sectors_kb limitation for the mptsas driver to 1MB, or the true hardware maximum, rather than a driver limit; however, the hardware may still be limited to sizes 1MB. Also, to clarify the sizes: the smallest bucket = transfer_size is the one incremented, so a 320KB IO increments the 512KB bucket. Since your HW says it can only do a 320KB IO, there will never be a 1MB IO. You may want to instrument your HBA driver to see what is going on (ie, why the max_hw_sectors_kb is 1024). Kevin Kevin Hildebrand wrote: Hi, I'm having some performance issues on my Lustre filesystem and it looks to me like it's related to I/Os getting fragmented before being written to disk, but I can't figure out why. This system is RHEL5, running Lustre 1.8.4. All of my OSTs look pretty much the same- read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 88811 38 38 | 46375 17 17 2:1497 0 38 | 7733 2 20 4:1161 0 39 | 1840 0 21 8:1168 0 39 | 7148 2 24 16:922 0 40 | 3297 1 25 32:979 0 40 | 7602 2 28 64: 1576 0 41 | 9046 3 31 128: 7063 3 44 | 16284 6 37 256:129282 55 100 | 162090 62 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 0: 51181 22 22 |0 0 0 1: 45280 19 42 | 82206 31 31 2: 16615 7 49 | 29108 11 42 3:3425 1 50 | 17392 6 49 4: 110445 48 98 | 129481 49 98 5:1661 0 99 | 2702 1 99 read | write disk I/O size ios % cum % | ios % cum % 4K: 45889 8 8 | 56240 7 7 8K: 3658 0 8 | 6416 0 8 16K: 7956 1 10 | 4703 0 9 32K: 4527 0 11 | 11951 1 10 64K:114369 20 31 | 134128 18 29 128K: 5095 0 32 | 17229 2 31 256K: 7164 1 33 | 30826 4 35 512K: 369512 66 100 | 465719 64 100 Oddly, there's no 1024K row in the I/O size table... ...and these seem small to me as well, but I can't seem to change them. Writing new values to either doesn't change anything. # cat /sys/block/sdb/queue/max_hw_sectors_kb 320 # cat /sys/block/sdb/queue/max_sectors_kb 320 Hardware in question is DELL PERC 6/E and DELL PERC H800 RAID controllers, with MD1000 and MD1200 arrays, respectively. Any clues on where I should look next? Thanks, Kevin Kevin Hildebrand University of Maryland, College Park Office of Information Technology ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
