Re: [Lustre-discuss] how to reuse OST indices (EADDRINUSE)
Hello, Did you backup old magic files (last_rcvd, LAST_ID, CONFIG/*) from the original OSTs and put them back before trying to mount them? You probably didn't do that. So when you remount the OSTs with existing index, the MGS will refuse to add them without being told to writeconf, hence -EADDRINUSE. The proper ways to replace an OST are described in bug 24128. 在 2010-12-21，上午8:33， Craig Prescott 写道： Hello list, We recently evacuated several OSTs on a single OSS, replaced RAID controllers, re-initialized RAIDs for new OSTs, and made new lustre filesystems for them, using the same OST indices as we had before. The filesystem and all its clients have been up and running the whole time. We disabled the OSTs we were working on on all clients and our MGS/MDS (lctl dl shows them as IN everywhere). Now we want to bring the newly-formatted OSTs back online. When we try to mount the new OSTs, we get this for each one in this syslog of the OSS that has been under maintenance: Lustre: mgc10.13.28@o2ib: Reactivating import LustreError: 11-0: an error occurred while communicating with 10.13.28@o2ib. The mgs_target_reg operation failed with -98 LustreError: 6065:0:(obd_mount.c:1097:server_start_targets()) Required registration failed for cms-OST0006: -98 LustreError: 6065:0:(obd_mount.c:1655:server_fill_super()) Unable to start targets: -98 LustreError: 6065:0:(obd_mount.c:1438:server_put_super()) no obd cms-OST0006 LustreError: 6065:0:(obd_mount.c:147:server_deregister_mount()) cms-OST0006 not registered What do we need to do to get these OSTs back into the filesystem? We really want to reuse the original indices. This is Lustre 1.8.4, btw. Thanks, Craig Prescott UF HPC Center ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： Ok, so, i'll try this : - One mds - 2 physical oss with one drive (1 GB) (one ost on one oss) On client, mount mds on /home.. lfs setstripe -c2 /home I upload (in sftp) one file, 300 MB - On Oss 1, he use 150 Mb of 1000 - On Oss 2, he use 150 Mb of 1000 All right! I continue... copy my first file 4 times (so there is 5 * 300 MB = 1500 MB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 * Now, I add a new oss server, with one ost (1GB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 - On Oss 3, he use 0 Mb of 1000 lfs setstripe -c3 /home on client I upload a big file, 1.3 Go He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop (Couldn't write to remote file /home/big_log.log: Failure) ** All files in a directory inherits its parent dir's stripe attributes. As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. So now, - On Oss 1, he use 1000 Mb of 1000 - On Oss 2, he use 1000 Mb of 1000 - On Oss 3, he use 250 Mb of 1000 I upload again, just for see, my first file (300 MB), he copy the file only on Oss3 (oss 1 and 2 are full of course), it's ok :) Is there a solution for this problem? If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). Regards -Message d'origine- De : Andreas Dilger [mailto:andreas.dil...@oracle.com] Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want Lustre itself can not do RAID over the network, if that is what you are looking for... For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) The problem, is, when sdb is full, he stop the copy, and don't continue on ost2 If you create your file to be striped over both OSTs, then it should work. Use lfs setstripe -c2 /home/newfile to specify a stripe count of 2. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
Lustre has its own load-balancing algorithms (either round-robin or weighted) depending on the space usage of the OSTs. 在 2010-12-16，下午11:25， Eudes PHILIPPE 写道： I was wrong about what I say If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok (sftp said nothing), but the file is different (the md5sum si different!) It's very dangerous!! Is there a solution, if I see all ossX are almost full, when I add (some) new oss, distribute data on these new oss to always have the same poucentage use on all oss? -Message d'origine- De : Wang Yibin [mailto:wang.yi...@oracle.com] Envoyé : jeudi 16 décembre 2010 16:09 À : Eudes PHILIPPE Cc : lustre-discuss Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? 在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： Ok, so, i'll try this : - One mds - 2 physical oss with one drive (1 GB) (one ost on one oss) On client, mount mds on /home.. lfs setstripe -c2 /home I upload (in sftp) one file, 300 MB - On Oss 1, he use 150 Mb of 1000 - On Oss 2, he use 150 Mb of 1000 All right! I continue... copy my first file 4 times (so there is 5 * 300 MB = 1500 MB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 * Now, I add a new oss server, with one ost (1GB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 - On Oss 3, he use 0 Mb of 1000 lfs setstripe -c3 /home on client I upload a big file, 1.3 Go He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop (Couldn't write to remote file /home/big_log.log: Failure) ** All files in a directory inherits its parent dir's stripe attributes. As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. So now, - On Oss 1, he use 1000 Mb of 1000 - On Oss 2, he use 1000 Mb of 1000 - On Oss 3, he use 250 Mb of 1000 I upload again, just for see, my first file (300 MB), he copy the file only on Oss3 (oss 1 and 2 are full of course), it's ok :) Is there a solution for this problem? If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). Regards -Message d'origine- De : Andreas Dilger [mailto:andreas.dil...@oracle.com] Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want Lustre itself can not do RAID over the network, if that is what you are looking for... For my first test, I upload on client a file (1.8 Go) (each ost have 1 Go) The problem, is, when sdb is full, he stop the copy, and don't continue on ost2 If you create your file to be striped over both OSTs, then it should work. Use lfs setstripe -c2 /home/newfile to specify a stripe count of 2. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Help
在 2010-11-20，上午8:39， Nihir Parikh 写道： Hello Wang Yibin, Thanks for getting back to me. Yes, S2 and S3 can ping each other using lctl ping. This indicates that your routing is work as expected. I was using nuttcp test and I also tried ib tests that comes with the IB utilities. I will lnet-selftest. These utilities do not understand lnet protocol so they won't work. My goal was to measure the bandwidth when it has to reach across different network. Are there any such tests specific to lustre? Lnet has its own testsuite which is called lnet self-test. To measure the bandwidth, you can load lnet_selftest module on your nodes and execute lst in brw mode. Thanks Nihir From: Wang Yibin [mailto:wang.yi...@oracle.com] Sent: Thursday, November 18, 2010 6:32 AM To: Nihir Parikh Cc: lustre-discuss@lists.lustre.org Subject: Re: [Lustre-discuss] Help Hi, 在 2010-11-17，上午9:17， Nihir Parikh 写道： Now my problem is to run some network tests from S2 à S3 and S3 à S2 to measure the bandwidth but somehow both S2 and S3 complain that network is unreachable. What am I doing wrong? Your configuration seems OK to me. Can S2 and S3 ping each other using 'lctl ping'? What kind of network test did you do? Note that only lustre LNET can do the routing. There's a script in lustre testsuite that's specifically for testing the network connectivity - lnet-selftest.sh. Thanks Nihir ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Broken client
Could you elaborate about how broken the files are? From your description and the error message you provide, I suspect that one(or some) of the OSTs went down. What does `lctl dl` show? 在 2010-11-18，下午8:18， Herbert Fruchtl 写道： I have a Lustre (1.6.7) system that looks OKish (as far as I can see) from the mds and most of the clients. From one client however (the users' login machine) it looks broken. Some files are missing, some seem broken, and the df command hangs. Rebooting the client doesn't change anything. Is it broken, or is there some persistent information that I need to flush? When I do an ls on a partially broken directory, I get the following two lines in /var/log/messages: Nov 18 12:13:53 mhdc kernel: [ 7093.751196] LustreError: 10919:0:(file.c:999:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO Nov 18 12:13:53 mhdc kernel: [ 7093.761098] LustreError: 10919:0:(file.c:999:ll_glimpse_size()) Skipped 9 previous similar messages Any ideas how to proceed with the least disruption? Thanks in advance, Herbert -- Herbert Fruchtl Senior Scientific Computing Officer School of Chemistry, School of Mathematics and Statistics University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Help
Hi, 在 2010-11-17，上午9:17， Nihir Parikh 写道： Now my problem is to run some network tests from S2 à S3 and S3 à S2 to measure the bandwidth but somehow both S2 and S3 complain that network is unreachable. What am I doing wrong? Your configuration seems OK to me. Can S2 and S3 ping each other using 'lctl ping'? What kind of network test did you do? Note that only lustre LNET can do the routing. There's a script in lustre testsuite that's specifically for testing the network connectivity - lnet-selftest.sh. Thanks Nihir ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Broken client
Hello, 在 2010-11-18，下午10:03， Herbert Fruchtl 写道： I was wrong about only one client having problems. It seems to be all of them, except the mds server (see below), so it is a problem of the filesystem (not the client) after all. Could you elaborate about how broken the files are? When I do an 'ls', the filenames are flashing in red (this is for example the case for broken symbolic links). Permissions, date and owner are missing, like in the middle of the next three lines: -rw--- 1 root root18308319 Jul 16 2009 stat_1247756353.gz ?- ? ?? ?? stat_1248125742.gz drwxr-xr-x 2 stephane ukmhd 4096 Jul 8 2009 stephane Attempting to access the file more closely results in an I/O error: [r...@mhdc ~]# ls -l /workspace/ls-lR_2009-01-20 ls: /workspace/ls-lR_2009-01-20: Input/output error [r...@mhdc ~]# cp /workspace/ls-lR_2009-01-20 /tmp cp: cannot stat `/workspace/ls-lR_2009-01-20': Input/output error This looks very much like some OSTs are failing. From your description and the error message you provide, I suspect that one(or some) of the OSTs went down. What does `lctl dl` show? The files are accessible from the mds server, and the OSTs seem visible from the broken clients: [r...@mhdc ~]# lctl dl 0 UP mgc mgc192.168.101@tcp 63568484-f714-da05-c5c2-b96db1b22962 5 1 UP lov home-clilov-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 4 2 UP mdc home-MDT-mdc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 3 UP osc home-OST0001-osc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 4 UP osc home-OST0003-osc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 5 UP osc home-OST0002-osc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 6 UP osc home-OST0005-osc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 7 UP osc home-OST0004-osc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 8 UP osc home-OST-osc-8100d7ecf000 651d7044-988f-f324-6896-3e09edf8a90b 5 Does this help? I mean 'lctl dl' output on the OSS servers. Make sure that your OSTs are all mounted and running well. Herbert 在 2010-11-18，下午8:18， Herbert Fruchtl 写道： I have a Lustre (1.6.7) system that looks OKish (as far as I can see) from the mds and most of the clients. From one client however (the users' login machine) it looks broken. Some files are missing, some seem broken, and the df command hangs. Rebooting the client doesn't change anything. Is it broken, or is there some persistent information that I need to flush? When I do an ls on a partially broken directory, I get the following two lines in /var/log/messages: Nov 18 12:13:53 mhdc kernel: [ 7093.751196] LustreError: 10919:0:(file.c:999:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO Nov 18 12:13:53 mhdc kernel: [ 7093.761098] LustreError: 10919:0:(file.c:999:ll_glimpse_size()) Skipped 9 previous similar messages Any ideas how to proceed with the least disruption? Thanks in advance, Herbert -- Herbert Fruchtl Senior Scientific Computing Officer School of Chemistry, School of Mathematics and Statistics University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- Herbert Fruchtl Senior Scientific Computing Officer School of Chemistry, School of Mathematics and Statistics University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] fstab mount fails often
Hi, 在 2010-11-16，下午7:25， Arne Brutschy 写道： Hello, From the log, we can see that either your MGS node was not ready for connection yet, or there's network error between client and the MGS node. No error on the server nor on the client. What else can it be? Maybe the switch is bad, I can see RX errors on most of it's interfaces. The switch could be the culprit - error message shows client failed to send request to MGS. Network sending status was -EHOSTUNREACH. I suggest you reexamine the network of your system. Were you rebooting the MGS at the moment? No. It's something that happenes regularly. Since you said there's no errors on the interface, you need to check the lnet connection and also verify that the MGS/MDT are up running. As far as I can tell, everything seems to be set up correctly. I have quite a simple setup (single network, single interface gbe). Thanks Arne 在 2010-11-15，下午11:32， Arne Brutschy 写道： Hi all, I am mounting lustre through an fstab entry. This fails quite often, the nodes end up without the lustre mount. Even when I log in, it take 2-3 tries to get it to mount. This is what I get: mount /lustre mount.lustre: mount 10.1@tcp0:/lustre at /lustre failed: Cannot send after transport endpoint shutdown This is /var/log/messages: Nov 15 16:27:43 compute-1-10 kernel: LustreError: 2124:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-10.1@tcp: -113 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 2124:0:(events.c:66:request_out_callback()) @@@ type 4, status -113 r...@d73d7c00 x1352468062535684/t0 o250-m...@mgc10.1.1.1@tcp_0:26/25 lens 368/584 e 0 to 1 dl 1289834868 ref 2 fl Rpc:N/0/0 rc 0/0 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 29069:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID r...@d73d7800 x1352468062535685/t0 o101-m...@mgc10.1.1.1@tcp_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 15c-8: mgc10.1@tcp: The configuration from log 'lustre-client' failed (-108). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Nov 15 16:27:43 compute-1-10 kernel: LustreError: 29069:0:(llite_lib.c:1176:ll_fill_super()) Unable to process log: -108 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 29069:0:(obd_mount.c:2045:lustre_fill_super()) Unable to mount (-108) I have no errors on the interface, so I assume this is a timing problem. Can I improve this through some timeout setting? Cheers, Arne ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- Arne Brutschy Ph.D. StudentEmailarne.brutschy(AT)ulb.ac.be IRIDIA CP 194/6 Web iridia.ulb.ac.be/~abrutschy Universite' Libre de Bruxelles Tel +32 2 650 2273 Avenue Franklin Roosevelt 50 Fax +32 2 650 2715 1050 Bruxelles, Belgium (Fax at IRIDIA secretary) ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] High CPU load, only on 1 OSS
Hello, Normally when stripe_offset is set to -1, MDS will do load/space balancing automatically. What is your use pattern of the filesystem? It sounds like that your applications are doing extensive I/O on that particular OSS. To find out why the load on the OSS is so high, please - find what processes are hogging the CPUs using top(1). - get the stripe info of your in-use files to see whether most of them reside on the same OSS. If the files in use are not distributed among the OSS servers, or your file usage pattern is one-OSS bound, you may want to consider tuning the stripe_count/stripe_size. 在 2010-11-16，下午10:38， Ronald K Long 写道： We recently setup a lustre config. 1 MDS 4 OSS's. Everything is running fine except on the first OSS we are experiencing very high cpu load. The first OSS is running a CPU load in the high 50's. The other 3 OSS's are steady at around 8. Everything is the same between all of the OSS's. The stripe is setup stripe_count: 1 stripe_offset: -1 Red Hat 5 64bit kernel-2.6.18-194.3.1.el5_lustre.1.8.4 kernel-devel-2.6.18-194.3.1.el5_lustre.1.8.4 lustre-ldiskfs-3.1.3-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-modules-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 Any thing I can check on the problem OSS to rectify this issue. Thank you in advance Rocky ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] fstab mount fails often
From the log, we can see that either your MGS node was not ready for connection yet, or there's network error between client and the MGS node. Were you rebooting the MGS at the moment? Since you said there's no errors on the interface, you need to check the lnet connection and also verify that the MGS/MDT are up running. 在 2010-11-15，下午11:32， Arne Brutschy 写道： Hi all, I am mounting lustre through an fstab entry. This fails quite often, the nodes end up without the lustre mount. Even when I log in, it take 2-3 tries to get it to mount. This is what I get: mount /lustre mount.lustre: mount 10.1@tcp0:/lustre at /lustre failed: Cannot send after transport endpoint shutdown This is /var/log/messages: Nov 15 16:27:43 compute-1-10 kernel: LustreError: 2124:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-10.1@tcp: -113 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 2124:0:(events.c:66:request_out_callback()) @@@ type 4, status -113 r...@d73d7c00 x1352468062535684/t0 o250-m...@mgc10.1.1.1@tcp_0:26/25 lens 368/584 e 0 to 1 dl 1289834868 ref 2 fl Rpc:N/0/0 rc 0/0 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 29069:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID r...@d73d7800 x1352468062535685/t0 o101-m...@mgc10.1.1.1@tcp_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 15c-8: mgc10.1@tcp: The configuration from log 'lustre-client' failed (-108). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Nov 15 16:27:43 compute-1-10 kernel: LustreError: 29069:0:(llite_lib.c:1176:ll_fill_super()) Unable to process log: -108 Nov 15 16:27:43 compute-1-10 kernel: LustreError: 29069:0:(obd_mount.c:2045:lustre_fill_super()) Unable to mount (-108) I have no errors on the interface, so I assume this is a timing problem. Can I improve this through some timeout setting? Cheers, Arne ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] non-consecutive OST ordering
This sounds like bug 20183 which was caused by race in object creation.
Unfortunately the fix is not in Lustre release 1.6.6.
You need to upgrade your filesystem to avoid hitting this LBUG again.
To fix dangling inodes, if you are very sure that these objects are not useful,
use '-d' option to delete them.
Else use lfsck with '-c' option to create 0-length objects on the OSTs (for
this particular case, OST 0x30).
在 2010-11-14，下午12:21， Christopher Walker 写道：
Thanks again for the advice, and for putting this patch together so
quickly. I put your patch into 1.41.6.sun1 and it works perfectly, at
least in read-only mode.
Unfortunately, when I run lfsck -l on this array, it gets to ost_idx 30,
runs through part of the check, but then hangs while checking for orphan
objects:
lfsck: ost_idx 30: pass2 ERROR: 46700 dangling inodes found (682467
files total)
lfsck: ost_idx 30: pass3: check for orphan objects
it hung right around the same time that an LBUG appeared on the MDS:
Nov 13 21:58:39 aegamds1 kernel: LustreError:
21864:0:(lov_pack.c:179:lov_packmd()) ASSERTION(loi-loi_id)
failed:lmm_oid 78132063 stripe 0/1 idx 30
Nov 13 21:58:39 aegamds1 kernel: LustreError:
21864:0:(lov_pack.c:179:lov_packmd()) LBUG
Judging from the number of dangling inodes, there's looks to be
something quite wrong with this OST (although it produces no errors when
run through e2fsck -fn). The last few lines of the lustre-log are:
0004:0008:1:1289703502.347853:0:21853:0:(obd.h:1191:obd_transno_commit_cb())
aegalfs-MDT: transno 4267034979 committed
0200:0008:0:1289703519.748478:0:21877:0:(upcall_cache.c:185:refresh_entry())
aegalfs-MDT: invoked upcall /usr/sbin/l_getgroups aegalfs-MDT 0
0002:0004:4:1289703519.876199:0:21864:0:(lov_pack.c:179:lov_packmd())
ASSERTION(loi-loi_id) failed:lmm_oid 78132063 stripe 0/1 idx 30
:0004:4:1289703519.876652:0:21864:0:(lov_pack.c:179:lov_packmd())
LBUG
0400:0400:4:1289703519.876877:0:21864:0:(linux-debug.c:185:libcfs_debug_dumpstack())
showing stack for process 21864
Debug log: 212116 lines, 212116 kept, 0 dropped.
[r...@aegamds1 ~]#
is 78132063 an objid that lfsck doesn't like? Is deleting the associated
file (assuming that I can find it) the best path forward?
Thanks again,
Chris
On 11/12/10 11:02 PM, Wang Yibin wrote:
For the moment, without investigation, I am not sure about this - There may
or may not be compatibility issue.
Please checkout the version of the e2fsprogs which is identical with that on
your system and patch against the lfsck.c accordingly.
Then you can compile against 1.6.6.
在 2010-11-13，上午11:20， Christopher Walker 写道：
Thanks again for this patch. I just have one quick question about this
-- 1.41.12.2.ora1 seems to require lustre_user.h from 1.8.x -- is OK to
use a version of lfsck compiled against 1.8.x on a 1.6.6 filesystem, and
with {mds,ost}db that were created with 1.41.6?
Best,
Chris
On 11/12/10 3:17 AM, Wang Yibin wrote:
This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST
index when there are sparse OST(s).
Please file a bug for this.
Before this bug can be fixed, you can apply the following patch to
e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified
though).
--- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.0 +0800
+++ lfsck.c 2010-11-12 12:14:38.0 +0800
@@ -1226,6 +1226,12 @@
__u64 last_id;
int i, rc;
+ /* skip empty UUID OST */
+ if(!strlen(lfsck_uuid[ost_idx].uuid)) {
+ log_write(index %d UUID is empty(sparse OST index?).
Skipping.\n, ost_idx);
+ return(0);
+ }
+
sprintf(dbname, %s.%d, MDS_OSTDB, ost_idx);
VERBOSE(2, testing ost_idx %d\n, ost_idx);
@@ -1279,11 +1284,20 @@
ost_hdr-ost_uuid.uuid);
if (obd_uuid_equals(lfsck_uuid[ost_idx], ost_hdr-ost_uuid)) {
+/* must be sparse ost index */
if (ost_hdr-ost_index != ost_idx) {
log_write(Requested ost_idx %u doesn't match
index %u found in %s\n, ost_idx,
ost_hdr-ost_index, ost_files[i]);
- continue;
+
+ log_write(Moving the index/uuid to the right place...\n);
+/* zero the original uuid entry */
+ memset(lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid));
+/* copy it to the right place */
+ost_idx = ost_hdr-ost_index;
+
strcpy(lfsck_uuid[ost_hdr-ost_index].uuid,ost_hdr-ost_uuid.uuid);
+ /* skip this round */
+ goto out;
}
break;
在 2010-11-12，上午10:53， Christopher Walker 写道：
Thanks very much for your reply. I've tried remaking the mdsdb and all
of the ostdb's, but I still get the same error -- it checks the first 34
osts without a problem, but can't find the ostdb file for the 35th
(which
Re: [Lustre-discuss] non-consecutive OST ordering
This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST
index when there are sparse OST(s).
Please file a bug for this.
Before this bug can be fixed, you can apply the following patch to
e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though).
--- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.0 +0800
+++ lfsck.c 2010-11-12 12:14:38.0 +0800
@@ -1226,6 +1226,12 @@
__u64 last_id;
int i, rc;
+ /* skip empty UUID OST */
+ if(!strlen(lfsck_uuid[ost_idx].uuid)) {
+ log_write(index %d UUID is empty(sparse OST index?). Skipping.\n,
ost_idx);
+ return(0);
+ }
+
sprintf(dbname, %s.%d, MDS_OSTDB, ost_idx);
VERBOSE(2, testing ost_idx %d\n, ost_idx);
@@ -1279,11 +1284,20 @@
ost_hdr-ost_uuid.uuid);
if (obd_uuid_equals(lfsck_uuid[ost_idx], ost_hdr-ost_uuid)) {
+/* must be sparse ost index */
if (ost_hdr-ost_index != ost_idx) {
log_write(Requested ost_idx %u doesn't match
index %u found in %s\n, ost_idx,
ost_hdr-ost_index, ost_files[i]);
- continue;
+
+ log_write(Moving the index/uuid to the right place...\n);
+/* zero the original uuid entry */
+ memset(lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid));
+/* copy it to the right place */
+ost_idx = ost_hdr-ost_index;
+
strcpy(lfsck_uuid[ost_hdr-ost_index].uuid,ost_hdr-ost_uuid.uuid);
+ /* skip this round */
+ goto out;
}
break;
在 2010-11-12，上午10:53， Christopher Walker 写道：
Thanks very much for your reply. I've tried remaking the mdsdb and all
of the ostdb's, but I still get the same error -- it checks the first 34
osts without a problem, but can't find the ostdb file for the 35th
(which has ost_idx 42):
...
lfsck: ost_idx 34: pass3 OK (676803 files total)
lfsck: can't find file for ost_idx 35
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 36
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 37
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 38
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 39
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 40
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 41
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 42
Files affected by missing ost info are : -
...
e2fsck claims to be making the ostdb without a problem:
Pass 6: Acquiring information for lfsck
OST: 'aegalfs-OST002a_UUID' ost idx 42: compat 0x2 rocomp 0 incomp 0x2
OST: num files = 676803
OST: last_id = 858163
and with the filesystem up I can see files on this OST:
[cwal...@iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c
OBDS:
0: aegalfs-OST_UUID ACTIVE
...
33: aegalfs-OST0021_UUID ACTIVE
42: aegalfs-OST002a_UUID ACTIVE
predict.c
obdidx objid objid group
42 10 0xa 0
lfsck identifies several hundred GB of orphan data that we'd like to
recover, so we'd really like to run lfsck on this array. We're willing
to forgo the recovery on the 35th ost, but I want to make sure that
running lfsck -l with the current configuration won't make things worse.
Thanks again for your reply; any further advice is very much appreciated!
Best,
Chris
On 11/10/10 12:10 AM, Wang Yibin wrote:
The error message indicates that the UUID of OST #35 does not match between
the live filesystem and the ostdb file.
Is this ostdb obsolete?
在 2010-11-9，下午11:45， Christopher Walker 写道：
For reasons that I can't recall, our OSTs are not in consecutive order
-- we have 35 OSTs, which are numbered consecutively from
-0021
and then there's one last OST at
002a
When I try to run lfsck on this array, it works fine for the first 34
OSTs, but it can't seem to find the last OST db file:
lfsck: ost_idx 34: pass3 OK (680045 files total)
lfsck: can't find file for ost_idx 35
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 36
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 37
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 38
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 39
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 40
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 41
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 42
Files affected by missing ost info are : -
/n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base
and then lists all of the files
Re: [Lustre-discuss] non-consecutive OST ordering
For the moment, without investigation, I am not sure about this - There may or
may not be compatibility issue.
Please checkout the version of the e2fsprogs which is identical with that on
your system and patch against the lfsck.c accordingly.
Then you can compile against 1.6.6.
在 2010-11-13，上午11:20， Christopher Walker 写道：
Thanks again for this patch. I just have one quick question about this
-- 1.41.12.2.ora1 seems to require lustre_user.h from 1.8.x -- is OK to
use a version of lfsck compiled against 1.8.x on a 1.6.6 filesystem, and
with {mds,ost}db that were created with 1.41.6?
Best,
Chris
On 11/12/10 3:17 AM, Wang Yibin wrote:
This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST
index when there are sparse OST(s).
Please file a bug for this.
Before this bug can be fixed, you can apply the following patch to
e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified
though).
--- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.0 +0800
+++ lfsck.c 2010-11-12 12:14:38.0 +0800
@@ -1226,6 +1226,12 @@
__u64 last_id;
int i, rc;
+ /* skip empty UUID OST */
+ if(!strlen(lfsck_uuid[ost_idx].uuid)) {
+ log_write(index %d UUID is empty(sparse OST index?). Skipping.\n,
ost_idx);
+ return(0);
+ }
+
sprintf(dbname, %s.%d, MDS_OSTDB, ost_idx);
VERBOSE(2, testing ost_idx %d\n, ost_idx);
@@ -1279,11 +1284,20 @@
ost_hdr-ost_uuid.uuid);
if (obd_uuid_equals(lfsck_uuid[ost_idx], ost_hdr-ost_uuid)) {
+/* must be sparse ost index */
if (ost_hdr-ost_index != ost_idx) {
log_write(Requested ost_idx %u doesn't match
index %u found in %s\n, ost_idx,
ost_hdr-ost_index, ost_files[i]);
- continue;
+
+ log_write(Moving the index/uuid to the right place...\n);
+/* zero the original uuid entry */
+ memset(lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid));
+/* copy it to the right place */
+ost_idx = ost_hdr-ost_index;
+
strcpy(lfsck_uuid[ost_hdr-ost_index].uuid,ost_hdr-ost_uuid.uuid);
+ /* skip this round */
+ goto out;
}
break;
在 2010-11-12，上午10:53， Christopher Walker 写道：
Thanks very much for your reply. I've tried remaking the mdsdb and all
of the ostdb's, but I still get the same error -- it checks the first 34
osts without a problem, but can't find the ostdb file for the 35th
(which has ost_idx 42):
...
lfsck: ost_idx 34: pass3 OK (676803 files total)
lfsck: can't find file for ost_idx 35
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 36
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 37
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 38
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 39
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 40
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 41
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 42
Files affected by missing ost info are : -
...
e2fsck claims to be making the ostdb without a problem:
Pass 6: Acquiring information for lfsck
OST: 'aegalfs-OST002a_UUID' ost idx 42: compat 0x2 rocomp 0 incomp 0x2
OST: num files = 676803
OST: last_id = 858163
and with the filesystem up I can see files on this OST:
[cwal...@iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c
OBDS:
0: aegalfs-OST_UUID ACTIVE
...
33: aegalfs-OST0021_UUID ACTIVE
42: aegalfs-OST002a_UUID ACTIVE
predict.c
obdidx objid objid group
42 10 0xa 0
lfsck identifies several hundred GB of orphan data that we'd like to
recover, so we'd really like to run lfsck on this array. We're willing
to forgo the recovery on the 35th ost, but I want to make sure that
running lfsck -l with the current configuration won't make things worse.
Thanks again for your reply; any further advice is very much appreciated!
Best,
Chris
On 11/10/10 12:10 AM, Wang Yibin wrote:
The error message indicates that the UUID of OST #35 does not match
between the live filesystem and the ostdb file.
Is this ostdb obsolete?
在 2010-11-9，下午11:45， Christopher Walker 写道：
For reasons that I can't recall, our OSTs are not in consecutive order
-- we have 35 OSTs, which are numbered consecutively from
-0021
and then there's one last OST at
002a
When I try to run lfsck on this array, it works fine for the first 34
OSTs, but it can't seem to find the last OST db file:
lfsck: ost_idx 34: pass3 OK (680045 files total)
lfsck: can't find file for ost_idx 35
Files affected by missing ost info are : -
lfsck: can't find file for ost_idx 36
Files affected
Re: [Lustre-discuss] very slow directory access (lstat taking 20s to return)
在 2010-11-9，下午10:35， Frederik Ferner 写道： I assume on a busy OST this could still take a while, though? Of course the load on the OST will affect the directory stats. I suggest you tune the maximum statahead count if there usually are a lot of files in a normal directory: # echo n /proc/fs/lustre/llite/*/statahead_max where n = 8192. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] non-consecutive OST ordering
The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. Is this ostdb obsolete? 在 2010-11-9，下午11:45， Christopher Walker 写道： For reasons that I can't recall, our OSTs are not in consecutive order -- we have 35 OSTs, which are numbered consecutively from -0021 and then there's one last OST at 002a When I try to run lfsck on this array, it works fine for the first 34 OSTs, but it can't seem to find the last OST db file: lfsck: ost_idx 34: pass3 OK (680045 files total) lfsck: can't find file for ost_idx 35 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 36 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 37 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 38 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 39 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 40 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 41 Files affected by missing ost info are : - lfsck: can't find file for ost_idx 42 Files affected by missing ost info are : - /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base and then lists all of the files that live on OST 002a. This db file definitely does exist -- it lives in the same directory as all of the other db files, and e2fsck for this OST ran without problems. Is there some way of forcing lfsck to recognize this OST db? Or, failing that, is it dangerous to run lfsck on the first 34 OSTs only? We're using e2fsck 1.41.6.sun1 (30-May-2009) Thanks very much! Chris ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre FS in Infiniband - client mount problem
As the error message said, the lustre modules probably were not loaded when you were trying to mount lustre client. Please provide information - specifically, the loaded modules and more dmesg log. 在 2010-11-4，上午2:30， ren yufei 写道： Dear all, I have setup some nodes (include MDT/MGS/OSS/Client) with Mellanox 40G RNIC connected via Mellanox MTS3600 switch, and deployed lustre FS in this cluster. The MDT/MGS node and OSS nodes works but client could not mount on this FS. The error information is as follows. Client: 192.168.1.23 MDS: 192.168.1.11 -- error information Client # mount -t lustre 192.168.1...@o2ib0:/lustre /mnt/lustre mount.lustre: mount 192.168.1...@o2ib0:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf Client # ls -l /mnt/lustre total 0 Client # dmesg | tail LustreError: 165-2: Nothing registered for client mount! Is the 'lustre' module loaded? LustreError: 5116:0:(obd_mount.c:2045:lustre_fill_super()) Unable to mount (-19) -- environment information. Client: # lctl list_nids 192.168.1...@o2ib # lctl ping 192.168.1...@o2ib0 1234...@lo 12345-192.168.1...@o2ib MDS: # lctl list_nids 192.168.1...@o2ib lctl device_list 0 UP mgs MGS MGS 13 1 UP mgc mgc192.168.1...@o2ib bb9cf87d-fd14-b679-85ce-f0fa1a866aff 5 2 UP mdt MDS MDS_uuid 3 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4 4 UP mds lustre-MDT lustre-MDT_UUID 3 5 UP osc lustre-OST-osc lustre-mdtlov_UUID 5 6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5 ... By the way, All these nodes could connect or access to each other via ping/iperf(TCP)/ibv_rc_pingpong/ibv_ud_pingpong/ib_write_lat. However, the 'rping' client, based on librdmacm, could not connect to the server side. error is: # rping -c 192.168.1.11 cq completion failed status 5 wait for CONNECTED state 10 connect error -1 cma event RDMA_CM_EVENT_REJECTED, error 8 Thank you very much. Yufei ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Intermittent obd_ping operation failed errors
It looks like that the network of str-ter-OST003(IP: 172.16.103.26) is quite flaky. Can you please check the stability of this particular OST? 在 2009-03-23一的 11:46 -0400，Roger Spellman写道： I am seeing the following error, multiple times on clients trying to talk to a particular OST. The errors are intermittent: I get five to ten every few seconds, then none for several hours (or even several days), then five to ten again. The servers are running Lustre 1.6.6, and are configured for IB and tcp (over eth0). There are 20 IB clients, and 200 tcp/eth0 clients. The system was relatively quiet while these errors were occurring. modprobe.conf contains: options ib_mthca msi_x=1 options lnet networks=tcp0(eth0),o2ib(ib0) options ko2iblnd ipif_name=ib0 Any idea what causes this, and how to resolve it? Thanks. Mar 18 10:26:39 ts-nrel-01 kernel: LustreError: 11-0: an error occurred while communicating with 172.16.103...@tcp. The obd_ping operation failed with -107 Mar 18 10:26:39 ts-nrel-01 kernel: Lustre: lstr-ter-OST0003-osc: Connection to service lstr-ter-OST0003 via nid 172.16.103...@tcp was lost; in progress operations using this service will wait for recovery to complete. Mar 18 10:26:39 ts-nrel-01 kernel: LustreError: 167-0: This client was evicted by lstr-ter-OST0003; in progress operations using this service will fail. Mar 18 10:26:39 ts-nrel-01 kernel: Lustre: lstr-ter-OST0003-osc: Connection restored to service lstr-ter-OST0003 using nid 172.16.103...@tcp. Mar 18 10:26:39 ts-nrel-01 kernel: Lustre: MDS lstr-ter-MDT: lstr-ter-OST0003_UUID now active, resetting orphans Mar 18 10:30:49 ts-nrel-01 kernel: LustreError: 11-0: an error occurred while communicating with 172.16.103...@tcp. The obd_ping operation failed with -107 Mar 18 10:30:49 ts-nrel-01 kernel: Lustre: lstr-ter-OST0003-osc: Connection to service lstr-ter-OST0003 via nid 172.16.103...@tcp was lost; in progress operations using this service will wait for recovery to complete. Mar 18 10:30:49 ts-nrel-01 kernel: LustreError: 167-0: This client was evicted by lstr-ter-OST0003; in progress operations using this service will fail. Mar 18 10:30:49 ts-nrel-01 kernel: Lustre: lstr-ter-OST0003-osc: Connection restored to service lstr-ter-OST0003 using nid 172.16.103...@tcp. Mar 18 10:30:49 ts-nrel-01 kernel: Lustre: MDS lstr-ter-MDT: lstr-ter-OST0003_UUID now active, resetting orphans Mar 18 10:34:59 ts-nrel-01 kernel: LustreError: 11-0: an error occurred while communicating with 172.16.103...@tcp. The obd_ping operation failed with -107 Mar 18 10:34:59 ts-nrel-01 kernel: Lustre: lstr-ter-OST0003-osc: Connection to service lstr-ter-OST0003 via nid 172.16.103...@tcp was lost; in progress operations using this service will wait for recovery to complete. Mar 18 10:34:59 ts-nrel-01 kernel: LustreError: 167-0: This client was evicted by lstr-ter-OST0003; in progress operations using this service will fail. Mar 18 10:34:59 ts-nrel-01 kernel: Lustre: lstr-ter-OST0003-osc: Connection restored to service lstr-ter-OST0003 using nid 172.16.103...@tcp. Mar 18 10:34:59 ts-nrel-01 kernel: Lustre: MDS lstr-ter-MDT: lstr-ter-OST0003_UUID now active, resetting orphans Roger Spellman Staff Engineer Terascala, Inc. 508-588-1501 www.terascala.com http://www.terascala.com/ ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Job fails opening 24k files and keeps them open during execution
Lustre does not impose maximum number of open files, but practically it depends on amount of RAM on the MDS. There are no tables for open files on the MDS, as they are only linked in a list to a given client's export. Each client process probably has a limit of several thousands of open files which depends on the ulimit. 在 2009-03-04三的 16:27 -0500，Osvaldo Rentas写道： Hello, I am working with a user that has Fortran code that opens 24.000 files and keeps them open during execution. We had to adjust our kernel parameters to allow this to happen, since Linux cuts you off at 1024 by default. This is job runs successfully for him on the local disk of a Linux machine, but when he moves the job to Lustre, it fails. The metadata servers are running Red Hat …do they impose their own user limitations as well? Or is there a limitation within Lustre or a config file? Thanks in advance, Oz No virus found in this outgoing message. Checked by AVG. Version: 7.5.557 / Virus Database: 270.11.3/1975 - Release Date: 2/27/2009 7:05 AM ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
