Re: [Lustre-discuss] Lustre Thumper Fault Tolerance

[Andreas Dilger]

On Mar 10, 2008  15:48 -0700, Klaus Steden wrote:
> Is the mmp feature already in the existing Lustre distribution? If so, what
> versions are mmp-aware? If not, which version will be the first to
> incorporate it?

It's in Lustre 1.6.2+ and 1.4.12, and e2fsprogs-1.40.2+.

> On 3/10/08 2:15 PM, "Andreas Dilger" <[EMAIL PROTECTED]>did etch on stone
> tablets:
> 
> > On Mar 10, 2008  09:09 -0600, Colin Faber wrote:
> >> Is this true even in the case of mounting the OSS as a read only node?
> > 
> > Yes, definitely even a "read only" mount can cause serious corruption.
> > There are several issues involved, the most dangerous is that even for
> > read-only mounting the journal is replayed by the kernel or otherwise
> > the filesystem may appear to be corrupted.
> > 
> > In addition, there is the problem that (meta)data that is cached on the
> > read-only mounting node will become incorrect as the writing node is
> > changing the filesystem.  The ext3 filesystem is not cluster aware.
> > 
> > In order to prevent situations like this, the newer releases of ldiskfs
> > and e2fsprogs have an "mmp" (multi-mount protection) feature which will
> > prevent the filesystem to be mounted on another node if it is active
> > on one node (either mounted, or running e2fsck).
> > 
> > This will be enabled by default on newly-formatted filesystems which
> > are created with the "--failover" flag, and can also be enabled by
> > hand with "tune2fs -O mmp /dev/" (replace with MDT or OST device
> > names as appropriate).  This will prevent the filesystem from being
> > mounted or e2fsck'd by old kernels/e2fsprogs so it isn't enabled by
> > default on existing filesystems.
> > 
> >> Andreas Dilger wrote:
> >>> On Mar 07, 2008  00:04 +0530, Neeladri Bose wrote:
> >>>   
>  To address the performance hit (whatever be the %age) if we setup DRDB in
>  active-passive mode across the 4500's but have the LustreFS points to
>  separate raid sets from the network across the DRDB pair of 4500's & thus
>  become an active-active solution which may actually increase the
>  throughput of the LustreFS.
>  
>  Can it be a possible scenario using DRDB on Linux with ext3 & LustreFS?
>  
> >>> 
> >>> No, Lustre does not support active-active export of backing filesystems.
> >>> This doesn't work because the backing filesystems (ext3/ZFS) are not
> >>> themselves cluster-aware and mounting them on two nodes will quickly
> >>> lead to whole filesystem corruption.
> > 
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Sr. Staff Engineer, Lustre Group
> > Sun Microsystems of Canada, Inc.
> > 
> > ___
> > Lustre-discuss mailing list
> > Lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] about lustre and virtual ip

[Felix New]

Hello all,
   i am running lustre 1.6.4 on my redhat advanced server 4 update
4.as all know, the mgs/mds(out mgs/mds on the same server) server is
the center server, and the cluster is unusable if the mgs/mds is dead.
so i setup heartbeat between the two mgs/mds server2, and use
drbd(network raid1) as the backend storage to sync data each other.

  to make the setup easily, i use a virtual ip(fload ip) between two
mgs servers(assume mgs0: 10.0.0.1,mgs1:10.0.0.2, vip: 10.0.0.3,
oss0:10.0.0.11, oss1:10.0.0.12, etc.). mgs0 as the primary mgs, and
mgs1 as sencondary mgs. when use mgs0, the vip 10.0.0.3 is a alias
addr on server0, and when mgs0 dead, the vip 10.0.0.3 will move to
mgs1(controled by heartbeat). so the mgs hight avalibal is transparent
to all the oss servers, clients, and they can only know the mgs
10.0.0.3, and mkfs with param --mgsnode=10.0.0.3 on oss and mount
cluster with nid of vip on client.

  now the problem is: when i add the vip to the mgs0, and mkfs.lustre
with --mgsnode=10.0.0.3, but the ost can't found the mgs when mount
the ost.the same error whe mount the cluster,but but when use
10.0.0.1.
client# mount -t lustre [EMAIL PROTECTED]:/mycfs /cfs/client/
client# mount.lustre: mount [EMAIL PROTECTED]:/mycfs at /cfs/client
failed: Cannot send after transport endpoint shutdown


-- 
Best regards
Felix New
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Lustre SNMP module

[Kilian CAVALOTTI]

Hi Brian, 

On Monday 10 March 2008 03:04:33 pm Brian J. Murrell wrote:
> I can't disagree with that, especially as Lustre installations get
> bigger and bigger.  Apart from writing custom monitoring tools,
> there's not a lot of "pre-emptive" monitoring options available. 
> There are a few tools out there like collectl (never seen it, just
> heard about it) 

collectl is very nice, but as dstat and such, it has to run on each and
every host. It can provide its results via sockets though, so it could
be used as a centralized monitoring system for a Lustre installation.

And it provides detailled statistics too:

# collectl -sL -O R
waiting for 1 second sample...

# LUSTRE CLIENT DETAIL: READAHEAD
#Filsys   Reads ReadKB  Writes WriteKB  Pend  Hits Misses NotCon MisWin LckFal  
Discrd ZFile ZerWin RA2Eof HitMax
home100192   0   0 0 0100  0  0  0  
0  0100  0  0
scratch 100192   0   0 0 0100  0  0  0  
0  0100  0  0
home102   6294  23 233 0 0 87  0  0  0  
0  0 87  0  0
scratch 102   6294  23 233 0 0 87  0  0  0  
0  0 87  0  0
home 95158  22 222 0 0 81  0  0  0  
0  0 81  0  0
scratch  95158  22 222 0 0 81  0  0  0  
0  0 81  0  0

# collectl -sL -O M
waiting for 1 second sample...

# LUSTRE CLIENT DETAIL: METADATA
#Filsys   Reads ReadKB  Writes WriteKB  Open Close GAttr SAttr  Seek Fsync 
DrtHit DrtMis
home  0  0   0   0 0 0 0 0 0 0  
0  0
scratch   0  0   0   0 0 0 2 0 0 0  
0  0
home  0  0   0   0 0 0 0 0 0 0  
0  0
scratch   0  0   0   0 0 0 0 0 0 0  
0  0
home  0  0   0   0 0 0 0 0 0 0  
0  0
scratch   0  0   0   0 0 0 1 0 0 0  
0  0

# collectl -sL -O B
waiting for 1 second sample...

# LUSTRE FILESYSTEM SINGLE OST STATISTICS
#Ost  Rds  RdK   1K   2K   4K   8K  16K  32K  64K 128K 256K Wrts 
WrtK   1K   2K   4K   8K  16K  32K  64K 128K 256K
home-OST0007000000000000
0000000000
scratch-OST0007 00900000000   12 
3075900000003
home-OST0007000000000000
0000000000
scratch-OST0007 001000000001
2100000000
home-OST0007000000000000
0000000000
scratch-OST0007 001000000001
2100000000


> and LLNL have one on sourceforge, 

Last time I checked, it only supported 1.4 versions, but it's been a while, 
so I'm probably a bit behind.

> but I can certainly  
> see the attraction at being able to monitor Lustre on your servers
> with the same tools as you are using to monitor the servers' health
> themselves.

Yes, that'd be a strong selling point.

> This could wind becoming a lustre-devel@ discussion, but for now, it
> would be interesting to extend the interface(s) we use to
> introduce /proc (and what will soon be it's replacement/augmentation)
> stats files so that they are automagically provided via SNMP.

That sounds like the way to proceed, indeed.

> You know, given the discussion in this thread:
> http://lists.lustre.org/pipermail/lustre-devel/2008-January/001475.ht
>ml now would be a good time for the the community (that perhaps might
> want to contribute) desiring SNMP access to get their foot in the
> door. Ideally, you get SNMP into the generic interface and then SNMP
> access to all current and future variables comes more or less free.

Oh, thanks for pointing this. It looks like major underlying changes 
are coming. I think I'll subscribe to the lustre-devel ML to try to 
follow them.

> That all said, there are some /proc files which provide a copious
> amount of information, like brw_stats for instance.  I don't know how
> well that sort of thing maps to SNMP, but having an SNMP manager
> watching something as useful as brw_stats for trends over time could
> be quite interesting.

Add some RRD graphs to keep historical variations, and you got the 
all-in-one Lustre monitoring tool we sysadmins are all waiting for. ;)

Cheers,
-- 
Kilian
___
Lustre-discuss mailing list
Lustre-

Re: [Lustre-discuss] Lustre Thumper Fault Tolerance

[Klaus Steden]

Andreas,

Is the mmp feature already in the existing Lustre distribution? If so, what
versions are mmp-aware? If not, which version will be the first to
incorporate it?

thanks,
Klaus 

On 3/10/08 2:15 PM, "Andreas Dilger" <[EMAIL PROTECTED]>did etch on stone
tablets:

> On Mar 10, 2008  09:09 -0600, Colin Faber wrote:
>> Is this true even in the case of mounting the OSS as a read only node?
> 
> Yes, definitely even a "read only" mount can cause serious corruption.
> There are several issues involved, the most dangerous is that even for
> read-only mounting the journal is replayed by the kernel or otherwise
> the filesystem may appear to be corrupted.
> 
> In addition, there is the problem that (meta)data that is cached on the
> read-only mounting node will become incorrect as the writing node is
> changing the filesystem.  The ext3 filesystem is not cluster aware.
> 
> In order to prevent situations like this, the newer releases of ldiskfs
> and e2fsprogs have an "mmp" (multi-mount protection) feature which will
> prevent the filesystem to be mounted on another node if it is active
> on one node (either mounted, or running e2fsck).
> 
> This will be enabled by default on newly-formatted filesystems which
> are created with the "--failover" flag, and can also be enabled by
> hand with "tune2fs -O mmp /dev/" (replace with MDT or OST device
> names as appropriate).  This will prevent the filesystem from being
> mounted or e2fsck'd by old kernels/e2fsprogs so it isn't enabled by
> default on existing filesystems.
> 
>> Andreas Dilger wrote:
>>> On Mar 07, 2008  00:04 +0530, Neeladri Bose wrote:
>>>   
 To address the performance hit (whatever be the %age) if we setup DRDB in
 active-passive mode across the 4500's but have the LustreFS points to
 separate raid sets from the network across the DRDB pair of 4500's & thus
 become an active-active solution which may actually increase the
 throughput of the LustreFS.
 
 Can it be a possible scenario using DRDB on Linux with ext3 & LustreFS?
 
>>> 
>>> No, Lustre does not support active-active export of backing filesystems.
>>> This doesn't work because the backing filesystems (ext3/ZFS) are not
>>> themselves cluster-aware and mounting them on two nodes will quickly
>>> lead to whole filesystem corruption.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
> 
> ___
> Lustre-discuss mailing list
> Lustre-discuss@lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Lustre SNMP module

[Brian J. Murrell]

On Mon, 2008-03-10 at 13:11 -0700, Kilian CAVALOTTI wrote:
> I think it would also be useful to get "activity metrics", the same kind 
> of information which is in /proc/fs/lustre/llite/*/stats on clients (so 
> we can see reads/writes and fs operations rates), 
> in /proc/fs/lustre/obdfilter/*/stats on OSSes and 
> in /proc/fs/lustre/mds/*/stats on MDSes.

I can't disagree with that, especially as Lustre installations get
bigger and bigger.  Apart from writing custom monitoring tools, there's
not a lot of "pre-emptive" monitoring options available.  There are a
few tools out there like collectl (never seen it, just heard about it)
and LLNL have one on sourceforge, but I can certainly see the attraction
at being able to monitor Lustre on your servers with the same tools as
you are using to monitor the servers' health themselves.

> Actually, all the /proc/fs/lustre/*/**/stats could be useful, but I 
> guess what precise metric is the most useful heavily depends on what 
> you want to see. :)

This could wind becoming a lustre-devel@ discussion, but for now, it
would be interesting to extend the interface(s) we use to
introduce /proc (and what will soon be it's replacement/augmentation)
stats files so that they are automagically provided via SNMP.

You know, given the discussion in this thread:
http://lists.lustre.org/pipermail/lustre-devel/2008-January/001475.html
now would be a good time for the the community (that perhaps might want
to contribute) desiring SNMP access to get their foot in the door.
Ideally, you get SNMP into the generic interface and then SNMP access to
all current and future variables comes more or less free.

That all said, there are some /proc files which provide a copious amount
of information, like brw_stats for instance.  I don't know how well that
sort of thing maps to SNMP, but having an SNMP manager watching
something as useful as brw_stats for trends over time could be quite
interesting.

b.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Lustre Thumper Fault Tolerance

[Andreas Dilger]

On Mar 10, 2008  09:09 -0600, Colin Faber wrote:
> Is this true even in the case of mounting the OSS as a read only node?

Yes, definitely even a "read only" mount can cause serious corruption.
There are several issues involved, the most dangerous is that even for
read-only mounting the journal is replayed by the kernel or otherwise
the filesystem may appear to be corrupted.

In addition, there is the problem that (meta)data that is cached on the
read-only mounting node will become incorrect as the writing node is
changing the filesystem.  The ext3 filesystem is not cluster aware.

In order to prevent situations like this, the newer releases of ldiskfs
and e2fsprogs have an "mmp" (multi-mount protection) feature which will
prevent the filesystem to be mounted on another node if it is active
on one node (either mounted, or running e2fsck).

This will be enabled by default on newly-formatted filesystems which
are created with the "--failover" flag, and can also be enabled by
hand with "tune2fs -O mmp /dev/" (replace with MDT or OST device
names as appropriate).  This will prevent the filesystem from being
mounted or e2fsck'd by old kernels/e2fsprogs so it isn't enabled by
default on existing filesystems.

> Andreas Dilger wrote:
>> On Mar 07, 2008  00:04 +0530, Neeladri Bose wrote:
>>   
>>> To address the performance hit (whatever be the %age) if we setup DRDB in 
>>> active-passive mode across the 4500's but have the LustreFS points to 
>>> separate raid sets from the network across the DRDB pair of 4500's & thus 
>>> become an active-active solution which may actually increase the 
>>> throughput of the LustreFS.
>>>
>>> Can it be a possible scenario using DRDB on Linux with ext3 & LustreFS?
>>> 
>>
>> No, Lustre does not support active-active export of backing filesystems.
>> This doesn't work because the backing filesystems (ext3/ZFS) are not
>> themselves cluster-aware and mounting them on two nodes will quickly
>> lead to whole filesystem corruption.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[Jack Chen]

> Jack,
>  
> Thanks, I had copied the output from that in a previous e-mail and 
> here it is again:
>  
> [EMAIL PROTECTED] ~]# ls 
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre
> llite_lloop.ko  lov.ko  lquota.ko  lustre.ko  lvfs.ko  mdc.ko  mgc.ko  
> obdclass.ko  obdecho.ko  osc.ko  ptlrpc.ko
>  
> and:
>
> [EMAIL PROTECTED] ~]# rpm -ql lustre-modules
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/llite_lloop.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lov.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lquota.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lustre.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lvfs.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/mdc.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/mgc.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/obdclass.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/obdecho.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/osc.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/ptlrpc.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/ksocklnd.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/libcfs.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/lnet.ko
> /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/lnet_selftest.ko
> /usr/share/doc/lustre-modules-1.6.4.3
> /usr/share/doc/lustre-modules-1.6.4.3/COPYING
> And here is the output for modprobing mgs:
>  
> [EMAIL PROTECTED] ~]# modprobe mgs
> FATAL: Module mgs not found.
> [EMAIL PROTECTED] ~]# dmesg
> [EMAIL PROTECTED] ~]#
>  
>  
Seems you just built patchless lustre modules, for patched lustre modues 
as follows:

 #ls /lib/modules/2.6.18-53.1.13.el5_lustre.1.6.4smp/kernel/fs/lustre/
fsfilt_ldiskfs.ko  lov.ko lvfs.ko  mgc.ko   obdecho.koost.ko
llite_lloop.ko lquota.ko  mdc.ko   mgs.ko   obdfilter.ko  ptlrpc.ko
llog_test.ko   lustre.ko  mds.ko   obdclass.ko  osc.ko

I'm not sure what commands you use to compile lustre(configure/lbuild), 
I'd recommend you re-compile lustre with patched lustre modules.
Normally, you don't specify any flags, but try to the following commands 
or by steps as Lustre_manual document described:

./configure --disable-modules --disable-utils --disable-liblustre 
--disable-tests --disable-doc
make clean 
make dist 
sh -x $CUR_LUSTRE/build/lbuild --target=2.6-rhel5 --tag=b1_6 
--kerneltree=/path/to/kernel 
--target-arch=$ARCH --lustre=$CUR_LUSTRE/lustre-$VERSION.tar.gz --release 
...


> Apparently it is not on my system. Looking in the lustre source, it 
> looks like mgs should have been compiled with everything else. Is 
> there a special flag to enable it?
>  
> Thank you,
> Mitchel
>
>  

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Lustre SNMP module

[Kilian CAVALOTTI]

Hi Klaus,

On Friday 07 March 2008 05:52:51 pm Klaus Steden wrote:
> I was asking that same question a few months ago.

Yes, I remember you haven't been overwhelmed by answers. :\

> I can send you my 
> 1.6.2 spec file for reference ... That version also did not bundle
> the SNMP library, so I ended up building it by recompiling the whole
> set of Lustre RPMs to get what I needed, and then just dropped the
> DSO in place.

That's exactly what I did, finally.

> I'm curious as to what metrics you see to be useful -- I wasn't sure
> what to look for, so while I installed the module, I haven't yet
> thought of good things to ask of it.

So, from what I've seen in the MIB, the current SNMP module mainly 
report version numbers and free space information.

I think it would also be useful to get "activity metrics", the same kind 
of information which is in /proc/fs/lustre/llite/*/stats on clients (so 
we can see reads/writes and fs operations rates), 
in /proc/fs/lustre/obdfilter/*/stats on OSSes and 
in /proc/fs/lustre/mds/*/stats on MDSes.

Actually, all the /proc/fs/lustre/*/**/stats could be useful, but I 
guess what precise metric is the most useful heavily depends on what 
you want to see. :)

Cheers,
-- 
Kilian
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[Jack Chen]

> Hmm. I did run into this while trying llmount.sh.
> [EMAIL PROTECTED] tests]# pwd
> /usr/src/lustre-1.6.4.3/lustre/tests 
> [EMAIL PROTECTED] tests]# sh llmount.sh
> Loading modules from /usr/src/lustre-1.6.4.3/lustre/tests/ 
> ..
> lnet options: 'networks=tcp0'
> FATAL: Module mgs not found.
> [EMAIL PROTECTED] tests]# dmesg -c
> [EMAIL PROTECTED] tests]#
> Does this mean I should add a ",mgs" to "networks=tcp0"?
Can you verify if mgs module is exist?

Run command by Isaac mentioned:

ls /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre
rpm -ql lustre-modules

If so, please try to "modprobe mgs" manually to see if any messages 
displayed.

Jack
> 
>
> ___
> Lustre-discuss mailing list
> Lustre-discuss@lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[mitcheloc]

Hmm. I did run into this while trying llmount.sh.
[EMAIL PROTECTED] tests]# pwd
/usr/src/lustre-1.6.4.3/lustre/tests
[EMAIL PROTECTED] tests]# sh llmount.sh
Loading modules from /usr/src/lustre-1.6.4.3/lustre/tests/..
lnet options: 'networks=tcp0'
FATAL: Module mgs not found.
[EMAIL PROTECTED] tests]# dmesg -c
[EMAIL PROTECTED] tests]#
Does this mean I should add a ",mgs" to "networks=tcp0"?
On Mon, Mar 10, 2008 at 1:32 PM, mitcheloc <[EMAIL PROTECTED]> wrote:

> Isaac,
>
> I checked my ethernet card and it didn't look like Quadrics hardware.
>
>  [EMAIL PROTECTED] ~]# lspci | grep Ethernet
> 00:19.0 Ethernet controller: Intel Corporation 82566DM Gigabit Network
> Connection (rev 02)
>
> So I removed the parameter, rebooted and it worked like a charm! I wonder
> how that setting got into my modules.conf file. I checked on another
> CentOS system I set up and it is not there. It was probably inserted by some
> other DFS I was trying out.
>
> After changing modules.conf and rebooting:
>
> [EMAIL PROTECTED] ~]# modprobe lustre
> [EMAIL PROTECTED] ~]# dmesg
> Lustre: OBD class driver, [EMAIL PROTECTED]
> Lustre Version: 1.6.4.3
> Build Version:
> 1.6.4.3-1969123116-PRISTINE-.usr.src.linux-2.6.18-53.1.14.el5.lustre
> Lustre: Added LNI [EMAIL PROTECTED] [8/256]
> Lustre: Accept secure, port 988
> Lustre: Lustre Client File System; [EMAIL PROTECTED]
>
> Thanks & hopefully I don't run into any other issues.
>
> Cheers,
> Mitchel
>
>   On Mon, Mar 10, 2008 at 12:48 PM, Isaac Huang <[EMAIL PROTECTED]> wrote:
>
> > On Mon, Mar 10, 2008 at 11:38:33AM -0500, mitcheloc wrote:
> > >From modprobe.conf:
> > >
> > >options lnet networks=tcp0,elan0
> >
> > If you don't have Quadrics Elan hardware, you can change it to:
> > options lnet networks=tcp0
> >
> > Otherwise,
> >
> > >Where should kqswlnd.ko be coming from?
> >
> > you need to compile lustre with proper QsNet support.
> >
> > Isaac
> >
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[mitcheloc]

Isaac,

I checked my ethernet card and it didn't look like Quadrics hardware.

 [EMAIL PROTECTED] ~]# lspci | grep Ethernet
00:19.0 Ethernet controller: Intel Corporation 82566DM Gigabit Network
Connection (rev 02)

So I removed the parameter, rebooted and it worked like a charm! I wonder
how that setting got into my modules.conf file. I checked on another CentOS
system I set up and it is not there. It was probably inserted by some other
DFS I was trying out.

After changing modules.conf and rebooting:

[EMAIL PROTECTED] ~]# modprobe lustre
[EMAIL PROTECTED] ~]# dmesg
Lustre: OBD class driver, [EMAIL PROTECTED]
Lustre Version: 1.6.4.3
Build Version:
1.6.4.3-1969123116-PRISTINE-.usr.src.linux-2.6.18-53.1.14.el5.lustre
Lustre: Added LNI [EMAIL PROTECTED] [8/256]
Lustre: Accept secure, port 988
Lustre: Lustre Client File System; [EMAIL PROTECTED]

Thanks & hopefully I don't run into any other issues.

Cheers,
Mitchel

On Mon, Mar 10, 2008 at 12:48 PM, Isaac Huang <[EMAIL PROTECTED]> wrote:

> On Mon, Mar 10, 2008 at 11:38:33AM -0500, mitcheloc wrote:
> >From modprobe.conf:
> >
> >options lnet networks=tcp0,elan0
>
> If you don't have Quadrics Elan hardware, you can change it to:
> options lnet networks=tcp0
>
> Otherwise,
>
> >Where should kqswlnd.ko be coming from?
>
> you need to compile lustre with proper QsNet support.
>
> Isaac
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[Isaac Huang]

On Mon, Mar 10, 2008 at 11:38:33AM -0500, mitcheloc wrote:
>From modprobe.conf:
> 
>options lnet networks=tcp0,elan0

If you don't have Quadrics Elan hardware, you can change it to:
options lnet networks=tcp0

Otherwise,

>Where should kqswlnd.ko be coming from?

you need to compile lustre with proper QsNet support.

Isaac
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[mitcheloc]

>From modprobe.conf:

options lnet networks=tcp0,elan0

Where should kqswlnd.ko be coming from?

On Mon, Mar 10, 2008 at 11:35 AM, Isaac Huang <[EMAIL PROTECTED]> wrote:

> On Mon, Mar 10, 2008 at 11:19:54AM -0500, mitcheloc wrote:
> >Isaac,
> >
> >Thanks for the quick response. A quick google search didn't tell me
> how
> >I can check the module parameters. What command or file should I
> check
> >for this?
> >
>
> It shall be in /etc/modprobe.conf or some file under /etc/modprobe.d.
> Exact location depends on your distribution. Look for a line that
> starts with "options lnet ".
>
> >
> >And as you requested:
> >
> >[EMAIL PROTECTED] ~]# ls
> >/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre
> >ksocklnd.ko  libcfs.ko  lnet.ko  lnet_selftest.ko
> >
>
> The kqswlnd.ko is missing.
>
> Isaac
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[Isaac Huang]

On Mon, Mar 10, 2008 at 11:19:54AM -0500, mitcheloc wrote:
>Isaac,
> 
>Thanks for the quick response. A quick google search didn't tell me how
>I can check the module parameters. What command or file should I check
>for this?
> 

It shall be in /etc/modprobe.conf or some file under /etc/modprobe.d.
Exact location depends on your distribution. Look for a line that
starts with "options lnet ".

> 
>And as you requested:
> 
>[EMAIL PROTECTED] ~]# ls
>/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre
>ksocklnd.ko  libcfs.ko  lnet.ko  lnet_selftest.ko
> 

The kqswlnd.ko is missing.

Isaac
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[mitcheloc]

Isaac,

Thanks for the quick response. A quick google search didn't tell me how I
can check the module parameters. What command or file should I check for
this?

And as you requested:

[EMAIL PROTECTED] ~]# ls /lib/modules/2.6.18-53.1.14.el5.lustre
/kernel/net/lustre
ksocklnd.ko  libcfs.ko  lnet.ko  lnet_selftest.ko

[EMAIL PROTECTED] ~]# rpm -ql lustre-modules
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/llite_lloop.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lov.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lquota.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lustre.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lvfs.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/mdc.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/mgc.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/obdclass.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/obdecho.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/osc.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/ptlrpc.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/ksocklnd.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/libcfs.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/lnet.ko
/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre/lnet_selftest.ko
/usr/share/doc/lustre-modules-1.6.4.3
/usr/share/doc/lustre-modules-1.6.4.3/COPYING

Thank you!
On Mon, Mar 10, 2008 at 11:02 AM, Isaac Huang <[EMAIL PROTECTED]> wrote:

> On Mon, Mar 10, 2008 at 10:04:50AM -0500, mitcheloc wrote:
> >
> >[EMAIL PROTECTED] ~]# dmesg
> >Lustre: OBD class driver, [EMAIL PROTECTED]
> >Lustre Version: [3]1.6.4.3
> >Build Version:
> >
> 1.6.4.3-1969123116-PRISTINE-.usr.src.linux-2.6.18-53.1.14.el5.lustr
> >e
> >Lustre: Added LNI [EMAIL PROTECTED] [8/256]
> >LustreError: 2359:0:(api-ni.c:1025:lnet_startup_lndnis()) Can't load
> >LND elan, module kqswlnd, rc=256
>
> LNet couldn't load the driver module (kqswlnd) for elan. What's your
> lnet module parameters?
>
> Please also run:
> ls /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre
> rpm -ql lustre-modules
>
> Thanks,
> Isaac
>
> >Lustre: Removed LNI [EMAIL PROTECTED]
>  >LustreError: 2359:0:(events.c:654:ptlrpc_init_portals()) network
> >initialisation failed
>
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Modprobe lustre fails

[Isaac Huang]

On Mon, Mar 10, 2008 at 10:04:50AM -0500, mitcheloc wrote:
> 
>[EMAIL PROTECTED] ~]# dmesg
>Lustre: OBD class driver, [EMAIL PROTECTED]
>Lustre Version: [3]1.6.4.3
>Build Version:
>1.6.4.3-1969123116-PRISTINE-.usr.src.linux-2.6.18-53.1.14.el5.lustr
>e
>Lustre: Added LNI [EMAIL PROTECTED] [8/256]
>LustreError: 2359:0:(api-ni.c:1025:lnet_startup_lndnis()) Can't load
>LND elan, module kqswlnd, rc=256

LNet couldn't load the driver module (kqswlnd) for elan. What's your
lnet module parameters? 

Please also run:
ls /lib/modules/2.6.18-53.1.14.el5.lustre/kernel/net/lustre
rpm -ql lustre-modules

Thanks,
Isaac

>Lustre: Removed LNI [EMAIL PROTECTED]
>LustreError: 2359:0:(events.c:654:ptlrpc_init_portals()) network
>initialisation failed
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] Modprobe lustre fails

[mitcheloc]

Hello,

I'm attempting to get Lustre set up on my machine. I am running CentOS and
I've patched, recompiled and booted into my new kernel using the rhel5
patches. I then compiled Lustre, created the RPMs and installed them. I've
attached as much info below as I could. I ran into this same issue using the
rhel5 rpms from the lustre download site.

[EMAIL PROTECTED] ~]# yum list lustre*
Installed Packages
lustre.i386  1.6.4.3-2.6.18_53.1.14 installed
lustre-modules.i386  1.6.4.3-2.6.18_53.1.14 installed

[EMAIL PROTECTED] ~]# uname -a
Linux catapult 2.6.18-53.1.14.el5.lustre #1 SMP Sun Mar 9 23:49:12 PDT 2008
i686 i686 i386 GNU/Linux

[EMAIL PROTECTED] ~]# modinfo lustre
filename:   /lib/modules/2.6.18-53.1.14.el5.lustre
/kernel/fs/lustre/lustre.ko
license:GPL
description:Lustre Lite Client File System
author: Cluster File Systems, Inc. <[EMAIL PROTECTED]>
srcversion: 39D35F54FE0ABEDEB6EE0A6
depends:obdclass,mdc,ptlrpc,libcfs,lvfs,lov,lnet
vermagic:   2.6.18-53.1.14.el5.lustre SMP mod_unload 686 REGPARM
4KSTACKS gcc-4.1
[EMAIL PROTECTED] ~]# modprobe lustre
WARNING: Error inserting ptlrpc
(/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/ptlrpc.ko):
Input/output error
WARNING: Error inserting mdc
(/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/mdc.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lov
(/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lov.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting lustre
(/lib/modules/2.6.18-53.1.14.el5.lustre/kernel/fs/lustre/lustre.ko):
Unknown symbol in module, or unknown parameter (see dmesg)

[EMAIL PROTECTED] ~]# ls /lib/modules/2.6.18-53.1.14.el5.lustre
/kernel/fs/lustre
llite_lloop.ko  lov.ko  lquota.ko  lustre.ko  lvfs.ko  mdc.ko  mgc.ko
obdclass.ko  obdecho.ko  osc.ko  ptlrpc.ko

[EMAIL PROTECTED] ~]# dmesg
Lustre: OBD class driver, [EMAIL PROTECTED]
Lustre Version: 1.6.4.3
Build Version:
1.6.4.3-1969123116-PRISTINE-.usr.src.linux-2.6.18-53.1.14.el5.lustre
Lustre: Added LNI [EMAIL PROTECTED] [8/256]
LustreError: 2359:0:(api-ni.c:1025:lnet_startup_lndnis()) Can't load LND
elan, module kqswlnd, rc=256
Lustre: Removed LNI [EMAIL PROTECTED]
LustreError: 2359:0:(events.c:654:ptlrpc_init_portals()) network
initialisation failed
mdc: Unknown symbol ldlm_prep_enqueue_req
mdc: Unknown symbol ldlm_resource_get
mdc: Unknown symbol lustre_msg_get_last_xid
mdc: Unknown symbol _ldlm_lock_debug
mdc: Unknown symbol ptlrpcd_addref
mdc: Unknown symbol lustre_msg_get_magic
mdc: Unknown symbol ptlrpc_check_set
mdc: Unknown symbol lustre_msg_get_last_committed
mdc: Unknown symbol ptlrpc_queue_wait
mdc: Unknown symbol client_import_del_conn
mdc: Unknown symbol ptlrpc_request_addref
mdc: Unknown symbol lustre_msg_bufcount
mdc: Unknown symbol ptlrpc_invalidate_import
mdc: Unknown symbol ldlm_completion_ast
mdc: Unknown symbol lustre_msg_add_flags
mdc: Unknown symbol client_obd_setup
mdc: Unknown symbol ldlm_cancel_resource_local
mdc: Unknown symbol ldlm_lock_match
mdc: Unknown symbol ptlrpc_set_import_active
mdc: Unknown symbol client_obd_cleanup
mdc: Unknown symbol ptlrpc_prep_bulk_imp
mdc: Unknown symbol ptlrpc_set_add_req
mdc: Unknown symbol lustre_swab_mds_body
mdc: Unknown symbol __ldlm_handle2lock
mdc: Unknown symbol ldlm_cli_cancel_list
mdc: Unknown symbol ptlrpcd_add_req
mdc: Unknown symbol llog_client_ops
mdc: Unknown symbol ptlrpc_prep_bulk_page
mdc: Unknown symbol lustre_msg_buf
mdc: Unknown symbol lustre_msg_buflen
mdc: Unknown symbol ldlm_lock_put
mdc: Unknown symbol lustre_swab_obd_statfs
mdc: Unknown symbol client_import_add_conn
mdc: Unknown symbol ldlm_lock_addref
mdc: Unknown symbol ldlm_lock_decref_and_cancel
mdc: Unknown symbol ldlm_resource_iterate
mdc: Unknown symbol unlock_res_and_lock
mdc: Unknown symbol ldlm_cli_enqueue
mdc: Unknown symbol lock_res_and_lock
mdc: Unknown symbol client_disconnect_export
mdc: Unknown symbol ptlrpc_free_rq_pool
mdc: Unknown symbol lustre_msg_get_opc
mdc: Unknown symbol ptlrpc_import_setasync
mdc: Unknown symbol lprocfs_wr_ping
mdc: Unknown symbol ldlm_namespace_cleanup
mdc: Unknown symbol lustre_msg_get_status
mdc: Unknown symbol ldlm_resource_putref
mdc: Unknown symbol lustre_msg_size
mdc: Unknown symbol ldlm_it2str
mdc: Unknown symbol _debug_req
mdc: Unknown symbol lustre_msg_get_type
mdc: Unknown symbol lustre_swab_repbuf
mdc: Unknown symbol ptlrpc_recover_import
mdc: Unknown symbol ptlrpc_prep_req
mdc: Unknown symbol client_connect_import
mdc: Unknown symbol ptlrpcd_decref
mdc: Unknown symbol ldlm_cli_enqueue_fini
mdc: Unknown symbol lustre_msg_set_buflen
mdc: Unknown symbol ptlrpc_req_finished
mdc: Unknown symbol ldlm_lock_decref
lov: Unknown symbol ptlrpc_set_destroy
lov: Unknown symbol ptlrpc_prep_set
lov: Unknown symbol _ldlm_lock_debug
lov: Unknown symbol lustre_swab_lov_user_md
lov: Unknown symbol __ldlm_handle2lock

Re: [Lustre-discuss] yet another lustre error

[Brock Palen]

On Mar 9, 2008, at 10:01 PM, Aaron Knister wrote:

> Hi! I have a few questions for you-
>
> 1. How many nodes was his job running on?

around 64 serial jobs accessing the same directory (not the same files).

> 2. What version of lustre and linux kernel are you running on your  
> servers/clients?

Lustre servers:
2.6.9-55.0.9.EL_lustre.1.6.4.1smp

Clients:
2.6.9-67.0.1.ELsmp


> 3. What ethernet module are you using on the servers/clients?

Most use the tg3, some use e1000.

>
> I honestly am not sure what the RPC errors mean but I've had  
> similar issues caused by ethernet-level errors.

Over the weekend the MDS/MGS went into a unhealthy state forced a  
reboot+fsck and when it came back up the directory was accessible  
again and jobs started working again.

>
> -Aaron
>
> On Mar 7, 2008, at 6:45 PM, Brock Palen wrote:
>
>> On a file system thats been up for only 57 days,  I have:
>>
>> 505 lustre-log.   dumps.
>>
>> THe problem at hand is a user has many jobs where his jobs are now
>> hung trying to create a directory from his pbs script.  On the
>> clients i see:
>>
>> LustreError: 11-0: an error occurred while communicating with
>> [EMAIL PROTECTED] The mds_connect operation failed with -16
>> LustreError: Skipped 2 previous similar messages
>>
>> On every client his jobs are on.
>>
>> In the most recent /tmp/lustre-log.  on the MDS/MGS I see this  
>> message:
>>
>> @@@ processing error (-16)  [EMAIL PROTECTED] x12808293/t0 o38-
>>> [EMAIL PROTECTED]:-1
>> lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
>> ldlm_lib.c
>> target_handle_reconnect
>> nobackup-MDT: 34b4fbea-200b-1f7c-dac0-516b8ce786fc reconnecting
>> ldlm_lib.c
>> target_handle_connect
>> nobackup-MDT: refuse reconnection from 34b4fbea-200b-1f7c-
>> [EMAIL PROTECTED]@tcp to 0x0100069a7000; still busy
>> with 2 active RPCs
>> ldlm_lib.c
>> target_send_reply_msg
>> @@@ processing error (-16)  [EMAIL PROTECTED] x11199816/t0 o38-
>>> [EMAIL PROTECTED]:-1
>> lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
>>
>>
>> What I see messages about active rpc's in other logs.  What would
>> this mean?  Is something suck someplace ?
>>
>>
>>
>> Brock Palen
>> Center for Advanced Computing
>> [EMAIL PROTECTED]
>> (734)936-1985
>>
>>
>> ___
>> Lustre-discuss mailing list
>> Lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
> Aaron Knister
> Associate Systems Analyst
> Center for Ocean-Land-Atmosphere Studies
>
> (301) 595-7000
> [EMAIL PROTECTED]
>
>
>
>
>
>

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lustre dstat plugin

[Brock Palen]

On Mar 9, 2008, at 10:03 PM, Aaron Knister wrote:

> Just wondering if either of you have used collectl if/and which you  
> prefer- dstat or collectl.

Never used it, Looks like they solve the same problem.  I like dstat  
for the simple plugins. (if your a better python programer than me).   
And how you can pull our results, like I use the following on our  
lustre OSS with two OST's sda and sdb.

dstat -D sda,sdb,total

That gives me per disk stats and a total.

Similar tools could be made for collectl I'm sure.

Brock

>
> -Aaron
>
> On Mar 7, 2008, at 7:03 PM, Brock Palen wrote:
>
>> On Mar 7, 2008, at 6:58 PM, Kilian CAVALOTTI wrote:
>>
>>> Hi Brock,
>>>
>>> On Wednesday 05 March 2008 05:21:51 pm Brock Palen wrote:
 I have wrote a lustre dstat plugin.  You can find it on my blog:
>>>
>>> That's cool! Very useful for my daily work, thanks!
>>
>> Thanks!  Its the first python I ever wrote.
>>
>>>
 It only works on clients, and has not been tested on multiple  
 mounts,
 Its very simple just reads /proc/
>>>
>>> It indeed doesn't read stats for multiple mounts. I slightly
>>> modified it
>>> so it can display read/write numbers for all the mounts it founds  
>>> (see
>>> the attached patch).
>>
>> This is great idea
>>
>>>
>>> Here's a typical output for a rsync transfer from scrath to home:
>>>
>>> -- 8<  
>>> ---
>>> $ dstat -M lustre
>>>
>>> Module dstat_lustre is still experimental.
>>> --scratch---home---
>>> read write: read write
>>> 110M0 :   0   110M
>>> 183M0 :   0   183M
>>> 184M0 :   0   184M
>>> -- 8<  
>>> ---
>>>
>>> Maybe it could be useful to also add the other metrics from the stat
>>> file, but I'm not sure which ones would be the more relevant. And it
>>> would probably be wise to do that in a separate module, like
>>> lustre_stats, to avoid clutter.
>>
>> Yes,  dstat comes with plugins for nfsv3  and has two modules,
>>
>> dstat_nfs3  and dstat_nfs3op  which has extended details.  So I think
>> this would be a good idea to follow that model.
>>
>>>
>>> Anyway, great job, and thanks for sharing it!
>>
>> Thanks again.
>>
>>> Cheers,
>>> -- 
>>> Kilian
>>
>> ___
>> Lustre-discuss mailing list
>> Lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
> Aaron Knister
> Associate Systems Analyst
> Center for Ocean-Land-Atmosphere Studies
>
> (301) 595-7000
> [EMAIL PROTECTED]
>
>
>
>
>
>

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lustre and small files overhead

[Balagopal Pillai]

Hi,

   I tried gfs and ocfs2 before i came back to Lustre as the 
choice for hpc cluster filesystem
and they were both quite good for small files. I wasn't happy with
the stability of either at the time and also didn't want to deal with 
the quorum issues when all
i wanted was a cluster file system and not the high availability part of 
the solution.

  gfs and ocfs2 are worth trying for benchmarks with small 
files. Please note that gfs2 is
not considered production ready. So you would need to stick with rhel4 
instead of rhel5 if sticking
with a redhat solution.


Regards
Balagopal


Joe Barjo wrote:
> Could it be tuned one day for small files?
> Which filesystem would you suggest for me?
> I already tried nfs, afs
> I will now try glusterfs
>
> Thanks for your support
>
> 
>
> ___
> Lustre-discuss mailing list
> Lustre-discuss@lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>   
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] socknal_sd00 100% lower?

[Isaac Huang]

On Fri, Mar 07, 2008 at 03:26:23PM -0700, Andreas Dilger wrote:
> 
> Maxim, Isaac, what are your thoughts about disabling IRQ affinity
> by default?  In the past this was important for maximizing performance
> with N CPUs and N ethernet NICs, but the CPUs have gotten much faster
> and more cores and I believe other customers have found better performance
> with irq_affinity disabled.
> 

Agree, and with ksocklnd bonding feature deprecated it's now more
common to configure lnet with a single NIC. I've committed the change
and filed a documentation bug to update the manuals accordingly.

Thanks,
Isaac
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] lustre and small files overhead

[Joe Barjo]

Andreas Dilger a écrit :
> On Mar 07, 2008  12:49 +0100, Joe Barjo wrote:
>   
>> I made some more tests, and have setup a micro lustre cluster on lvm
>> volumes.
>> node a: MDS
>> node b and c: OST
>> node a,b,c,d,e,f: clients
>> Gigabit ethernet network.
>> Made the optimizations: lnet.debug=0, lru_size to 1, max_dirty_mb to
>> 1024
>> 
>
> For high RPC-rate operations using an interconnect like Infiniband is
> better than ethernet.
>
>   
infiniband is not in our budget...
>> The svn checkout takes 50s ( 15s on a localdisk, 25s on a local lustre
>> demo (with debug=0))
>> Launching gkrellm, a single svn checkout consumes about 20% of the MDS
>> system cpu with about 2.4mbyte/sec ethernet communication.
>> 
>
>   
>> About 6MByte/s disk bandwidth on OST1, up to 12-16MB/s on OST2 disk
>> bandwidth, network bandwidth on OST is about 10 to 20 times under disk
>> bandwidth.
>> Why so much disk bandwidth on OSTs, is it a readahead problem?
>> 
>
> That does seem strange, I can't really say why.  There is some metadata
> overhead, and that is higher with small files, but I don't think it
> would be 10-20x overhead.
>
>   
The checkouted source is only 65 megabytes. So much OST disk bandwidth
is probably not normal.
Maybe you should verify this point.
Are you sure there isn't an optimazation for this? This looks like
readahead or something like that.
>> I launched a compilation distributed on the 6 clients:
>> MDS system cpu goes up to 60% system ressource (athlon 64 3500+)
>> 12MByte/s on the ethernet, OST goes up to the same level as previous test.
>>
>> How come is there so much network communications on the MDT?
>> 
>
> Because every metadata operation has to be done on the MDS currently.
> We are working toward having metadata writeback cache operations on
> the client, but it doesn't happen currently.  For operations like
> compilation it is basically entirely metadata overhead.
>
>   
>> As I understood that the MDS can not be load balanced, I don't see how
>> lustre is scalable to thousands of clients...
>> 
>
> Because in many HPC environments there are very few metadata operations
> in comparison to the amount of data being read/written.  Average file
> sizes are 20-30MB instead of 20-30kB.
>
>   
>> It looks like lustre is not made for this kind of application
>> 
>
> No, it definitely isn't tuned for small files.
>   
Could it be tuned one day for small files?
Which filesystem would you suggest for me?
I already tried nfs, afs
I will now try glusterfs

Thanks for your support

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] yet another lustre error

[Alex Lyashkov]

On Fri, 2008-03-07 at 18:45 -0500, Brock Palen wrote:
> On a file system thats been up for only 57 days,  I have:
> 

> target_handle_reconnect
> nobackup-MDT: 34b4fbea-200b-1f7c-dac0-516b8ce786fc reconnecting
> ldlm_lib.c
> target_handle_connect
> nobackup-MDT: refuse reconnection from 34b4fbea-200b-1f7c- 
> [EMAIL PROTECTED]@tcp to 0x0100069a7000; still busy  
> with 2 active RPCs
> ldlm_lib.c
> target_send_reply_msg
> @@@ processing error (-16)  [EMAIL PROTECTED] x11199816/t0 o38- 
>  >[EMAIL PROTECTED]:-1  
> lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
> 
> 
> What I see messages about active rpc's in other logs.  What would  
> this mean?  Is something suck someplace ?
> 
-16 = EBUSY. This say client reconnected to server which already work on
different request from this client. After old rpc from this client will
be finished - client will be reconnected.

-- 
Alex Lyashkov <[EMAIL PROTECTED]>
Lustre Group, Sun Microsystems

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss