[lustre-discuss] MDS cases with ldlm_flock_deadlock error

[Lixin Liu via lustre-discuss]

Hi,

Starting from yesterday, we see frequent MDS crashes, all of them are showing 
ldlm_flock_deadlock.
Servers are running Lustre 2.15.4, MDT and MGT are on LDISKFS and OSTs are on 
ZFS. AlmaLinux 8.9.
Clients are mostly CentOS 7.9 with Lustre client 2.15.4.

In one of these crashes, we have a complete coredump in case if someone wants 
to check.

Thanks,

Lixin.

[15817.464501] LustreError: 22687:0:(ldlm_flock.c:230:ldlm_flock_deadlock()) 
ASSERTION( req != lock ) failed:
[15817.474247] LustreError: 22687:0:(ldlm_flock.c:230:ldlm_flock_deadlock()) 
LBUG
[15817.481497] Pid: 22687, comm: mdt01_003 4.18.0-513.9.1.el8_lustre.x86_64 #1 
SMP Sat Dec 23 05:23:32 UTC 2023
[15817.491318] Call Trace TBD:
[15817.494137] [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[15817.499297] [<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[15817.504097] [<0>] ldlm_flock_deadlock.isra.10+0x1fb/0x240 [ptlrpc]
[15817.510398] [<0>] ldlm_process_flock_lock+0x289/0x1f90 [ptlrpc]
[15817.516402] [<0>] ldlm_lock_enqueue+0x2a5/0xaa0 [ptlrpc]
[15817.521813] [<0>] ldlm_handle_enqueue0+0x634/0x1520 [ptlrpc]
[15817.527562] [<0>] tgt_enqueue+0xa4/0x220 [ptlrpc]
[15817.532368] [<0>] tgt_request_handle+0xccd/0x1a20 [ptlrpc]
[15817.537949] [<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]
[15817.544311] [<0>] ptlrpc_main+0xbec/0x1530 [ptlrpc]
[15817.549294] [<0>] kthread+0x134/0x150
[15817.552966] [<0>] ret_from_fork+0x1f/0x40
[15817.556980] Kernel panic - not syncing: LBUG
[15817.561248] CPU: 23 PID: 22687 Comm: mdt01_003 Kdump: loaded Tainted: G  
 OE- -  - 4.18.0-513.9.1.el8_lustre.x86_64 #1
[15817.573669] Hardware name: Dell Inc. PowerEdge R640/0CRT1G, BIOS 2.19.1 
06/04/2023
[15817.581235] Call Trace:
[15817.583687]  dump_stack+0x41/0x60
[15817.587007]  panic+0xe7/0x2ac
[15817.589979]  ? ret_from_fork+0x1f/0x40
[15817.593733]  lbug_with_loc.cold.8+0x18/0x18 [libcfs]
[15817.598714]  ldlm_flock_deadlock.isra.10+0x1fb/0x240 [ptlrpc]
[15817.604557]  ldlm_process_flock_lock+0x289/0x1f90 [ptlrpc]
[15817.610121]  ? lustre_msg_get_flags+0x2a/0x90 [ptlrpc]
[15817.615346]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
[15817.620745]  ldlm_lock_enqueue+0x2a5/0xaa0 [ptlrpc]
[15817.625702]  ldlm_handle_enqueue0+0x634/0x1520 [ptlrpc]
[15817.631007]  tgt_enqueue+0xa4/0x220 [ptlrpc]
[15817.635365]  tgt_request_handle+0xccd/0x1a20 [ptlrpc]
[15817.640503]  ? ptlrpc_nrs_req_get_nolock0+0xff/0x1f0 [ptlrpc]
[15817.646337]  ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]
[15817.652256]  ptlrpc_main+0xbec/0x1530 [ptlrpc]
[15817.656791]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
[15817.661928]  kthread+0x134/0x150
[15817.665161]  ? set_kthread_struct+0x50/0x50
[15817.669346]  ret_from_fork+0x1f/0x40


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS hardware - NVME?

[Thomas Roth via lustre-discuss]

Dear all,

considering NVME storage for the next MDS.

As I understand, NVME disks are bundled in software, not by a hardware raid 
controller.
This would be done using Linux software raid, mdadm, correct?

We have some experience with ZFS, which we use on our OSTs.
But I would like to stick to ldiskfs for the MDTs, and a zpool with a zvol on 
top which is then formatted with ldiskfs - to much voodoo...

How is this handled elsewhere? Any experiences?


The available devices are quite large. If I create a raid-10 out of 4 disks, e.g. 7 TB each, my MDT will be 14 TB - already close to the 16 TB limit. 
So no need for a box with lots of U.3 slots.


But for MDS operations, we will still need a powerful dual-CPU system with lots 
of RAM.
Then the NVME devices should be distributed between the CPUs?
Is there a way to pinpoint this in a call for tender?


Best regards,
Thomas


Thomas Roth

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
State Secretary / Staatssekretär Dr. Volkmar Dietz


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashes, lustre version 2.15.3

[Lixin Liu via lustre-discuss]

Hi Aurelien,

Thanks, I guess we will have to rebuild our own 2.15.x server. I see other 
crashes have different dump, usually like these:

[36664.403408] BUG: unable to handle kernel NULL pointer dereference at 

[36664.411237] PGD 0 P4D 0
[36664.413776] Oops:  [#1] SMP PTI
[36664.417268] CPU: 28 PID: 11101 Comm: qmt_reba_cedar_ Kdump: loaded Tainted: 
G  IOE- -  - 4.18.0-477.10.1.el8_lustre.x86_64 #1
[36664.430293] Hardware name: Dell Inc. PowerEdge R640/0CRT1G, BIOS 2.19.1 
06/04/2023
[36664.437860] RIP: 0010:qmt_id_lock_cb+0x69/0x100 [lquota]
[36664.443199] Code: 48 8b 53 20 8b 4a 0c 85 c9 74 78 89 c1 48 8b 42 18 83 78 
10 02 75 0a 83 e1 01 b8 01 00 00 00 74 17 48 63 44 24 04 48 c1 e0 04 <48> 03 45 
00 f6 40 08 0c 0f 95 c0 0f b6 c0 48 8b 4c 24 08 65 48 33
[36664.461942] RSP: 0018:aa2e303f3df0 EFLAGS: 00010246
[36664.467169] RAX:  RBX: 98722c74b700 RCX: 
[36664.474301] RDX: 9880415ce660 RSI: 0010 RDI: 9881240b5c64
[36664.481435] RBP:  R08:  R09: 0004
[36664.488566] R10: 0010 R11: f000 R12: 98722c74b700
[36664.495697] R13: 9875fc07a320 R14: 9878444d3d10 R15: 9878444d3cc0
[36664.502832] FS:  () GS:987f20f8() 
knlGS:
[36664.510917] CS:  0010 DS:  ES:  CR0: 80050033
[36664.516664] CR2:  CR3: 002065a10004 CR4: 007706e0
[36664.523794] DR0:  DR1:  DR2: 
[36664.530927] DR3:  DR6: fffe0ff0 DR7: 0400
[36664.538058] PKRU: 5554
[36664.540772] Call Trace:
[36664.543231]  ? cfs_cdebug_show.part.3.constprop.23+0x20/0x20 [lquota]
[36664.549699]  qmt_glimpse_lock.isra.20+0x1e7/0xfa0 [lquota]
[36664.555204]  qmt_reba_thread+0x5cd/0x9b0 [lquota]
[36664.559927]  ? qmt_glimpse_lock.isra.20+0xfa0/0xfa0 [lquota]
[36664.565602]  kthread+0x134/0x150
[36664.568834]  ? set_kthread_struct+0x50/0x50
[36664.573021]  ret_from_fork+0x1f/0x40
[36664.576603] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) 
mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) mbcache jbd2 lustre(OE) 
lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ksocklnd(OE) ko2iblnd(OE) 
ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) 8021q garp mrp stp llc dell_rbu 
vfat fat dm_round_robin dm_multipath rpcrdma sunrpc rdma_ucm ib_srpt ib_isert 
iscsi_target_mod target_core_mod ib_iser libiscsi opa_vnic scsi_transport_iscsi 
ib_umad rdma_cm ib_ipoib iw_cm ib_cm intel_rapl_msr intel_rapl_common 
isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp 
coretemp kvm_intel dell_smbios iTCO_wdt iTCO_vendor_support wmi_bmof 
dell_wmi_descriptor dcdbas kvm ipmi_ssif irqbypass crct10dif_pclmul hfi1 
mgag200 crc32_pclmul drm_shmem_helper ghash_clmulni_intel rdmavt qla2xxx 
drm_kms_helper rapl ib_uverbs nvme_fc intel_cstate syscopyarea nvme_fabrics 
sysfillrect sysimgblt nvme_core intel_uncore fb_sys_fops pcspkr acpi_ipmi 
ib_core scsi_transport_fc igb
[36664.576699]  drm ipmi_si i2c_algo_bit mei_me dca ipmi_devintf mei i2c_i801 
lpc_ich wmi ipmi_msghandler acpi_power_meter xfs libcrc32c sd_mod t10_pi sg 
ahci libahci crc32c_intel libata megaraid_sas dm_mirror dm_region_hash dm_log 
dm_mod
[36664.684758] CR2: 

Is this also related to the same bug?

Thanks,

Lixin.

From: Aurelien Degremont 
Date: Wednesday, November 29, 2023 at 8:31 AM
To: lustre-discuss , Lixin Liu 
Subject: RE: MDS crashes, lustre version 2.15.3

You are likely hitting that bug https://jira.whamcloud.com/browse/LU-15207 
which is fixed in (not yet released) 2.16.0

Aurélien

De : lustre-discuss  de la part de 
Lixin Liu via lustre-discuss 
Envoyé : mercredi 29 novembre 2023 17:18
À : lustre-discuss 
Objet : [lustre-discuss] MDS crashes, lustre version 2.15.3

External email: Use caution opening links or attachments


Hi,

We built our 2.15.3 environment a few months ago. MDT is using ldiskfs and OSTs 
are using ZFS.
The system seems to perform well at the beginning, but recently, we see 
frequent MDS crashes.
The vmcore-dmesg.txt shows the following:

[26056.031259] LustreError: 69513:0:(hash.c:1469:cfs_hash_for_each_tight()) 
ASSERTION( !cfs_hash_is_rehashing(hs) ) failed:
[26056.043494] LustreError: 69513:0:(hash.c:1469:cfs_hash_for_each_tight()) LBUG
[26056.051460] Pid: 69513, comm: lquota_wb_cedar 
4.18.0-477.10.1.el8_lustre.x86_64 #1 SMP Tue Jun 20 00:12:13 UTC 2023
[26056.063099] Call Trace TBD:
[26056.066221] [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[26056.071970] [<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[26056.077322] [<0>] cfs_hash_for_each_tight+0x301/0x310 [libcfs]
[26056.083839] [<0>] qsd_start_reint_thread+0x561/0xcc0 [lquota]
[26056.090265] [<0>] qsd_upd_thread+0xd43/0x1040 [lquota]
[26056.096008] [<0>] kthread+0x13

[lustre-discuss] mds and mst are lost

[Sergey Astafyev via lustre-discuss]

Hello.
Please help.
It doesn’t really matter how, but MDS and MDT servers are  lost.
I connected old OSTs to the new lustre, but I don’t see any information.
lfs find tells me cannot get lov name : inappropriate ioctl for device

Is there any mechanism for recovering information from orphaned OST partitions?

best regards

S.Astafev
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS kernel taunt when accessing snapshot files

[Martin Rehr via lustre-discuss]

Greetings,

We are running a lustre setup on top of ZFS.

Mounting a lustre snapshot on a client and thereafter running:
'lfs data_version -n SNAPSHOT_FILEPATH'

Makes the MDS server fail with a kernel taunt (dump_stack call) thrown from 
'lustre/osd-zfs/osd_handler.c : osd_trans_create':

if (dt->dd_rdonly) {
CERROR("%s: someone try to start transaction under "
   "readonly mode, should be disabled.\n",
   osd_name(osd_dt_dev(dt)));
dump_stack();
RETURN(ERR_PTR(-EROFS));
}


In the above (lfs dataversion) example we can prevent the kernel taunt by 
mounting the snapshot (client side) with 'noatime', but that doesn't help us if 
we try to run:
`lfs getstripe -d SNAPSHOT_FILEPATH'

We have traced the problem and found that it is the function calls 'mdt_open.c 
: mo_attr_set' and 'mdt_open.c : mo_xattr_set' that triggers the kernel taunt.

Our MDS crashed hard after running the above tests on millions of (snapshot) 
files simultaneously and we had to restore it from generic ZFS snapshots.

Is this a know issue ?

Best Regards,
   Martin


signature.asc
Description: Message signed with OpenPGP
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS using D3710 DAS - partially Solved

[Sid Young]

After some investigation it looks like a timeout issue in the smartpqi
kernel module is causing the disks to be removed soon after they are
initially added based on what is reported in "dmesg"

This issue first occurred in RHEL/Centos 7.4 and should have been resolved
by centos 7.7. I've emailed the maintainer of the module and he's come
back to me with an offer to create a test driver to see if increasing the
timeout fixes the issue. There is an existing patch but its version is less
than the one in Centos 7.9.

On the bright side, I've built and rebuilt the Lustre MDS and OSS config
several times as I optimise the installation while running under
Pacemaker and have been able to mount /lustre and /home on the Compute
nodes so this new system is 50% of the way there :)


Sid Young


> Today's Topics:
>
>1. Re: MDS using D3710 DAS (Sid Young)
>2. Re: MDS using D3710 DAS (Christopher Mountford)
>
>
>
> -- Forwarded message --
> From: Sid Young 
> To: Christopher Mountford ,
> lustre-discuss@lists.lustre.org
> Cc:
> Bcc:
> Date: Mon, 15 Feb 2021 08:42:43 +1000
> Subject: Re: [lustre-discuss] MDS using D3710 DAS
> Hi Christopher,
>
> Just some background, all servers are DL385's all servers are running the
> same image of Centos 7.9, The MDS HA pair have a SAS connected D3710 and
> the dual OSS HA pair have a D8000 each with 45 disks in each of them.
>
> The D3710 (which has 24x 960G SSD's) seams a bit hit and miss at
> presenting two LV's, I had setup a /lustre and /home which I was going to
> use ldiskfs rather than zfs however I am finding that the disks MAY present
> to both servers after some reboots but usually the first server to reboot
> see's the LV presented and the other only see's its local internal disks
> only, so the array appears to only present the LV's to one host most of the
> time.
>
> With the 4 OSS servers. i see the same issue, sometimes the LV's present
> and sometimes they don't.
>
> I was planning on setting up the OST's as ldiskfs as well, but I could
> also go zfs, my test bed system and my current HPC uses ldsikfs.
>
> Correct me if I am wrong, but disks should present to both servers all the
> time and using PCS I should be able to mount up a /lustre and /home one the
> first server while the disks present on the second server but no software
> is mounting them so there should be no issues?
>
>
> Sid Young
>
> On Fri, Feb 12, 2021 at 7:27 PM Christopher Mountford <
> cj...@leicester.ac.uk> wrote:
>
>> Hi Sid,
>>
>> We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair which
>> each consist of 2 DL360 connected to a single D3700. However we are using
>> Lustre on ZFS with each array split into 2 or 4 zpools (depending on the
>> usage) and haven't seen any problems of this sort. Are you using ldiskfs?
>>
>> - Chris
>>
>>
>> On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:
>> >G'day all,
>> >Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
>> >Configuration? If so, is your D3710 presenting LV's to both servers
>> at
>> >the same time AND are you using PCS with the Lustre PCS Resources?
>> >I've just received new kit and cannot get disk to present to the MDS
>> >servers at the same time. :(
>> >Sid Young
>>
>> > ___
>> > lustre-discuss mailing list
>> > lustre-discuss@lists.lustre.org
>> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>>
>
>
> -- Forwarded message --
> From: Christopher Mountford 
> To: Sid Young 
> Cc: Christopher Mountford ,
> lustre-discuss@lists.lustre.org
> Bcc:
> Date: Mon, 15 Feb 2021 10:44:10 +
> Subject: Re: [lustre-discuss] MDS using D3710 DAS
>
> Hi Sid.
>
> We use the D3700s (and our D8000s) as JBODS with zfs providing the
> redundancy - do you have some kind of hardware RAID? If so, are your raid
> controller the array corntrollers or on the HBAs? Off the top of my head,
> if the latter, there might be an issue with multiple HBAs trying to
> assemble the same RAID array?
>
> - Chris.
>
> On Mon, Feb 15, 2021 at 08:42:43AM +1000, Sid Young wrote:
> >Hi Christopher,
> >Just some background, all servers are DL385's all servers are running
> >the same image of Centos 7.9, The MDS HA pair have a SAS connected
> >D3710 and the dual OSS HA pair have a D8000 each with 45 disks in each
> >of them.
> >The D3710 (which has 24x 960G SSD's) seams a bit hit and miss at
> >presenting two LV's, I had setup a /

Re: [lustre-discuss] MDS using D3710 DAS

[Christopher Mountford]

Hi Sid.

We use the D3700s (and our D8000s) as JBODS with zfs providing the redundancy - 
do you have some kind of hardware RAID? If so, are your raid controller the 
array corntrollers or on the HBAs? Off the top of my head, if the latter, there 
might be an issue with multiple HBAs trying to assemble the same RAID array? 

- Chris.

On Mon, Feb 15, 2021 at 08:42:43AM +1000, Sid Young wrote:
>Hi Christopher,
>Just some background, all servers are DL385's all servers are running
>the same image of Centos 7.9, The MDS HA pair have a SAS connected
>D3710 and the dual OSS HA pair have a D8000 each with 45 disks in each
>of them.
>The D3710 (which has 24x 960G SSD's) seams a bit hit and miss at
>presenting two LV's, I had setup a /lustre and /home which I was going
>to use ldiskfs rather than zfs however I am finding that the disks MAY
>present to both servers after some reboots but usually the first server
>to reboot see's the LV presented and the other only see's its local
>internal disks only, so the array appears to only present the LV's to
>one host most of the time.
>With the 4 OSS servers. i see the same issue, sometimes the LV's
>present and sometimes they don't.
>I was planning on setting up the OST's as ldiskfs as well, but I could
>also go zfs, my test bed system and my current HPC uses ldsikfs.
>Correct me if I am wrong, but disks should present to both servers all
>the time and using PCS I should be able to mount up a /lustre and /home
>one the first server while the disks present on the second server but
>no software is mounting them so there should be no issues?
>Sid Young
> 
>On Fri, Feb 12, 2021 at 7:27 PM Christopher Mountford
><[1]cj...@leicester.ac.uk> wrote:
> 
>  Hi Sid,
>  We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair
>  which each consist of 2 DL360 connected to a single D3700. However
>  we are using Lustre on ZFS with each array split into 2 or 4 zpools
>  (depending on the usage) and haven't seen any problems of this sort.
>  Are you using ldiskfs?
>  - Chris
>  On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:
>  >G'day all,
>  >Is anyone using a HPe D3710 with two HPeDL380/385 servers in a
>  MDS HA
>  >Configuration? If so, is your D3710 presenting LV's to both
>  servers at
>  >the same time AND are you using PCS with the Lustre PCS
>  Resources?
>  >I've just received new kit and cannot get disk to present to
>  the MDS
>  >servers at the same time. :(
>  >Sid Young
>  > ___
>  > lustre-discuss mailing list
>  > [2]lustre-discuss@lists.lustre.org
>  > [3]http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
> References
> 
>1. mailto:cj...@leicester.ac.uk
>2. mailto:lustre-discuss@lists.lustre.org
>3. 
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org=04%7C01%7Ccjm14%40leicester.ac.uk%7C4d86239b31b545d327db08d8d139f050%7Caebecd6a31d44b0195ce8274afe853d9%7C0%7C0%7C637489394067185599%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=x1PMOvlWp3bocS%2Bub1mpvE1Mn59Q0EU0M18NQbj1wOk%3D=0
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS using D3710 DAS

[Sid Young]

Hi Christopher,

Just some background, all servers are DL385's all servers are running the
same image of Centos 7.9, The MDS HA pair have a SAS connected D3710 and
the dual OSS HA pair have a D8000 each with 45 disks in each of them.

The D3710 (which has 24x 960G SSD's) seams a bit hit and miss at presenting
two LV's, I had setup a /lustre and /home which I was going to use ldiskfs
rather than zfs however I am finding that the disks MAY present to both
servers after some reboots but usually the first server to reboot see's the
LV presented and the other only see's its local internal disks only, so the
array appears to only present the LV's to one host most of the time.

With the 4 OSS servers. i see the same issue, sometimes the LV's present
and sometimes they don't.

I was planning on setting up the OST's as ldiskfs as well, but I could also
go zfs, my test bed system and my current HPC uses ldsikfs.

Correct me if I am wrong, but disks should present to both servers all the
time and using PCS I should be able to mount up a /lustre and /home one the
first server while the disks present on the second server but no software
is mounting them so there should be no issues?


Sid Young

On Fri, Feb 12, 2021 at 7:27 PM Christopher Mountford 
wrote:

> Hi Sid,
>
> We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair which
> each consist of 2 DL360 connected to a single D3700. However we are using
> Lustre on ZFS with each array split into 2 or 4 zpools (depending on the
> usage) and haven't seen any problems of this sort. Are you using ldiskfs?
>
> - Chris
>
>
> On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:
> >G'day all,
> >Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
> >Configuration? If so, is your D3710 presenting LV's to both servers at
> >the same time AND are you using PCS with the Lustre PCS Resources?
> >I've just received new kit and cannot get disk to present to the MDS
> >servers at the same time. :(
> >Sid Young
>
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS using D3710 DAS

[Daniel Kobras]

Hi!

Am 12.02.21 um 06:14 schrieb Sid Young:
> Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
> Configuration? If so, is your D3710 presenting LV's to both servers at
> the same time AND are you using PCS with the Lustre PCS Resources?
> 
> I've just received new kit and cannot get disk to present to the MDS
> servers at the same time. :(

It's been a while, but I've seen this once with an enclosure that had
been equipped with single- instead of dual-ported drives by mistake.
Might be worthwhile to double-check the specs for your system.

Kind regards,

Daniel
-- 
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de

-- 
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft: Eisenbahnstraße 1, 72072 
Tübingen

Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer: 
Lukas Kallies, Daniel Kobras, Mark Pröhl

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS using D3710 DAS

[Peter Jones]

Sorry about the spam everyone. We get a constant stream of would-be spammers 
trying to post to the lists but when non-members try and post it has to go 
through administrator approval before reaching the list. This account was a 
first in that it actually registered to the list before posting (I've now 
blocked it). If we see more of this kind of thing we'll update it so that new 
registrants have to be approved before being able to post but I would prefer to 
do this if possible because it can cause delays for legitimate community 
members to be able to participate.

On 2021-02-12, 5:29 AM, "lustre-discuss on behalf of lino" 
 wrote:

Hola Mar,

Estoy online en zoom, cuando quieras:

https://ucph-ku.zoom.us/j/5939783343

Lino

On 12/2/21 10:27, Christopher Mountford wrote:
> Hi Sid,
>
> We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair which 
each consist of 2 DL360 connected to a single D3700. However we are using 
Lustre on ZFS with each array split into 2 or 4 zpools (depending on the usage) 
and haven't seen any problems of this sort. Are you using ldiskfs?
>
> - Chris
>
>
> On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:
>> G'day all,
>> Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
>> Configuration? If so, is your D3710 presenting LV's to both servers 
at
>> the same time AND are you using PCS with the Lustre PCS Resources?
>> I've just received new kit and cannot get disk to present to the MDS
>> servers at the same time. :(
>> Sid Young
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS using D3710 DAS

[lino]

Hola Mar,

Estoy online en zoom, cuando quieras:

https://ucph-ku.zoom.us/j/5939783343

Lino

On 12/2/21 10:27, Christopher Mountford wrote:

Hi Sid,

We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair which each 
consist of 2 DL360 connected to a single D3700. However we are using Lustre on 
ZFS with each array split into 2 or 4 zpools (depending on the usage) and 
haven't seen any problems of this sort. Are you using ldiskfs?

- Chris


On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:

G'day all,
Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
Configuration? If so, is your D3710 presenting LV's to both servers at
the same time AND are you using PCS with the Lustre PCS Resources?
I've just received new kit and cannot get disk to present to the MDS
servers at the same time. :(
Sid Young
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS using D3710 DAS

[Christopher Mountford]

Hi Sid,

We've a similar hardware configuration - 2 MDS pairs and 1 OSS pair which each 
consist of 2 DL360 connected to a single D3700. However we are using Lustre on 
ZFS with each array split into 2 or 4 zpools (depending on the usage) and 
haven't seen any problems of this sort. Are you using ldiskfs?

- Chris


On Fri, Feb 12, 2021 at 03:14:58PM +1000, Sid Young wrote:
>G'day all,
>Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
>Configuration? If so, is your D3710 presenting LV's to both servers at
>the same time AND are you using PCS with the Lustre PCS Resources?
>I've just received new kit and cannot get disk to present to the MDS
>servers at the same time. :(
>Sid Young

> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS using D3710 DAS

[Sid Young]

G'day all,

Is anyone using a HPe D3710 with two HPeDL380/385 servers in a MDS HA
Configuration? If so, is your D3710 presenting LV's to both servers at the
same time AND are you using PCS with the Lustre PCS Resources?

I've just received new kit and cannot get disk to present to the MDS
servers at the same time. :(


Sid Young
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS HIT LBUG

[肖正刚]

Hi，
Our MDS hit a bug(Red Hat issue) as described in :
https://jira.whamcloud.com/browse/LU-10678
https://jira.whamcloud.com/browse/LU-11786

Our kernel version : 3.10.0-957.10.1.el7_lustre.x86_64
lustre version: 2.12.2
OS version: CentOS 7.6

RHEL said the kernel bug was resolved in kernel-3.10.0-957.12.1.el7 (
https://access.redhat.com/errata/RHSA-2019:0818)

but in LU-11786, Mahmoud Hanafi said they hit this bug twice  with
3.10.0-957.21.3.el7 and lustre2.12.2,  so It does appear to be unresolved.

Does anyone hit this bug and fixed?

Thanks.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS/MDT

[Abe Asraoui]

Adding lustre-devel alias
From: "Abe Asraoui (System)" 
Date: Friday, February 14, 2020 at 3:43 PM
To: "lustre-discuss@lists.lustre.org" , "Abe 
Asraoui (System)" 
Subject: MDS/MDT

 Hi All,

Has anyone done any meta data performance with dual ported nvmes or regular 
nvmes..

Can you share what IOPs were obtained vs sata ssds ?


Thanks in Advance,
abe


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS/MDT

[Abe Asraoui]

 Hi All,

Has anyone done any meta data performance with dual ported nvmes or regular 
nvmes..

Can you share what IOPs were obtained vs sata ssds ?


Thanks in Advance,
abe


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS/MGS has a block storage device mounted and it does not have any permissions (no read , no write, no execute)

[Andreas Dilger]

On Feb 5, 2019, at 15:39, Pinkesh Valdria  wrote:
> 
> Hello All,
>  
> I am new to Lustre.   I started by using the docs on this page to deploy 
> Lustre on Virtual machines running CentOS 7.x (CentOS-7-2018.08.15-0).
> Included below are the content of the scripts I used and the error I get.  
> I have not done any setup for “o2ib0(ib0)” and lnet is using tcp.   All the 
> nodes are on the same network & subnet and cannot communicate on my protocol 
> and port #. 
>  
> Thanks for your help.  I am completely blocked and looking for ideas. 
> (already did google search ☹).  
>  
> I have 2 questions:  
>   • The MDT mounted on MDS has no permissions (no read , no write, no 
> execute), even for root user on MDS/MGS node.   Is that expected? .   See 
> “MGS/MDS node setup” section for more details on what I did. 
> [root@lustre-mds-server-1 opc]# mount -t lustre /dev/sdb /mnt/mdt
>  
> [root@lustre-mds-server-1 opc]# ll /mnt
> total 0
> d-. 1 root root 0 Jan  1  1970 mdt

The mountpoint on the MDS is just there for "df" to work and to manage the 
block device. It does not provide access to filesystem.  You need to do a 
client mount for that (typically on another node, but it also works on the 
MDS), like:

mount -t lustre lustre-mds-server-1:/lustrewt /mnt/lustrewt

or similar.

> [root@lustre-mds-server-1 opc]#
>   • Assuming if the above is not an issue,  after setting up OSS/OST and 
> Client node,  When my client tries to mount, I get the below error: 
> [root@lustre-client-1 opc]# mount -t lustre 10.0.2.4@tcp:/lustrewt /mnt
> mount.lustre: mount 10.0.2.4@tcp:/lustrewt at /mnt failed: Input/output error
> Is the MGS running?
> [root@lustre-client-1 opc]#

Can you do "lctl ping" from the client to the MGS node?  Most commonly this 
happens because the client still has a firewall configured, or it is defined to 
have "127.0.0.1" as the local node address.

> dmesg shows the below error on the client node:  
> [root@lustre-client-1 opc]#  dmesg
> [35639.535862] Lustre: 11730:0:(client.c:2114:ptlrpc_expire_one_request()) 
> @@@ Request sent has timed out for slow reply: [sent 1549386846/real 
> 1549386846]  req@9259bb518c00 x1624614953288208/t0(0) 
> o250->MGC10.0.2.4@tcp@10.0.2.4@tcp:26/25 lens 520/544 e 0 to 1 dl 1549386851 
> ref 1 fl Rpc:XN/0/ rc 0/-1
> [35640.535877] LustreError: 7718:0:(mgc_request.c:251:do_config_log_add()) 
> MGC10.0.2.4@tcp: failed processing log, type 1: rc = -5
> [35669.535028] Lustre: 11730:0:(client.c:2114:ptlrpc_expire_one_request()) 
> @@@ Request sent has timed out for slow reply: [sent 1549386871/real 
> 1549386871]  req@9259bb428f00 x1624614953288256/t0(0) 
> o250->MGC10.0.2.4@tcp@10.0.2.4@tcp:26/25 lens 520/544 e 0 to 1 dl 1549386881 
> ref 1 fl Rpc:XN/0/ rc 0/-1
> [35670.546671] LustreError: 15c-8: MGC10.0.2.4@tcp: The configuration from 
> log 'lustrewt-client' failed (-5). This may be the result of communication 
> errors between this node and the MGS, a bad configuration, or other errors. 
> See the syslog for more information.
> [35670.557472] Lustre: Unmounted lustrewt-client
> [35670.560432] LustreError: 7718:0:(obd_mount.c:1582:lustre_fill_super()) 
> Unable to mount  (-5)
> [root@lustre-client-1 opc]#

Nothing here except that the client can't communicate with the MGS.  There 
might be some
more useful messages earlier on in the logs.
 
> I have firewall turned off on all nodes (client, mds/mgs, oss),  selinux is 
> disabled/setenforce=0 .  I can telnet to the MDS/MGS node from client 
> machine.  
>  
>  
> Given below is the setup I have on different nodes: 
>  
> MGS/MDS node setup 
> #!/bin/bash
> service firewalld stop
> chkconfig firewalld off
>  
> cat > /etc/yum.repos.d/lustre.repo << EOF
> [hpddLustreserver]
> name=CentOS- - Lustre
> baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.6.1810/server/
> gpgcheck=0
>  
> [e2fsprogs]
> name=CentOS- - Ldiskfs
> baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/
> gpgcheck=0
>  
> [hpddLustreclient]
> name=CentOS- - Lustre
> baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.6.1810/client/
> gpgcheck=0
> EOF
>  
> sudo yum install lustre-tests -y
>  
> cp /etc/selinux/config /etc/selinux/config.backup
> sed 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
>  
> setenforce 0
>  
> echo "complete.  rebooting now"
> reboot
>  
>  
>  
> After reboot is complete,  I login to the MGS/MDS node as root and run the 
> following steps: 
>  
> The node has a block storage device attached:  /dev/sdb
> Run the below command: 
> pvcreate -y  /dev/sdb
> mkfs.xfs -f /dev/sdb

The "mkfs.xfs" command here is useless.  Lustre uses only ext4 or ZFS for the 
MDT and OSTs,
and reformats the filesystem with mkfs.lustre in any case.

> [root@lustre-mds-server-1 opc]# setenforce 0
> [root@lustre-mds-server-1 opc]# mkfs.lustre --fsname=lustrewt --index=0 --mgs 
> --mdt /dev/sdb
>Permanent disk data:
> Target: 

[lustre-discuss] MDS/MGS has a block storage device mounted and it does not have any permissions (no read , no write, no execute)

[Pinkesh Valdria]

Hello All,

 

I am new to Lustre.   I started by using the docs on this page to deploy Lustre 
on Virtual machines running CentOS 7.x (CentOS-7-2018.08.15-0).    Included 
below are the content of the scripts I used and the error I get.  

I have not done any setup for “o2ib0(ib0)” and lnet is using tcp.   All the 
nodes are on the same network & subnet and cannot communicate on my protocol 
and port #. 

 

Thanks for your help.  I am completely blocked and looking for ideas. (already 
did google search ☹).  

 

I have 2 questions:  

The MDT mounted on MDS has no permissions (no read , no write, no execute), 
even for root user on MDS/MGS node.   Is that expected? .   See “MGS/MDS node 
setup” section for more details on what I did. 

[root@lustre-mds-server-1 opc]# mount -t lustre /dev/sdb /mnt/mdt

 

[root@lustre-mds-server-1 opc]# ll /mnt

total 0

d-. 1 root root 0 Jan  1  1970 mdt

[root@lustre-mds-server-1 opc]#

Assuming if the above is not an issue,  after setting up OSS/OST and Client 
node,  When my client tries to mount, I get the below error: 

[root@lustre-client-1 opc]# mount -t lustre 10.0.2.4@tcp:/lustrewt /mnt

mount.lustre: mount 10.0.2.4@tcp:/lustrewt at /mnt failed: Input/output error

Is the MGS running?

[root@lustre-client-1 opc]#

 

dmesg shows the below error on the client node:  

[root@lustre-client-1 opc]#  dmesg

[35639.535862] Lustre: 11730:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ 
Request sent has timed out for slow reply: [sent 1549386846/real 1549386846]  
req@9259bb518c00 x1624614953288208/t0(0) 
o250->MGC10.0.2.4@tcp@10.0.2.4@tcp:26/25 lens 520/544 e 0 to 1 dl 1549386851 
ref 1 fl Rpc:XN/0/ rc 0/-1

[35640.535877] LustreError: 7718:0:(mgc_request.c:251:do_config_log_add()) 
MGC10.0.2.4@tcp: failed processing log, type 1: rc = -5

[35669.535028] Lustre: 11730:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ 
Request sent has timed out for slow reply: [sent 1549386871/real 1549386871]  
req@9259bb428f00 x1624614953288256/t0(0) 
o250->MGC10.0.2.4@tcp@10.0.2.4@tcp:26/25 lens 520/544 e 0 to 1 dl 1549386881 
ref 1 fl Rpc:XN/0/ rc 0/-1

[35670.546671] LustreError: 15c-8: MGC10.0.2.4@tcp: The configuration from log 
'lustrewt-client' failed (-5). This may be the result of communication errors 
between this node and the MGS, a bad configuration, or other errors. See the 
syslog for more information.

[35670.557472] Lustre: Unmounted lustrewt-client

[35670.560432] LustreError: 7718:0:(obd_mount.c:1582:lustre_fill_super()) 
Unable to mount  (-5)

[root@lustre-client-1 opc]#

 

I have firewall turned off on all nodes (client, mds/mgs, oss),  selinux is 
disabled/setenforce=0 .  I can telnet to the MDS/MGS node from client machine.  

 

 

Given below is the setup I have on different nodes: 

 

MGS/MDS node setup 

#!/bin/bash

service firewalld stop

chkconfig firewalld off

 

cat > /etc/yum.repos.d/lustre.repo << EOF

[hpddLustreserver]

name=CentOS- - Lustre

baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.6.1810/server/

gpgcheck=0

 

[e2fsprogs]

name=CentOS- - Ldiskfs

baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/

gpgcheck=0

 

[hpddLustreclient]

name=CentOS- - Lustre

baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.6.1810/client/

gpgcheck=0

EOF

 

sudo yum install lustre-tests -y

 

cp /etc/selinux/config /etc/selinux/config.backup

sed 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

 

setenforce 0

 

echo "complete.  rebooting now"

reboot

 

 

 

After reboot is complete,  I login to the MGS/MDS node as root and run the 
following steps: 

 

The node has a block storage device attached:  /dev/sdb

Run the below command: 

pvcreate -y  /dev/sdb

mkfs.xfs -f /dev/sdb

 

 

[root@lustre-mds-server-1 opc]# setenforce 0

[root@lustre-mds-server-1 opc]# mkfs.lustre --fsname=lustrewt --index=0 --mgs 
--mdt /dev/sdb

   Permanent disk data:

Target: lustrewt:MDT

Index:  0

Lustre FS:  lustrewt

Mount type: ldiskfs

Flags:  0x65

  (MDT MGS first_time update )

Persistent mount opts: user_xattr,errors=remount-ro

Parameters:

 

checking for existing Lustre data: not found

device size = 51200MB

formatting backing filesystem ldiskfs on /dev/sdb

    target name   lustrewt:MDT

    4k blocks 13107200

    options    -J size=2048 -I 1024 -i 2560 -q -O 
dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E 
lazy_journal_init -F

mkfs_cmd = mke2fs -j -b 4096 -L lustrewt:MDT  -J size=2048 -I 1024 -i 2560 
-q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E 
lazy_journal_init -F /dev/sdb 13107200

 

 

[root@lustre-mds-server-1 opc]# mkdir -p /mnt/mdt

[root@lustre-mds-server-1 opc]# mount -t lustre /dev/sdb /mnt/mdt

[root@lustre-mds-server-1 opc]# modprobe lnet

[root@lustre-mds-server-1 opc]# lctl network up

LNET configured

Re: [lustre-discuss] MDS often overload

[Zeeshan Ali Shah]

I have fixed the issue

some how misconf two systems had the same IP of client .. after it change
all are ok

/Zee

On Mon, Oct 8, 2018 at 12:51 PM Zeeshan Ali Shah 
wrote:

> We are getting the following error when run rsync .
>
> Background: We have three filesystem on same MDS , MDT are different zfs
> pools.. would that be an issue ?
>
> any advice ?
>
> error below
> 
> [Mon Oct  8 12:29:11 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
> Connection restored to 172.100.120.25@o2ib (at 172.100.120.25@o2ib)
> [Mon Oct  8 12:29:11 2018] Lustre: Skipped 14 previous similar messages
> [Mon Oct  8 12:31:25 2018] LNet:
> 31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Abort reconnection of
> 172.100.120.25@o2ib: connected
> [Mon Oct  8 12:31:25 2018] LNet:
> 31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Skipped 9 previous
> similar messages
> [Mon Oct  8 12:31:32 2018] Lustre:
> 54961:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has
> timed out for slow reply: [sent 1538991905/real 1538991905]
> req@8819acf68f00 x1611847503061104/t0(0)
> o36->sgp-MDT-mdc-883ffc2c3000@172.100.120.25@o2ib:12/10 lens
> 608/33520 e 0 to 1 dl 1538991912 ref 2 fl Rpc:X/0/ rc 0/-1
> [Mon Oct  8 12:31:32 2018] Lustre:
> 54961:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous
> similar messages
> [Mon Oct  8 12:31:32 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
> Connection to sgp-MDT (at 172.100.120.25@o2ib) was lost; in progress
> operations using this service will wait for recovery to complete
> [Mon Oct  8 12:31:32 2018] Lustre: Skipped 2 previous similar messages
> [Mon Oct  8 12:31:32 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
> Connection restored to 172.100.120.25@o2ib (at 172.100.120.25@o2ib)
> [Mon Oct  8 12:31:32 2018] Lustre: Skipped 2 previous similar messages
> [Mon Oct  8 12:34:01 2018] LNet:
> 25934:0:(o2iblnd_cb.c:2307:kiblnd_passive_connect()) Stale connection
> request
> [Mon Oct  8 12:34:01 2018] LNet:
> 25934:0:(o2iblnd_cb.c:2307:kiblnd_passive_connect()) Skipped 2 previous
> similar messages
> [Mon Oct  8 12:34:01 2018] LNet:
> 31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Abort reconnection of
> 172.100.120.25@o2ib: connected
> [Mon Oct  8 12:34:01 2018] LNet:
> 31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Skipped 3 previous
> similar messages
> [Mon Oct  8 12:34:08 2018] Lustre:
> 54961:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has
> timed out for slow reply: [sent 1538992061/real 1538992061]
> req@881ad060f500 x1611847503440304/t0(0)
> o101->sgp-MDT-mdc-883ffc2c3000@172.100.120.25@o2ib:12/10 lens
> 880/33728 e 0 to 1 dl 1538992068 ref 2 fl Rpc:X/0/ rc 0/-1
> [Mon Oct  8 12:34:08 2018] Lustre:
> 54961:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous
> similar message
> [Mon Oct  8 12:34:08 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
> Connection to sgp-MDT (at 172.100.120.25@o2ib) was lost; in progress
> operations using this service will wait for recovery to complete
> [Mon Oct  8 12:34:08 2018] Lustre: Skipped 1 previous similar message
> [Mon Oct  8 12:34:08 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
> Connection restored to 172.100.120.25@o2ib (at 172.100.120.25@o2ib)
> [Mon Oct  8 12:34:08 2018] Lustre: Skipped 1 previous similar message
> --
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS often overload

[Zeeshan Ali Shah]

We are getting the following error when run rsync .

Background: We have three filesystem on same MDS , MDT are different zfs
pools.. would that be an issue ?

any advice ?

error below

[Mon Oct  8 12:29:11 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
Connection restored to 172.100.120.25@o2ib (at 172.100.120.25@o2ib)
[Mon Oct  8 12:29:11 2018] Lustre: Skipped 14 previous similar messages
[Mon Oct  8 12:31:25 2018] LNet:
31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Abort reconnection of
172.100.120.25@o2ib: connected
[Mon Oct  8 12:31:25 2018] LNet:
31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Skipped 9 previous
similar messages
[Mon Oct  8 12:31:32 2018] Lustre:
54961:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has
timed out for slow reply: [sent 1538991905/real 1538991905]
req@8819acf68f00 x1611847503061104/t0(0)
o36->sgp-MDT-mdc-883ffc2c3000@172.100.120.25@o2ib:12/10 lens
608/33520 e 0 to 1 dl 1538991912 ref 2 fl Rpc:X/0/ rc 0/-1
[Mon Oct  8 12:31:32 2018] Lustre:
54961:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous
similar messages
[Mon Oct  8 12:31:32 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
Connection to sgp-MDT (at 172.100.120.25@o2ib) was lost; in progress
operations using this service will wait for recovery to complete
[Mon Oct  8 12:31:32 2018] Lustre: Skipped 2 previous similar messages
[Mon Oct  8 12:31:32 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
Connection restored to 172.100.120.25@o2ib (at 172.100.120.25@o2ib)
[Mon Oct  8 12:31:32 2018] Lustre: Skipped 2 previous similar messages
[Mon Oct  8 12:34:01 2018] LNet:
25934:0:(o2iblnd_cb.c:2307:kiblnd_passive_connect()) Stale connection
request
[Mon Oct  8 12:34:01 2018] LNet:
25934:0:(o2iblnd_cb.c:2307:kiblnd_passive_connect()) Skipped 2 previous
similar messages
[Mon Oct  8 12:34:01 2018] LNet:
31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Abort reconnection of
172.100.120.25@o2ib: connected
[Mon Oct  8 12:34:01 2018] LNet:
31340:0:(o2iblnd_cb.c:1350:kiblnd_reconnect_peer()) Skipped 3 previous
similar messages
[Mon Oct  8 12:34:08 2018] Lustre:
54961:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has
timed out for slow reply: [sent 1538992061/real 1538992061]
req@881ad060f500 x1611847503440304/t0(0)
o101->sgp-MDT-mdc-883ffc2c3000@172.100.120.25@o2ib:12/10 lens
880/33728 e 0 to 1 dl 1538992068 ref 2 fl Rpc:X/0/ rc 0/-1
[Mon Oct  8 12:34:08 2018] Lustre:
54961:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous
similar message
[Mon Oct  8 12:34:08 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
Connection to sgp-MDT (at 172.100.120.25@o2ib) was lost; in progress
operations using this service will wait for recovery to complete
[Mon Oct  8 12:34:08 2018] Lustre: Skipped 1 previous similar message
[Mon Oct  8 12:34:08 2018] Lustre: sgp-MDT-mdc-883ffc2c3000:
Connection restored to 172.100.120.25@o2ib (at 172.100.120.25@o2ib)
[Mon Oct  8 12:34:08 2018] Lustre: Skipped 1 previous similar message
--
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Sangeetha Banavathi Srinivasa]

I checked the output of dmesg and found
lnet: Unknown parameter `#’

The error was caused due to a comment that was inserted in 
/etc/modprobe.d/lustre.conf

Thanks a lot for your time and your help.

> On Jun 22, 2016, at 4:03 PM, Patrick Farrell  wrote:
> 
> OK, that's not the error we started with.  You were having a problem loading 
> modules, but aren't now. (probably)
> 
> The error output from that command is bad.  It asks if the modules are 
> loaded, but that doesn't mean they aren't.
> 
> "No such device" is probably the key here.  I'd look at the arguments to the 
> mount command.  Does /dev/vg00/mdt1 exist and is it set up correctly?  How 
> about the mount point, /mdt?
> 
> On 06/22/2016 03:00 PM, Sangeetha Banavathi Srinivasa wrote:
>> The error is in the last step when I have to run MD
>> 
>> 3. mount -t lustre /dev/vg00/mdt1 /mdt
>> mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
>> Are the lustre modules loaded?
>> Check /etc/modprobe.conf and /proc/filesystems
>> 
>>> On Jun 22, 2016, at 3:58 PM, Patrick Farrell >> > wrote:
>>> 
>>> Ah, now that I look more closely - that warning may not be fatal.  Did you 
>>> get a failure, or just the warning?
>>> 
>>> (The guide looks good)
>>> On 06/22/2016 02:52 PM, Sangeetha Banavathi Srinivasa wrote:
 Patrick,
 
 I am following the steps on 
 http://lustre.ornl.gov/lustre101-courses/content/C1/L3/LustreBasicInstall.pdf
  
 
 Except that I am installing 2.8 instead of 2.7.
 Could you recommend any other article/blog that I can follow in order to 
 successfully install lustre.
  
> On Jun 22, 2016, at 3:43 PM, Patrick Farrell  > wrote:
> 
> Hm.  It sounds like your RPMs are not installed correctly, but I vaguely 
> recall something about a packaging issue with the Lustre 2.8 release RPMs.
> 
> Perhaps someone else on the list can help further.
> - Patrick
> 
> On 06/22/2016 02:40 PM, Sangeetha Banavathi Srinivasa wrote:
>> I have done that already and re-booted my system.
>> But I still get the error.
>> 
>> When I did run that command, I got the below error:
>> WARNING: Can't read module 
>> /lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/weak-updates/kernel/fs/lustre/llog_test.ko:
>>  No such file or directory
>> egrep: 
>> /lib/modules/2.6.32-573.12.1.el6_lustre.x86_64//weak-updates/kernel/fs/lustre/llog_test.ko:
>>  No such file or directory
>> 
>> 
>>> On Jun 22, 2016, at 3:38 PM, Patrick Farrell >> > wrote:
>>> 
>>> Well, that's not patching the kernel.  It's making the initrd and and 
>>> updating the boot stuff so your system will boot that kernel.
>>> 
>>> But, yes, that should be what you need to do.
>>> 
>>> - Patrick
>>> On 06/22/2016 02:36 PM, Sangeetha Banavathi Srinivasa wrote:
 When you say install the associated kernel do you mean patching the 
 kernel like below
 sudo /sbin/new-kernel-pkg --package kernel --mkinitrd  --dracut 
 --depmod --install 2.6.32-573.12.1.el6_lustre.x86_64
 
 I have already done rpm -ivh 
 kernel-2.6.32-573.12.1.el6_lustre.x86_64.rpm
 
 
> On Jun 22, 2016, at 2:51 PM, Patrick Farrell  > wrote:
> 
> The kernel version is found with uname -r or cat /proc/version.
> 
> That needs to match the kernel version in the Lustre RPMs.  (The 
> version string is the 2.6.32-573.12.1.el6_lustre) part.
> 
> In fact, it looks like you've got Lustre server RPMs, given the 
> _lustre in the name of the kernel.  That means you need to install 
> the associated kernel.
> 
> - Patrick
> 
> On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:
>> Patrick,
>> 
>> The CentOS version on the machines is 6.8
>> 
>> 
>>   
>> [lustre@hulk2 ~]$ cat /etc/redhat-release 
>> 
>>   CentOS 
>> release 6.8 (Final)
>> 
>> 
>> The kernel version that I have installed is what I found along with 
>> the lustre RPMs
>> [lustre@amaranth3 ~]$ uname -r
>> 2.6.32-573.12.1.el6_lustre.x86_64
>> 
>> 
>> And I am trying to install lustre 2.8.
>> 
>> Should the CentOS release version be 2.7??
>> 
>> 
>> 
>>> On Jun 22, 2016, at 2:17 PM, Patrick Farrell >> > wrote:

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Patrick Farrell]

OK, that's not the error we started with.  You were having a problem 
loading modules, but aren't now. (probably)


The error output from that command is bad.  It asks if the modules are 
loaded, but that doesn't mean they aren't.


"No such device" is probably the key here.  I'd look at the arguments to 
the mount command.  Does /dev/vg00/mdt1 exist and is it set up 
correctly?  How about the mount point, /mdt?


On 06/22/2016 03:00 PM, Sangeetha Banavathi Srinivasa wrote:

The error is in the last step when I have to run MD

3. mount -t lustre /dev/vg00/mdt1 /mdt
mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems

On Jun 22, 2016, at 3:58 PM, Patrick Farrell > wrote:


Ah, now that I look more closely - that warning may not be fatal.  
Did you get a failure, or just the warning?


(The guide looks good)
On 06/22/2016 02:52 PM, Sangeetha Banavathi Srinivasa wrote:

Patrick,

I am following the steps on 
http://lustre.ornl.gov/lustre101-courses/content/C1/L3/LustreBasicInstall.pdf

Except that I am installing 2.8 instead of 2.7.
Could you recommend any other article/blog that I can follow in 
order to successfully install lustre.


On Jun 22, 2016, at 3:43 PM, Patrick Farrell > wrote:


Hm.  It sounds like your RPMs are not installed correctly, but I 
vaguely recall something about a packaging issue with the Lustre 
2.8 release RPMs.


Perhaps someone else on the list can help further.
- Patrick

On 06/22/2016 02:40 PM, Sangeetha Banavathi Srinivasa wrote:

I have done that already and re-booted my system.
But I still get the error.

When I did run that command, I got the below error:
WARNING: Can't read module 
/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/weak-updates/kernel/fs/lustre/llog_test.ko: 
No such file or directory
egrep: 
/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64//weak-updates/kernel/fs/lustre/llog_test.ko: 
No such file or directory



On Jun 22, 2016, at 3:38 PM, Patrick Farrell > wrote:


Well, that's not patching the kernel.  It's making the initrd and 
and updating the boot stuff so your system will boot that kernel.


But, yes, that should be what you need to do.

- Patrick
On 06/22/2016 02:36 PM, Sangeetha Banavathi Srinivasa wrote:
When you say install the associated kernel do you mean patching 
the kernel like below
sudo /sbin/new-kernel-pkg --package kernel --mkinitrd  --dracut 
--depmod --install 2.6.32-573.12.1.el6_lustre.x86_64


I have already done rpm -ivh 
kernel-2.6.32-573.12.1.el6_lustre.x86_64.rpm



On Jun 22, 2016, at 2:51 PM, Patrick Farrell > wrote:


The kernel version is found with uname -r or cat /proc/version.

That needs to match the kernel version in the Lustre RPMs. (The 
version string is the 2.6.32-573.12.1.el6_lustre) part.


In fact, it looks like you've got Lustre server RPMs, given the 
_lustre in the name of the kernel.  That means you need to 
install the associated kernel.


- Patrick

On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:

Patrick,

The CentOS version on the machines is 6.8

[lustre@hulk2 ~]$ cat /etc/redhat-release
CentOS release 6.8 (Final)


The kernel version that I have installed is what I found along 
with the lustre RPMs

[lustre@amaranth3 ~]$ uname -r
2.6.32-573.12.1.el6_lustre.x86_64


And I am trying to install lustre 2.8.

Should the CentOS release version be 2.7??



On Jun 22, 2016, at 2:17 PM, Patrick Farrell > wrote:


Well, it says to check dmesg... Though that will just list 
the symbols. In general, this usually means you've got a 
mismatch between the version of the kernel (or OFED) Lustre 
is built against and your version of either the kernel or 
OFED.  Given the number and locations of the mismatches, I'm 
guessing it's the kernel.


So, compare your kernel version and your Lustre RPMs.

Regards,
- Patrick
On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:

When I try modprobe lustre, I see the below errors:
[*05:44:03*] *root@hulk2**lustre #*modprobe lustre
WARNING: Error inserting obdclass 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ptlrpc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fld 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lmv 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fid 

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Sangeetha Banavathi Srinivasa]

The error is in the last step when I have to run MD

3. mount -t lustre /dev/vg00/mdt1 /mdt
mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems

> On Jun 22, 2016, at 3:58 PM, Patrick Farrell  wrote:
> 
> Ah, now that I look more closely - that warning may not be fatal.  Did you 
> get a failure, or just the warning?
> 
> (The guide looks good)
> On 06/22/2016 02:52 PM, Sangeetha Banavathi Srinivasa wrote:
>> Patrick,
>> 
>> I am following the steps on 
>> http://lustre.ornl.gov/lustre101-courses/content/C1/L3/LustreBasicInstall.pdf
>>  
>> 
>> Except that I am installing 2.8 instead of 2.7.
>> Could you recommend any other article/blog that I can follow in order to 
>> successfully install lustre.
>>  
>>> On Jun 22, 2016, at 3:43 PM, Patrick Farrell >> > wrote:
>>> 
>>> Hm.  It sounds like your RPMs are not installed correctly, but I vaguely 
>>> recall something about a packaging issue with the Lustre 2.8 release RPMs.
>>> 
>>> Perhaps someone else on the list can help further.
>>> - Patrick
>>> 
>>> On 06/22/2016 02:40 PM, Sangeetha Banavathi Srinivasa wrote:
 I have done that already and re-booted my system.
 But I still get the error.
 
 When I did run that command, I got the below error:
 WARNING: Can't read module 
 /lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/weak-updates/kernel/fs/lustre/llog_test.ko:
  No such file or directory
 egrep: 
 /lib/modules/2.6.32-573.12.1.el6_lustre.x86_64//weak-updates/kernel/fs/lustre/llog_test.ko:
  No such file or directory
 
 
> On Jun 22, 2016, at 3:38 PM, Patrick Farrell  > wrote:
> 
> Well, that's not patching the kernel.  It's making the initrd and and 
> updating the boot stuff so your system will boot that kernel.
> 
> But, yes, that should be what you need to do.
> 
> - Patrick
> On 06/22/2016 02:36 PM, Sangeetha Banavathi Srinivasa wrote:
>> When you say install the associated kernel do you mean patching the 
>> kernel like below
>> sudo /sbin/new-kernel-pkg --package kernel --mkinitrd  --dracut --depmod 
>> --install 2.6.32-573.12.1.el6_lustre.x86_64
>> 
>> I have already done rpm -ivh kernel-2.6.32-573.12.1.el6_lustre.x86_64.rpm
>> 
>> 
>>> On Jun 22, 2016, at 2:51 PM, Patrick Farrell >> > wrote:
>>> 
>>> The kernel version is found with uname -r or cat /proc/version.
>>> 
>>> That needs to match the kernel version in the Lustre RPMs.  (The 
>>> version string is the 2.6.32-573.12.1.el6_lustre) part.
>>> 
>>> In fact, it looks like you've got Lustre server RPMs, given the _lustre 
>>> in the name of the kernel.  That means you need to install the 
>>> associated kernel.
>>> 
>>> - Patrick
>>> 
>>> On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:
 Patrick,
 
 The CentOS version on the machines is 6.8
 
  [lustre@hulk2 ~]$ cat /etc/redhat-release 
  CentOS release 6.8 (Final)
 
 
 The kernel version that I have installed is what I found along with 
 the lustre RPMs
 [lustre@amaranth3 ~]$ uname -r
 2.6.32-573.12.1.el6_lustre.x86_64
 
 
 And I am trying to install lustre 2.8.
 
 Should the CentOS release version be 2.7??
 
 
 
> On Jun 22, 2016, at 2:17 PM, Patrick Farrell  > wrote:
> 
> Well, it says to check dmesg...  Though that will just list the 
> symbols.  In general, this usually means you've got a mismatch 
> between the version of the kernel (or OFED) Lustre is built against 
> and your version of either the kernel or OFED.  Given the number and 
> locations of the mismatches, I'm guessing it's the kernel.
> 
> So, compare your kernel version and your Lustre RPMs.
> 
> Regards,
> - Patrick
> On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:
>> When I try modprobe lustre, I see the below errors:
>> [05:44:03] root@hulk2 lustre # modprobe lustre
>> WARNING: Error inserting obdclass 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko):
>>  Unknown symbol in module, or unknown parameter (see 
>>   dmesg)
>> WARNING: Error inserting ptlrpc 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko):
>>  Unknown symbol in module, or unknown parameter (see 

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Patrick Farrell]

Ah, now that I look more closely - that warning may not be fatal. Did 
you get a failure, or just the warning?


(The guide looks good)
On 06/22/2016 02:52 PM, Sangeetha Banavathi Srinivasa wrote:

Patrick,

I am following the steps on 
http://lustre.ornl.gov/lustre101-courses/content/C1/L3/LustreBasicInstall.pdf

Except that I am installing 2.8 instead of 2.7.
Could you recommend any other article/blog that I can follow in order 
to successfully install lustre.


On Jun 22, 2016, at 3:43 PM, Patrick Farrell > wrote:


Hm.  It sounds like your RPMs are not installed correctly, but I 
vaguely recall something about a packaging issue with the Lustre 2.8 
release RPMs.


Perhaps someone else on the list can help further.
- Patrick

On 06/22/2016 02:40 PM, Sangeetha Banavathi Srinivasa wrote:

I have done that already and re-booted my system.
But I still get the error.

When I did run that command, I got the below error:
WARNING: Can't read module 
/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/weak-updates/kernel/fs/lustre/llog_test.ko: 
No such file or directory
egrep: 
/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64//weak-updates/kernel/fs/lustre/llog_test.ko: 
No such file or directory



On Jun 22, 2016, at 3:38 PM, Patrick Farrell > wrote:


Well, that's not patching the kernel.  It's making the initrd and 
and updating the boot stuff so your system will boot that kernel.


But, yes, that should be what you need to do.

- Patrick
On 06/22/2016 02:36 PM, Sangeetha Banavathi Srinivasa wrote:
When you say install the associated kernel do you mean patching 
the kernel like below
sudo /sbin/new-kernel-pkg --package kernel --mkinitrd  --dracut 
--depmod --install 2.6.32-573.12.1.el6_lustre.x86_64


I have already done rpm -ivh 
kernel-2.6.32-573.12.1.el6_lustre.x86_64.rpm



On Jun 22, 2016, at 2:51 PM, Patrick Farrell > wrote:


The kernel version is found with uname -r or cat /proc/version.

That needs to match the kernel version in the Lustre RPMs. (The 
version string is the 2.6.32-573.12.1.el6_lustre) part.


In fact, it looks like you've got Lustre server RPMs, given the 
_lustre in the name of the kernel.  That means you need to 
install the associated kernel.


- Patrick

On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:

Patrick,

The CentOS version on the machines is 6.8

[lustre@hulk2 ~]$ cat /etc/redhat-release
CentOS release 6.8 (Final)


The kernel version that I have installed is what I found along 
with the lustre RPMs

[lustre@amaranth3 ~]$ uname -r
2.6.32-573.12.1.el6_lustre.x86_64


And I am trying to install lustre 2.8.

Should the CentOS release version be 2.7??



On Jun 22, 2016, at 2:17 PM, Patrick Farrell > wrote:


Well, it says to check dmesg...  Though that will just list the 
symbols. In general, this usually means you've got a mismatch 
between the version of the kernel (or OFED) Lustre is built 
against and your version of either the kernel or OFED.  Given 
the number and locations of the mismatches, I'm guessing it's 
the kernel.


So, compare your kernel version and your Lustre RPMs.

Regards,
- Patrick
On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:

When I try modprobe lustre, I see the below errors:
[*05:44:03*] *root@hulk2**lustre #*modprobe lustre
WARNING: Error inserting obdclass 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ptlrpc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fld 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lmv 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fid 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting mdc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lov 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting lustre 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)


On Jun 22, 2016, at 1:44 PM, Sangeetha Banavathi Srinivasa 
> wrote:


Hi,

Below are the steps I executed in configuring MDS.

1. mkfs.lustre --fsname lustre 

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Sangeetha Banavathi Srinivasa]

Patrick,

I am following the steps on 
http://lustre.ornl.gov/lustre101-courses/content/C1/L3/LustreBasicInstall.pdf
Except that I am installing 2.8 instead of 2.7.
Could you recommend any other article/blog that I can follow in order to 
successfully install lustre.
 
> On Jun 22, 2016, at 3:43 PM, Patrick Farrell  wrote:
> 
> Hm.  It sounds like your RPMs are not installed correctly, but I vaguely 
> recall something about a packaging issue with the Lustre 2.8 release RPMs.
> 
> Perhaps someone else on the list can help further.
> - Patrick
> 
> On 06/22/2016 02:40 PM, Sangeetha Banavathi Srinivasa wrote:
>> I have done that already and re-booted my system.
>> But I still get the error.
>> 
>> When I did run that command, I got the below error:
>> WARNING: Can't read module 
>> /lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/weak-updates/kernel/fs/lustre/llog_test.ko:
>>  No such file or directory
>> egrep: 
>> /lib/modules/2.6.32-573.12.1.el6_lustre.x86_64//weak-updates/kernel/fs/lustre/llog_test.ko:
>>  No such file or directory
>> 
>> 
>>> On Jun 22, 2016, at 3:38 PM, Patrick Farrell >> > wrote:
>>> 
>>> Well, that's not patching the kernel.  It's making the initrd and and 
>>> updating the boot stuff so your system will boot that kernel.
>>> 
>>> But, yes, that should be what you need to do.
>>> 
>>> - Patrick
>>> On 06/22/2016 02:36 PM, Sangeetha Banavathi Srinivasa wrote:
 When you say install the associated kernel do you mean patching the kernel 
 like below
 sudo /sbin/new-kernel-pkg --package kernel --mkinitrd  --dracut --depmod 
 --install 2.6.32-573.12.1.el6_lustre.x86_64
 
 I have already done rpm -ivh kernel-2.6.32-573.12.1.el6_lustre.x86_64.rpm
 
 
> On Jun 22, 2016, at 2:51 PM, Patrick Farrell  > wrote:
> 
> The kernel version is found with uname -r or cat /proc/version.
> 
> That needs to match the kernel version in the Lustre RPMs.  (The version 
> string is the 2.6.32-573.12.1.el6_lustre) part.
> 
> In fact, it looks like you've got Lustre server RPMs, given the _lustre 
> in the name of the kernel.  That means you need to install the associated 
> kernel.
> 
> - Patrick
> 
> On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:
>> Patrick,
>> 
>> The CentOS version on the machines is 6.8
>> 
>>  [lustre@hulk2 ~]$ cat /etc/redhat-release 
>>  CentOS release 6.8 (Final)
>> 
>> 
>> The kernel version that I have installed is what I found along with the 
>> lustre RPMs
>> [lustre@amaranth3 ~]$ uname -r
>> 2.6.32-573.12.1.el6_lustre.x86_64
>> 
>> 
>> And I am trying to install lustre 2.8.
>> 
>> Should the CentOS release version be 2.7??
>> 
>> 
>> 
>>> On Jun 22, 2016, at 2:17 PM, Patrick Farrell >> > wrote:
>>> 
>>> Well, it says to check dmesg...  Though that will just list the 
>>> symbols.  In general, this usually means you've got a mismatch between 
>>> the version of the kernel (or OFED) Lustre is built against and your 
>>> version of either the kernel or OFED.  Given the number and locations 
>>> of the mismatches, I'm guessing it's the kernel.
>>> 
>>> So, compare your kernel version and your Lustre RPMs.
>>> 
>>> Regards,
>>> - Patrick
>>> On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:
 When I try modprobe lustre, I see the below errors:
 [05:44:03] root@hulk2 lustre # modprobe lustre
 WARNING: Error inserting obdclass 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko):
  Unknown symbol in module, or unknown parameter (see dmesg)
 WARNING: Error inserting ptlrpc 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko):
  Unknown symbol in module, or unknown parameter (see dmesg)
 WARNING: Error inserting fld 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko):
  Unknown symbol in module, or unknown parameter (see dmesg)
 WARNING: Error inserting lmv 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko):
  Unknown symbol in module, or unknown parameter (see dmesg)
 WARNING: Error inserting fid 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko):
  Unknown symbol in module, or unknown parameter (see dmesg)
 WARNING: Error inserting mdc 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko):
  Unknown symbol in module, or unknown parameter (see dmesg)
 WARNING: Error inserting lov 
 (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko):

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Patrick Farrell]

Well, that's not patching the kernel.  It's making the initrd and and 
updating the boot stuff so your system will boot that kernel.


But, yes, that should be what you need to do.

- Patrick
On 06/22/2016 02:36 PM, Sangeetha Banavathi Srinivasa wrote:
When you say install the associated kernel do you mean patching the 
kernel like below
sudo /sbin/new-kernel-pkg --package kernel --mkinitrd  --dracut 
--depmod --install 2.6.32-573.12.1.el6_lustre.x86_64


I have already done rpm -ivh kernel-2.6.32-573.12.1.el6_lustre.x86_64.rpm


On Jun 22, 2016, at 2:51 PM, Patrick Farrell > wrote:


The kernel version is found with uname -r or cat /proc/version.

That needs to match the kernel version in the Lustre RPMs.  (The 
version string is the 2.6.32-573.12.1.el6_lustre) part.


In fact, it looks like you've got Lustre server RPMs, given the 
_lustre in the name of the kernel.  That means you need to install 
the associated kernel.


- Patrick

On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:

Patrick,

The CentOS version on the machines is 6.8

[lustre@hulk2 ~]$ cat /etc/redhat-release
CentOS release 6.8 (Final)


The kernel version that I have installed is what I found along with 
the lustre RPMs

[lustre@amaranth3 ~]$ uname -r
2.6.32-573.12.1.el6_lustre.x86_64


And I am trying to install lustre 2.8.

Should the CentOS release version be 2.7??



On Jun 22, 2016, at 2:17 PM, Patrick Farrell > wrote:


Well, it says to check dmesg...  Though that will just list the 
symbols.  In general, this usually means you've got a mismatch 
between the version of the kernel (or OFED) Lustre is built against 
and your version of either the kernel or OFED.  Given the number 
and locations of the mismatches, I'm guessing it's the kernel.


So, compare your kernel version and your Lustre RPMs.

Regards,
- Patrick
On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:

When I try modprobe lustre, I see the below errors:
[*05:44:03*] *root@hulk2**lustre #*modprobe lustre
WARNING: Error inserting obdclass 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ptlrpc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fld 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lmv 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fid 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting mdc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lov 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting lustre 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)


On Jun 22, 2016, at 1:44 PM, Sangeetha Banavathi Srinivasa 
> wrote:


Hi,

Below are the steps I executed in configuring MDS.

1. mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt

2. mkdir /mdt

3. mount -t lustre /dev/vg00/mdt /mdt

I see the below error after step 3:

mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such 
device

Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
[root@amaranth3 lustre]#


I tried modprobe lustre and retried mounting but it resulted in 
the same error.  Any help will be appreciated.



For your info, the lustre modules that I have installed are:
[root@amaranth3 lustre]# rpm -qa |grep lustre
lustre-osd-ldiskfs-mount-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-tests-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-modules-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-iokit-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
kernel-2.6.32-573.12.1.el6_lustre.x86_64
lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64

[root@amaranth3 lustre]# rpm -qa |grep e2fs
e2fsprogs-libs-1.42.13.wc5-7.el6.x86_64
e2fsprogs-1.42.13.wc5-7.el6.x86_64

[root@amaranth3 lustre]# rpm -qa |grep libss
libss-1.42.13.wc5-7.el6.x86_64

root@amaranth3 lustre]# rpm -qa |grep libcom
libcom_err-1.42.13.wc5-7.el6.x86_64


[root@amaranth3 lustre]# uname -a
Linux amaranth3 2.6.32-573.12.1.el6_lustre.x86_64 #1 SMP Tue Feb 
23 22:41:09 PST 2016 x86_64 

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Patrick Farrell]

The kernel version is found with uname -r or cat /proc/version.

That needs to match the kernel version in the Lustre RPMs.  (The version 
string is the 2.6.32-573.12.1.el6_lustre) part.


In fact, it looks like you've got Lustre server RPMs, given the _lustre 
in the name of the kernel.  That means you need to install the 
associated kernel.


- Patrick

On 06/22/2016 01:43 PM, Sangeetha Banavathi Srinivasa wrote:

Patrick,

The CentOS version on the machines is 6.8

[lustre@hulk2 ~]$ cat /etc/redhat-release
CentOS release 6.8 (Final)


The kernel version that I have installed is what I found along with 
the lustre RPMs

[lustre@amaranth3 ~]$ uname -r
2.6.32-573.12.1.el6_lustre.x86_64


And I am trying to install lustre 2.8.

Should the CentOS release version be 2.7??



On Jun 22, 2016, at 2:17 PM, Patrick Farrell > wrote:


Well, it says to check dmesg...  Though that will just list the 
symbols.  In general, this usually means you've got a mismatch 
between the version of the kernel (or OFED) Lustre is built against 
and your version of either the kernel or OFED.  Given the number and 
locations of the mismatches, I'm guessing it's the kernel.


So, compare your kernel version and your Lustre RPMs.

Regards,
- Patrick
On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:

When I try modprobe lustre, I see the below errors:
[*05:44:03*] *root@hulk2**lustre #*modprobe lustre
WARNING: Error inserting obdclass 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ptlrpc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fld 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lmv 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fid 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting mdc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lov 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting lustre 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)


On Jun 22, 2016, at 1:44 PM, Sangeetha Banavathi Srinivasa 
> wrote:


Hi,

Below are the steps I executed in configuring MDS.

1. mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt

2. mkdir /mdt

3. mount -t lustre /dev/vg00/mdt /mdt

I see the below error after step 3:

mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such 
device

Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
[root@amaranth3 lustre]#


I tried modprobe lustre and retried mounting but it resulted in the 
same error.  Any help will be appreciated.



For your info, the lustre modules that I have installed are:
[root@amaranth3 lustre]# rpm -qa |grep lustre
lustre-osd-ldiskfs-mount-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-tests-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-modules-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-iokit-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
kernel-2.6.32-573.12.1.el6_lustre.x86_64
lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64

[root@amaranth3 lustre]# rpm -qa |grep e2fs
e2fsprogs-libs-1.42.13.wc5-7.el6.x86_64
e2fsprogs-1.42.13.wc5-7.el6.x86_64

[root@amaranth3 lustre]# rpm -qa |grep libss
libss-1.42.13.wc5-7.el6.x86_64

root@amaranth3 lustre]# rpm -qa |grep libcom
libcom_err-1.42.13.wc5-7.el6.x86_64


[root@amaranth3 lustre]# uname -a
Linux amaranth3 2.6.32-573.12.1.el6_lustre.x86_64 #1 SMP Tue Feb 23 
22:41:09 PST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@amaranth3 lustre]#



Thanks,
Sangeetha




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org 
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Sangeetha Banavathi Srinivasa]

Patrick,

The CentOS version on the machines is 6.8

[lustre@hulk2 ~]$ cat /etc/redhat-release 
CentOS release 6.8 (Final)


The kernel version that I have installed is what I found along with the lustre 
RPMs
[lustre@amaranth3 ~]$ uname -r
2.6.32-573.12.1.el6_lustre.x86_64


And I am trying to install lustre 2.8.

Should the CentOS release version be 2.7??



> On Jun 22, 2016, at 2:17 PM, Patrick Farrell  wrote:
> 
> Well, it says to check dmesg...  Though that will just list the symbols.  In 
> general, this usually means you've got a mismatch between the version of the 
> kernel (or OFED) Lustre is built against and your version of either the 
> kernel or OFED.  Given the number and locations of the mismatches, I'm 
> guessing it's the kernel.
> 
> So, compare your kernel version and your Lustre RPMs.
> 
> Regards,
> - Patrick
> On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:
>> When I try modprobe lustre, I see the below errors:
>> [05:44:03] root@hulk2 lustre # modprobe lustre
>> WARNING: Error inserting obdclass 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> WARNING: Error inserting ptlrpc 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> WARNING: Error inserting fld 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> WARNING: Error inserting lmv 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> WARNING: Error inserting fid 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> WARNING: Error inserting mdc 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> WARNING: Error inserting lov 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> FATAL: Error inserting lustre 
>> (/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko):
>>  Unknown symbol in module, or unknown parameter (see dmesg)
>> 
>>> On Jun 22, 2016, at 1:44 PM, Sangeetha Banavathi Srinivasa >> > wrote:
>>> 
>>> Hi,
>>> 
>>> Below are the steps I executed in configuring MDS.
>>> 
>>> 1. mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt
>>> 2. mkdir /mdt
>>> 
>>> 3. mount -t lustre /dev/vg00/mdt /mdt
>>> 
>>> I see the below error after step 3:
>>> 
>>> mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
>>> Are the lustre modules loaded?
>>> Check /etc/modprobe.conf and /proc/filesystems
>>> [root@amaranth3 lustre]# 
>>> 
>>> 
>>> I tried modprobe lustre and retried mounting but it resulted in the same 
>>> error.  Any help will be appreciated.
>>> 
>>> 
>>> For your info, the lustre modules that I have installed are:
>>> [root@amaranth3 lustre]# rpm -qa |grep lustre
>>> lustre-osd-ldiskfs-mount-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
>>> lustre-tests-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
>>> lustre-modules-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
>>> lustre-osd-ldiskfs-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
>>> lustre-iokit-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
>>> kernel-2.6.32-573.12.1.el6_lustre.x86_64
>>> lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
>>> 
>>> [root@amaranth3 lustre]# rpm -qa |grep e2fs
>>> e2fsprogs-libs-1.42.13.wc5-7.el6.x86_64
>>> e2fsprogs-1.42.13.wc5-7.el6.x86_64
>>> 
>>> [root@amaranth3 lustre]# rpm -qa |grep libss
>>> libss-1.42.13.wc5-7.el6.x86_64
>>> 
>>> root@amaranth3 lustre]# rpm -qa |grep libcom
>>> libcom_err-1.42.13.wc5-7.el6.x86_64
>>> 
>>> 
>>> [root@amaranth3 lustre]# uname -a
>>> Linux amaranth3 2.6.32-573.12.1.el6_lustre.x86_64 #1 SMP Tue Feb 23 
>>> 22:41:09 PST 2016 x86_64 x86_64 x86_64 GNU/Linux
>>> [root@amaranth3 lustre]# 
>>> 
>>> 
>>> 
>>> Thanks,
>>> Sangeetha 
>> 
>> 
>> 
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org 
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org 
>> 
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Patrick Farrell]

Well, it says to check dmesg...  Though that will just list the 
symbols.  In general, this usually means you've got a mismatch between 
the version of the kernel (or OFED) Lustre is built against and your 
version of either the kernel or OFED.  Given the number and locations of 
the mismatches, I'm guessing it's the kernel.


So, compare your kernel version and your Lustre RPMs.

Regards,
- Patrick
On 06/22/2016 12:47 PM, Sangeetha Banavathi Srinivasa wrote:

When I try modprobe lustre, I see the below errors:
[*05:44:03*] *root@hulk2**lustre #*modprobe lustre
WARNING: Error inserting obdclass 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ptlrpc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fld 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lmv 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fid 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting mdc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lov 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting lustre 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)


On Jun 22, 2016, at 1:44 PM, Sangeetha Banavathi Srinivasa 
> wrote:


Hi,

Below are the steps I executed in configuring MDS.

1. mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt

2. mkdir /mdt

3. mount -t lustre /dev/vg00/mdt /mdt

I see the below error after step 3:

mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
[root@amaranth3 lustre]#


I tried modprobe lustre and retried mounting but it resulted in the 
same error.  Any help will be appreciated.



For your info, the lustre modules that I have installed are:
[root@amaranth3 lustre]# rpm -qa |grep lustre
lustre-osd-ldiskfs-mount-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-tests-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-modules-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-iokit-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
kernel-2.6.32-573.12.1.el6_lustre.x86_64
lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64

[root@amaranth3 lustre]# rpm -qa |grep e2fs
e2fsprogs-libs-1.42.13.wc5-7.el6.x86_64
e2fsprogs-1.42.13.wc5-7.el6.x86_64

[root@amaranth3 lustre]# rpm -qa |grep libss
libss-1.42.13.wc5-7.el6.x86_64

root@amaranth3 lustre]# rpm -qa |grep libcom
libcom_err-1.42.13.wc5-7.el6.x86_64


[root@amaranth3 lustre]# uname -a
Linux amaranth3 2.6.32-573.12.1.el6_lustre.x86_64 #1 SMP Tue Feb 23 
22:41:09 PST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@amaranth3 lustre]#



Thanks,
Sangeetha




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Sangeetha Banavathi Srinivasa]

When I try modprobe lustre, I see the below errors:
[05:44:03] root@hulk2 lustre # modprobe lustre
WARNING: Error inserting obdclass 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko):
 Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ptlrpc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko):
 Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fld 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lmv 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lmv.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting fid 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting mdc 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting lov 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting lustre 
(/lib/modules/2.6.32-573.12.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko):
 Unknown symbol in module, or unknown parameter (see dmesg)

> On Jun 22, 2016, at 1:44 PM, Sangeetha Banavathi Srinivasa  
> wrote:
> 
> Hi,
> 
> Below are the steps I executed in configuring MDS.
> 
> 1. mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt
> 2. mkdir /mdt
> 
> 3. mount -t lustre /dev/vg00/mdt /mdt
> 
> I see the below error after step 3:
> 
> mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
> Are the lustre modules loaded?
> Check /etc/modprobe.conf and /proc/filesystems
> [root@amaranth3 lustre]# 
> 
> 
> I tried modprobe lustre and retried mounting but it resulted in the same 
> error.  Any help will be appreciated.
> 
> 
> For your info, the lustre modules that I have installed are:
> [root@amaranth3 lustre]# rpm -qa |grep lustre
> lustre-osd-ldiskfs-mount-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
> lustre-tests-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
> lustre-modules-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
> lustre-osd-ldiskfs-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
> lustre-iokit-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
> kernel-2.6.32-573.12.1.el6_lustre.x86_64
> lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
> 
> [root@amaranth3 lustre]# rpm -qa |grep e2fs
> e2fsprogs-libs-1.42.13.wc5-7.el6.x86_64
> e2fsprogs-1.42.13.wc5-7.el6.x86_64
> 
> [root@amaranth3 lustre]# rpm -qa |grep libss
> libss-1.42.13.wc5-7.el6.x86_64
> 
> root@amaranth3 lustre]# rpm -qa |grep libcom
> libcom_err-1.42.13.wc5-7.el6.x86_64
> 
> 
> [root@amaranth3 lustre]# uname -a
> Linux amaranth3 2.6.32-573.12.1.el6_lustre.x86_64 #1 SMP Tue Feb 23 22:41:09 
> PST 2016 x86_64 x86_64 x86_64 GNU/Linux
> [root@amaranth3 lustre]# 
> 
> 
> 
> Thanks,
> Sangeetha 

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS Config Help: Are the lustre modules loaded?

[Sangeetha Banavathi Srinivasa]

Hi,

Below are the steps I executed in configuring MDS.

1. mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt
2. mkdir /mdt

3. mount -t lustre /dev/vg00/mdt /mdt

I see the below error after step 3:

mount.lustre: mount /dev/mapper/vg00-mdt1 at /mdt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
[root@amaranth3 lustre]# 


I tried modprobe lustre and retried mounting but it resulted in the same error. 
 Any help will be appreciated.


For your info, the lustre modules that I have installed are:
[root@amaranth3 lustre]# rpm -qa |grep lustre
lustre-osd-ldiskfs-mount-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-tests-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-modules-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
lustre-iokit-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64
kernel-2.6.32-573.12.1.el6_lustre.x86_64
lustre-2.8.0-2.6.32_573.12.1.el6_lustre.x86_64.x86_64

[root@amaranth3 lustre]# rpm -qa |grep e2fs
e2fsprogs-libs-1.42.13.wc5-7.el6.x86_64
e2fsprogs-1.42.13.wc5-7.el6.x86_64

[root@amaranth3 lustre]# rpm -qa |grep libss
libss-1.42.13.wc5-7.el6.x86_64

root@amaranth3 lustre]# rpm -qa |grep libcom
libcom_err-1.42.13.wc5-7.el6.x86_64


[root@amaranth3 lustre]# uname -a
Linux amaranth3 2.6.32-573.12.1.el6_lustre.x86_64 #1 SMP Tue Feb 23 22:41:09 
PST 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@amaranth3 lustre]# 



Thanks,
Sangeetha ___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Apr 13, 2016, at 2:53 PM, Mark Hahn  wrote:
> thanks, we'll be trying the LU-5726 patch and cpu_npartitions things.
> it's quite a long thread - do I understand correctly that periodic
> vm.drop_caches=1 can postpone the issue?

Not really.  I was periodically dropping the caches as a way to monitor how 
fast the memory was leaking.  I was trying to distinguish between normal cache 
usage (which allows memory to be reclaimed) and some other non-reclaimable 
usage.  The rate at which the memory leak grows depends upon how many file 
unlinks are happening (and based on my testing, sometimes the pattern in which 
they are unlinked).  But in general, the memory usage will just continue to 
gradually grow.

> It seems odd that if this is purely a memory balance problem,
> it manifests as a 0xdeadbeef panic, rather than OOM.  While I understand
> that the oom-killer path itself need memory to operate, does this also imply 
> that some allocation in the kernel or filesystem is not checking a return 
> value?

Not sure about that one, but in my experience when nodes start to run out of 
memory, they can fail in all sorts of new and exciting ways.  IIRC, SDSC also 
had an issue with LU-5726 but the symptoms they saw were not identical to mine. 
 So maybe you are seeing the same problem manifest itself in a different way.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mark Hahn]

We had to use lustre-2.5.3.90 on the MDS servers because of memory leak.

https://jira.hpdd.intel.com/browse/LU-5726


Mark,

If you don?t have the patch for LU-5726, then you should definitely try to get that 
one.  If nothing else, reading through the bug report might be useful.  It details 
some of the MDS OOM problems I had and mentions setting vm.zone_reclaim_mode=0.  It 
also has Robin Humble?s suggestion of setting "options libcfs 
cpu_npartitions=1? (which is something that I started doing as well).


thanks, we'll be trying the LU-5726 patch and cpu_npartitions things.
it's quite a long thread - do I understand correctly that periodic
vm.drop_caches=1 can postpone the issue?  I can replicate the warning
signs mentioned in the thread (growth of Inactive(file) to dizzying 
heights when doing a lot of unlinks).


It seems odd that if this is purely a memory balance problem,
it manifests as a 0xdeadbeef panic, rather than OOM.  While I understand
that the oom-killer path itself need memory to operate, does this 
also imply that some allocation in the kernel or filesystem is not 
checking a return value?


thanks, mark hahn.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Apr 13, 2016, at 8:02 AM, Tommi T  wrote:
> 
> We had to use lustre-2.5.3.90 on the MDS servers because of memory leak.
> 
> https://jira.hpdd.intel.com/browse/LU-5726

Mark,

If you don’t have the patch for LU-5726, then you should definitely try to get 
that one.  If nothing else, reading through the bug report might be useful.  It 
details some of the MDS OOM problems I had and mentions setting 
vm.zone_reclaim_mode=0.  It also has Robin Humble’s suggestion of setting 
"options libcfs cpu_npartitions=1” (which is something that I started doing as 
well).

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Apr 12, 2016, at 6:46 PM, Mark Hahn  wrote:
> 
> all our existing Lustre MDSes run happily with vm.zone_reclaim_mode=0,
> and making this one consistent appears to have resolved a problem
> (in which one family of lustre kernel threads would appear to spin,
> "perf top" showing nearly all time spent in spinlock_irq.  iirc.)
> 
> might your system have had a *lot* of memory?  ours tend to be fairly modest 
> (32-64G, dual-socket intel.)

I have 64 GB on my servers.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Robin Humble]

Hi Mark,

On Tue, Apr 12, 2016 at 04:49:10PM -0400, Mark Hahn wrote:
>One of our MDSs is crashing with the following:
>
>BUG: unable to handle kernel paging request at deadbeef
>IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
>PGD 0
>Oops: 0002 [#1] SMP
>
>The MDS is running 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
>with about 2k clients ranging from 1.8.8 to 2.6.0

I saw an identical crash in Sep 2014 when the MDS was put under memory
pressure.

>to be related to vm.zone_reclaim_mode=1.  We also enabled quotas

zone_reclaim_mode should always be 0. 1 is broken. hung processes
perpetually 'scanning' in one zone in /proc/zoneinfo whilst plenty of
pages are free in another zone is a sure sign of this issue.

however if you have vm.zone_reclaim_mode=0 now and are still seeing the
issue, then I would suspect that lustre's overly agresssive memory
affinity code is partially to blame. at the very least it is most
likely stopping you from making use of half your MDS ram.

see
  https://jira.hpdd.intel.com/browse/LU-5050

set
  options libcfs cpu_npartitions=1
to fix it. that's what I use on OSS and MDS for all my clusters.

cheers,
robin
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Tommi T]

Hi,

We had to use lustre-2.5.3.90 on the MDS servers because of memory leak.

https://jira.hpdd.intel.com/browse/LU-5726

I'm not sure if it's related but worthwhile to check.

BR,
Tommi



- Original Message -
From: Mark Hahn <h...@mcmaster.ca>
To: lustre-discuss@lists.lustre.org
Sent: Tuesday, April 12, 2016 11:49 PM
Subject: [lustre-discuss] MDS crashing: unable to handle kernel paging request 
at deadbeef (iam_container_init+0x18/0x70)

One of our MDSs is crashing with the following:

BUG: unable to handle kernel paging request at deadbeef
IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
PGD 0
Oops: 0002 [#1] SMP

The MDS is running 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
with about 2k clients ranging from 1.8.8 to 2.6.0

I'd appreciate any comments on where to point fingers: google doesn't
provide anything suggestive about iam_container_init.

Our problem seems to correlate with an unintentional creation of a tree 
of >500M files.  Some of the crashes we've had since then appeared
to be related to vm.zone_reclaim_mode=1.  We also enabled quotas 
right after the 500M file thing, and were thinking that inconsistent
quota records might cause this sort of crash.

But 0xdeadbeef is usually added as a canary for allocation issues;
is it used this way in Lustre?

thanks,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
   | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
   | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Patrick Farrell]

Ew.  Well, that doesn't help. :)

Can you configure kdump on the node?  That would get you both dmesg and a dump. 
 Dmesg would include the rest of the stack trace.  (I'm hoping to give you a 
better idea whether or not the quota code is involved.)  A dump would let a 
developer type dig deeper as well, though it would also contain private info 
from your server.

- Patrick

From: Mark Hahn [h...@mcmaster.ca]
Sent: Tuesday, April 12, 2016 4:39 PM
To: Patrick Farrell
Cc: lustre-discuss@lists.lustre.org
Subject: RE: [lustre-discuss] MDS crashing: unable to handle kernel paging 
request at deadbeef (iam_container_init+0x18/0x70)

> Giving the rest of the back trace of the crash would help for developers
>looking at it.
>
> It's a lot easier to tell what code is involved with the whole trace.

thanks.  I'm sure that's the case, but these oopsen are truncated.
well, one was slightly longer:

BUG: unable to handle kernel paging request at deadbeef
IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
PGD 0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 14
Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) 
fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) 
mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) 
sha512_generic sha256_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd 
fscache auth_rpcgss nfs_acl sunrpc mlx4_en ipt_REJECT nf_conntrack_ipv4 
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 iTCO_wdt 
iTCO_vendor_support serio_raw raid10 i2c_i801 lpc_ich mfd_core ipmi_devintf 
mlx4_core sg acpi_pad igb dca i2c_algo_bit i2c_core ptp pps_core shpchp ext4 
jbd2 mbcache raid1 sr_mod cdrom sd_mod crc_t10dif isci libsas mpt2sas 
scsi_transport_sas raid_class ahci wmi dm_mirror dm_region_hash dm_log dm_mod 
[last unloaded: scsi_wait_scan]
Pid: 7768, comm: mdt00_039 Not tainted 2.6.32-431.23.3.el6_lustre.x86_64 #1 
Supermicro SYS-2027R-WRF/X9DRW

by way of straw-grasping, I'll mention two other very frequent messages
we're seeing on the MDS in question:

Lustre: 17673:0:(mdt_xattr.c:465:mdt_reint_setxattr()) covework-MDT: client 
miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object 
[0x200031f84:0x1cad0:0x0]] [valid 68719476736]

(which seems to be https://jira.hpdd.intel.com/browse/LU-532 and a
consequence of some of our very old clients.  but not MDS-crash-able.)

LustreError: 22970:0:(tgt_lastrcvd.c:813:tgt_last_rcvd_update()) 
covework-MDT: trying to overwrite bigger transno:on-disk: 197587694105, 
new: 197587694104 replay: 0. see LU-617.

perplexing because the MDS is 2.5.3 and
https://jira.hpdd.intel.com/browse/LU-617 shows fixed circa 2.2.0/2.1.2.
(and our problem isn't with recovery afaikt.)

thanks!

regards,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
   | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
   | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mark Hahn]

Our problem seems to correlate with an unintentional creation of a tree of 
>500M files.  Some of the crashes we've had since then appeared
to be related to vm.zone_reclaim_mode=1.  We also enabled quotas right after 
the 500M file thing, and were thinking that inconsistent
quota records might cause this sort of crash.


Have you set vm.zone_reclaim_mode=0 yet?  I had an issue with this on my
file system a while back when it was set to 1.


all our existing Lustre MDSes run happily with vm.zone_reclaim_mode=0,
and making this one consistent appears to have resolved a problem
(in which one family of lustre kernel threads would appear to spin,
"perf top" showing nearly all time spent in spinlock_irq.  iirc.)

might your system have had a *lot* of memory?  ours tend to be 
fairly modest (32-64G, dual-socket intel.)


thanks,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
  | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
  | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Apr 12, 2016, at 4:49 PM, Mark Hahn  wrote:
> 
> Our problem seems to correlate with an unintentional creation of a tree of 
> >500M files.  Some of the crashes we've had since then appeared
> to be related to vm.zone_reclaim_mode=1.  We also enabled quotas right after 
> the 500M file thing, and were thinking that inconsistent
> quota records might cause this sort of crash.

Have you set vm.zone_reclaim_mode=0 yet?  I had an issue with this on my file 
system a while back when it was set to 1.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mark Hahn]

Giving the rest of the back trace of the crash would help for developers
looking at it. 


It's a lot easier to tell what code is involved with the whole trace.


thanks.  I'm sure that's the case, but these oopsen are truncated.
well, one was slightly longer:

BUG: unable to handle kernel paging request at deadbeef
IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
PGD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 14 
Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_en ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 iTCO_wdt iTCO_vendor_support serio_raw raid10 i2c_i801 lpc_ich mfd_core ipmi_devintf mlx4_core sg acpi_pad igb dca i2c_algo_bit i2c_core ptp pps_core shpchp ext4 jbd2 mbcache raid1 sr_mod cdrom sd_mod crc_t10dif isci libsas mpt2sas scsi_transport_sas raid_class ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 7768, comm: mdt00_039 Not tainted 2.6.32-431.23.3.el6_lustre.x86_64 #1 
Supermicro SYS-2027R-WRF/X9DRW

by way of straw-grasping, I'll mention two other very frequent messages
we're seeing on the MDS in question:

Lustre: 17673:0:(mdt_xattr.c:465:mdt_reint_setxattr()) covework-MDT: client 
miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object 
[0x200031f84:0x1cad0:0x0]] [valid 68719476736]

(which seems to be https://jira.hpdd.intel.com/browse/LU-532 and a 
consequence of some of our very old clients.  but not MDS-crash-able.)


LustreError: 22970:0:(tgt_lastrcvd.c:813:tgt_last_rcvd_update()) 
covework-MDT: trying to overwrite bigger transno:on-disk: 197587694105, 
new: 197587694104 replay: 0. see LU-617.

perplexing because the MDS is 2.5.3 and 
https://jira.hpdd.intel.com/browse/LU-617 shows fixed circa 2.2.0/2.1.2.

(and our problem isn't with recovery afaikt.)

thanks!

regards,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
  | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
  | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Patrick Farrell]

Mark,

Giving the rest of the back trace of the crash would help for developers 
looking at it.

It's a lot easier to tell what code is involved with the whole trace.

Thanks,
- Patrick

From: lustre-discuss [lustre-discuss-boun...@lists.lustre.org] on behalf of 
Mark Hahn [h...@mcmaster.ca]
Sent: Tuesday, April 12, 2016 3:49 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] MDS crashing: unable to handle kernel paging request 
at deadbeef (iam_container_init+0x18/0x70)

One of our MDSs is crashing with the following:

BUG: unable to handle kernel paging request at deadbeef
IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
PGD 0
Oops: 0002 [#1] SMP

The MDS is running 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
with about 2k clients ranging from 1.8.8 to 2.6.0

I'd appreciate any comments on where to point fingers: google doesn't
provide anything suggestive about iam_container_init.

Our problem seems to correlate with an unintentional creation of a tree
of >500M files.  Some of the crashes we've had since then appeared
to be related to vm.zone_reclaim_mode=1.  We also enabled quotas
right after the 500M file thing, and were thinking that inconsistent
quota records might cause this sort of crash.

But 0xdeadbeef is usually added as a canary for allocation issues;
is it used this way in Lustre?

thanks,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
   | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
   | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

[Mark Hahn]

One of our MDSs is crashing with the following:

BUG: unable to handle kernel paging request at deadbeef
IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
PGD 0
Oops: 0002 [#1] SMP

The MDS is running 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
with about 2k clients ranging from 1.8.8 to 2.6.0

I'd appreciate any comments on where to point fingers: google doesn't
provide anything suggestive about iam_container_init.

Our problem seems to correlate with an unintentional creation of a tree 
of >500M files.  Some of the crashes we've had since then appeared
to be related to vm.zone_reclaim_mode=1.  We also enabled quotas 
right after the 500M file thing, and were thinking that inconsistent

quota records might cause this sort of crash.

But 0xdeadbeef is usually added as a canary for allocation issues;
is it used this way in Lustre?

thanks,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
  | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
  | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] MDS kernel panics - 2.5.3 - waking for gap in transno

[Steve Barnet]

Hi all,

  Earlier today, one of our MDSes kernel panicked. I was not able
to grab the stack trace, but it appeared to be in the locking code
path.

The last entries prior to the crash on our remote syslog server
looked like this:

Sep 14 10:17:19 lfs-us-mds kernel: LustreError: 
6864:0:(mdt_handler.c:1444:mdt_getattr_name_lock()) ASSERTION( lock != 
NULL ) failed: Invalid lock handle 0x55755bac33ca291d
Sep 14 10:17:19 lfs-us-mds kernel: LustreError: 
6864:0:(mdt_handler.c:1444:mdt_getattr_name_lock()) LBUG



After a reboot and e2fsck of the MDT, we saw many errors that look
like this:


Sep 14 12:40:35 lfs-us-mds kernel: LustreError: 
3307:0:(ldlm_lib.c:1751:check_for_next_transno()) lfs3-MDT: waking 
for gap in transno, VBR is OFF (skip: 66055727404, ql: 15, comp: 172, 
conn: 187, next: 66055727409, last_committed: 66055720625)



The output from the e2fsck is attached, but the quick summary is
that it was entirely QUOTA WARNINGs. Shortly after recovery
completed, the machine panicked again (trace attached).

After a reboot, it appears that the system has stabilized, but
as you might expect, it does not leave me with a warm, fuzzy
feeling.

Some of my searching indicates that an lfsck may be needed, but
before I start too far down that path, I'd like to have some idea
that this is indeed reasonable.

We are using 2.5.3 on all servers and clients:

lfs-us-mds ~ # cat /proc/fs/lustre/version
lustre: 2.5.3
kernel: patchless_client
build:  2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64


OS is Scientific Linux 6.4:

Linux lfs-us-mds 2.6.32-431.23.3.el6_lustre.x86_64 #1 SMP Thu Aug 28 
20:20:13 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux



Any advice?

Thanks much, in advance!

Best,

---Steve

--
Steve Barnet
UW-Madison - IceCube
bar...@icecube.wisc.edu


lfs-us-mds ~ # e2fsck -fp /dev/sdb
lfs3-MDT: recovering journal
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15023759360, 226810) != 
expected (19607052288, 245133)
[QUOTA WARNING] Usage inconsistent for ID 10210:actual (78381056, 230125) != 
expected (75276288, 192681)
[QUOTA WARNING] Usage inconsistent for ID 33480:actual (449388544, 724085) != 
expected (449351680, 724040)
[QUOTA WARNING] Usage inconsistent for ID 50330:actual (458280960, 312174) != 
expected (458280960, 312173)
[QUOTA WARNING] Usage inconsistent for ID 33416:actual (4321316864, 3795704) != 
expected (4044886016, 3506498)
[QUOTA WARNING] Usage inconsistent for ID 17437:actual (339566592, 487621) != 
expected (1359523840, 8241900)
[QUOTA WARNING] Usage inconsistent for ID 15329:actual (318918656, 657583) != 
expected (269213696, 496130)
[QUOTA WARNING] Usage inconsistent for ID 37701:actual (1103691776, 1146625) != 
expected (971554816, 969972)
[QUOTA WARNING] Usage inconsistent for ID 23129:actual (579133440, 4448732) != 
expected (579379200, 4449891)
[QUOTA WARNING] Usage inconsistent for ID 31251:actual (17772544, 213487) != 
expected (17752064, 215409)
[QUOTA WARNING] Usage inconsistent for ID 50225:actual (189779968, 216721) != 
expected (189005824, 234420)
[QUOTA WARNING] Usage inconsistent for ID 41003:actual (550715392, 3005115) != 
expected (323002368, 1068804)
[QUOTA WARNING] Usage inconsistent for ID 51509:actual (739606528, 575118) != 
expected (739540992, 574880)
[QUOTA WARNING] Usage inconsistent for ID 41279:actual (76873728, 192292) != 
expected (20619264, 50497)
[QUOTA WARNING] Usage inconsistent for ID 1823:actual (135487488, 175342) != 
expected (134180864, 167683)
[QUOTA WARNING] Usage inconsistent for ID 46376:actual (25743360, 35712) != 
expected (25350144, 31629)
[QUOTA WARNING] Usage inconsistent for ID 501:actual (9920512, 37723) != 
expected (9920512, 37713)
[QUOTA WARNING] Usage inconsistent for ID 37735:actual (69378048, 897135) != 
expected (65445888, 864621)
[QUOTA WARNING] Usage inconsistent for ID 6204:actual (27136000, 29948) != 
expected (27136000, 29924)
[QUOTA WARNING] Usage inconsistent for ID 13092:actual (4096, 20178) != 
expected (4096, 15903)
[QUOTA WARNING] Usage inconsistent for ID 17657:actual (244539392, 234381) != 
expected (244539392, 234368)
[QUOTA WARNING] Usage inconsistent for ID 5427:actual (27328512, 214542) != 
expected (27324416, 214507)
[QUOTA WARNING] Usage inconsistent for ID 47558:actual (303357952, 713320) != 
expected (303357952, 714298)
[QUOTA WARNING] Usage inconsistent for ID 27572:actual (218345472, 996620) != 
expected (218345472, 996619)
[QUOTA WARNING] Usage inconsistent for ID 46314:actual (457404416, 3944252) != 
expected (0, 8709)
[QUOTA WARNING] Usage inconsistent for ID 26165:actual (22102016, 189712) != 
expected (22102016, 206743)
[QUOTA WARNING] Usage inconsistent for ID 40893:actual (1303056384, 3147456) != 
expected (1210523648, 3677756)
[QUOTA WARNING] Usage inconsistent for ID 26992:actual (1749823488, 5134252) != 
expected (1749684224, 5133980)
[QUOTA WARNING] Usage inconsistent for ID 5408:actual (151408640, 204432) != 
expected (151339008, 203525)
[QUOTA WARNING] Usage inconsistent for 

[Lustre-discuss] MDS kernel panic

[Pardo Diaz, Alfonso]

Hello, 

Since I update my lustre 2.2 to 2.5.1 (Centos6.5) and copy the MDT to a new SSD 
disk. I get random kernel panics in the MDS (both HA pairs). The last kernel 
panic I get this log:

4Lustre: MGS: non-config logname received: params
3LustreError: 11-0: cetafs-MDT-lwp-MDT: Communicating with 0@lo, 
operation mds_connect failed with -11.
4Lustre: MGS: non-config logname received: params
4Lustre: cetafs-MDT: Will be in recovery for at least 5:00, or until 102 
clients reconnect
4Lustre: MGS: non-config logname received: params
4Lustre: MGS: non-config logname received: params
4Lustre: Skipped 5 previous similar messages
4Lustre: MGS: non-config logname received: params
4Lustre: Skipped 9 previous similar messages
4Lustre: MGS: non-config logname received: params
4Lustre: Skipped 2 previous similar messages
4Lustre: MGS: non-config logname received: params
4Lustre: Skipped 23 previous similar messages
4Lustre: MGS: non-config logname received: params
4Lustre: Skipped 8 previous similar messages
3LustreError: 3461:0:(ldlm_lib.c:1751:check_for_next_transno()) 
cetafs-MDT: waking for gap in transno, VBR is OFF (skip: 17188113481, ql: 
1, comp: 101, conn: 102, next: 17188113493, last_committed: 17188113480)
6Lustre: cetafs-MDT: Recovery over after 1:13, of 102 clients 102 
recovered and 0 were evicted.
1BUG: unable to handle kernel NULL pointer dereference at (null)
1IP: [a0c3b6a0] __iam_path_lookup+0x70/0x1f0 [osd_ldiskfs]
4PGD 106c0bf067 PUD 106c0be067 PMD 0 
4Oops: 0002 [#1] SMP 
4last sysfs file: /sys/devices/system/cpu/online
4CPU 0 
4Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) 
fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) 
mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) 
lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) ipmi_devintf 
cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm 
ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 dm_multipath microcode 
iTCO_wdt iTCO_vendor_support sb_edac edac_core lpc_ich mfd_core i2c_i801 igb 
i2c_algo_bit i2c_core ptp pps_core ioatdma dca mlx4_ib ib_sa ib_mad ib_core 
mlx4_en mlx4_core sg ext4 jbd2 mbcache sd_mod crc_t10dif ahci isci libsas 
mpt2sas scsi_transport_sas raid_class megaraid_sas dm_mirror dm_region_hash 
dm_log dm_mod [last unloaded: scsi_wait_scan]
4
4Pid: 3362, comm: mdt00_001 Not tainted 2.6.32-431.5.1.el6_lustre.x86_64 #1 
Bull SAS bullx/X9DRH-7TF/7F/iTF/iF
4RIP: 0010:[a0c3b6a0]  [a0c3b6a0] 
__iam_path_lookup+0x70/0x1f0 [osd_ldiskfs]
4RSP: 0018:88085e2754b0  EFLAGS: 00010246
4RAX: fffb RBX: 88085e275600 RCX: 0009c93c
4RDX:  RSI: 0009c93b RDI: 88106bcc32f0
4RBP: 88085e275500 R08:  R09: 
4R10:  R11:  R12: 88085e2755c8
4R13: 5250 R14: 8810569bf308 R15: 0001
4FS:  () GS:88002820() knlGS:
4CS:  0010 DS: 0018 ES: 0018 CR0: 8005003b
4CR2:  CR3: 00106dd9b000 CR4: 000407f0
4DR0:  DR1:  DR2: 
4DR3:  DR6: 0ff0 DR7: 0400
4Process mdt00_001 (pid: 3362, threadinfo 88085e274000, task 
88085f55c080)
4Stack:
4  88085e2755d8 8810569bf288 a00fd2c4
4d 88085e275660 88085e2755c8 88085e2756c8 
4d  88085db2a480 88085e275530 a0c3ba6c
4Call Trace:
4 [a00fd2c4] ? do_get_write_access+0x3b4/0x520 [jbd2]
4 [a0c3ba6c] iam_lookup_lock+0x7c/0xb0 [osd_ldiskfs]
4 [a0c3bad4] __iam_it_get+0x34/0x160 [osd_ldiskfs]
4 [a0c3be1e] iam_it_get+0x2e/0x150 [osd_ldiskfs]
4 [a0c3bf4e] iam_it_get_exact+0xe/0x30 [osd_ldiskfs]
4 [a0c3d47f] iam_insert+0x4f/0xb0 [osd_ldiskfs]
4 [a0c366ea] osd_oi_iam_refresh+0x18a/0x330 [osd_ldiskfs]
4 [a0c3ea40] ? iam_lfix_ipd_alloc+0x0/0x20 [osd_ldiskfs]
4 [a0c386dd] osd_oi_insert+0x11d/0x480 [osd_ldiskfs]
4 [811ae522] ? generic_setxattr+0xa2/0xb0
4 [a0c25021] ? osd_ea_fid_set+0xf1/0x410 [osd_ldiskfs]
4 [a0c33595] osd_object_ea_create+0x5b5/0x700 [osd_ldiskfs]
4 [a0e173bf] lod_object_create+0x13f/0x260 [lod]
4 [a0e756c0] mdd_object_create_internal+0xa0/0x1c0 [mdd]
4 [a0e86428] mdd_create+0xa38/0x1730 [mdd]
4 [a0c2af37] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs]
4 [a0e14770] ? lod_index_lookup+0x0/0x30 [lod]
4 [a0d50358] mdo_create+0x18/0x50 [mdt]
4 [a0d5a64c] mdt_reint_open+0x13ac/0x21a0 [mdt]
4 [a065983c] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc]
4 [a04f4600] ? lu_ucred_key_init+0x160/0x1a0 [obdclass]
4 [a0d431f1] mdt_reint_rec+0x41/0xe0 [mdt]
4 [a0d2add3] mdt_reint_internal+0x4c3/0x780 [mdt]
4 

Re: [Lustre-discuss] MDS files attribute

[Alfonso Pardo]

I found it!!!

http://wiki.lustre.org/manual/LustreManual20_HTML/LustreMonitoring.html



From: Alfonso Pardo 
Sent: Thursday, November 29, 2012 8:39 AM
To: lustre-discuss@lists.lustre.org 
Cc: wc-disc...@whamcloud.com 
Subject: [Lustre-discuss] MDS files attribute

Hello,

I need to monitoring a directory in the lustre file system. I want to change 
the group id for each new file written in the directory. 

Is it possible to ask to the MDS the file attributes to obtain more speed in my 
chown script? One more thing, is it possible to ask to the MDS the last wrote 
files? And then apply the new group id only to the new files. 



Thanks in advance
 Confidencialidad: Este mensaje y sus ficheros 
adjuntos se dirige exclusivamente a su destinatario y puede contener 
información privilegiada o confidencial. Si no es vd. el destinatario indicado, 
queda notificado de que la utilización, divulgación y/o copia sin autorización 
está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje 
por error, le rogamos que nos lo comunique inmediatamente respondiendo al 
mensaje y proceda a su destrucción. Disclaimer: This message and its attached 
files is intended exclusively for its recipients and may contain confidential 
information. If you received this e-mail in error you are hereby notified that 
any dissemination, copy or disclosure of this communication is strictly 
prohibited and may be unlawful. In this case, please notify us by a reply and 
delete this email and its contents immediately.  



___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS files attribute

[Dilger, Andreas]

On Nov 29, 2012, at 12:39 AM, Alfonso Pardo wrote:
 I need to monitoring a directory in the lustre file system.

We don't have any inotify or dnotify support for Lustre.  In Lustre 2.x it is 
possible to monitor the whole filesystem with the Lustre ChangeLog 
functionality.  It might be an interesting project for someone to wire up the 
ChangeLog to the kernel inotify/dnotify, though that would obviously take some 
significant effort.

 I want to change the group id for each new file written in the directory.

You could just use the BSD-style directory group ownership to force all new 
files created there to have a specific group ID?

chgrp {group} /path/to/dir
chmod g+s /path/to/dir

Now all new files will have the group {group}.  This is not Lustre-specific, 
it works for other filesystems as well.

 Is it possible to ask to the MDS the file attributes to obtain more speed in 
 my chown script?

The lfs find code does fetch attributes from just the MDS.  There was a 
proposal for a new Linux syscall to be able to get specific attributes of a 
file instead of all of them:

http://lists.openwall.net/linux-ext4/2010/07/01/11

but it never quite made it into the kernel.

 One more thing, is it possible to ask to the MDS the last wrote files? And 
 then apply the new group id only to the new files.
  
  
  
 Thanks in advance
  Confidencialidad: Este mensaje y sus ficheros 
 adjuntos se dirige exclusivamente a su destinatario y puede contener 
 información privilegiada o confidencial. Si no es vd. el destinatario 
 indicado, queda notificado de que la utilización, divulgación y/o copia sin 
 autorización está prohibida en virtud de la legislación vigente. Si ha 
 recibido este mensaje por error, le rogamos que nos lo comunique 
 inmediatamente respondiendo al mensaje y proceda a su destrucción. 
 Disclaimer: This message and its attached files is intended exclusively for 
 its recipients and may contain confidential information. If you received this 
 e-mail in error you are hereby notified that any dissemination, copy or 
 disclosure of this communication is strictly prohibited and may be unlawful. 
 In this case, please notify us by a reply and delete this email and its 
 contents immediately. 
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel Corporation






___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS files attribute

[Alfonso Pardo]

Hello,

I need to monitoring a directory in the lustre file system. I want to change 
the group id for each new file written in the directory. 

Is it possible to ask to the MDS the file attributes to obtain more speed in my 
chown script? One more thing, is it possible to ask to the MDS the last wrote 
files? And then apply the new group id only to the new files. 



Thanks in advance

Confidencialidad: 
Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario 
y puede contener información privilegiada o confidencial. Si no es vd. el 
destinatario indicado, queda notificado de que la utilización, divulgación y/o 
copia sin autorización está prohibida en virtud de la legislación vigente. Si 
ha recibido este mensaje por error, le rogamos que nos lo comunique 
inmediatamente respondiendo al mensaje y proceda a su destrucción.

Disclaimer: 
This message and its attached files is intended exclusively for its recipients 
and may contain confidential information. If you received this e-mail in error 
you are hereby notified that any dissemination, copy or disclosure of this 
communication is strictly prohibited and may be unlawful. In this case, please 
notify us by a reply and delete this email and its contents immediately. 


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] mds-survey on Lustre-1.8

[Andreas Dilger]

On 2012-11-13, at 4:59 PM, Christopher J. Walker wrote:On 13/11/12 18:49, Richard Henwood wrote:On Tue, Nov 13, 2012 at 12:33 PM, Christopher J. Walkerc.j.wal...@qmul.ac.uk wrote:In the Lustre 2.1 manual,http://wiki.whamcloud.com/display/PUB/Documentationhas a link titled Lustre Manual 2.1 - though in a row titled 2.xhttp://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre-iokit/mds-survey;hb=HEADIt doesn't seem to work on Lustre 1.8. Should it?No, mds-survey arrived with Lustre 2.2:http://wiki.whamcloud.com/display/PUB/Lustre+2.2I was hoping (obviously incorrectly) that whilst it arrived in 2.2, itwould work with 1.8.The problem is that the mds-survey tool needs hooks in the Lustre kernel code inorder to generate the metadata operations. While this is driven from userspace,the kernel hooks are not in Lustre 1.8.Cheers, AndreasWhat I'm actually trying to do is migrate an MDS to new hardware - andI'd like to perform some basic tests - partly to see if using LVM has asignificant performance impact.mds-survey appeals to me because you don't need any OSTs. I don't knowif a 'OST-free' performance setup is possible for your 1.8 testing -others may know differently?That's a clear advantage, and being a standard Lustre tool give someconfidence too.Chris___Lustre-discuss mailing listLustre-discuss@lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas--Andreas DilgerWhamcloud, Inc.Principal Lustre Engineer  http://www.whamcloud.com/___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] mds-survey on Lustre-1.8

[Christopher J. Walker]

In the Lustre 2.1 manual, mds-survey is mentioned, and google finds the
source at:

http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre-iokit/mds-survey;hb=HEAD

It doesn't seem to work on Lustre 1.8. Should it?

The error I get is:

[root@mds03 client]# thrhi=64 file_count=20 sh /root/mds-survey
error: setup: Operation not supported
Can't setup echo-client
[root@mds03 client]# thrhi=64 file_count=20 sh /root/mds-survey
error: attach: LCFG_ATTACH File exists
Can't setup echo-client


What I'm actually trying to do is migrate an MDS to new hardware - and
I'd like to perform some basic tests - partly to see if using LVM has a
significant performance impact.

Chris
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] mds-survey on Lustre-1.8

[Richard Henwood]

On Tue, Nov 13, 2012 at 12:33 PM, Christopher J. Walker
c.j.wal...@qmul.ac.uk wrote:
 In the Lustre 2.1 manual, mds-survey is mentioned, and google finds the
 source at:


Can you provide a link to the manual text? The manual may need
clarification on this point (see below.)

 http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre-iokit/mds-survey;hb=HEAD

 It doesn't seem to work on Lustre 1.8. Should it?


No, mds-survey arrived with Lustre 2.2:
http://wiki.whamcloud.com/display/PUB/Lustre+2.2

snip

 What I'm actually trying to do is migrate an MDS to new hardware - and
 I'd like to perform some basic tests - partly to see if using LVM has a
 significant performance impact.


mds-survey appeals to me because you don't need any OSTs. I don't know
if a 'OST-free' performance setup is possible for your 1.8 testing -
others may know differently?

cheers,
Richard
-- 
richard.henw...@intel.com
High Performance Data Division
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] mds-survey on Lustre-1.8

[Christopher J. Walker]

On 13/11/12 18:49, Richard Henwood wrote:
 On Tue, Nov 13, 2012 at 12:33 PM, Christopher J. Walker
 c.j.wal...@qmul.ac.uk  wrote:
 In the Lustre 2.1 manual,

http://wiki.whamcloud.com/display/PUB/Documentation

has a link titled Lustre Manual 2.1 - though in a row titled 2.x


 mds-survey is mentioned, and google finds the
 source at:


 Can you provide a link to the manual text? The manual may need
 clarification on this point (see below.)

http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.html#mds_survey_ref


 http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre-iokit/mds-survey;hb=HEAD

 It doesn't seem to work on Lustre 1.8. Should it?


 No, mds-survey arrived with Lustre 2.2:
 http://wiki.whamcloud.com/display/PUB/Lustre+2.2

I was hoping (obviously incorrectly) that whilst it arrived in 2.2, it 
would work with 1.8.


 snip

 What I'm actually trying to do is migrate an MDS to new hardware - and
 I'd like to perform some basic tests - partly to see if using LVM has a
 significant performance impact.


 mds-survey appeals to me because you don't need any OSTs. I don't know
 if a 'OST-free' performance setup is possible for your 1.8 testing -
 others may know differently?

That's a clear advantage, and being a standard Lustre tool give some 
confidence too.

Chris
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS read-only

[wanglu]

Dear all, 
   Two of our MDS have got repeatedly read-only error recently after once 
e2fsck on lustre 1.8.5.  After the MDT mounted for a while, the kernel will 
reports errors like:
Oct  8 20:16:44 mainmds kernel: LDISKFS-fs error (device cciss!c0d1): 
ldiskfs_ext_check_inode: bad header/extent in inode #50736178: invalid magic - 
magic 0, entries 0, max 0(0), depth 0(0)
Oct  8 20:16:44 mainmds kernel: Aborting journal on device cciss!c0d1-8.
And make the MDS read-only.  
   This problem has made  about 1PB data, 0.1 billion files unavailable  to 
access.  We believe there is some structure wrong in the local file system of 
MDT, so we have tried to use e2fsck to fix it follow the process in lustre 
manual. However, with the loop always goes like this: 
   1.  run e2fsck,   fixed or not fixed some errors
   2.  mount  MDT, report read-only after some client operations, and the 
whole system became unusable. 
   3.  e2fsck again. 

We have tried with three different version lustre:  1.8.5, 1.8.6, and 
1.8.8-wc and their corresponding e2fsprog, the problem still exists.  
Currently, We can only use lustre with all the clients mounted in read-only 
mode,  and tried to copy the whole file system. However, It takes a long period 
to generate all the directory structure and file list for 0.1 billion files. 

Can any one give us some suggestions? Thank you very much!

Lu Wang 
CC-IHEP
   


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS read-only

[wanglu]

By the way,  we have also tried to dd the MDT device and mount the 
replica, the problem still exists. Besides, we have not seen any error reported 
on hardware monitor.  It is much more like an ldiskfs error than hardware error.

Lu


在 2012-10-9，下午12:04， wanglu 写道：

 Dear all, 
   Two of our MDS have got repeatedly read-only error recently after once 
 e2fsck on lustre 1.8.5.  After the MDT mounted for a while, the kernel will 
 reports errors like:
 Oct  8 20:16:44 mainmds kernel: LDISKFS-fs error (device cciss!c0d1): 
 ldiskfs_ext_check_inode: bad header/extent in inode #50736178: invalid magic 
 - magic 0, entries 0, max 0(0), depth 0(0)
 Oct  8 20:16:44 mainmds kernel: Aborting journal on device cciss!c0d1-8.
And make the MDS read-only.  
   This problem has made  about 1PB data, 0.1 billion files unavailable  
 to access.  We believe there is some structure wrong in the local file system 
 of MDT, so we have tried to use e2fsck to fix it follow the process in lustre 
 manual. However, with the loop always goes like this: 
   1.  run e2fsck,   fixed or not fixed some errors
   2.  mount  MDT, report read-only after some client operations, and the 
 whole system became unusable. 
   3.  e2fsck again. 
 
We have tried with three different version lustre:  1.8.5, 1.8.6, and 
 1.8.8-wc and their corresponding e2fsprog, the problem still exists.  
 Currently, We can only use lustre with all the clients mounted in read-only 
 mode,  and tried to copy the whole file system. However, It takes a long 
 period to generate all the directory structure and file list for 0.1 billion 
 files. 
 
Can any one give us some suggestions? Thank you very much!
 
 Lu Wang 
 CC-IHEP
 
 
 
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] mds backup question

[Jason Brooks]

Hello,

I am running a file-level backup of an 1.8 mds system.  At the moment, I am 
running the getfattr command as mentioned on page 15-3 in the lustre fs manual. 
 The biggest thing i a noticing, is that the attributes are as large or larger 
than the actual filesystem.  I expect that when converting bits to ascii data, 
the amount of space used will increase.  The question though is how much?  It 
doesn't seem reasonable for the bits to inflate THAT much.

--jason


Jason Brooks
971-238-3924
brook...@ohsu.edumailto:brook...@ohsu.edu
Oregon Health Sciences University
Center for Spoken Language Understanding

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] mds backup question

[Jason Brooks]

Hello,

Now that the getfattr operation is done, here are the specs:

the mds filesystem (mounted as type ldiskfs) has 7.7 gigabytes in use.

however, the getfattr operation produced an ascii file that is 20 gigabytes in 
size.

is this reasonable?


On Sep 19, 2012, at 10:06 AM, Jason Brooks wrote:

Hello,

I am running a file-level backup of an 1.8 mds system.  At the moment, I am 
running the getfattr command as mentioned on page 15-3 in the lustre fs manual. 
 The biggest thing i a noticing, is that the attributes are as large or larger 
than the actual filesystem.  I expect that when converting bits to ascii data, 
the amount of space used will increase.  The question though is how much?  It 
doesn't seem reasonable for the bits to inflate THAT much.

--jason


Jason Brooks
971-238-3924
brook...@ohsu.edumailto:brook...@ohsu.edu
Oregon Health Sciences University
Center for Spoken Language Understanding

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.orgmailto:Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Jason Brooks
971-238-3924
brook...@ohsu.edumailto:brook...@ohsu.edu
Oregon Health Sciences University
Center for Spoken Language Understanding

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] mds backup question

[Andreas Dilger]

On 2012-09-19, at 9:36 PM, Jason Brooks wrote:
 Now that the getfattr operation is done, here are the specs:
 
 the mds filesystem (mounted as type ldiskfs) has 7.7 gigabytes in use.
 
 however, the getfattr operation produced an ascii file that is 20 gigabytes 
 in size.
 
 is this reasonable?

Depends on what you are storing in xattrs and striping.  The MDS files have no 
data in them, so tar will only store the pathname and POSIX attributes.  The 
xattr data will have at least 96 bytes of data * 2 for asciification + any 
additional data you are storing for ACLs or user xattrs.

Cheers, Andreas

 On Sep 19, 2012, at 10:06 AM, Jason Brooks wrote:
 
 Hello,
 
 I am running a file-level backup of an 1.8 mds system.  At the moment, I am 
 running the getfattr command as mentioned on page 15-3 in the lustre fs 
 manual.  The biggest thing i a noticing, is that the attributes are as large 
 or larger than the actual filesystem.  I expect that when converting bits to 
 ascii data, the amount of space used will increase.  The question though is 
 how much?  It doesn't seem reasonable for the bits to inflate THAT much.
 
 --jason
 
 
 Jason Brooks
 971-238-3924
 brook...@ohsu.edu
 Oregon Health Sciences University
 Center for Spoken Language Understanding
 
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 
 Jason Brooks
 971-238-3924
 brook...@ohsu.edu
 Oregon Health Sciences University
 Center for Spoken Language Understanding
 
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger   Whamcloud, Inc.
Principal Lustre Engineerhttp://www.whamcloud.com/




___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS performance: SMP Node Affinity Docs for OpenSFS NRE

[Richard Henwood]

Hi All,

OpenSFS and Whamcloud are engaged in delivering high-priority
enhancements to Lustre for the community.
The Solution Architecture and High Level Design for MDS Performance:
SMP Node Affinity have been completed and are available for comments
and feedback:
http://wiki.whamcloud.com/display/PUB/MDS+SMP+Node+Affinity+Solution+Architecture
http://wiki.whamcloud.com/display/PUB/MDS+SMP+Node+Affinity+High+Level+Design

With best regards,
Richard
-- 
richard.henw...@whamcloud.com
Whamcloud Inc.
tel: +1 512 410 9612
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS failover: SSD+DRDB or shared 15K-SAS-Storage RAID with approx. 10 disks

[Michael Kluge]

Hi,

I have been asked, which one of the two I would chose for two MDS 
servers (active/passive). Whether I would like to have SSDs, maybe two 
(mirrored) in both servers and DRDB for synching, or a RAID controller 
that has a 15K disks. I have not done benchmarks on this topic myself 
and would like to ask if anyone has an idea or numbers? The cluster will 
be pretty small, about 50 clients.


Regards, Michael

-- 
Dr.-Ing. Michael Kluge

Technische Universität Dresden
Center for Information Services and
High Performance Computing (ZIH)
D-01062 Dresden
Germany

Contact:
Willersbau, Room WIL A 208
Phone:  (+49) 351 463-34217
Fax:(+49) 351 463-37773
e-mail: michael.kl...@tu-dresden.de
WWW:http://www.tu-dresden.de/zih
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS failover: SSD+DRDB or shared 15K-SAS-Storage RAID with approx. 10 disks

[Carlos Thomaz]

Hi Michael,

In my experience SSDs didn't help much, since the MDS bottleneck is not
only a disk problem rather than the entire lustre metadata mechanism.
We've been configuring highly scalable systems, lots of them with a high
IOPS demand, using RAID controllers and fast disks as you stated (usually
SAS 15K RPM, usually RAID10). It's difficult to to size it without knowing
exactly your I/O pattern and workload, but if you would like more
information about current systems and metadata benchmark on raid
controllers, size and configuration, let us know.

One remark about DRDB: I've seen customers using it, but IMHO, if
Active/standby HA type configuration would be more reliable and will
provide you a better resilience. Again, don't know about your uptime and
reliability needs, but the customers I've worked with that requires
minimum downtime on production, always go for RAID controllers rather than
DRDB replication.

Regards,
Carlos.


--
Carlos Thomaz | Systems Architect
Mobile: +1 (303) 519-0578
ctho...@ddn.com | Skype ID: carlosthomaz
DataDirect Networks, Inc.
9960 Federal Dr., Ste 100 Colorado Springs, CO 80921
ddn.com http://www.ddn.com/ | Twitter: @ddn_limitless
http://twitter.com/ddn_limitless | 1.800.TERABYTE





On 1/22/12 12:04 PM, Michael Kluge michael.kl...@tu-dresden.de wrote:

Hi,

I have been asked, which one of the two I would chose for two MDS
servers (active/passive). Whether I would like to have SSDs, maybe two
(mirrored) in both servers and DRDB for synching, or a RAID controller
that has a 15K disks. I have not done benchmarks on this topic myself
and would like to ask if anyone has an idea or numbers? The cluster will
be pretty small, about 50 clients.


Regards, Michael

-- 
Dr.-Ing. Michael Kluge

Technische Universität Dresden
Center for Information Services and
High Performance Computing (ZIH)
D-01062 Dresden
Germany

Contact:
Willersbau, Room WIL A 208
Phone:  (+49) 351 463-34217
Fax:(+49) 351 463-37773
e-mail: michael.kl...@tu-dresden.de
WWW:http://www.tu-dresden.de/zih
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS failover: SSD+DRDB or shared 15K-SAS-Storage RAID with approx. 10 disks

[Michael Kluge]

Hi Carlos,

 In my experience SSDs didn't help much, since the MDS bottleneck is not
 only a disk problem rather than the entire lustre metadata mechanism.

Yes, but one does not need much space on the MDS and four SSDs (as MDT) 
are way cheaper than a RAID controller with 10 15K disks. So the 
question is basically how the DRDB latency will influence the MDT 
performance. I know sync/async makes a big difference here, but I have 
no idea about the performance impact of both or how the reliability is 
influenced.

 One remark about DRDB: I've seen customers using it, but IMHO, if
 Active/standby HA type configuration would be more reliable and will
 provide you a better resilience. Again, don't know about your uptime and
 reliability needs, but the customers I've worked with that requires
 minimum downtime on production, always go for RAID controllers rather than
 DRDB replication.

OK, thanks. That is a good information. So SSD+DRDB are considered to be 
the cheap solution. Even for small clusters?


Regards, Michael


 Regards,
 Carlos.


 --
 Carlos Thomaz | Systems Architect
 Mobile: +1 (303) 519-0578
 ctho...@ddn.com | Skype ID: carlosthomaz
 DataDirect Networks, Inc.
 9960 Federal Dr., Ste 100 Colorado Springs, CO 80921
 ddn.comhttp://www.ddn.com/  | Twitter: @ddn_limitless
 http://twitter.com/ddn_limitless  | 1.800.TERABYTE





 On 1/22/12 12:04 PM, Michael Klugemichael.kl...@tu-dresden.de  wrote:

 Hi,

 I have been asked, which one of the two I would chose for two MDS
 servers (active/passive). Whether I would like to have SSDs, maybe two
 (mirrored) in both servers and DRDB for synching, or a RAID controller
 that has a 15K disks. I have not done benchmarks on this topic myself
 and would like to ask if anyone has an idea or numbers? The cluster will
 be pretty small, about 50 clients.


 Regards, Michael

 --
 Dr.-Ing. Michael Kluge

 Technische Universität Dresden
 Center for Information Services and
 High Performance Computing (ZIH)
 D-01062 Dresden
 Germany

 Contact:
 Willersbau, Room WIL A 208
 Phone:  (+49) 351 463-34217
 Fax:(+49) 351 463-37773
 e-mail: michael.kl...@tu-dresden.de
 WWW:http://www.tu-dresden.de/zih
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss


-- 
Dr.-Ing. Michael Kluge

Technische Universität Dresden
Center for Information Services and
High Performance Computing (ZIH)
D-01062 Dresden
Germany

Contact:
Willersbau, Room WIL A 208
Phone:  (+49) 351 463-34217
Fax:(+49) 351 463-37773
e-mail: michael.kl...@tu-dresden.de
WWW:http://www.tu-dresden.de/zih
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS failover: SSD+DRDB or shared 15K-SAS-Storage RAID with approx. 10 disks

[Wojciech Turek]

Hi Michael,

As Carlos said the SSDs do not improve meta data rates to much due to the
bottle necks in Lustre code itself (at least this is the case in Lustre-1.8)

If you have a limited budget the DRBD solution seems like a good choice. I
have been running production Lustre filesystem which uses DRBD mirrored MDT
for 4 years now and never had problems with it. However if the budget is
not tight and your are thinking about adding more filesystems in the future
then you may consider an external RAID disk array for example with 24 small
2.5 SAS 15Kdisks. My bigger production Lustre filesystems use that type of
storage and it does the job very well.

For the information on both configurations and some matadata performance
figures please read through my white papers:

http://i.dell.com/sites/content/shared-content/solutions/en/Documents/lustre-storage-brick-white-paper.pdf
http://i.dell.com/sites/content/business/solutions/hpcc/en/Documents/Lustre-HPC-Whitepaper-10082011.pdf

Best regards,

Wojciech


On 22 January 2012 19:55, Michael Kluge michael.kl...@tu-dresden.de wrote:

 Hi Carlos,

  In my experience SSDs didn't help much, since the MDS bottleneck is not
  only a disk problem rather than the entire lustre metadata mechanism.

 Yes, but one does not need much space on the MDS and four SSDs (as MDT)
 are way cheaper than a RAID controller with 10 15K disks. So the
 question is basically how the DRDB latency will influence the MDT
 performance. I know sync/async makes a big difference here, but I have
 no idea about the performance impact of both or how the reliability is
 influenced.

  One remark about DRDB: I've seen customers using it, but IMHO, if
  Active/standby HA type configuration would be more reliable and will
  provide you a better resilience. Again, don't know about your uptime and
  reliability needs, but the customers I've worked with that requires
  minimum downtime on production, always go for RAID controllers rather
 than
  DRDB replication.

 OK, thanks. That is a good information. So SSD+DRDB are considered to be
 the cheap solution. Even for small clusters?


 Regards, Michael

 
  Regards,
  Carlos.
 
 
  --
  Carlos Thomaz | Systems Architect
  Mobile: +1 (303) 519-0578
  ctho...@ddn.com | Skype ID: carlosthomaz
  DataDirect Networks, Inc.
  9960 Federal Dr., Ste 100 Colorado Springs, CO 80921
  ddn.comhttp://www.ddn.com/  | Twitter: @ddn_limitless
  http://twitter.com/ddn_limitless  | 1.800.TERABYTE
 
 
 
 
 
  On 1/22/12 12:04 PM, Michael Klugemichael.kl...@tu-dresden.de
  wrote:
 
  Hi,
 
  I have been asked, which one of the two I would chose for two MDS
  servers (active/passive). Whether I would like to have SSDs, maybe two
  (mirrored) in both servers and DRDB for synching, or a RAID controller
  that has a 15K disks. I have not done benchmarks on this topic myself
  and would like to ask if anyone has an idea or numbers? The cluster will
  be pretty small, about 50 clients.
 
 
  Regards, Michael
 
  --
  Dr.-Ing. Michael Kluge
 
  Technische Universität Dresden
  Center for Information Services and
  High Performance Computing (ZIH)
  D-01062 Dresden
  Germany
 
  Contact:
  Willersbau, Room WIL A 208
  Phone:  (+49) 351 463-34217
  Fax:(+49) 351 463-37773
  e-mail: michael.kl...@tu-dresden.de
  WWW:http://www.tu-dresden.de/zih
  ___
  Lustre-discuss mailing list
  Lustre-discuss@lists.lustre.org
  http://lists.lustre.org/mailman/listinfo/lustre-discuss
 

 --
 Dr.-Ing. Michael Kluge

 Technische Universität Dresden
 Center for Information Services and
 High Performance Computing (ZIH)
 D-01062 Dresden
 Germany

 Contact:
 Willersbau, Room WIL A 208
 Phone:  (+49) 351 463-34217
 Fax:(+49) 351 463-37773
 e-mail: michael.kl...@tu-dresden.de
 WWW:http://www.tu-dresden.de/zih
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss




-- 
Wojciech Turek

Senior System Architect

High Performance Computing Service
University of Cambridge
Email: wj...@cam.ac.uk
Tel: (+)44 1223 763517
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS network traffic question

[Kevin Van Maren]

I would replace the 1GigE with the 10GigE: have all Ethernet traffic go over 
the 10GigE links, rather than add another tcp1 network for Lustre.  This will 
keep your configuration much simpler, and make the migration as painless as 
possible (just move the IP address to the 10GigE port on the servers).

The MDS traffic _volume_ is much lower than it is for the OSS nodes.  The big 
win from 10GigE would be the lower latency: if you approach 100MB/s of MDS 
traffic, you have much bigger problems than a 10GigE NIC can solve.

Kevin


On Oct 11, 2011, at 3:54 PM, James Robnett wrote:

 
   We have a small lustre install consisting of an MDS and 5 OSS servers.
 Historically the MDS and OSS servers had both a 1Gbit ethernet interface
 (tcp0) to workstations and a QDR IB interface (ib0) to our cluster.
 
   We're planning on adding a MTU 9000 10Gbit ethernet (tcp1) interface
 to the MDS and OSS nodes and workstations for faster access.  Our
 software has a pretty high IO to CPU component.
 
   I just discovered that our MDS can't in fact take another PCIe 8x
 card but it does have a spare GigE port.  The 10gbit Ethernet switch
 can support 1gbit and 10gbit interfaces.
 
   We'd then have 3 networks
 tcp0 at 1gbit to slow clients
 tcp1 at 10gbit to faster clients
 ib0 to cluster
 
   My question is:
 
   Is there a risk of congestion or overrunning that 2nd GigE MDS 
 interface if our workstations and OSS servers communicate over tcp1 at
 10gbit but the MDS tcp1 is connected at 1Gbit.  The bulk of our traffic
 will continue to be between the cluster and lustre over IB but the
 workstations can trivially over run ethernet hence the desire for
 10gbit between them and the OSSes.
 
   My gut feeling is it should be fine, particularly with the larger MTU,
 there's not that much traffic to the MDS but I'd easily believe it if
 somebody said it's risky thing to do.
 
   The alternative is to buy a new MDS and swap disks into it.
 
 James Robnett
 National Radio Astronomy Observatory
 Array Operations Center
 


Confidentiality Notice: This e-mail message, its contents and any attachments 
to it are confidential to the intended recipient, and may contain information 
that is privileged and/or exempt from disclosure under applicable law. If you 
are not the intended recipient, please immediately notify the sender and 
destroy the original e-mail message and any attachments (and any copies that 
may have been made) from your system or otherwise. Any unauthorized use, 
copying, disclosure or distribution of this information is strictly prohibited.
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS network traffic question

[James Robnett]

   We have a small lustre install consisting of an MDS and 5 OSS servers.
Historically the MDS and OSS servers had both a 1Gbit ethernet interface
(tcp0) to workstations and a QDR IB interface (ib0) to our cluster.

   We're planning on adding a MTU 9000 10Gbit ethernet (tcp1) interface
to the MDS and OSS nodes and workstations for faster access.  Our
software has a pretty high IO to CPU component.

   I just discovered that our MDS can't in fact take another PCIe 8x
card but it does have a spare GigE port.  The 10gbit Ethernet switch
can support 1gbit and 10gbit interfaces.

   We'd then have 3 networks
tcp0 at 1gbit to slow clients
tcp1 at 10gbit to faster clients
ib0 to cluster

   My question is:

   Is there a risk of congestion or overrunning that 2nd GigE MDS 
interface if our workstations and OSS servers communicate over tcp1 at
10gbit but the MDS tcp1 is connected at 1Gbit.  The bulk of our traffic
will continue to be between the cluster and lustre over IB but the
workstations can trivially over run ethernet hence the desire for
10gbit between them and the OSSes.

   My gut feeling is it should be fine, particularly with the larger MTU,
there's not that much traffic to the MDS but I'd easily believe it if
somebody said it's risky thing to do.

   The alternative is to buy a new MDS and swap disks into it.

James Robnett
National Radio Astronomy Observatory
Array Operations Center





___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS LBUG: ASSERTION(!mds_inode_is_orphan(dchild-d_inode)) failed

[Frederik Ferner]

All,

I'd like to follow up on this as I can now repeatedly reproduce this on 
our test file system. I've managed to reproduce it on an version up to 
lustre 1.8.6-wc1 on the MDS that I've tried so far.

I've also reported it as LU-534 
(http://jira.whamcloud.com/browse/LU-534) and included current stack 
traces etc.

I'll repeat the basic instructions how to reproduce here:

Export a Lustre file system via NFS(v3) from a Lustre client, mount it 
on one other system over NFS, run racer on the file system over NFS, 
after a few minutes (sometimes one or two hours) the MDS LBUGs with the 
ASSERTION in the subject.

If anyone has any suggestions of debug flags to enable or other ideas 
how to track down the exact problem, I'd like to hear them.

Kind regards,
Frederik

On 08/07/11 14:26, Frederik Ferner wrote:
 All,

 we are experiencing what looks like the same MDS LBUG with increasing
 frequency, see below for a sample stack trace. This seems to affect only
 one client at a time and even this client will recover after some time
 (usually minutes but sometimes longer) and continue to work even without
 requiring immediate MDS reboots.

 In the recent past, it seems to have affected one specific client more
 often than others. This client is mainly a NFS exporter for the Lustre
 file system. All attempts to trigger the LBUG with known actions have
 been unsuccessful so far. Attempts to trigger it on the test file system
 have equally not been successful but we are still working on this.

 As far as I can see, this could be this bug
 https://bugzilla.lustre.org/show_bug.cgi?id=17764 but there has been no
 recent activity. And I'm not entirely sure this is the same bug.

 As far as I can see the log dumps don't contain any useful information,
 but I'm happy to provide as sample file if someone offers to look at
 it.

 I'm also looking for suggestions how to go about debugging this problem,
 ideally initially with as little performance impact as possible so we
 might apply it on the productions system until we can reproduce it on a
 test file system. Once we can reproduce it on the test file system,
 debugging with performance implications should be possible as well.

 The MDS and clients are currently running Lustre 1.8.3.ddn3.3 on Red Hat
 Enterprise 5.

 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: LustreError: 
 4037:0:(mds_open.c:1295:mds_open()) 
 ASSERTION(!mds_inode_is_orphan(dchild-d_inode)) failed: dchild 
 8ad94b2:0cae8d46 (8101995b0300) inode 81041d4e8548/145593522/21276602
 2
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: LustreError: 
 4037:0:(mds_open.c:1295:mds_open()) LBUG
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: Lustre: 
 4037:0:(linux-debug.c:264:libcfs_debug_dumpstack()) showing stack for 
 process 4037
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: ll_mdt_49 R  running task 
   0  4037  1  4038  4036 (L-TLB)
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  810226da0d00 
 81024712 0286 0082
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  00811400 
 8101db219ef8 0001 0001
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  8101ead74db8 
  810423223e10 8008aee7
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: Call Trace:
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8008aee7] 
 __wake_up_common+0x3e/0x68
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [887acee8] 
 :ptlrpc:ptlrpc_main+0x1258/0x1420
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8008cabd] 
 default_wake_function+0x0/0xe
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [800b7310] 
 audit_syscall_exit+0x336/0x362
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8005dfb1] 
 child_rip+0xa/0x11
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [887abc90] 
 :ptlrpc:ptlrpc_main+0x0/0x1420
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8005dfa7] 
 child_rip+0x0/0x11
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: LustreError: dumping log to 
 /tmp/lustre-log.1309949645.4037

 Kind regards,
 Frederik


-- 
Frederik Ferner
Computer Systems Administrator  phone: +44 1235 77 8624
Diamond Light Source Ltd.   mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. 

[Lustre-discuss] MDS LBUG: ASSERTION(!mds_inode_is_orphan(dchild-d_inode)) failed

[Frederik Ferner]

All,

we are experiencing what looks like the same MDS LBUG with increasing 
frequency, see below for a sample stack trace. This seems to affect only 
one client at a time and even this client will recover after some time 
(usually minutes but sometimes longer) and continue to work even without 
requiring immediate MDS reboots.

In the recent past, it seems to have affected one specific client more 
often than others. This client is mainly a NFS exporter for the Lustre 
file system. All attempts to trigger the LBUG with known actions have 
been unsuccessful so far. Attempts to trigger it on the test file system 
have equally not been successful but we are still working on this.

As far as I can see, this could be this bug 
https://bugzilla.lustre.org/show_bug.cgi?id=17764 but there has been no 
recent activity. And I'm not entirely sure this is the same bug.

As far as I can see the log dumps don't contain any useful information, 
   but I'm happy to provide as sample file if someone offers to look at 
it.

I'm also looking for suggestions how to go about debugging this problem, 
ideally initially with as little performance impact as possible so we 
might apply it on the productions system until we can reproduce it on a 
test file system. Once we can reproduce it on the test file system, 
debugging with performance implications should be possible as well.

The MDS and clients are currently running Lustre 1.8.3.ddn3.3 on Red Hat 
Enterprise 5.

 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: LustreError: 
 4037:0:(mds_open.c:1295:mds_open()) 
 ASSERTION(!mds_inode_is_orphan(dchild-d_inode)) failed: dchild 
 8ad94b2:0cae8d46 (8101995b0300) inode 81041d4e8548/145593522/21276602
 2
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: LustreError: 
 4037:0:(mds_open.c:1295:mds_open()) LBUG
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: Lustre: 
 4037:0:(linux-debug.c:264:libcfs_debug_dumpstack()) showing stack for process 
 4037
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: ll_mdt_49 R  running task  
  0  4037  1  4038  4036 (L-TLB)
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  810226da0d00 
 81024712 0286 0082
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  00811400 
 8101db219ef8 0001 0001
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  8101ead74db8 
  810423223e10 8008aee7
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: Call Trace:
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8008aee7] 
 __wake_up_common+0x3e/0x68
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [887acee8] 
 :ptlrpc:ptlrpc_main+0x1258/0x1420
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8008cabd] 
 default_wake_function+0x0/0xe
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [800b7310] 
 audit_syscall_exit+0x336/0x362
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8005dfb1] 
 child_rip+0xa/0x11
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [887abc90] 
 :ptlrpc:ptlrpc_main+0x0/0x1420
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel:  [8005dfa7] 
 child_rip+0x0/0x11
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: 
 Jul  6 11:54:05 cs04r-sc-mds01-01-10ge kernel: LustreError: dumping log to 
 /tmp/lustre-log.1309949645.4037

Kind regards,
Frederik
-- 
Frederik Ferner
Computer Systems Administrator  phone: +44 1235 77 8624
Diamond Light Source Ltd.   mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

-- 
This e-mail and any attachments may contain confidential, copyright and or 
privileged material, and are for the use of the intended addressee only. If you 
are not the intended addressee or an authorised recipient of the addressee 
please notify us of receipt by returning the e-mail and do not use, copy, 
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not 
necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments 
are free from viruses and we cannot accept liability for any damage which you 
may sustain as a result of software viruses which may be transmitted in or with 
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and 
Wales with its registered office at Diamond House, Harwell Science and 
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 



___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS can't recover OSTs

[Johann Lombardi]

On Thu, Apr 28, 2011 at 12:47:02PM -0400, Charles Taylor wrote:
 Apr 28 11:37:54 crnmds kernel: Lustre: 31982:0:(import.c: 
 1137:ptlrpc_connect_interpret()) recovery of crn-OST0013_UUID on  
 10.13.24.92@o2ib failed (-16)
 Apr 28 11:37:54 crnmds kernel: Lustre: 31982:0:(import.c: 
 1137:ptlrpc_connect_interpret()) recovery of crn-OST0007_UUID on  
 10.13.24.91@o2ib failed (-16)

Both OST0007  OST0013 return EBUSY. Any messages or watchdogs in the OSS logs 
(i.e. 10.13.24.9{1,2}@o2ib)?

Johann

-- 
Johann Lombardi
Whamcloud, Inc.
www.whamcloud.com
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS can't recover OSTs

[Charles Taylor]

We had a RAID array barf this morning resulting in some OST corruption  
which appeared to be successfully repaired with a combination of fsck  
and ll_recover_lost_found_objs.   The OSTs mounted OK but the MDS  
can't seem to recover its connection to two of the OSTs as we are  
seeing a continuing stream of the following in the MDS syslog.

Apr 28 11:37:54 crnmds kernel: Lustre: 31983:0:(recover.c: 
67:ptlrpc_initiate_recovery()) crn-OST0013_UUID: starting recovery
Apr 28 11:37:54 crnmds kernel: Lustre: 31983:0:(import.c: 
608:ptlrpc_connect_import()) 810117426000 crn-OST0013_UUID:  
changing import state from DISCONN to CONNECTING
Apr 28 11:37:54 crnmds kernel: Lustre: 31983:0:(import.c: 
470:import_select_connection()) crn-OST0013-osc: connect to NID  
10.13.24.92@o2ib last attempt 22689204132
Apr 28 11:37:54 crnmds kernel: Lustre: 31983:0:(import.c: 
544:import_select_connection()) crn-OST0013-osc: import  
810117426000 using connection 10.13.24.92@o2ib/10.13.24.92@o2ib
Apr 28 11:37:54 crnmds kernel: Lustre: 31982:0:(import.c: 
1091:ptlrpc_connect_interpret()) 810117426000 crn-OST0013_UUID:  
changing import state from CONNECTING to DISCONN
Apr 28 11:37:54 crnmds kernel: Lustre: 31982:0:(import.c: 
1137:ptlrpc_connect_interpret()) recovery of crn-OST0013_UUID on  
10.13.24.92@o2ib failed (-16)
Apr 28 11:37:54 crnmds kernel: Lustre: 31982:0:(import.c: 
1091:ptlrpc_connect_interpret()) 81012e50d000 crn-OST0007_UUID:  
changing import state from CONNECTING to DISCONN
Apr 28 11:37:54 crnmds kernel: Lustre: 31982:0:(import.c: 
1137:ptlrpc_connect_interpret()) recovery of crn-OST0007_UUID on  
10.13.24.91@o2ib failed (-16)

It seems that we never see a  'oscc recovery finished' message on  
crnmds for OST0007 or OST0013.

We have not seen this problem before so we are trying to figure out  
how to get the MDT reconnected to these two OSTs.

Any one else been through this before?

Thanks,

Charlie Taylor
UF HPC Center



___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS activity monitoring

[Andreas Dilger]

On 2011-03-20, at 9:09 PM, Andreas Davour wrote:
 I've decided I want to keep an eye on the activity on my MDS, and poking 
 about 
 in /proc I found both /proc/fs/lustre/mds/fsname-MDT/stats and 
 /proc/fs/lustre/mdt/MDS/mds/stats which both contain stats about metaops, but 
 while similar they are slightly different. 
 
 Now, I'm not really sure I see how they are different, and why there are two 
 /proc/fs/lustre/md* directories with stats like that? 

I believe there is actaully a patch in bugzilla that consolidates these two 
/proc files, and removes one of them.  I think one is just left over for 
compatibility reasons.

 Can somebody who knows something about lustre internals maybe shed some light 
 on the layout of the /proc files and where to best look to get a grip on 
 what's 
 going on in the file system?

There are two main different types of stats files - ones for the network RPC 
service (e.g. MDS/{mds,mds_readpage}, OSS/{ost,ost_io,ost_create}), and one for 
the target device (e.g. mds/fsname-MDT/, obdfilter/fsname-OST).

Each have slightly different statistics, because they have access to different 
information.  I think some of this is documented in the manual.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.



___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS activity monitoring

[Andreas Davour]

On Monday, March 21, 2011 08:32:59 Andreas Dilger wrote:
 On 2011-03-20, at 9:09 PM, Andreas Davour wrote:
  I've decided I want to keep an eye on the activity on my MDS, and poking
  about in /proc I found both /proc/fs/lustre/mds/fsname-MDT/stats
  and /proc/fs/lustre/mdt/MDS/mds/stats which both contain stats about
  metaops, but while similar they are slightly different.
  
  Now, I'm not really sure I see how they are different, and why there are
  two /proc/fs/lustre/md* directories with stats like that?
 
 I believe there is actaully a patch in bugzilla that consolidates these two
 /proc files, and removes one of them.  I think one is just left over for
 compatibility reasons.

That would be a good thing, since they are confusing. Any hint on where to find 
that bugzilla entry?

  Can somebody who knows something about lustre internals maybe shed some
  light on the layout of the /proc files and where to best look to get a
  grip on what's going on in the file system?
 
 There are two main different types of stats files - ones for the network
 RPC service (e.g. MDS/{mds,mds_readpage}, OSS/{ost,ost_io,ost_create}),
 and one for the target device (e.g. mds/fsname-MDT/,
 obdfilter/fsname-OST).
 
 Each have slightly different statistics, because they have access to
 different information.  I think some of this is documented in the manual.

Thanks. I'll take a closer look at the documentation.

/andreas
-- 
Systems Engineer
PDC Center for High Performance Computing
CSC School of Computer Science and Communication
KTH Royal Institute of Technology
SE-100 44 Stockholm, Sweden
Phone: 087906658
A satellite, an earring, and a dust bunny are what made America great!
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS activity monitoring

[Andreas Davour]

Hi

I've decided I want to keep an eye on the activity on my MDS, and poking about 
in /proc I found both /proc/fs/lustre/mds/fsname-MDT/stats and 
/proc/fs/lustre/mdt/MDS/mds/stats which both contain stats about metaops, but 
while similar they are slightly different. 

Now, I'm not really sure I see how they are different, and why there are two 
/proc/fs/lustre/md* directories with stats like that? 

Can somebody who knows something about lustre internals maybe shed some light 
on the layout of the /proc files and where to best look to get a grip on what's 
going on in the file system?

/andreas
-- 
Systems Engineer
PDC Center for High Performance Computing
CSC School of Computer Science and Communication
KTH Royal Institute of Technology
SE-100 44 Stockholm, Sweden
Phone: 087906658
A satellite, an earring, and a dust bunny are what made America great!
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS hangs with OFED

[Kevin Hildebrand]

We've been seeing occasional hangs on our MDS and I'd like to see if 
anyone else is seeing this or can provide suggestions on where to look.
This might not even be a Lustre problem at all.

We're running Lustre 1.8.4 with OFED 1.5.2, and kernel version 
2.6.18-194.3.1.el5_lustre.1.8.4.

The problem is that at some point it appears that something in the IB 
stack is going out to lunch- pings to the IPoIB interface time out, and 
anything that touches IB (perfquery, etc) goes into a hard hang and cannot 
be killed.

The only solution to the problem once it occurs is to power-cycle the 
machine, as shutdown/reboot hang as well.

From what I can see, the first abnormal entries in the system logs on 
the MDS are messages showing that connections to the OSSes are timing out.

Any insight would be appreciated.

Thanks,

Kevin
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS hangs with OFED

[Cliff White]

Unfortunately, we've had lot's of reports of IB instability.  It does appear
to happen
quite a bit, and generally is not a Lustre problem at all.
- Check all mechanical connections, cables, etc. - replace if need be - many
issues have been cable-related.
- Check firmware versions of all IB cards, find the best version for yours.
- Make sure your IB cards are in the proper (best performing) slots in your
backplane.
- If you have an IB switch with monitoring/error reporting you may be able
to get more data.
cliffw


On Thu, Mar 17, 2011 at 10:54 AM, Kevin Hildebrand ke...@umd.edu wrote:


 We've been seeing occasional hangs on our MDS and I'd like to see if
 anyone else is seeing this or can provide suggestions on where to look.
 This might not even be a Lustre problem at all.

 We're running Lustre 1.8.4 with OFED 1.5.2, and kernel version
 2.6.18-194.3.1.el5_lustre.1.8.4.

 The problem is that at some point it appears that something in the IB
 stack is going out to lunch- pings to the IPoIB interface time out, and
 anything that touches IB (perfquery, etc) goes into a hard hang and cannot
 be killed.

 The only solution to the problem once it occurs is to power-cycle the
 machine, as shutdown/reboot hang as well.

 From what I can see, the first abnormal entries in the system logs on
 the MDS are messages showing that connections to the OSSes are timing out.

 Any insight would be appreciated.

 Thanks,

 Kevin
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss




-- 
cliffw
Support Guy
WhamCloud, Inc.
www.whamcloud.com
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS mount cannot assign requested address

[Mike Hanby]

Here are the entries from syslog

Feb  2 11:14:21 lustre-mds-0-0 kernel: kjournald starting.  Commit interval 5 
seconds
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS FS on sdb, internal journal
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS-fs: mounted filesystem with 
ordered data mode.
Feb  2 11:14:21 lustre-mds-0-0 kernel: kjournald starting.  Commit interval 5 
seconds
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS FS on sdb, internal journal
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS-fs: mounted filesystem with 
ordered data mode.
Feb  2 11:14:21 lustre-mds-0-0 kernel: Lustre: MGS MGS started
Feb  2 11:14:21 lustre-mds-0-0 kernel: Lustre: MGC192.168.2.20@o2ib: 
Reactivating import
Feb  2 11:14:21 lustre-mds-0-0 kernel: Lustre: Denying initial registration 
attempt from nid 192.168.2.20@o2ib, specified as failover
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:1097:server_start_targets()) Required registration failed 
for lustre-MDT: -99
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:1655:server_fill_super()) Unable to start targets: -99
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:1438:server_put_super()) no obd lustre-MDT
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:147:server_deregister_mount()) lustre-MDT not registered
Feb  2 11:14:22 lustre-mds-0-0 kernel: Lustre: MGS has stopped.
Feb  2 11:14:22 lustre-mds-0-0 kernel: Lustre: server umount lustre-MDT 
complete
Feb  2 11:14:22 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:2050:lustre_fill_super()) Unable to mount  (-99)

-Original Message-
From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Mike Hanby
Sent: Wednesday, February 02, 2011 11:06 AM
To: lustre-discuss@lists.lustre.org
Subject: [Lustre-discuss] MDS mount cannot assign requested address

Howdy,

We recently changed our ip scheme for our cluster. I disconnected all of the 
clients, ost's, and unmounted the combined mdt/mgs.

changed the ip addresses for eth0, ib0 and ib1 on the MDS, and changed the LNET 
parameters in modprobe.ddn to reflect the new ip's.

Next I ran the tunefs.lustre to make the changes (see below for the full syntax 
and output).

Now, when I attempt to mount the mdt/mgs I get:

# mount /mnt/mdt-mtpt 
mount.lustre: mount /dev/sdb at /mnt/mdt-mtpt failed: Cannot assign requested 
address

I've tried a reboot, thinking maybe a port might be in use, still the same 
problem. I've searched the docs and found the error message in the source code, 
but no useful tech docs. Have you encountered this or do you have any 
suggestions?

Here's the tunefs command I ran to change the nids to our new IP scheme:

# hostname
lustre-mds-0-0

# lctl list_nids
192.168.2.20@o2ib
192.168.3.20@o2ib1
172.20.0.20@tcp

# tunefs.lustre --writeconf --erase-params 
--param=failover.node=192.168.2.20@o2ib 
--param=failover.node=192.168.3.20@o2ib1 
--param=failover.node=172.20.0.20@tcp --param=mdt.group_upcall=NONE /dev/sdb
checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata

   Read previous values:
Target: lustre-MDT
Index:  0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:  0x545
  (MDT MGS update writeconf )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=192.168.2.20@o2ib failover.node=192.168.3.20@o2ib1 
failover.node=172.20.0.20@tcp mdt.group_upcall=NONE

# ifconfig
eth0  Link encap:Ethernet  HWaddr 84:2B:  
  inet addr:172.20.0.20  Bcast:172.20.255.255  Mask:255.255.0.0

ib0   Link encap:InfiniBand  HWaddr 80:00:x
  inet addr:192.168.2.20  Bcast:192.168.2.255  Mask:255.255.255.0

ib1   Link encap:InfiniBand  HWaddr 80:00:
  inet addr:192.168.3.20  Bcast:192.168.3.255  Mask:255.255.255.0

=
Mike Hanby
mha...@uab.edu
UAB School of Engineering
Information Systems Specialist II
IT HPCS / Research Computing


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS mount cannot assign requested address

[Mike Hanby]

Aha, my bad, I was specifying the MDS's own IPs as failover, removing those 
resolved the issue:

# tunefs.lustre --writeconf --erase-params --param=mdt.group_upcall=NONE 
/dev/sdb

I need to do that on the failover, not the primary I guess.


-Original Message-
From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Mike Hanby
Sent: Wednesday, February 02, 2011 11:31 AM
To: lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] MDS mount cannot assign requested address

Here are the entries from syslog

Feb  2 11:14:21 lustre-mds-0-0 kernel: kjournald starting.  Commit interval 5 
seconds
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS FS on sdb, internal journal
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS-fs: mounted filesystem with 
ordered data mode.
Feb  2 11:14:21 lustre-mds-0-0 kernel: kjournald starting.  Commit interval 5 
seconds
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS FS on sdb, internal journal
Feb  2 11:14:21 lustre-mds-0-0 kernel: LDISKFS-fs: mounted filesystem with 
ordered data mode.
Feb  2 11:14:21 lustre-mds-0-0 kernel: Lustre: MGS MGS started
Feb  2 11:14:21 lustre-mds-0-0 kernel: Lustre: MGC192.168.2.20@o2ib: 
Reactivating import
Feb  2 11:14:21 lustre-mds-0-0 kernel: Lustre: Denying initial registration 
attempt from nid 192.168.2.20@o2ib, specified as failover
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:1097:server_start_targets()) Required registration failed 
for lustre-MDT: -99
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:1655:server_fill_super()) Unable to start targets: -99
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:1438:server_put_super()) no obd lustre-MDT
Feb  2 11:14:21 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:147:server_deregister_mount()) lustre-MDT not registered
Feb  2 11:14:22 lustre-mds-0-0 kernel: Lustre: MGS has stopped.
Feb  2 11:14:22 lustre-mds-0-0 kernel: Lustre: server umount lustre-MDT 
complete
Feb  2 11:14:22 lustre-mds-0-0 kernel: LustreError: 
8493:0:(obd_mount.c:2050:lustre_fill_super()) Unable to mount  (-99)

-Original Message-
From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Mike Hanby
Sent: Wednesday, February 02, 2011 11:06 AM
To: lustre-discuss@lists.lustre.org
Subject: [Lustre-discuss] MDS mount cannot assign requested address

Howdy,

We recently changed our ip scheme for our cluster. I disconnected all of the 
clients, ost's, and unmounted the combined mdt/mgs.

changed the ip addresses for eth0, ib0 and ib1 on the MDS, and changed the LNET 
parameters in modprobe.ddn to reflect the new ip's.

Next I ran the tunefs.lustre to make the changes (see below for the full syntax 
and output).

Now, when I attempt to mount the mdt/mgs I get:

# mount /mnt/mdt-mtpt 
mount.lustre: mount /dev/sdb at /mnt/mdt-mtpt failed: Cannot assign requested 
address

I've tried a reboot, thinking maybe a port might be in use, still the same 
problem. I've searched the docs and found the error message in the source code, 
but no useful tech docs. Have you encountered this or do you have any 
suggestions?

Here's the tunefs command I ran to change the nids to our new IP scheme:

# hostname
lustre-mds-0-0

# lctl list_nids
192.168.2.20@o2ib
192.168.3.20@o2ib1
172.20.0.20@tcp

# tunefs.lustre --writeconf --erase-params 
--param=failover.node=192.168.2.20@o2ib 
--param=failover.node=192.168.3.20@o2ib1 
--param=failover.node=172.20.0.20@tcp --param=mdt.group_upcall=NONE /dev/sdb
checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata

   Read previous values:
Target: lustre-MDT
Index:  0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:  0x545
  (MDT MGS update writeconf )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=192.168.2.20@o2ib failover.node=192.168.3.20@o2ib1 
failover.node=172.20.0.20@tcp mdt.group_upcall=NONE

# ifconfig
eth0  Link encap:Ethernet  HWaddr 84:2B:  
  inet addr:172.20.0.20  Bcast:172.20.255.255  Mask:255.255.0.0

ib0   Link encap:InfiniBand  HWaddr 80:00:x
  inet addr:192.168.2.20  Bcast:192.168.2.255  Mask:255.255.255.0

ib1   Link encap:InfiniBand  HWaddr 80:00:
  inet addr:192.168.3.20  Bcast:192.168.3.255  Mask:255.255.255.0

=
Mike Hanby
mha...@uab.edu
UAB School of Engineering
Information Systems Specialist II
IT HPCS / Research Computing


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] mds/t bug

[Oleg Drokin]

Hello!

On Sep 3, 2010, at 4:52 PM, John White wrote:

   Can someone help me out figuring out what's wrong here?  We have an 
 MDS/T that keeps causing problems.  I have 2 dozen or so dumps from threads 
 crashing.  The threads in question all appear to be either ll_mdt_rdpg_ ?  or 
 ll_mdt_??
 
 I can provide dumps on request, any help would be greatly appreciated.

Please include at least some information (like the error messages, stack traces 
and such) so that we can actually try to help you.
Alternatively you can try to open a bug in bugzilla and include dumps there.

Bye,
Oleg
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] mds/t bug

[John White]

Hello Folks,
Can someone help me out figuring out what's wrong here?  We have an 
MDS/T that keeps causing problems.  I have 2 dozen or so dumps from threads 
crashing.  The threads in question all appear to be either ll_mdt_rdpg_ ?  or 
ll_mdt_??

I can provide dumps on request, any help would be greatly appreciated.

John White
High Performance Computing Services (HPCS)
(510) 486-7307
One Cyclotron Rd, MS: 50B-3209C
Lawrence Berkeley National Lab
Berkeley, CA 94720

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS memory usage

[Bernd Schubert]

Hello Frederik,


On Wednesday, August 25, 2010, Frederik Ferner wrote:
 Hi Bernd,
 
 thanks for your reply.
 
 Bernd Schubert wrote:
  On Tuesday, August 24, 2010, Frederik Ferner wrote:
  on our MDS we noticed that all memory seems to be used. (And it's not
  just normal buffers/cache as far as I can tell.)
  
  When we put load on the machine, for example by starting rsync
  on a few clients, generating file lists to copy data from Lustre to
  local disks or just running a MDT backup locally using dd/gzip to copy a
  LVM snapshot to a remote server, kswapd starts using a lot of CPU
  time, sometimes up to 100% of one CPU core.
  
  This is on a Lustre 1.6.7.2.ddn3.5 based file system with about 200TB,
  the MDT is 800GB with 200M inodes, ACLs enabled.
  
  Did you recompile it, or did you use the binaries from my home page (or
  those you got from CV)?
 
 This is a recompiled Lustre version to include the patch from bug 
 22820.
 
  Possibly it is a LRU auto-resize problem, but which has been disabled in
  DDN builds. As our 1.6 releases didn't include a patch for that, you
  would need to specify the correct command options if you recompiled it.
 
 I guess it's likely that I have not specified the correct option. So the
   binaries on your home page are compiled with '--disable-lru-resize'?
 Any other options that you used?

I always enable the health-write, which will help pacemaker to detect IO 
errors (by monitoring /proc/fs/lustre/health_check)

 --enable-health-write 

 
  Another reason might be bug 22771, although that should only come up on
  MDS with more memory you have.
 
 I had a look at that bug and while we have a default stripe count of 1
 so the stripe count should fit into the inode. On the other hand we use
 ACLs in quite a few places, so it seems we might hit this bug if we
 increase the memory from the 16GB currently, correct?

Yeah and I think 16GB should be sufficient for the MDS. 



-- 
Bernd Schubert
DataDirect Networks
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS memory usage

[Frederik Ferner]

Hi Bernd,

thanks for your reply.

Bernd Schubert wrote:
 On Tuesday, August 24, 2010, Frederik Ferner wrote:
 on our MDS we noticed that all memory seems to be used. (And it's not
 just normal buffers/cache as far as I can tell.)

 When we put load on the machine, for example by starting rsync
 on a few clients, generating file lists to copy data from Lustre to
 local disks or just running a MDT backup locally using dd/gzip to copy a
 LVM snapshot to a remote server, kswapd starts using a lot of CPU
 time, sometimes up to 100% of one CPU core.

 This is on a Lustre 1.6.7.2.ddn3.5 based file system with about 200TB,
 the MDT is 800GB with 200M inodes, ACLs enabled.
 
 Did you recompile it, or did you use the binaries from my home page (or those 
 you got from CV)?

This is a recompiled Lustre version to include the patch from bug22820.

 Possibly it is a LRU auto-resize problem, but which has been disabled in DDN 
 builds. As our 1.6 releases didn't include a patch for that, you would need 
 to 
 specify the correct command options if you recompiled it.

I guess it's likely that I have not specified the correct option. So the 
  binaries on your home page are compiled with '--disable-lru-resize'? 
Any other options that you used?

 Another reason might be bug 22771, although that should only come up on MDS 
 with more memory you have.

I had a look at that bug and while we have a default stripe count of 1 
so the stripe count should fit into the inode. On the other hand we use 
ACLs in quite a few places, so it seems we might hit this bug if we 
increase the memory from the 16GB currently, correct?

Cheers,
Frederik
-- 
Frederik Ferner
Computer Systems Administrator  phone: +44 1235 77 8624
Diamond Light Source Ltd.   mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS memory usage

[Frederik Ferner]

Hi List,

on our MDS we noticed that all memory seems to be used. (And it's not
just normal buffers/cache as far as I can tell.)

When we put load on the machine, for example by starting rsync
on a few clients, generating file lists to copy data from Lustre to
local disks or just running a MDT backup locally using dd/gzip to copy a
LVM snapshot to a remote server, kswapd starts using a lot of CPU
time, sometimes up to 100% of one CPU core.

This is on a Lustre 1.6.7.2.ddn3.5 based file system with about 200TB,
the MDT is 800GB with 200M inodes, ACLs enabled.

The memory seems mostly used by the kernel and that quite a lot of it
is ldlm_locks, ldlm_resource according to slabtop. Some details of this
are below, but the main question that we now have is whether or not this
is normal and expected.

Is there a tunable to restrict Lustre to use a bit less slab memory
than it currently is?

Will adding more memory to this machine solve the problem that there 
seems to be not enough memory to run normal processes or will it just 
delay the occurrences of this?

Kind regards,
Frederik

Memory details:
snip
[r...@cs04r-sc-mds01-01 proc]# free
  total   used   free sharedbuffers cached
Mem:  16497436   16146416 351020  0 257624  17836
-/+ buffers/cache:   15870956 626480
Swap:  2031608 3227681708840
[r...@cs04r-sc-mds01-01 proc]# cat /proc/meminfo
MemTotal: 16497436 kB
MemFree:352004 kB
Buffers:256084 kB
Cached:  17688 kB
SwapCached: 149544 kB
Active: 200764 kB
Inactive:   255344 kB
HighTotal:   0 kB
HighFree:0 kB
LowTotal: 16497436 kB
LowFree:352004 kB
SwapTotal: 2031608 kB
SwapFree:  1708840 kB
Dirty: 268 kB
Writeback:   0 kB
AnonPages:  182272 kB
Mapped:  17528 kB
Slab: 15248816 kB
PageTables:   6984 kB
NFS_Unstable:0 kB
Bounce:  0 kB
CommitLimit:  10280324 kB
Committed_AS:  1321284 kB
VmallocTotal: 34359738367 kB
VmallocUsed:330740 kB
VmallocChunk: 34359394255 kB
HugePages_Total: 0
HugePages_Free:  0
HugePages_Rsvd:  0
Hugepagesize: 2048 kB
[r...@cs04r-sc-mds01-01 proc]# slabtop --once |head -15
  Active / Total Objects (% used): 30350433 / 38705406 (78.4%)
  Active / Total Slabs (% used)  : 3801362 / 3801369 (100.0%)
  Active / Total Caches (% used) : 114 / 168 (67.9%)
  Active / Total Size (% used)   : 12325021.07K / 14610074.85K (84.4%)
  Minimum / Average / Maximum Object : 0.02K / 0.38K / 128.00K

   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
15657800 14362022  91%0.50K 19572258   7828900K ldlm_locks
10165900 9719990  95%0.38K 1016590   10   4066360K ldlm_resources
3650979 1038530  28%0.06K  61881   59247524K size-64
3646620 3159662  86%0.12K 121554   30486216K size-128
3099906 863841  27%0.21K 172217   18688868K dentry_cache
1679436 859267  51%0.83K 4198594   1679436K ldiskfs_inode_cache
460725 133164  28%0.25K  30715   15122860K size-256
122440  65022  53%0.09K   3061   40 12244K buffer_head


-- 
Frederik Ferner
Computer Systems Administrator  phone: +44 1235 77 8624
Diamond Light Source Ltd.   mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS memory usage

[Bernd Schubert]

On Tuesday, August 24, 2010, Frederik Ferner wrote:
 Hi List,
 
 on our MDS we noticed that all memory seems to be used. (And it's not
 just normal buffers/cache as far as I can tell.)
 
 When we put load on the machine, for example by starting rsync
 on a few clients, generating file lists to copy data from Lustre to
 local disks or just running a MDT backup locally using dd/gzip to copy a
 LVM snapshot to a remote server, kswapd starts using a lot of CPU
 time, sometimes up to 100% of one CPU core.
 
 This is on a Lustre 1.6.7.2.ddn3.5 based file system with about 200TB,
 the MDT is 800GB with 200M inodes, ACLs enabled.

Did you recompile it, or did you use the binaries from my home page (or those 
you got from CV)?

Possibly it is a LRU auto-resize problem, but which has been disabled in DDN 
builds. As our 1.6 releases didn't include a patch for that, you would need to 
specify the correct command options if you recompiled it.

Another reason might be bug 22771, although that should only come up on MDS 
with more memory you have.


Cheers,
Bernd

-- 
Bernd Schubert
DataDirect Networks
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS high cpu usage

[Frans-Paul Ruzius]

Hello,

On the lustre system I have set up I use one MDS and 5 OSTs.
The MDS has almoast 100% CPU usage when I check top.
Sometimes the filesystem responds slow.
There are two processes running with over 35% cpu usage.

socknal_sd00 and socknal_sd01

Are these the main processes of the lfs?
Is it better to move the MDS to a faster node (with more CPU power)?
The MDS has now a 2Ghz dualcore AMD Athlon processor.

There area around 30 clients accessing the filesystem, somtimes all at once.

Regards,

Frans Ruzius
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS inode allocation question

[Gary Molenkamp]

Thanx for the details on the Inode number, but I'm still having an issue
where I'm not getting the number I expected from the MDS creation, but I
suspect its not a reporting error from lfs.

When I create the MDS, I specified '-i 1024' and I can see (locally)
800M inodes, but only part of the available space is allocated.  Also,
when the client mounts the filesystem,  the MDS only has 400M blocks
available:

gulfwork-MDT_UUID 430781784500264 3872740840% /gulfwork[MDT:0]

As we were creating files for testing, I saw that each inode allocation
on the MDS was consuming 4k of space, so even though I have 800M inodes
available on actual mds partition, it appears that the actual space
available was only allowing 100M inodes in the lustre fs.  Am I
understanding that correctly?

I tried to force the MDS creation to use a smaller size per inode but
that produced an error:

mkfs.lustre --fsname gulfwork --mdt --mgs --mkfsoptions='-i 1024 -I
1024' --reformat --failnode=10.18.12.1 /dev/sda
...
   mke2fs: inode_size (1024) * inodes_count (860148736) too big for a
filesystem with 215037184 blocks, specify higher inode_ratio
(-i) or lower inode count (-N).
...

yet the actual drive has many more blocks available:

SCSI device sda: 1720297472 512-byte hdwr sectors (880792 MB)

Is this ext4 setting the block size limit?


FYI, I am using:
  lustre-1.8.2-2.6.18_164.11.1.el5-ext4_lustre.1.8.2.x86_64.rpm
  lustre-ldiskfs-3.0.9-2.6.18_164.11.1.el5-ext4_lustre.1.8.2.x86_64.rpm
  e2fsprogs-1.41.6.sun1-0redhat.rhel5.x86_64.rpm




-- 
Gary Molenkamp  SHARCNET
Systems Administrator   University of Western Ontario
g...@sharcnet.cahttp://www.sharcnet.ca
(519) 661-2111 x88429   (519) 661-4000
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS inode allocation question

[Andreas Dilger]

On 2010-04-28, at 7:44, Gary Molenkamp g...@sharcnet.ca wrote:

 When I create the MDS, I specified '-i 1024' and I can see (locally)
 800M inodes, but only part of the available space is allocated.

This is to be expected. There needs to be free space on the MDS for  
directories, striping and other internal usage.

  Also, when the client mounts the filesystem,  the MDS only has 400M  
 blocks available:

 gulfwork-MDT_UUID 430781784500264 3872740840% /gulfwork 
 [MDT:0]

 As we were creating files for testing, I saw that each inode  
 allocation
 on the MDS was consuming 4k of space,

That depends on how you are striping your files.  If the striping is  
larger than will fit inside the inode (13 stripes for 512-byte inodes  
IIRC) them each inode will also consume a block for the striping, and  
some step-wise fraction of a block for each directory entry. That is  
why 'df -i' will return min(free blocks, free inodes), though the  
common case is that files do not need an external xattr block for the  
striping (see stripe hint argument for mkfs.lustre) and the number of  
'free' inodes will remain constant as files are being created, until  
the number of free blocks exceeds the free inode count.

 so even though I have 800M inodes available on actual mds partition,  
 it appears that the actual space available was only allowing 100M  
 inodes in the lustre fs.  Am I
 understanding that correctly?

Possibly, yes. If you are striping all files widely by default it can  
happen as you write.

 I tried to force the MDS creation to use a smaller size per inode but
 that produced an error:

 mkfs.lustre --fsname gulfwork --mdt --mgs --mkfsoptions='-i 1024 -I
 1024' --reformat --failnode=10.18.12.1 /dev/sda
 ...
   mke2fs: inode_size (1024) * inodes_count (860148736) too big for a
filesystem with 215037184 blocks, specify higher inode_ratio
(-i) or lower inode count (-N).
 ...

You can't fill the filesystem 100% full of inodes (1 inode per 1024  
bytes and each inode is 1024 bytes in size). If you ARE striping  
widely you may try -i 1536 -I 1024 but please make sure this is  
actually needed or it will reduce you MDS performance due to 2x larger  
inodes.

 yet the actual drive has many more blocks available:

 SCSI device sda: 1720297472 512-byte hdwr sectors (880792 MB)

 Is this ext4 setting the block size limit?


 FYI, I am using:
  lustre-1.8.2-2.6.18_164.11.1.el5-ext4_lustre.1.8.2.x86_64.rpm
  lustre-ldiskfs-3.0.9-2.6.18_164.11.1.el5-ext4_lustre.1.8.2.x86_64.rpm
  e2fsprogs-1.41.6.sun1-0redhat.rhel5.x86_64.rpm




 -- 
 Gary MolenkampSHARCNET
 Systems AdministratorUniversity of Western Ontario
 g...@sharcnet.cahttp://www.sharcnet.ca
 (519) 661-2111 x88429(519) 661-4000
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS inode allocation question

[Kevin Van Maren]

Andreas Dilger wrote:
 On 2010-04-23, at 13:30, Kevin Van Maren wrote:
   
 Not sure if it was fixed, but there was a bug in Lustre returning the 
 wrong values here.  If you create a bunch of files, the number of inodes 
 reported should go up until you get where you expect it to be.
 

 It depends what you mean by wrong values.  The number reported by df is 
 the number of new files you are guaranteed to be able to create at that time 
 in the filesystem in the worst case scenario.  The returned value is limited 
 by both the number of objects on the OSTs, as well as the number of blocks 
 (for wide striped files) on the MDT.  As the MDT filesystem has files created 
 in it, the number of files that can be created (i.e. IFree) will usually 
 stay constant, because the worst case is not the common case.

 Since we wanted IFree and IUsed to reflect the actual values, the 
 Inodes value had by necessity to be variable, because the Unix statfs() 
 interface only supplies Inodes and IFree, not IUsed.
   

By wrong value I meant that the return values from df -i show an 
increasing number of inodes as more files are created, which did not 
decrease when the files were removed (ie, I have more free inodes after 
creating and deleting a bunch of files than I had to start with).


 Note that the number of inodes on the OSTs also limits the number of 
 creatable files:
 each file requires an inodes on at least one OST (number depends on how 
 many OSTs each file is striped across).
 

 Right.  If you don't have enough OST objects, then you will never be able to 
 hit this limit.  However, it is relatively easy to add more OSTs if you ever 
 get close to running out of objects.  Most people run out of space first, but 
 then adding more OSTs for space also gives you proportionately more objects, 
 so the available objects are rarely the issue.

   
 Gary Molenkamp wrote:
 
 When creating the MDS filesystem, I used  '-i 1024' on a 860GB logical
 drive to provide approx 800M inodes in the lustre filesystem.  This was
 then verified with 'df -i' on the server:

  /dev/sda86016  130452 8600295481% /data/mds

 Later, after completing the OST creation and mounting the full
 filesystem on a client, I noticed that 'df -i' on the client mount is
 only showing 108M inodes in the lfs:

 10.18.1...@tcp:10.18.1...@tcp:/gulfwork
 107454606  130452 1073241541% /gulfwork

 A check with 'lfs df -i' shows the MDT only has 108M inodes:

 gulfwork-MDT_UUID 107454606130452 1073241540%
 /gulfwork[MDT:0]

 Is there a preallocation mechanism in play here, or did I miss something
 critical in the initial setup?  My concern is that modifications to the
 inodes are not reconfigurable, so it must be correct before the
 filesystem goes into production.

 FYI,  the filesystem was created with:

 MDS/MGS on 880G logical drive:
 mkfs.lustre --fsname gulfwork --mdt --mgs --mkfsoptions='-i 1024'
 --failnode=10.18.12.1 /dev/sda

 OSSs on 9.1TB logical drives:
 /usr/sbin/mkfs.lustre --fsname gulfwork --ost --mgsnode=10.18.1...@tcp
 --mgsnode=10.18.1...@tcp /dev/cciss/c0d0

 Thanks.


   
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
 


 Cheers, Andreas
 --
 Andreas Dilger
 Lustre Technical Lead
 Oracle Corporation Canada Inc.

   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS inode allocation question

[Gary Molenkamp]

When creating the MDS filesystem, I used  '-i 1024' on a 860GB logical
drive to provide approx 800M inodes in the lustre filesystem.  This was
then verified with 'df -i' on the server:

  /dev/sda86016  130452 8600295481% /data/mds

Later, after completing the OST creation and mounting the full
filesystem on a client, I noticed that 'df -i' on the client mount is
only showing 108M inodes in the lfs:

10.18.1...@tcp:10.18.1...@tcp:/gulfwork
 107454606  130452 1073241541% /gulfwork

A check with 'lfs df -i' shows the MDT only has 108M inodes:

gulfwork-MDT_UUID 107454606130452 1073241540%
/gulfwork[MDT:0]

Is there a preallocation mechanism in play here, or did I miss something
critical in the initial setup?  My concern is that modifications to the
inodes are not reconfigurable, so it must be correct before the
filesystem goes into production.

FYI,  the filesystem was created with:

MDS/MGS on 880G logical drive:
mkfs.lustre --fsname gulfwork --mdt --mgs --mkfsoptions='-i 1024'
--failnode=10.18.12.1 /dev/sda

OSSs on 9.1TB logical drives:
/usr/sbin/mkfs.lustre --fsname gulfwork --ost --mgsnode=10.18.1...@tcp
--mgsnode=10.18.1...@tcp /dev/cciss/c0d0

Thanks.

-- 
Gary Molenkamp  SHARCNET
Systems Administrator   University of Western Ontario
g...@sharcnet.cahttp://www.sharcnet.ca
(519) 661-2111 x88429   (519) 661-4000
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS inode allocation question

[Kevin Van Maren]

Not sure if it was fixed, but there was a bug in Lustre returning the 
wrong values here.
If you create a bunch of files, the number of inodes reported should go 
up until you get
where you expect it to be.

Note that the number of inodes on the OSTs also limits the number of 
creatable files:
each file requires an inodes on at least one OST (number depends on how 
many OSTs
each file is striped across).

Kevin


Gary Molenkamp wrote:
 When creating the MDS filesystem, I used  '-i 1024' on a 860GB logical
 drive to provide approx 800M inodes in the lustre filesystem.  This was
 then verified with 'df -i' on the server:

   /dev/sda86016  130452 8600295481% /data/mds

 Later, after completing the OST creation and mounting the full
 filesystem on a client, I noticed that 'df -i' on the client mount is
 only showing 108M inodes in the lfs:

 10.18.1...@tcp:10.18.1...@tcp:/gulfwork
  107454606  130452 1073241541% /gulfwork

 A check with 'lfs df -i' shows the MDT only has 108M inodes:

 gulfwork-MDT_UUID 107454606130452 1073241540%
   /gulfwork[MDT:0]

 Is there a preallocation mechanism in play here, or did I miss something
 critical in the initial setup?  My concern is that modifications to the
 inodes are not reconfigurable, so it must be correct before the
 filesystem goes into production.

 FYI,  the filesystem was created with:

 MDS/MGS on 880G logical drive:
 mkfs.lustre --fsname gulfwork --mdt --mgs --mkfsoptions='-i 1024'
   --failnode=10.18.12.1 /dev/sda

 OSSs on 9.1TB logical drives:
 /usr/sbin/mkfs.lustre --fsname gulfwork --ost --mgsnode=10.18.1...@tcp
   --mgsnode=10.18.1...@tcp /dev/cciss/c0d0

 Thanks.

   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Brian J. Murrell]

On Sun, 2010-01-24 at 22:54 -0700, Andreas Dilger wrote: 
 
 If they are call traces due to the watchdog timer, then this is somewhat
 expected for extremely high load.

Andreas,

Do you know, does adaptive timeouts take care of setting the timeout
appropriately on watchdogs?

b.



signature.asc
Description: This is a digitally signed message part
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Johann Lombardi]

On Mon, Jan 25, 2010 at 08:51:59AM -0500, Brian J. Murrell wrote:
 Do you know, does adaptive timeouts take care of setting the timeout
 appropriately on watchdogs?

Yes, the watchdog timer is updated based on the estimated rpc service
time (multiplied by a factor which is usually 2).

Johann
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Christopher J.Walker]

Brian J. Murrell wrote:
 On Sun, 2010-01-24 at 22:54 -0700, Andreas Dilger wrote: 
 If they are call traces due to the watchdog timer, then this is somewhat
 expected for extremely high load.
 
 Andreas,
 
 Do you know, does adaptive timeouts take care of setting the timeout
 appropriately on watchdogs?
 

I don't think this is quite what you are asking, but some details on our 
setup.

We have a mixture of 1.6.7.2 clients and 1.8.1.1 clients. The 1.6.7.2 
clients were not using adaptive timeouts when the problem occurred[1]. 
At least one of the 1.6 machines gets regularly swamped with network 
traffic - leading to packet loss.

It was 40 1.8.1.1 clients running updatedb that caused the problem.

Chris

[1] One machine is the interface to the outside world - and runs 
1.6.7.2. I see packet loss to this machine at times and have observed 
lustre  hanging for a while. I suspect the problem is that it is 
occasionally overloaded with network packets, lustre packets are then 
lost (probably at the router), followed by a timeout and recovery. I've 
now enabled adaptive timeouts on this machine - and will install a 
10GigE card too.
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Brian J. Murrell]

On Mon, 2010-01-25 at 15:09 +0100, Johann Lombardi wrote: 
 
 Yes, the watchdog timer is updated based on the estimated rpc service
 time (multiplied by a factor which is usually 2).

Ahhh.  Great.  It would be interesting to know which Lustre release the
poster seeing the stack traces was using.

b.



signature.asc
Description: This is a digitally signed message part
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Andreas Dilger]

On 2010-01-23, at 15:12, Christopher J. Walker wrote:
 I don't see an LBUG in my logs, but there are several Call Traces.  
 Would
 it be useful if I filed a bug too or I could add to David's bug if  
 you'd
 prefer  - if so, can you let me know the bug number as I can't find it
 in bugzilla.  Would you like /tmp/lustre-log.* too?


If they are call traces due to the watchdog timer, then this is somewhat
expected for extremely high load.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Christopher J. Walker]

Brian J. Murrell wrote:
 On Wed, 2010-01-06 at 11:25 +0200, David Cohen wrote: 
 I was indeed the *locate update, a simple edit of /etc/updatedb.conf on the 
 clients and the system is stable again.
 

I've just encountered the same thing - the mds crashing at the same time 
several times this week. It's just after the *locate update - I've added 
lustre to the excluded filesystems, and so far so good.

 Great.  But as Andreas said previously, load should not have caused the
 LBUG that you got.  Could you open a bug on our bugzilla about that?
 Please attach to that bug an excerpt from the tech-mds log that covers a
 window of time of 12h hours prior to the the LBUG and an hour after.
 

I don't see an LBUG in my logs, but there are several Call Traces. Would 
it be useful if I filed a bug too or I could add to David's bug if you'd 
prefer  - if so, can you let me know the bug number as I can't find it 
in bugzilla.  Would you like /tmp/lustre-log.* too?

Chris
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Andreas Dilger]

On 2010-01-06, at 04:25, David Cohen wrote:
 On Monday 04 January 2010 20:42:12 Andreas Dilger wrote:
 On 2010-01-04, at 03:02, David Cohen wrote:
 I'm using a mixed environment of 1.8.0.1 MDS and 1.6.6 OSS's (had a
 problem with qlogic drivers and rolled back to 1.6.6).
 My MDS get unresponsive each day at 4-5 am local time, no kernel
 panic or error messages before.

 I was indeed the *locate update, a simple edit of /etc/updatedb.conf  
 on the
 clients and the system is stable again.

I asked the upstream Fedora/RHEL maintainer of mlocate to add lustre  
to the exception list in updatedb.conf, and he has already done so for  
Fedora.  There is also a bug filed for RHEL5 to do the same, if anyone  
is interested in following it:

https://bugzilla.redhat.com/show_bug.cgi?id=557712

 Judging by the time, I'd guess this is slocate or mlocate running
 on all of your clients at the same time.  This used to be a source of
 extremely high load back in the old days, but I thought that Lustre
 was in the exclude list in newer versions of *locate.  Looking at the
 installed mlocate on my system, that doesn't seem to be the case...
 strange.

 Some errors and an LBUG appear in the log after force booting the
 MDS and
 mounting the MDT and then the log is clear until next morning:

 Jan  4 06:33:31 tech-mds kernel: LustreError: 6357:0:
 (class_hash.c:225:lustre_hash_findadd_unique_hnode())
 ASSERTION(hlist_unhashed(hnode)) failed
 Jan  4 06:33:31 tech-mds kernel: LustreError: 6357:0:
 (class_hash.c:225:lustre_hash_findadd_unique_hnode()) LBUG
 Jan  4 06:33:31 tech-mds kernel: Lustre: 6357:0:(linux-
 debug.c:222:libcfs_debug_dumpstack()) showing stack for process 6357
 Jan  4 06:33:31 tech-mds kernel: ll_mgs_02 R  running task
 0  6357
 16340 (L-TLB)
 Jan  4 06:33:31 tech-mds kernel: Call Trace:
 Jan  4 06:33:31 tech-mds kernel: thread_return+0x62/0xfe
 Jan  4 06:33:31 tech-mds kernel: __wake_up_common+0x3e/0x68
 Jan  4 06:33:31 tech-mds kernel: :ptlrpc:ptlrpc_main+0x1218/0x13e0
 Jan  4 06:33:31 tech-mds kernel: default_wake_function+0x0/0xe
 Jan  4 06:33:31 tech-mds kernel: audit_syscall_exit+0x31b/0x336
 Jan  4 06:33:31 tech-mds kernel: child_rip+0xa/0x11
 Jan  4 06:33:31 tech-mds kernel: :ptlrpc:ptlrpc_main+0x0/0x13e0
 Jan  4 06:33:31 tech-mds kernel: child_rip+0x0/0x11

 It shouldn't LBUG during recovery, however.

 Cheers, Andreas
 --
 Andreas Dilger
 Sr. Staff Engineer, Lustre Group
 Sun Microsystems of Canada, Inc.


 -- 
 David Cohen
 Grid Computing
 Physics Department
 Technion - Israel Institute of Technology
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[David Cohen]

On Monday 04 January 2010 20:42:12 Andreas Dilger wrote:
 On 2010-01-04, at 03:02, David Cohen wrote:
  I'm using a mixed environment of 1.8.0.1 MDS and 1.6.6 OSS's (had a
  problem
  with qlogic drivers and rolled back to 1.6.6).
  My MDS get unresponsive each day at 4-5 am local time, no kernel
  panic or
  error messages before.

I was indeed the *locate update, a simple edit of /etc/updatedb.conf on the 
clients and the system is stable again.
Many Thanks.


 
 Judging by the time, I'd guess this is slocate or mlocate running
 on all of your clients at the same time.  This used to be a source of
 extremely high load back in the old days, but I thought that Lustre
 was in the exclude list in newer versions of *locate.  Looking at the
 installed mlocate on my system, that doesn't seem to be the case...
 strange.
 
  Some errors and an LBUG appear in the log after force booting the
  MDS and
  mounting the MDT and then the log is clear until next morning:
 
  Jan  4 06:33:31 tech-mds kernel: LustreError: 6357:0:
  (class_hash.c:225:lustre_hash_findadd_unique_hnode())
  ASSERTION(hlist_unhashed(hnode)) failed
  Jan  4 06:33:31 tech-mds kernel: LustreError: 6357:0:
  (class_hash.c:225:lustre_hash_findadd_unique_hnode()) LBUG
  Jan  4 06:33:31 tech-mds kernel: Lustre: 6357:0:(linux-
  debug.c:222:libcfs_debug_dumpstack()) showing stack for process 6357
  Jan  4 06:33:31 tech-mds kernel: ll_mgs_02 R  running task
  0  6357
  16340 (L-TLB)
  Jan  4 06:33:31 tech-mds kernel: Call Trace:
  Jan  4 06:33:31 tech-mds kernel: thread_return+0x62/0xfe
  Jan  4 06:33:31 tech-mds kernel: __wake_up_common+0x3e/0x68
  Jan  4 06:33:31 tech-mds kernel: :ptlrpc:ptlrpc_main+0x1218/0x13e0
  Jan  4 06:33:31 tech-mds kernel: default_wake_function+0x0/0xe
  Jan  4 06:33:31 tech-mds kernel: audit_syscall_exit+0x31b/0x336
  Jan  4 06:33:31 tech-mds kernel: child_rip+0xa/0x11
  Jan  4 06:33:31 tech-mds kernel: :ptlrpc:ptlrpc_main+0x0/0x13e0
  Jan  4 06:33:31 tech-mds kernel: child_rip+0x0/0x11
 
 It shouldn't LBUG during recovery, however.
 
 Cheers, Andreas
 --
 Andreas Dilger
 Sr. Staff Engineer, Lustre Group
 Sun Microsystems of Canada, Inc.
 

-- 
David Cohen
Grid Computing
Physics Department
Technion - Israel Institute of Technology
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Brian J. Murrell]

On Wed, 2010-01-06 at 11:25 +0200, David Cohen wrote: 
 
 I was indeed the *locate update, a simple edit of /etc/updatedb.conf on the 
 clients and the system is stable again.

Great.  But as Andreas said previously, load should not have caused the
LBUG that you got.  Could you open a bug on our bugzilla about that?
Please attach to that bug an excerpt from the tech-mds log that covers a
window of time of 12h hours prior to the the LBUG and an hour after.

Thanx,
b.




signature.asc
Description: This is a digitally signed message part
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS crashes daily at the same hour

[David Cohen]

Hi,
I'm using a mixed environment of 1.8.0.1 MDS and 1.6.6 OSS's (had a problem 
with qlogic drivers and rolled back to 1.6.6).
My MDS get unresponsive each day at 4-5 am local time, no kernel panic or 
error messages before.
Some errors and an LBUG appear in the log after force booting the MDS and 
mounting the MDT and then the log is clear until next morning:

Jan  4 06:27:32 tech-mds kernel: LustreError: 6290:0:
(ldlm_lib.c:884:target_handle_connect()) technion-MDT: denying connection 
for new client 192.114.101...@tcp (ab671897-b1e2-76d3-b661-7b87e82d23e7): 34 
clients in recovery for 337s

  
Jan  4 06:27:32 tech-mds kernel: LustreError: 6290:0:
(ldlm_lib.c:1826:target_send_reply_msg()) @@@ processing error (-16)  
r...@81006f99cc00 x1323646107950586/t0 o38-?@?:0/0 lens 368/264 e 0 to 
0 
dl 1262579352 ref 1 fl Interpret:/0/0 rc -16/0  

  
Jan  4 06:27:41 tech-mds kernel: Lustre: 6280:0:
(ldlm_lib.c:1718:target_queue_last_replay_reply()) technion-MDT: 33 
recoverable clients remain   
Jan  4 06:27:57 tech-mds kernel: LustreError: 6284:0:
(ldlm_lib.c:884:target_handle_connect()) technion-MDT: denying connection 
for new client 192.114.101...@tcp (ab671897-b1e2-76d3-b661-7b87e82d23e7): 33 
clients in recovery for 312s

  
Jan  4 06:27:57 tech-mds kernel: LustreError: 6284:0:
(ldlm_lib.c:1826:target_send_reply_msg()) @@@ processing error (-16)  
r...@81011c69d400 x1323646107950600/t0 o38-?@?:0/0 lens 368/264 e 0 to 
0 
dl 1262579377 ref 1 fl Interpret:/0/0 rc -16/0  

  
Jan  4 06:28:22 tech-mds kernel: LustreError: 6302:0:
(ldlm_lib.c:884:target_handle_connect()) technion-MDT: denying connection 
for new client 192.114.101...@tcp (ab671897-b1e2-76d3-b661-7b87e82d23e7): 33 
clients in recovery for 287s

  
Jan  4 06:28:22 tech-mds kernel: LustreError: 6302:0:
(ldlm_lib.c:1826:target_send_reply_msg()) @@@ processing error (-16)  
r...@81006fa4e000 x1323646107950612/t0 o38-?@?:0/0 lens 368/264 e 0 to 
0 
dl 1262579402 ref 1 fl Interpret:/0/0 rc -16/0  

  
Jan  4 06:28:47 tech-mds kernel: LustreError: 6305:0:
(ldlm_lib.c:884:target_handle_connect()) technion-MDT: denying connection 
for new client 192.114.101...@tcp (ab671897-b1e2-76d3-b661-7b87e82d23e7): 33 
clients in recovery for 262s

  
Jan  4 06:28:47 tech-mds kernel: LustreError: 6305:0:
(ldlm_lib.c:1826:target_send_reply_msg()) @@@ processing error (-16)  
r...@81011c69d800 x1323646107950624/t0 o38-?@?:0/0 lens 368/264 e 0 to 
0 
dl 1262579427 ref 1 fl Interpret:/0/0 rc -16/0  

  
Jan  4 06:29:01 tech-mds ntpd[5999]: synchronized to 132.68.238.40, stratum 2   

 
Jan  4 06:29:01 tech-mds ntpd[5999]: kernel time sync enabled 0001  

 
Jan  4 06:29:12 tech-mds kernel: LustreError: 6278:0:
(ldlm_lib.c:884:target_handle_connect()) technion-MDT: denying connection 
for new client 192.114.101...@tcp (ab671897-b1e2-76d3-b661-7b87e82d23e7): 33 
clients in recovery for 237s

  
Jan  4 06:29:12 tech-mds kernel: LustreError: 6278:0:
(ldlm_lib.c:1826:target_send_reply_msg()) @@@ processing error (-16)  
r...@81007053ac00 x1323646107950636/t0 o38-?@?:0/0 lens 368/264 e 0 to 
0 
dl 1262579452 ref 1 fl Interpret:/0/0 rc -16/0  

  
Jan  4 06:29:37 tech-mds kernel: LustreError: 6293:0:
(ldlm_lib.c:884:target_handle_connect()) technion-MDT: denying connection 
for new client 192.114.101...@tcp (ab671897-b1e2-76d3-b661-7b87e82d23e7): 33 
clients in recovery for 212s
 

Re: [Lustre-discuss] MDS crashes daily at the same hour

[Andreas Dilger]

On 2010-01-04, at 03:02, David Cohen wrote:
 I'm using a mixed environment of 1.8.0.1 MDS and 1.6.6 OSS's (had a  
 problem
 with qlogic drivers and rolled back to 1.6.6).
 My MDS get unresponsive each day at 4-5 am local time, no kernel  
 panic or
 error messages before.

Judging by the time, I'd guess this is slocate or mlocate running  
on all of your clients at the same time.  This used to be a source of  
extremely high load back in the old days, but I thought that Lustre  
was in the exclude list in newer versions of *locate.  Looking at the  
installed mlocate on my system, that doesn't seem to be the case...   
strange.

 Some errors and an LBUG appear in the log after force booting the  
 MDS and
 mounting the MDT and then the log is clear until next morning:

 Jan  4 06:33:31 tech-mds kernel: LustreError: 6357:0:
 (class_hash.c:225:lustre_hash_findadd_unique_hnode())
 ASSERTION(hlist_unhashed(hnode)) failed
 Jan  4 06:33:31 tech-mds kernel: LustreError: 6357:0:
 (class_hash.c:225:lustre_hash_findadd_unique_hnode()) LBUG
 Jan  4 06:33:31 tech-mds kernel: Lustre: 6357:0:(linux-
 debug.c:222:libcfs_debug_dumpstack()) showing stack for process 6357
 Jan  4 06:33:31 tech-mds kernel: ll_mgs_02 R  running task
 0  6357
 16340 (L-TLB)
 Jan  4 06:33:31 tech-mds kernel: Call Trace:
 Jan  4 06:33:31 tech-mds kernel: thread_return+0x62/0xfe
 Jan  4 06:33:31 tech-mds kernel: __wake_up_common+0x3e/0x68
 Jan  4 06:33:31 tech-mds kernel: :ptlrpc:ptlrpc_main+0x1218/0x13e0
 Jan  4 06:33:31 tech-mds kernel: default_wake_function+0x0/0xe
 Jan  4 06:33:31 tech-mds kernel: audit_syscall_exit+0x31b/0x336
 Jan  4 06:33:31 tech-mds kernel: child_rip+0xa/0x11
 Jan  4 06:33:31 tech-mds kernel: :ptlrpc:ptlrpc_main+0x0/0x13e0
 Jan  4 06:33:31 tech-mds kernel: child_rip+0x0/0x11

It shouldn't LBUG during recovery, however.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS performance

[Francois Chassaing]

Hi list, 
I know this question has been asked zillion times, but I'll ask it again, 
because as most lustre-builder wannabe, I'm worried about MDS being a 
bottleneck. 
I'm planning to build a 60TB lustre install separated in two pools : 
- one big / slow / cheap 50 TB archive based on Dell MD3000i+MD1000 RAID 5 OST 
with two cheap OSS in front of it 
- one beefier (=smaller / faster / more expensive) 10 TB based on 2 servers 
being both OSS+OST each composed of 8 cores, 12 GB RAM, 8 NL SAS spindles RAID 
10. 
the big / slow archive is not really a problem but i'm asking myself about the 
smaller one, given the fact that this storage (if good enough in terms of 
performance) could double pretty soon by adding more OSS+OST 
I've been planning to have one redundant MDS over 2 Dell R610 sharing a common 
sas-attached MD3000 RAID 10 MDT. 
I've noticed that this array is not the best on the market for perf. but is 
rather good at $$... 
I plan on connecting all this on DDR infiniband (along with the main clients -4 
of them-). 

So now for my questions about MDS : 
- Should I do better having two lustre installs instead of a single install 
with two pools ? 
- Should I consider buying a better array for MDT ? 
- Should I be better using even beefier MDS with internal storage and DRBD ? 

and about OSS : 
- should I take smaller OSS+OST to improve perf ? 
- should I split my OST storage inside the OSSes for the 10GB storage pool ? 

I'll be sooo grateful if someone could answer those, I'll be glad to provide 
any other details one would need to help me out. 

Thanks. 
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] MDS doesn't switch to failover OST node

[Dam Thanh Tung]

Hi list

I am encountering a problem with OST-MDS connecting. Because of RAID card
hanging, our OST went down this morning and when i tried to mount the faill
over node of that OST, problem occurred :

MDS only sent request to the OST which was down and didn't connect to our
backup (failover) OST, so our backup solution was useless, we lost all data
from that OST. It's really a disaster for me because we even lost all of our
data before with the same kind of problem: OST can't connect to MDS 

We use drbd between OSTs to synchronize data. The backup (failover node) was
mounted successfully without any error but didn't have any client to recover
like this:

cat /proc/fs/lustre/obdfilter/lustre-OST0006/recovery_status
status: RECOVERING
recovery_start: 0
time_remaining: 0
connected_clients: 0/1
delayed_clients: 0/1
completed_clients: 0/1
replayed_requests: 0*/??*
queued_requests: 0
next_transno: 30064771073

In MDS's message log, we only saw the connection to our dead OST:

Nov 18 22:44:03 MDS1 kernel: Lustre: Request x1314965674069373 sent from
lustre-OST0006-osc to NID 192.168.1...@tcp 56s ago has timed out (limit
56s).
..

The output of* **lctl dl *command from MDS

lctl dl
  0 UP mgs MGS MGS 25
  1 UP mgc mgc192.168.1...@tcp 0681a267-849f-350c-5b2c-6869c794550f 5
  2 UP mdt MDS MDS_uuid 3
  3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
  4 UP mds lustre-MDT lustre-MDT_UUID 15
  5 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
  6 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
  7 IN osc lustre-OST0006-osc lustre-mdtlov_UUID 5
  8 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
  9 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5

I did activated OST6 ( lctl --device 7 activate ) but it couldn't help



Could anyone tell me how to route MDS to connect to our backup OST ( with ip
address 192.168.1.67 , for example ) ? , to bring our OST up ?

Any help would be really appreciated !

Hope that i can receive your answers or suggestions as soon as possible

Best Regards
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss