[lustre-discuss] Error on a zpool underlying an OST

2016-07-11 Thread Kevin Abbey 

Hi,

Can anyone advise how to clean up 1000s of zfs level permanent errors 
and the lustre level too?


A similar question was presented on the list but I did not see an answer.
https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg12454.html

As I was testing new hardware I discovered an LSI HBA was bad.  On a 
single combined MDS/OSS there were 8 OSTs split across 2 jbod and 2 LSI 
HBA.  The mdt was on a 3rd jbod downlinked on the jbod connected with 
the bad controller.  The zpools connected to the good HBA were scrubed 
clean after unmounting and stopping lustre.  The zpools on the bad 
controller continued to have errors while connected to the bad 
controller.  One of these OSTs reported a disk failure during the scrub 
and began resilvering yet autoreplace was off.This is a very bad 
event considering the card was causing all of the errors.  Neither a 
scrub or resilver would ever complete.  I stopped the scrub on the 3 
other osts and detached the spare from the ost in resilver process.  
After narrowing down the bad HBA (initially it was not clear if cables 
or jbod backplanes were bad), I use the good HBA to scrub the jbod 1 
again, then shutdown disconnected the jbod1.  Then proceeded to connect 
the jbod2 to the good controller to scrub the jbod 2 zpools which had 
previously been attached to the bad LSI controller.  The 3 zpools which 
had scrub stopped previously did complete successfully.  The one which 
had begun resilvering began again to resilver after I initiated a 
replace of the failed disk with the spare.  The resilver completed but 
many permanent errors were discovered on the zpool.  Since this is a 
test pool I was interested to know if zfs would recover.  In a real 
scenario with HW problems I'll shutdown and disconnect the data drives 
prior to HW testing.


The status listed below shows a new scrub in process after the resilver 
completed.  The cache drive is missing because the 3rd jbod is 
disconnected temporarily.



===

ZFS:   v0.6.5.7-1
lustre 2.8.55
kernel 2.6.32_642.1.1.el6.x86_64.x86_64
Centos 6.8


===
  ~]# zpool status -v test-ost4
  pool: test-ost4
 state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption.  Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub in progress since Mon Jul 11 22:29:09 2016
689G scanned out of 12.4T at 711M/s, 4h49m to go
40K repaired, 5.41% done
config:

NAME   STATE READ WRITE CKSUM
test-ost4  ONLINE 0 0   180
  raidz2-0 ONLINE 0 0   360
ata-ST4000NM0033-9ZM170_Z1Z7GYXY   ONLINE 0 0 2  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7KKPQ   ONLINE 0 0 3  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7L5E7   ONLINE 0 0 3  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7KGQT   ONLINE 0 0 0  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7LA8K   ONLINE 0 0 4  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7KB0X   ONLINE 0 0 3  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7JSMN   ONLINE 0 0 2  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7KXRA   ONLINE 0 0 2  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7MLSN   ONLINE 0 0 2  
(repairing)
ata-ST4000NM0033-9ZM170_Z1Z7L4DT   ONLINE 0 0 7  
(repairing)

cache
  ata-D2CSTK251M20-0240_A19CV01122792  UNAVAIL 0 0 0

errors: Permanent errors have been detected in the following files:

test-ost4/test-ost4:<0xe00>
test-ost4/test-ost4:<0xe01>
test-ost4/test-ost4:<0xe02>
test-ost4/test-ost4:<0xe03>
test-ost4/test-ost4:<0xe04>
test-ost4/test-ost4:<0xe05>
test-ost4/test-ost4:<0xe06>...
...
...continues..
...
...
test-ost4/test-ost4:<0xdfe>
test-ost4/test-ost4:<0xdff>
===

Follow up questions,

Is is better to not have a spare attached to the pool to prevent 
resilvering in this scenario?  (bad HBA, disk failed during scrub, 
resilver began, yet auto relplace was off.  The spare was assigned to 
the zpool.)


In a dual path to the jbod would the bad HBA card be disabled 
automatically to prevent IO errors reaching the disk?  The current setup 
is single path only.



Thank you for any notes in advance,
Kevin

--
Kevin Abbey
Systems Administrator
Center for Computational and Integrative Biology (CCIB)
http://ccib.camden.rutgers.edu/

Rutgers University - Science Building
315 Penn St.
Camden, NJ 08102
Telephone: (856) 225-6770
Fax:(856) 225-6312
Email: kevin.ab...@rutgers.edu

Re: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

2016-07-11 Thread Riccardo Veraldi 

On 11/07/16 17:02, Faaland, Olaf P. wrote:

Riccardo,

If you are not booting from a zpool, you do not need the "zfs-dracut" package.  
This package causes ZFS to be loaded very early in the boot process, most likely before 
your /etc/modprobe.d/zfs.conf files is visible to the kernel.

As long as you are not booting from a zpool, remove that package (rpm -e) and 
boot again to see if your problem is fixed.

I did it I removed zfs-dracut since I do not boot from ZFS.

 cat /etc/modprobe.d/zfs.conf
options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug_unload=1

after reboot

cat /sys/module/zfs/parameters/zfs_prefetch_disable
0

cat /sys/module/zfs/parameters/zfs_txg_history
0

so it looks like the parameters inside zfs.conf are ignored

ls -la /etc/modprobe.d/zfs.conf
-rw-r--r-- 1 root root 103 Jul 11 17:18 /etc/modprobe.d/zfs.conf




-Olaf


From: Riccardo Veraldi [riccardo.vera...@cnaf.infn.it]
Sent: Monday, July 11, 2016 4:55 PM
To: Faaland, Olaf P.; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

On 11/07/16 16:15, Faaland, Olaf P. wrote:

1) What is the output of:

rpm -qa | grep zfs

libzfs2-0.6.5.7-1.el7.centos.x86_64
zfs-dkms-0.6.5.7-1.el7.centos.noarch
lustre-osd-zfs-mount-2.8.0-3.10.0_327.18.2.el7.x86_64.x86_64
zfs-0.6.5.7-1.el7.centos.x86_64
zfs-dracut-0.6.5.7-1.el7.centos.x86_64
lustre-osd-zfs-2.8.0-3.10.0_327.18.2.el7.x86_64.x86_64

from that system after it boots?

2) How do those values get into the /etc/modprobe.d/zfs.conf file?  Are they 
there before the node boots, or are modifying that file somehow during the boot 
process?

I edited those values myself inside zfs.conf then rebooted the system,
but they are not loaded

thank you

Riccardo



Olaf P. Faaland
Livermore Computing

From: lustre-discuss [lustre-discuss-boun...@lists.lustre.org] on behalf of 
Riccardo Veraldi [riccardo.vera...@cnaf.infn.it]
Sent: Monday, July 11, 2016 3:53 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

Hello,
I am tailoring my system for lustre on ZFS and I am not able to set
these parameters
writing the config file /etc/modprobe.d/zfs.conf with the following options

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug:unload=1

when I check the parameters after boot in /sys/module/zfs/parameters
they are still the defaults, and my modification are not taken.

zfs.conf is being ignored at boot time I do not know why.

I had to do a

echo 1 > /sys/module/zfs/parameters/options zfs zfs_prefetch_disable

inside rc.local

to have it working

I am running RHEL7 with latest kernel  and zfs 0.6.5.7-1

any hints ?

thank you

Riccardo

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

2016-07-11 Thread Faaland, Olaf P. 
Riccardo,

If you are not booting from a zpool, you do not need the "zfs-dracut" package.  
This package causes ZFS to be loaded very early in the boot process, most 
likely before your /etc/modprobe.d/zfs.conf files is visible to the kernel.

As long as you are not booting from a zpool, remove that package (rpm -e) and 
boot again to see if your problem is fixed.

-Olaf


From: Riccardo Veraldi [riccardo.vera...@cnaf.infn.it]
Sent: Monday, July 11, 2016 4:55 PM
To: Faaland, Olaf P.; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

On 11/07/16 16:15, Faaland, Olaf P. wrote:
> 1) What is the output of:
>
> rpm -qa | grep zfs
libzfs2-0.6.5.7-1.el7.centos.x86_64
zfs-dkms-0.6.5.7-1.el7.centos.noarch
lustre-osd-zfs-mount-2.8.0-3.10.0_327.18.2.el7.x86_64.x86_64
zfs-0.6.5.7-1.el7.centos.x86_64
zfs-dracut-0.6.5.7-1.el7.centos.x86_64
lustre-osd-zfs-2.8.0-3.10.0_327.18.2.el7.x86_64.x86_64
>
> from that system after it boots?
>
> 2) How do those values get into the /etc/modprobe.d/zfs.conf file?  Are they 
> there before the node boots, or are modifying that file somehow during the 
> boot process?
I edited those values myself inside zfs.conf then rebooted the system,
but they are not loaded

thank you

Riccardo


>
> Olaf P. Faaland
> Livermore Computing
> 
> From: lustre-discuss [lustre-discuss-boun...@lists.lustre.org] on behalf of 
> Riccardo Veraldi [riccardo.vera...@cnaf.infn.it]
> Sent: Monday, July 11, 2016 3:53 PM
> To: lustre-discuss@lists.lustre.org
> Subject: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs
>
> Hello,
> I am tailoring my system for lustre on ZFS and I am not able to set
> these parameters
> writing the config file /etc/modprobe.d/zfs.conf with the following options
>
> options zfs zfs_prefetch_disable=1
> options zfs zfs_txg_history=120
> options zfs metaslab_debug:unload=1
>
> when I check the parameters after boot in /sys/module/zfs/parameters
> they are still the defaults, and my modification are not taken.
>
> zfs.conf is being ignored at boot time I do not know why.
>
> I had to do a
>
> echo 1 > /sys/module/zfs/parameters/options zfs zfs_prefetch_disable
>
> inside rc.local
>
> to have it working
>
> I am running RHEL7 with latest kernel  and zfs 0.6.5.7-1
>
> any hints ?
>
> thank you
>
> Riccardo
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

2016-07-11 Thread Riccardo Veraldi 

On 11/07/16 16:15, Faaland, Olaf P. wrote:

1) What is the output of:

rpm -qa | grep zfs

libzfs2-0.6.5.7-1.el7.centos.x86_64
zfs-dkms-0.6.5.7-1.el7.centos.noarch
lustre-osd-zfs-mount-2.8.0-3.10.0_327.18.2.el7.x86_64.x86_64
zfs-0.6.5.7-1.el7.centos.x86_64
zfs-dracut-0.6.5.7-1.el7.centos.x86_64
lustre-osd-zfs-2.8.0-3.10.0_327.18.2.el7.x86_64.x86_64


from that system after it boots?

2) How do those values get into the /etc/modprobe.d/zfs.conf file?  Are they 
there before the node boots, or are modifying that file somehow during the boot 
process?
I edited those values myself inside zfs.conf then rebooted the system, 
but they are not loaded


thank you

Riccardo




Olaf P. Faaland
Livermore Computing

From: lustre-discuss [lustre-discuss-boun...@lists.lustre.org] on behalf of 
Riccardo Veraldi [riccardo.vera...@cnaf.infn.it]
Sent: Monday, July 11, 2016 3:53 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

Hello,
I am tailoring my system for lustre on ZFS and I am not able to set
these parameters
writing the config file /etc/modprobe.d/zfs.conf with the following options

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug:unload=1

when I check the parameters after boot in /sys/module/zfs/parameters
they are still the defaults, and my modification are not taken.

zfs.conf is being ignored at boot time I do not know why.

I had to do a

echo 1 > /sys/module/zfs/parameters/options zfs zfs_prefetch_disable

inside rc.local

to have it working

I am running RHEL7 with latest kernel  and zfs 0.6.5.7-1

any hints ?

thank you

Riccardo

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

2016-07-11 Thread Faaland, Olaf P. 
1) What is the output of:

rpm -qa | grep zfs

from that system after it boots?

2) How do those values get into the /etc/modprobe.d/zfs.conf file?  Are they 
there before the node boots, or are modifying that file somehow during the boot 
process?

Olaf P. Faaland
Livermore Computing

From: lustre-discuss [lustre-discuss-boun...@lists.lustre.org] on behalf of 
Riccardo Veraldi [riccardo.vera...@cnaf.infn.it]
Sent: Monday, July 11, 2016 3:53 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] problem loading zfs properties for a lustre/zfs fs

Hello,
I am tailoring my system for lustre on ZFS and I am not able to set
these parameters
writing the config file /etc/modprobe.d/zfs.conf with the following options

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug:unload=1

when I check the parameters after boot in /sys/module/zfs/parameters
they are still the defaults, and my modification are not taken.

zfs.conf is being ignored at boot time I do not know why.

I had to do a

echo 1 > /sys/module/zfs/parameters/options zfs zfs_prefetch_disable

inside rc.local

to have it working

I am running RHEL7 with latest kernel  and zfs 0.6.5.7-1

any hints ?

thank you

Riccardo

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] problem loading zfs properties for a lustre/zfs fs

2016-07-11 Thread Riccardo Veraldi 

Hello,
I am tailoring my system for lustre on ZFS and I am not able to set 
these parameters

writing the config file /etc/modprobe.d/zfs.conf with the following options

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug:unload=1

when I check the parameters after boot in /sys/module/zfs/parameters
they are still the defaults, and my modification are not taken.

zfs.conf is being ignored at boot time I do not know why.

I had to do a

echo 1 > /sys/module/zfs/parameters/options zfs zfs_prefetch_disable

inside rc.local

to have it working

I am running RHEL7 with latest kernel  and zfs 0.6.5.7-1

any hints ?

thank you

Riccardo

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] difficulties mounting client via an lnet router--SOLVED!

2016-07-11 Thread Jessica Otey 

All,

Thanks to the repliers who contributed to the solution. Here's a rundown:

First, here's a way to see if you have connectivity via the router 
between client and mdt, etc., using NIDS (to list NIDS, use lctl list_nids)


ltcl ping 

If the ping between client and the mdt works, you have connectivity. And 
indeed, in our case, the problem wasn't router configs or connectivity, 
but rather our lustre filesystem, which needed a --writeconf because the 
client had previously connected without the router (meaning directly via 
tcp rather than o2ib).


On 07/11/2016 12:11 PM, Oucharek, Doug S wrote:
The router is the not the issue and would be working fine “if” the 
file system were using the correct NID.  So, for example, before 
having an IB network for the servers, I suspect you had an MDT 
accessed via NID: 10.7.29.130@tcp.  When moving it to the IB network, 
it should become something like: 10.7.129.130@o2ib0.  The file system 
configuration will still try to get the clients to use 10.7.29.130@tcp 
until it is updated to the new NID.  But I can say that your LNet 
configurations are correct and will work once the file system starts 
using the correct NIDs. 


Here's how to do that (from this link: 
http://wiki.old.lustre.org/manual/LustreManual20_HTML/LustreMaintenance.html#50438199_31353) 



Section 14.5 Changing a Server NID
To change a server NID:

1.Update the LNET configuration in the /etc/modprobe.conf file so 
the list of server NIDs (lctl list_nids) is correct.The lctl list_nids 
command indicates which network(s) are configured to work with Lustre.

2.Shut down the file system in this order:
a.Unmount the clients.
b. Unmount the MDT.
c. Unmount all OSTs.
3.Run the writeconf command on all servers.
Run writeconf on the MDT first, and then the OSTs.
a.On the MDT, run:
$ tunefs.lustre --writeconf 
b. On each OST, run:
$ tunefs.lustre --writeconf 
c. If the NID on the MGS was changed, communicate the new MGS location 
to each server. Run:

tunefs.lustre --erase-param --mgsnode= --writeconf /dev/..
4.Restart the file system in this order:
a.Mount the MGS (or the combined MGS/MDT).
b. Mount the MDT.
c. Mount the OSTs.
d. Mount the clients.
After the writeconf command is run, the configuration logs are 
re-generated as servers restart, and server NIDs in the updated 
list_nids file are used.


This worked for us, and we were at last able to mount the client via the 
router!


Thanks lustre experts for being there!!!

Jessica

On 07/11/2016 10:34 AM, Jessica Otey wrote:

All,
I am, as before, working on a small test lustre setup (RHEL 6.8, 
lustre v. 2.4.3) to prepare for upgrading at 1.8.9 lustre production 
system to 2.4.3 (first the servers and lnet routers, then at a 
subsequent time, the clients). Lustre servers have IB connections, but 
the clients are 1G ethernet only.


For the life of me, I cannot get the client to mount via the router on 
this test system. (Client will mount fine when router is taken out of 
the equation.) This is the error I am seeing in the syslog from the 
mount attempt:


Jul 11 10:15:37 tlclient kernel: Lustre: 
3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent 
has timed out for slow reply: [sent 1468246532/real 1468246532]  
req@88032a3f9400 x1539566484848752/t0(0) 
o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 
400/544 e 0 to 1 dl 1468246537 ref 1 fl Rpc:XN/0/ rc 0/-1
Jul 11 10:16:07 tlclient kernel: Lustre: 
3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent 
has timed out for slow reply: [sent 1468246557/real 1468246557]  
req@880629819000 x1539566484848764/t0(0) 
o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 
400/544 e 0 to 1 dl 1468246567 ref 1 fl Rpc:XN/0/ rc 0/-1
Jul 11 10:16:37 tlclient kernel: Lustre: 
3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent 
has timed out for slow reply: [sent 1468246582/real 1468246582]  
req@88062a371000 x1539566484848772/t0(0) 
o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 
400/544 e 0 to 1 dl 1468246597 ref 1 fl Rpc:XN/0/ rc 0/-1
Jul 11 10:16:44 tlclient kernel: LustreError: 
2511:0:(lov_obd.c:937:lov_cleanup()) lov tgt 0 not cleaned! 
deathrow=0, lovrc=1

Jul 11 10:16:44 tlclient kernel: Lustre: Unmounted tlustre-client
Jul 11 10:16:44 tlclient kernel: LustreError: 
4881:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount (-4)


More than one pair of eyes has looked at the configs and confirmed 
they look okay. But frankly we've got to be missing something since 
this should (like lustre on a good day) 'just work'.


If anyone has seen this issue before and could give some advice, it'd 
be appreciated. One major question I have is whether the problem is a 
configuration issue or a procedure issue--perhaps the order in which I 
am doing things is causing the failure? The order I'm following 
currently is:


1) unmount/remove

Re: [lustre-discuss] difficulties mounting client via an lnet router

2016-07-11 Thread Oucharek, Doug S 
You mentioned that the servers are on the o2ib0 network, but the error messages 
indicate that the client is trying to communicate with the MDT on the tcp 
network.   The file system configuration needs to be updated to use the updated 
NIDs.  

Doug

> On Jul 11, 2016, at 7:34 AM, Jessica Otey  wrote:
> 
> All,
> I am, as before, working on a small test lustre setup (RHEL 6.8, lustre v. 
> 2.4.3) to prepare for upgrading at 1.8.9 lustre production system to 2.4.3 
> (first the servers and lnet routers, then at a subsequent time, the clients). 
> Lustre servers have IB connections, but the clients are 1G ethernet only.
> 
> For the life of me, I cannot get the client to mount via the router on this 
> test system. (Client will mount fine when router is taken out of the 
> equation.) This is the error I am seeing in the syslog from the mount attempt:
> 
> Jul 11 10:15:37 tlclient kernel: Lustre: 
> 3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed 
> out for slow reply: [sent 1468246532/real 1468246532]  req@88032a3f9400 
> x1539566484848752/t0(0) 
> o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 400/544 
> e 0 to 1 dl 1468246537 ref 1 fl Rpc:XN/0/ rc 0/-1
> Jul 11 10:16:07 tlclient kernel: Lustre: 
> 3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed 
> out for slow reply: [sent 1468246557/real 1468246557]  req@880629819000 
> x1539566484848764/t0(0) 
> o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 400/544 
> e 0 to 1 dl 1468246567 ref 1 fl Rpc:XN/0/ rc 0/-1
> Jul 11 10:16:37 tlclient kernel: Lustre: 
> 3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed 
> out for slow reply: [sent 1468246582/real 1468246582]  req@88062a371000 
> x1539566484848772/t0(0) 
> o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 400/544 
> e 0 to 1 dl 1468246597 ref 1 fl Rpc:XN/0/ rc 0/-1
> Jul 11 10:16:44 tlclient kernel: LustreError: 
> 2511:0:(lov_obd.c:937:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, 
> lovrc=1
> Jul 11 10:16:44 tlclient kernel: Lustre: Unmounted tlustre-client
> Jul 11 10:16:44 tlclient kernel: LustreError: 
> 4881:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount (-4)
> 
> More than one pair of eyes has looked at the configs and confirmed they look 
> okay. But frankly we've got to be missing something since this should (like 
> lustre on a good day) 'just work'.
> 
> If anyone has seen this issue before and could give some advice, it'd be 
> appreciated. One major question I have is whether the problem is a 
> configuration issue or a procedure issue--perhaps the order in which I am 
> doing things is causing the failure? The order I'm following currently is:
> 
> 1) unmount/remove modules on all boxes
> 2) bring up the lnet modules on the router, and bring up the network
> 3) On the mds: add the modules, bring up the network, mount the mdt
> 4) On the oss: add the modules, bring up the network, mount the oss
> 5) On the client: add the modules, bring up the network, attempt to mount 
> client (fails)
> 
> Configs follow below.
> 
> Thanks in advance,
> Jessica
> 
> tlnet (the router)
> [root@tlnet ~]# cat /etc/modprobe.d/lustre.conf
> # tlnet configuration
> alias ib0 ib_ipoib
> alias net-pf-27 ib_sdp
> options lnet networks="o2ib0(ib0),tcp0(em1)" forwarding="enabled"
> 
> [root@tlnet ~]# ifconfig #lo omitted
> em1   Link encap:Ethernet  HWaddr 78:2B:CB:25:A7:E2
>  inet addr:10.7.29.134  Bcast:10.7.29.255 Mask:255.255.255.0
>  UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1
>  RX packets:453441 errors:0 dropped:0 overruns:0 frame:0
>  TX packets:264313 errors:0 dropped:0 overruns:0 carrier:0
>  collisions:0 txqueuelen:1000
>  RX bytes:436188202 (415.9 MiB)  TX bytes:22274957 (21.2 MiB)
> ib0   Link encap:InfiniBand  HWaddr 
> 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
>  inet addr:10.7.129.134  Bcast:10.7.129.255 Mask:255.255.255.0
>  UP BROADCAST RUNNING MULTICAST  MTU:2044 Metric:1
>  RX packets:650 errors:0 dropped:0 overruns:0 frame:0
>  TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
>  collisions:0 txqueuelen:256
>  RX bytes:75376 (73.6 KiB)  TX bytes:2904 (2.8 KiB)
> 
> tlclient (the client)
> [root@tlclient ~]# cat /etc/modprobe.d/lustre.conf
> options lnet networks="tcp0(em1)" routes="o2ib0 10.7.29.134@tcp0" 
> live_router_check_interval=60 dead_router_check_interval=60
> 
> [root@tlclient ~]# ifconfig #lo omitted
> em1   Link encap:Ethernet  HWaddr 00:26:B9:35:B1:1A
>  inet addr:10.7.29.132  Bcast:10.7.29.255 Mask:255.255.255.0
>  UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1
>  RX packets:2817 errors:0 dropped:0 overruns:0 frame:0
>  TX packets:2233 errors:0 dropped:0 overruns:0 carrier:0
>  collisions:0

[lustre-discuss] Understanding quotas

2016-07-11 Thread Gibbins, Faye 
Hi,

Could someone please help me understand lustre quotas?

I've created a lustre filesystem called "scratch" using RHEL 7.2 and Lustre 
2.8. When I run "lctl get_param qmt.scratch-QMT.dt-0x0.*" on the MDT I see 
for my ID the following:

- id:  20977
  limits:  { hard:6, soft:5, granted:   
 117440512, time:0 }

Can someone tell me what the "granted" field is representing? I know it's not 
my used quota, that's only 86M and this value is nearer 112G.

Yours
Faye Gibbins

Snr Systems Administrator, Unix Lead Architect (Software)
Cirrus Logic | cirrus.com | +44 131 272 7398
[cid:image003.png@01CFC5F0.FE149B60]



This message and any attachments may contain privileged and confidential 
information that is intended solely for the person(s) to whom it is addressed. 
If you are not an intended recipient you must not: read; copy; distribute; 
discuss; take any action in or make any reliance upon the contents of this 
message; nor open or read any attachment. If you have received this message in 
error, please notify us as soon as possible on the following telephone number 
and destroy this message including any attachments. Thank you. Cirrus Logic 
International (UK) Ltd and Cirrus Logic International Semiconductor Ltd are 
companies registered in Scotland, with registered numbers SC089839 and SC495735 
respectively. Our registered office is at Westfield House, 26 Westfield Road, 
Edinburgh, EH11 2QB, UK. Tel: +44 (0)131 272 7000. cirrus.com
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] difficulties mounting client via an lnet router

2016-07-11 Thread Jessica Otey 

All,
I am, as before, working on a small test lustre setup (RHEL 6.8, lustre 
v. 2.4.3) to prepare for upgrading at 1.8.9 lustre production system to 
2.4.3 (first the servers and lnet routers, then at a subsequent time, 
the clients). Lustre servers have IB connections, but the clients are 1G 
ethernet only.


For the life of me, I cannot get the client to mount via the router on 
this test system. (Client will mount fine when router is taken out of 
the equation.) This is the error I am seeing in the syslog from the 
mount attempt:


Jul 11 10:15:37 tlclient kernel: Lustre: 
3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has 
timed out for slow reply: [sent 1468246532/real 1468246532]  
req@88032a3f9400 x1539566484848752/t0(0) 
o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 
400/544 e 0 to 1 dl 1468246537 ref 1 fl Rpc:XN/0/ rc 0/-1
Jul 11 10:16:07 tlclient kernel: Lustre: 
3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has 
timed out for slow reply: [sent 1468246557/real 1468246557]  
req@880629819000 x1539566484848764/t0(0) 
o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 
400/544 e 0 to 1 dl 1468246567 ref 1 fl Rpc:XN/0/ rc 0/-1
Jul 11 10:16:37 tlclient kernel: Lustre: 
3605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has 
timed out for slow reply: [sent 1468246582/real 1468246582]  
req@88062a371000 x1539566484848772/t0(0) 
o38->tlustre-MDT-mdc-88032ad20400@10.7.29.130@tcp:12/10 lens 
400/544 e 0 to 1 dl 1468246597 ref 1 fl Rpc:XN/0/ rc 0/-1
Jul 11 10:16:44 tlclient kernel: LustreError: 
2511:0:(lov_obd.c:937:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, 
lovrc=1

Jul 11 10:16:44 tlclient kernel: Lustre: Unmounted tlustre-client
Jul 11 10:16:44 tlclient kernel: LustreError: 
4881:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount (-4)


More than one pair of eyes has looked at the configs and confirmed they 
look okay. But frankly we've got to be missing something since this 
should (like lustre on a good day) 'just work'.


If anyone has seen this issue before and could give some advice, it'd be 
appreciated. One major question I have is whether the problem is a 
configuration issue or a procedure issue--perhaps the order in which I 
am doing things is causing the failure? The order I'm following 
currently is:


1) unmount/remove modules on all boxes
2) bring up the lnet modules on the router, and bring up the network
3) On the mds: add the modules, bring up the network, mount the mdt
4) On the oss: add the modules, bring up the network, mount the oss
5) On the client: add the modules, bring up the network, attempt to 
mount client (fails)


Configs follow below.

Thanks in advance,
Jessica

tlnet (the router)
[root@tlnet ~]# cat /etc/modprobe.d/lustre.conf
# tlnet configuration
alias ib0 ib_ipoib
alias net-pf-27 ib_sdp
options lnet networks="o2ib0(ib0),tcp0(em1)" forwarding="enabled"

[root@tlnet ~]# ifconfig #lo omitted
em1   Link encap:Ethernet  HWaddr 78:2B:CB:25:A7:E2
  inet addr:10.7.29.134  Bcast:10.7.29.255 Mask:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1
  RX packets:453441 errors:0 dropped:0 overruns:0 frame:0
  TX packets:264313 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:436188202 (415.9 MiB)  TX bytes:22274957 (21.2 MiB)
ib0   Link encap:InfiniBand  HWaddr 
80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00

  inet addr:10.7.129.134  Bcast:10.7.129.255 Mask:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:2044 Metric:1
  RX packets:650 errors:0 dropped:0 overruns:0 frame:0
  TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:256
  RX bytes:75376 (73.6 KiB)  TX bytes:2904 (2.8 KiB)

tlclient (the client)
[root@tlclient ~]# cat /etc/modprobe.d/lustre.conf
options lnet networks="tcp0(em1)" routes="o2ib0 10.7.29.134@tcp0" 
live_router_check_interval=60 dead_router_check_interval=60


[root@tlclient ~]# ifconfig #lo omitted
em1   Link encap:Ethernet  HWaddr 00:26:B9:35:B1:1A
  inet addr:10.7.29.132  Bcast:10.7.29.255 Mask:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1
  RX packets:2817 errors:0 dropped:0 overruns:0 frame:0
  TX packets:2233 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:354856 (346.5 KiB)  TX bytes:328782 (321.0 KiB)

[root@tlclient ~]# cat /etc/fstab | grep lustre
10.7.129.130@o2ib0:/tlustre/testlustrelustre 
defaults,noauto,user_xattr,flock  0 0


tlmds/tloss (mdt and oss)
[root@tloss ~]# cat /etc/modprobe.d/lustre.conf
alias ib0 ib_ipoib
alias net-pf-27 ib_sdp
options lnet networks="o2ib0(ib0)" routes="tcp0 10.7.129.134@o2ib0" 
live_router_check_interval="60" dead_router_check_interval="60"


tloss

Re: [lustre-discuss] Is live upgrade of 2.4 to 2.5 unproblematic?

2016-07-11 Thread Peter Bortas 
Hi Patrick,

Thanks for the additional input! I'll skip the exiting live upgrade
this time then.

Regards,
-- 
Peter Bortas, NSC

On Mon, Jul 11, 2016 at 1:39 AM, Patrick Farrell  wrote:
> Because of the issue highlighted by Andreas - a great number of possible
> states when a job is running - Cray does our upgrades with the system quiet.
> Live upgrades aren't something we even consider - The potential damage is
> too large for the time saved.  Especially since the actual *upgrade* usually
> doesn't take very long at all, generally speaking.  For 2.4 to 2.5, the
> 'clean' version is just stop activity to the filesystem, unmount it on
> clients, stop it/unmount it server side, install the new Lustre RPMs, and
> start it up again.  This is relatively quick.
>
> 
> From: lustre-discuss  on behalf of
> Dilger, Andreas 
> Sent: Sunday, July 10, 2016 5:53:38 PM
> To: Peter Bortas
> Cc: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] Is live upgrade of 2.4 to 2.5 unproblematic?
>
> We typically test 2.x->2.x+1 upgrades, both live and offline, for every
> version of Lustre. That said, there are a large number of possible states
> that may occur with a running job, so it isn't possible to test everything.
> If you are ready to abort the long-running job, then trying the live upgrade
> and having to restart if it fails isn't any worse.
>
> I'd always recommend to make a backup of the MDT, regardless of whether you
> are doing an upgrade or not, since it is a lot easier to restore only the
> MDT if there are problems than to restore the whole filesystem.
>
> Cheers, Andreas
>
>> On Jul 8, 2016, at 09:08, Peter Bortas  wrote:
>>
>> I'm upgrading a few ZFS backed filesystems from 2.4.2 to 2.5.3 (both
>> from the llnl chaos branch). Clients are already running 2.5EE. It's a
>> simple setup with no failover or mirroring of MDSs or OSSs. Originally
>> the plan was to do this with the filesystems unmounted on the clients,
>> but it looks like it will be hard to get a window to do that any time
>> soon.
>>
>> Are there any known problems just doing an online upgrade 2.4 -> 2.5?
>>
>> Is the recommended method still OSSs first and MDS last?
>>
>> (Obviously the clients will lock up if they access these filesystems,
>> but locking them up for a fraction of a day beats aborting a 7 day
>> compute job.)
>>
>> Regards,
>> --
>> Peter Bortas, NSC
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Is live upgrade of 2.4 to 2.5 unproblematic?

2016-07-11 Thread Peter Bortas 
Hi Andreas,

"Backing up" is easy enough. ZFS snapshots are nice and I'll make an
extra dump of the MDS.

The consensus seems to be that online upgrade should work but is
avoided in the field. So I'll skip working into that minefield this
time.

Thanks!
-- 
Peter Bortas, NSC


On Mon, Jul 11, 2016 at 12:53 AM, Dilger, Andreas
 wrote:
> We typically test 2.x->2.x+1 upgrades, both live and offline, for every 
> version of Lustre. That said, there are a large number of possible states 
> that may occur with a running job, so it isn't possible to test everything. 
> If you are ready to abort the long-running job, then trying the live upgrade 
> and having to restart if it fails isn't any worse.
>
> I'd always recommend to make a backup of the MDT, regardless of whether you 
> are doing an upgrade or not, since it is a lot easier to restore only the MDT 
> if there are problems than to restore the whole filesystem.
>
> Cheers, Andreas
>
>> On Jul 8, 2016, at 09:08, Peter Bortas  wrote:
>>
>> I'm upgrading a few ZFS backed filesystems from 2.4.2 to 2.5.3 (both
>> from the llnl chaos branch). Clients are already running 2.5EE. It's a
>> simple setup with no failover or mirroring of MDSs or OSSs. Originally
>> the plan was to do this with the filesystems unmounted on the clients,
>> but it looks like it will be hard to get a window to do that any time
>> soon.
>>
>> Are there any known problems just doing an online upgrade 2.4 -> 2.5?
>>
>> Is the recommended method still OSSs first and MDS last?
>>
>> (Obviously the clients will lock up if they access these filesystems,
>> but locking them up for a fraction of a day beats aborting a 7 day
>> compute job.)
>>
>> Regards,
>> --
>> Peter Bortas, NSC
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Understanding quotas

2016-07-11 Thread Gibbins, Faye 
Hi,

Could someone please help me understand lustre quotas?

I've created a lustre filesystem called "scratch" using RHEL 7.2 and Lustre 
2.8. When I run "lctl get_param qmt.scratch-QMT.dt-0x0.*" on the MDT I see 
for my ID the following:

- id:  20977
  limits:  { hard:6, soft:5, granted:   
 117440512, time:0 }

Can someone tell me what the "granted" field is representing? I know it's not 
my used quota, that's only 86M and this value is nearer 112G.

Yours
Faye Gibbins

Snr Systems Administrator, Unix Lead Architect (Software)
Cirrus Logic | cirrus.com | +44 131 272 7398
[cid:image003.png@01CFC5F0.FE149B60]



This message and any attachments may contain privileged and confidential 
information that is intended solely for the person(s) to whom it is addressed. 
If you are not an intended recipient you must not: read; copy; distribute; 
discuss; take any action in or make any reliance upon the contents of this 
message; nor open or read any attachment. If you have received this message in 
error, please notify us as soon as possible on the following telephone number 
and destroy this message including any attachments. Thank you. Cirrus Logic 
International (UK) Ltd and Cirrus Logic International Semiconductor Ltd are 
companies registered in Scotland, with registered numbers SC089839 and SC495735 
respectively. Our registered office is at Westfield House, 26 Westfield Road, 
Edinburgh, EH11 2QB, UK. Tel: +44 (0)131 272 7000. cirrus.com
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org