Re: [lustre-discuss] design to enable kernel updates

2017-02-10 Thread Vicker, Darby (JSC-EG311) 
Yeah, since this is our first experience using failover with lustre we are just 
doing manual failover for now. But we may implement the 
corosync/pacemaker/stonith setup in the future.

On Feb 10, 2017, at 11:57 PM, Jeff Johnson 
> wrote:

You're also leaving out the corosync/pacemaker/stonith configuration. That is 
unless you are doing manual export/import of pools.

On Fri, Feb 10, 2017 at 9:03 PM, Vicker, Darby (JSC-EG311) 
> wrote:
Sure.  Our hardware is very similar to this:

https://www.supermicro.com/solutions/Lustre.cfm

We are using twin servers instead two single chassis servers as shown there but 
functionally this is the same – we can just fit more stuff into a single rack 
with the twin servers.  We are using a single JBOB per twin server as shown in 
one of the configurations on the above page and are using ZFS as the backend.  
All servers are dual-homed on both Ethernet and IB.  A combined MGS/MDS is at 
10.148.0.30 address for IB and X.X.98.30 for Ethernet. The secondary MDS/MGS on 
the .31 address for both networks.  With the combined MDS/MGS, they both fail 
over together.  This did require a patch from LU-8397 to get the MGS failover 
to work properly so we are using 2.9.0 with the LU-8397 patch and are compiling 
our own server rpms.  But this is pretty simple with ZFS since you don't need a 
patched kernel.  The lustre formatting and configuration bits are below.  I'm 
leaving out the ZFS pool creation but I think you get the idea.

I hope that helps.

Darby



if [[ $HOSTNAME == *mds* ]] ; then

mkfs.lustre \
--fsname=hpfs-fsl \
--backfstype=zfs \
--reformat \
--verbose \
--mgs --mdt --index=0 \
--servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_IP}@o2ib0 \
--servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0 \
metadata/meta-fst

elif [[ $HOSTNAME == *oss* ]] ; then

   num=`hostname --short | sed 's/hpfs-fsl-//' | sed 's/oss//'`
   num=`printf '%g' $num`

   mkfs.lustre \
   --mgsnode=X.X.98.30@tcp0,10.148.0.30@o2ib0 \
   --mgsnode=X.X.98.31@tcp0,10.148.0.31@o2ib0 \
   --fsname=hpfs-fsl \
   --backfstype=zfs \
   --reformat \
   --verbose \
   --ost --index=$num \
   --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_IP}@o2ib0 \
   --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0 \
   $pool/ost-fsl
fi




/etc/ldev.conf:

#local  foreign/-  label   [md|zfs:]device-path   [journal-path]/- [raidtab]

hpfs-fsl-mds0  hpfs-fsl-mds1  hpfs-fsl-MDT  zfs:metadata/meta-fsl

hpfs-fsl-oss00 hpfs-fsl-oss01 hpfs-fsl-OST  zfs:oss00-0/ost-fsl
hpfs-fsl-oss01 hpfs-fsl-oss00 hpfs-fsl-OST0001  zfs:oss01-0/ost-fsl
hpfs-fsl-oss02 hpfs-fsl-oss03 hpfs-fsl-OST0002  zfs:oss02-0/ost-fsl
hpfs-fsl-oss03 hpfs-fsl-oss02 hpfs-fsl-OST0003  zfs:oss03-0/ost-fsl
hpfs-fsl-oss04 hpfs-fsl-oss05 hpfs-fsl-OST0004  zfs:oss04-0/ost-fsl
hpfs-fsl-oss05 hpfs-fsl-oss04 hpfs-fsl-OST0005  zfs:oss05-0/ost-fsl
hpfs-fsl-oss06 hpfs-fsl-oss07 hpfs-fsl-OST0006  zfs:oss06-0/ost-fsl
hpfs-fsl-oss07 hpfs-fsl-oss06 hpfs-fsl-OST0007  zfs:oss07-0/ost-fsl
hpfs-fsl-oss08 hpfs-fsl-oss09 hpfs-fsl-OST0008  zfs:oss08-0/ost-fsl
hpfs-fsl-oss09 hpfs-fsl-oss08 hpfs-fsl-OST0009  zfs:oss09-0/ost-fsl
hpfs-fsl-oss10 hpfs-fsl-oss11 hpfs-fsl-OST000a  zfs:oss10-0/ost-fsl
hpfs-fsl-oss11 hpfs-fsl-oss10 hpfs-fsl-OST000b  zfs:oss11-0/ost-fsl




/etc/modprobe.d/lustre.conf:

options lnet networks=tcp0(enp4s0),o2ib0(ib1)
options ko2iblnd map_on_demand=32

-Original Message-
From: Brian Andrus >
Date: Friday, February 10, 2017 at 12:07 AM
To: Darby Vicker >, Ben 
Evans >, 
"lustre-discuss@lists.lustre.org" 
>
Subject: Re: [lustre-discuss] design to enable kernel updates

Darby,

Do you mind if I inquire about the setup for your lustre systems?
I'm trying to understand how the MGS/MGT is setup for high availability.
I understand with OSTs and MDTs where all I really need is to have the
failnode set when I do the mkfs.lustre
However, as I understand it, you have to use something like pacemaker
and drbd to deal with the MGS/MGT. Is this how you approached it?

Brian Andrus



On 2/6/2017 12:58 PM, Vicker, Darby (JSC-EG311) wrote:
> Agreed.  We are just about to go into production on our next LFS with the
> setup described.  We had to get past a bug in the MGS failover for
> dual-homed servers but as of last week that is done and everything is
> working great (see "MGS failover problem" thread on this mailing list from
> this month and last).  We are in the process of syncing our existing LFS
> to this new one and

Re: [lustre-discuss] design to enable kernel updates

2017-02-10 Thread Jeff Johnson 
You're also leaving out the corosync/pacemaker/stonith configuration. That
is unless you are doing manual export/import of pools.

On Fri, Feb 10, 2017 at 9:03 PM, Vicker, Darby (JSC-EG311) <
darby.vicke...@nasa.gov> wrote:

> Sure.  Our hardware is very similar to this:
>
> https://www.supermicro.com/solutions/Lustre.cfm
>
> We are using twin servers instead two single chassis servers as shown
> there but functionally this is the same – we can just fit more stuff into a
> single rack with the twin servers.  We are using a single JBOB per twin
> server as shown in one of the configurations on the above page and are
> using ZFS as the backend.  All servers are dual-homed on both Ethernet and
> IB.  A combined MGS/MDS is at 10.148.0.30 address for IB and X.X.98.30 for
> Ethernet. The secondary MDS/MGS on the .31 address for both networks.  With
> the combined MDS/MGS, they both fail over together.  This did require a
> patch from LU-8397 to get the MGS failover to work properly so we are using
> 2.9.0 with the LU-8397 patch and are compiling our own server rpms.  But
> this is pretty simple with ZFS since you don't need a patched kernel.  The
> lustre formatting and configuration bits are below.  I'm leaving out the
> ZFS pool creation but I think you get the idea.
>
> I hope that helps.
>
> Darby
>
>
>
> if [[ $HOSTNAME == *mds* ]] ; then
>
> mkfs.lustre \
> --fsname=hpfs-fsl \
> --backfstype=zfs \
> --reformat \
> --verbose \
> --mgs --mdt --index=0 \
> --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_
> IB_IP}@o2ib0 \
> --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_
> IP}@o2ib0 \
> metadata/meta-fst
>
> elif [[ $HOSTNAME == *oss* ]] ; then
>
>num=`hostname --short | sed 's/hpfs-fsl-//' | sed 's/oss//'`
>num=`printf '%g' $num`
>
>mkfs.lustre \
>--mgsnode=X.X.98.30@tcp0,10.148.0.30@o2ib0 \
>--mgsnode=X.X.98.31@tcp0,10.148.0.31@o2ib0 \
>--fsname=hpfs-fsl \
>--backfstype=zfs \
>--reformat \
>--verbose \
>--ost --index=$num \
>--servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_
> IB_IP}@o2ib0 \
>--servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0
> \
>$pool/ost-fsl
> fi
>
>
>
>
> /etc/ldev.conf:
>
> #local  foreign/-  label   [md|zfs:]device-path   [journal-path]/-
> [raidtab]
>
> hpfs-fsl-mds0  hpfs-fsl-mds1  hpfs-fsl-MDT  zfs:metadata/meta-fsl
>
> hpfs-fsl-oss00 hpfs-fsl-oss01 hpfs-fsl-OST  zfs:oss00-0/ost-fsl
> hpfs-fsl-oss01 hpfs-fsl-oss00 hpfs-fsl-OST0001  zfs:oss01-0/ost-fsl
> hpfs-fsl-oss02 hpfs-fsl-oss03 hpfs-fsl-OST0002  zfs:oss02-0/ost-fsl
> hpfs-fsl-oss03 hpfs-fsl-oss02 hpfs-fsl-OST0003  zfs:oss03-0/ost-fsl
> hpfs-fsl-oss04 hpfs-fsl-oss05 hpfs-fsl-OST0004  zfs:oss04-0/ost-fsl
> hpfs-fsl-oss05 hpfs-fsl-oss04 hpfs-fsl-OST0005  zfs:oss05-0/ost-fsl
> hpfs-fsl-oss06 hpfs-fsl-oss07 hpfs-fsl-OST0006  zfs:oss06-0/ost-fsl
> hpfs-fsl-oss07 hpfs-fsl-oss06 hpfs-fsl-OST0007  zfs:oss07-0/ost-fsl
> hpfs-fsl-oss08 hpfs-fsl-oss09 hpfs-fsl-OST0008  zfs:oss08-0/ost-fsl
> hpfs-fsl-oss09 hpfs-fsl-oss08 hpfs-fsl-OST0009  zfs:oss09-0/ost-fsl
> hpfs-fsl-oss10 hpfs-fsl-oss11 hpfs-fsl-OST000a  zfs:oss10-0/ost-fsl
> hpfs-fsl-oss11 hpfs-fsl-oss10 hpfs-fsl-OST000b  zfs:oss11-0/ost-fsl
>
>
>
>
> /etc/modprobe.d/lustre.conf:
>
> options lnet networks=tcp0(enp4s0),o2ib0(ib1)
> options ko2iblnd map_on_demand=32
>
> -Original Message-
> From: Brian Andrus 
> Date: Friday, February 10, 2017 at 12:07 AM
> To: Darby Vicker , Ben Evans , "
> lustre-discuss@lists.lustre.org" 
> Subject: Re: [lustre-discuss] design to enable kernel updates
>
> Darby,
>
> Do you mind if I inquire about the setup for your lustre systems?
> I'm trying to understand how the MGS/MGT is setup for high availability.
> I understand with OSTs and MDTs where all I really need is to have the
> failnode set when I do the mkfs.lustre
> However, as I understand it, you have to use something like pacemaker
> and drbd to deal with the MGS/MGT. Is this how you approached it?
>
> Brian Andrus
>
>
>
> On 2/6/2017 12:58 PM, Vicker, Darby (JSC-EG311) wrote:
> > Agreed.  We are just about to go into production on our next LFS with the
> > setup described.  We had to get past a bug in the MGS failover for
> > dual-homed servers but as of last week that is done and everything is
> > working great (see "MGS failover problem" thread on this mailing list
> from
> > this month and last).  We are in the process of syncing our existing LFS
> > to this new one and I've failed over/rebooted/upgraded the new LFS
> servers
> > many times now to make sure we can do this in practice when the new LFS
> goes
> > into production.  Its working beautifully.
> >
> > Many thanks to the lustre developers for their continued efforts.  We
> have
> > been using and have been fans of

Re: [lustre-discuss] design to enable kernel updates

2017-02-10 Thread Vicker, Darby (JSC-EG311) 
Sure.  Our hardware is very similar to this:

https://www.supermicro.com/solutions/Lustre.cfm

We are using twin servers instead two single chassis servers as shown there but 
functionally this is the same – we can just fit more stuff into a single rack 
with the twin servers.  We are using a single JBOB per twin server as shown in 
one of the configurations on the above page and are using ZFS as the backend.  
All servers are dual-homed on both Ethernet and IB.  A combined MGS/MDS is at 
10.148.0.30 address for IB and X.X.98.30 for Ethernet. The secondary MDS/MGS on 
the .31 address for both networks.  With the combined MDS/MGS, they both fail 
over together.  This did require a patch from LU-8397 to get the MGS failover 
to work properly so we are using 2.9.0 with the LU-8397 patch and are compiling 
our own server rpms.  But this is pretty simple with ZFS since you don't need a 
patched kernel.  The lustre formatting and configuration bits are below.  I'm 
leaving out the ZFS pool creation but I think you get the idea.  

I hope that helps. 

Darby



if [[ $HOSTNAME == *mds* ]] ; then

mkfs.lustre \
--fsname=hpfs-fsl \
--backfstype=zfs \
--reformat \
--verbose \
--mgs --mdt --index=0 \
--servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_IP}@o2ib0 \
--servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0 \
metadata/meta-fst

elif [[ $HOSTNAME == *oss* ]] ; then

   num=`hostname --short | sed 's/hpfs-fsl-//' | sed 's/oss//'`
   num=`printf '%g' $num`

   mkfs.lustre \
   --mgsnode=X.X.98.30@tcp0,10.148.0.30@o2ib0 \
   --mgsnode=X.X.98.31@tcp0,10.148.0.31@o2ib0 \
   --fsname=hpfs-fsl \
   --backfstype=zfs \
   --reformat \
   --verbose \
   --ost --index=$num \
   --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_IP}@o2ib0 \
   --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0 \
   $pool/ost-fsl
fi




/etc/ldev.conf:

#local  foreign/-  label   [md|zfs:]device-path   [journal-path]/- [raidtab]

hpfs-fsl-mds0  hpfs-fsl-mds1  hpfs-fsl-MDT  zfs:metadata/meta-fsl

hpfs-fsl-oss00 hpfs-fsl-oss01 hpfs-fsl-OST  zfs:oss00-0/ost-fsl
hpfs-fsl-oss01 hpfs-fsl-oss00 hpfs-fsl-OST0001  zfs:oss01-0/ost-fsl
hpfs-fsl-oss02 hpfs-fsl-oss03 hpfs-fsl-OST0002  zfs:oss02-0/ost-fsl
hpfs-fsl-oss03 hpfs-fsl-oss02 hpfs-fsl-OST0003  zfs:oss03-0/ost-fsl
hpfs-fsl-oss04 hpfs-fsl-oss05 hpfs-fsl-OST0004  zfs:oss04-0/ost-fsl
hpfs-fsl-oss05 hpfs-fsl-oss04 hpfs-fsl-OST0005  zfs:oss05-0/ost-fsl
hpfs-fsl-oss06 hpfs-fsl-oss07 hpfs-fsl-OST0006  zfs:oss06-0/ost-fsl
hpfs-fsl-oss07 hpfs-fsl-oss06 hpfs-fsl-OST0007  zfs:oss07-0/ost-fsl
hpfs-fsl-oss08 hpfs-fsl-oss09 hpfs-fsl-OST0008  zfs:oss08-0/ost-fsl
hpfs-fsl-oss09 hpfs-fsl-oss08 hpfs-fsl-OST0009  zfs:oss09-0/ost-fsl
hpfs-fsl-oss10 hpfs-fsl-oss11 hpfs-fsl-OST000a  zfs:oss10-0/ost-fsl
hpfs-fsl-oss11 hpfs-fsl-oss10 hpfs-fsl-OST000b  zfs:oss11-0/ost-fsl




/etc/modprobe.d/lustre.conf:

options lnet networks=tcp0(enp4s0),o2ib0(ib1)
options ko2iblnd map_on_demand=32

-Original Message-
From: Brian Andrus 
Date: Friday, February 10, 2017 at 12:07 AM
To: Darby Vicker , Ben Evans , 
"lustre-discuss@lists.lustre.org" 
Subject: Re: [lustre-discuss] design to enable kernel updates

Darby,

Do you mind if I inquire about the setup for your lustre systems?
I'm trying to understand how the MGS/MGT is setup for high availability.
I understand with OSTs and MDTs where all I really need is to have the 
failnode set when I do the mkfs.lustre
However, as I understand it, you have to use something like pacemaker 
and drbd to deal with the MGS/MGT. Is this how you approached it?

Brian Andrus



On 2/6/2017 12:58 PM, Vicker, Darby (JSC-EG311) wrote:
> Agreed.  We are just about to go into production on our next LFS with the
> setup described.  We had to get past a bug in the MGS failover for
> dual-homed servers but as of last week that is done and everything is
> working great (see "MGS failover problem" thread on this mailing list from
> this month and last).  We are in the process of syncing our existing LFS
> to this new one and I've failed over/rebooted/upgraded the new LFS servers
> many times now to make sure we can do this in practice when the new LFS goes
> into production.  Its working beautifully.
>
> Many thanks to the lustre developers for their continued efforts.  We have
> been using and have been fans of lustre for quite some time now and it
> just keeps getting better.
>
> -Original Message-
> From: lustre-discuss  on behalf of 
> Ben Evans 
> Date: Monday, February 6, 2017 at 2:22 PM
> To: Brian Andrus , "lustre-discuss@lists.lustre.org" 
> 
> Subject: Re: [lustre-discuss] design to enable kernel updates
>
> It's

Re: [lustre-discuss] ZFS PANIC

2017-02-10 Thread Bob Ball 
Well, I find this odd, to say the least.  All of this below was from 
yesterday, and persisted through a couple of reboots.  Today, shortly 
after I sent this, I found all the disks idle, but this one OST out of 6 
totally unresponsive, so I power cycled the system, and it came up just 
fine.  No issues, no complaints, responsive  So I have no idea why 
this healed itself.


Can anyone enlighten me?

I _think_ that what triggered this was adding a few more client mounts 
of the lustre file system.  That's when it all went wrong. Is this 
helpful?  Or just a coincidence?  Current state:

 18 UP obdfilter umt3B-OST000f umt3B-OST000f_UUID 403

bob

On 2/10/2017 9:39 AM, Bob Ball wrote:

Hi,

I am getting this message

PANIC: zfs: accessing past end of object 29/7 (size=33792 
access=33792+128)


The affected OST seems to reject new mounts from clients now, and the 
lctl dl count of connections to the obdfilter process increases, but 
does not seem to decrease?


This is Lustre 2.7.58 with zfs 0.6.4.2

Can anyone help me diagnose and fix whatever is going wrong here? I've 
included the stack dump below.


Thanks,
bob


2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781874] 
Showing stack for process 24449
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781876] 
Pid: 24449, comm: ll_ost00_078 Tainted: P   ---
2.6.32.504.16.2.el6_lustre #7
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781878] 
Call Trace:
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.781902]  [] ? spl_dumpstack+0x3d/0x40 [spl]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.781908]  [] ? vcmn_err+0x8d/0xf0 [spl]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.781950]  [] ? RW_WRITE_HELD+0x66/0xb0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.781970]  [] ? 
dbuf_rele_and_unlock+0x268/0x3f0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.781991]  [] ? dbuf_read+0x5ca/0x8a0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782024]  [] ? zfs_panic_recover+0x52/0x60 
[zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782045]  [] ? 
dmu_buf_hold_array_by_dnode+0x41b/0x560 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782068]  [] ? dmu_buf_hold_array+0x65/0x90 
[zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782090]  [] ? dmu_write+0x68/0x1a0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782147]  [] ? lprocfs_oh_tally+0x2e/0x50 
[obdclass]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782173]  [] ? osd_write+0x1d1/0x390 [osd_zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782206]  [] ? dt_record_write+0x3d/0x130 
[obdclass]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782305]  [] ? 
tgt_client_data_write+0x165/0x1b0 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782347]  [] ? 
tgt_client_data_update+0x335/0x680 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782388]  [] ? tgt_client_new+0x3d8/0x6a0 
[ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782407]  [] ? ofd_obd_connect+0x363/0x400 
[ofd]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782443]  [] ? 
target_handle_connect+0xe58/0x2d30 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782450]  [] ? enqueue_entity+0x125/0x450
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782457]  [] ? check_preempt_curr+0x7c/0x90
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782462]  [] ? try_to_wake_up+0x24e/0x3e0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782481]  [] ? lc_watchdog_touch+0x7a/0x190 
[libcfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782524]  [] ? 
tgt_request_handle+0x5b2/0x1230 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782564]  [] ? ptlrpc_main+0xe41/0x1920 
[ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782570]  [] ? sched_clock+0x9/0x10
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782576]  [] ? thread_return+0x4e/0x7d0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782615]  [] ? ptlrpc_main+0x0/0x1920 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782622]  [] ? kthread+0x9e/0xc0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782626]  [] ? kthread+0x0/0xc0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782632]  [] ? child_rip+0xa/0x20
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782636]  [] ? kthread+0x0/0xc0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: 
[11630254.782641]  [] ? child_rip+0x0/0x20



Later, that same process showed:
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel:

[lustre-discuss] ZFS PANIC

2017-02-10 Thread Bob Ball 

Hi,

I am getting this message

PANIC: zfs: accessing past end of object 29/7 (size=33792 access=33792+128)

The affected OST seems to reject new mounts from clients now, and the 
lctl dl count of connections to the obdfilter process increases, but 
does not seem to decrease?


This is Lustre 2.7.58 with zfs 0.6.4.2

Can anyone help me diagnose and fix whatever is going wrong here? I've 
included the stack dump below.


Thanks,
bob


2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781874] 
Showing stack for process 24449
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781876] 
Pid: 24449, comm: ll_ost00_078 Tainted: P   ---
2.6.32.504.16.2.el6_lustre #7
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781878] 
Call Trace:
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781902]  
[] ? spl_dumpstack+0x3d/0x40 [spl]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781908]  
[] ? vcmn_err+0x8d/0xf0 [spl]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781950]  
[] ? RW_WRITE_HELD+0x66/0xb0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781970]  
[] ? dbuf_rele_and_unlock+0x268/0x3f0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.781991]  
[] ? dbuf_read+0x5ca/0x8a0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782024]  
[] ? zfs_panic_recover+0x52/0x60 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782045]  
[] ? dmu_buf_hold_array_by_dnode+0x41b/0x560 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782068]  
[] ? dmu_buf_hold_array+0x65/0x90 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782090]  
[] ? dmu_write+0x68/0x1a0 [zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782147]  
[] ? lprocfs_oh_tally+0x2e/0x50 [obdclass]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782173]  
[] ? osd_write+0x1d1/0x390 [osd_zfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782206]  
[] ? dt_record_write+0x3d/0x130 [obdclass]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782305]  
[] ? tgt_client_data_write+0x165/0x1b0 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782347]  
[] ? tgt_client_data_update+0x335/0x680 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782388]  
[] ? tgt_client_new+0x3d8/0x6a0 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782407]  
[] ? ofd_obd_connect+0x363/0x400 [ofd]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782443]  
[] ? target_handle_connect+0xe58/0x2d30 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782450]  
[] ? enqueue_entity+0x125/0x450
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782457]  
[] ? check_preempt_curr+0x7c/0x90
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782462]  
[] ? try_to_wake_up+0x24e/0x3e0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782481]  
[] ? lc_watchdog_touch+0x7a/0x190 [libcfs]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782524]  
[] ? tgt_request_handle+0x5b2/0x1230 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782564]  
[] ? ptlrpc_main+0xe41/0x1920 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782570]  
[] ? sched_clock+0x9/0x10
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782576]  
[] ? thread_return+0x4e/0x7d0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782615]  
[] ? ptlrpc_main+0x0/0x1920 [ptlrpc]
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782622]  
[] ? kthread+0x9e/0xc0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782626]  
[] ? kthread+0x0/0xc0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782632]  
[] ? child_rip+0xa/0x20
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782636]  
[] ? kthread+0x0/0xc0
2017-02-08T23:02:23-05:00 umdist01.aglt2.org kernel: [11630254.782641]  
[] ? child_rip+0x0/0x20



Later, that same process showed:
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: [11630454.773156] 
LNet: Service thread pid 24449 was inactive for 200.00s. The thread 
might be hung, or it might only be slow and will resume later. Dumping 
the stack trace for debugging purposes:
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: [11630454.773163] 
Pid: 24449, comm: ll_ost00_078

2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: [11630454.773164]
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: [11630454.773165] 
Call Trace:
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: [11630454.773181]  
[] ? show_trace_log_lvl+0x55/0x70
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: [11630454.773194]  
[] ? dump_stack+0x6f/0x76
2017-02-08T23:05:43-05:00 umdist01.aglt2.org kernel: