On Wed, 2009-08-05 at 18:22 -0700, Joe Eykholt wrote:
> Vasu Dev wrote:
> > On Wed, 2009-08-05 at 14:10 -0700, Joe Eykholt wrote:
> >> Hi All,
> >>
> >> I'm posting this problem description hoping someone sees the problem,
> >> or perhaps has run into it.
> >>
> >> Vasu, I'm hoping you'll take a look at this since you're so familiar with
> >> the exchange manager.
> >>
> >> I've been seeing an occasional problem where I'm trying to unload
> >> the libfc module, after all instances are destroyed and fcoe.ko and libfcoe
> >> have been removed, where one exchange is still not freed. If using the
> >> slab allocator, it results in a message from kmem_cache_destroy()
> >> "Can't free all objects".
> >>
> >
> > This issue could be similar as reported by you in this RFC
> >
> > http://www.open-fcoe.org/pipermail/devel/2009-July/003401.html
> >
> > So is this issue even with your patch in above RFC ?
>
> Yes, even with that patch.
>
> > It could be similar as in above RFC since an exch must have got
> > allocated after final EM reset. I'll look more into these debug info and
> > get back to you.
>
> I'm not sure it's after the final reset. I can add debug code to
> make sure it isn't allocated after that final reset.
>
> One clue is the XID is 0, and I don't think that would
> happen except by huge coincidence
> after the lport has been up for a while. It must've been the first
> exchange that was created. It must've timed out but I don't see
> any bugs in that path. I can print the xids in my trace, that
> may also help.
>
> I've refreshed to the latest patches and am testing it again.
>
Joe it would be great if you can repeat your test with my debug patch,
more details below on debug patch.
I think following cases could lead to pending exch/es while destroying
EM mempool:-
1. exch being allocated after final EM reset as we talked before.
2. currently i/f destroy code does't wait for all exches released to
mempool.
I added code for above two cases in debug patch to help us in root
causing this issue.
The debug patch adds code for case 1 to abandon exch alloc in case the
lport has entered into LPORT_ST_DISABLED state. I also added
dump_stack() for this case to help us track down source any such exch
allocation request while LPORT_ST_DISABLED.
Added code for case 2 to wait for all pending exches released to mempool
before destroying EM mempool. You can combine this code change with your
code printing pending exch details just before waiting for all pending
exches released in fc_exch_mgr_destroy. I think we need this fix any way
since currently destroy i/f code path doesn't wait for all exch released
to mempool but not sure if we can have better fix than waiting on
completion as my debug patch does with added stat counter for this.
I'm sending debug patch separately since I could not send attachment on
this mail list before.
I'll also try reproducing this issue here to root cause this issue.
Vasu
> Thanks for thinking about this.
>
> Joe
>
>
> > Vasu
> >
> >
> >> I added debug code to keep a list of all exchanges, even those that
> >> have been completed, until the last fc_exch_release() frees it.
> >>
> >> I hit this after creating and deleting 75 times, so it doesn't
> >> happen that often. After it does, though, you can't reload libfc anymore.
> >>
> >> In this case, it's an exchange for an outgoing FLOGI. Its refcnt is 1,
> >> and it
> >> happens to have XID of 0. I can't figure out any way that's special.
> >>
> >> Here's the exchange from kgdb:
> >>
> >>> $5 = {em = 0xffff88012c480000, state = 0x3, xid = 0x0, ex_list = {
> >>> next = 0x100100, prev = 0x200200}, ex_debug = {next =
> >>> 0xffff88012c480058,
> >>> prev = 0xffff88012c480058}, ex_lock = {raw_lock = {slock = 0x404},
> >>> magic = 0xdead4ead, owner_cpu = 0xffffffff, owner =
> >>> 0xffffffffffffffff,
> >>> dep_map = {key = 0xffffffffa05458d8, class_cache = 0x0,
> >>> name = 0xffffffffa053e72c}}, ex_refcnt = {counter = 0x1},
> >>> timeout_work = {work = {data = {counter = 0xffff88002809c601}, entry = {
> >>> next = 0xffff88002809c630, prev = 0xffff8801325a3e00},
> >>> func = 0xffffffffa05361d2, lockdep_map = {key = 0xffffffffa05458d0,
> >>> class_cache = 0x0, name = 0xffffffffa053e739}}, timer = {entry = {
> >>> next = 0x0, prev = 0x200200}, expires = 0x1001045aa,
> >>> function = 0xffffffff81053ba5, data = 0xffff88012c9276e8,
> >>> base = 0xffff88013faf0000, start_site = 0x0, start_comm = {
> >>> 0x0 <repeats 16 times>}, start_pid = 0xffffffff, lockdep_map = {
> >>> key = 0xffffffffa05458c8, class_cache = 0x0,
> >>> name = 0xffffffffa053e754}}}, lp = 0xffff88012c85c5d8, oxid = 0x0,
> >>> rxid = 0xffff, oid = 0x0, sid = 0x0, did = 0xfffffe, esb_stat =
> >>> 0x70000000,
> >>> r_a_tov = 0x2710, seq_id = 0x2, f_ctl = 0x90000, fh_type = 0x1,
> >>> class = 0x2e, seq = {id = 0x1, ssb_stat = 0x0, cnt = 0x0, rec_data =
> >>> 0x0},
> >>> resp = 0x0, destructor = 0x0, arg = 0xffff88012c85c5d8}
> >> I looked at the work.data and work.timer. work.data shows work is pending,
> >> and work.timer shows the timer isn't pending. I thought maybe the
> >> timer_cancel_sync in fc_exch_reset could be at fault, but I don't think so
> >> now.
> >>
> >> This is with the previous rebase, before the per-cpu exchange pools,
> >> and with some additional rport/lport changes by me.
> >>
> >> Also, in case you're interested, here's the log with lport and rport debug
> >> messages (with a temporarily-modified rport debug format).
> >>
> >> host75 is the one that has the leaked exchange.
> >>
> >>
> >>
> >> [ 1344.349639] scsi74 : FCoE Driver
> >> [ 1344.353338] host74: lport ffffffff: Entered RESET state from disabled
> >> state
> >> [ 1344.360517] libfc: Link up on port ( 0)
> >> [ 1344.364840] host74: lport 0: Entered FLOGI state from reset state
> >> [ 1344.371523] device eth0 entered promiscuous mode
> >> [ 1344.376393] scsi75 : FCoE Driver
> >> [ 1344.380010] host75: lport ffffffff: Entered RESET state from disabled
> >> state
> >> [ 1344.387119] libfc: Link up on port ( 0)
> >> [ 1344.391422] host75: lport 0: Entered FLOGI state from reset state
> >> [ 1346.371007] libfcoe: host74: FIP selected Fibre-Channel Forwarder MAC
> >> 00:0d:ec:6d:b2:c0
> >> [ 1346.371010] host74: lport 0: Received a FLOGI response timeout
> >> [ 1346.371014] host74: lport 0: Error -1 in state FLOGI, retries 0
> >> [ 1346.398012] host75: lport 0: Received a FLOGI response timeout
> >> [ 1346.404404] host75: lport 0: Error -1 in state FLOGI, retries 0
> >> [ 1348.371008] host74: lport 0: Entered FLOGI state from FLOGI state
> >> [ 1348.402343] host74: lport 0: Received a FLOGI accept
> >> [ 1348.407863] libfc: Assigned FID (6a0000) in FLOGI response
> >> [ 1348.410011] host75: lport 0: Entered FLOGI state from FLOGI state
> >> [ 1348.420127] host74: lport 6a0000: Entered DNS state from FLOGI state
> >> [ 1348.426583] host74: rport fffffc: ffff88012c8cf000 r 1 Init Login
> >> to port
> >> [ 1348.433865] host74: rport fffffc: ffff88012c8cf000 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1348.443719] host74: rport fffffc: ffff88012c8cf000 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1348.452141] host74: rport fffffc: ffff88012c8cf000 r 2 Ready Port is
> >> Ready
> >> [ 1348.459461] host74: rport fffffc: ffff88012c8cf000 r 1 Ready work
> >> event 1
> >> [ 1348.466964] host74: rport fffffc: ffff88012c8cf000 r 2 Ready
> >> callback ev 1
> >> [ 1348.474454] host74: lport 6a0000: Received a 1 event for port (fffffc)
> >> [ 1348.481110] host74: lport 6a0000: Entered RPN_ID state from dNS state
> >> [ 1348.487949] host74: lport 6a0000: Received a RPN_ID reject
> >> [ 1348.493590] host74: lport 6a0000: Entered RFT_ID state from RPN_ID state
> >> [ 1348.500838] host74: lport 6a0000: Received a RFT_ID reject
> >> [ 1348.506448] host74: lport 6a0000: Entered SCR state from RFT_ID state
> >> [ 1348.513341] host74: lport 6a0000: Received a SCR accept
> >> [ 1348.518679] host74: lport 6a0000: Entered READY from state SCR
> >> [ 1348.524641] host74: disc: Restarting discovery
> >> [ 1348.529793] host74: disc: Received a GPN_FT response
> >> [ 1348.534861] host74: disc: Discovery complete
> >> [ 1348.539205] host74: rport d10400: ffff88013d7d7800 r 1 Init Login
> >> to port
> >> [ 1348.546485] host74: rport d10400: ffff88013d7d7800 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1348.556188] host74: rport 35000f: ffff88013d7d7a00 r 1 Init Login
> >> to port
> >> [ 1348.562418] host74: rport d10400: ffff88013d7d7800 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1348.562421] host74: rport d10400: ffff88013d7d7800 r 2 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1348.569143] host74: rport d10400: ffff88013d7d7800 r 2 PRLI
> >> Received a PRLI accept
> >> [ 1348.569147] host74: rport d10400: ffff88013d7d7800 r 2 PRLI Port
> >> entered RTV state from PRLI state
> >> [ 1348.569461] host74: rport d10400: ffff88013d7d7800 r 2 RTV
> >> Received a RTV reject
> >> [ 1348.569464] host74: rport d10400: ffff88013d7d7800 r 2 Ready Port is
> >> Ready
> >> [ 1348.569475] host74: rport d10400: ffff88013d7d7800 r 1 Ready work
> >> event 1
> >> [ 1348.621266] host74: rport 35000f: ffff88013d7d7a00 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1348.630942] host74: rport 706e1: ffff88013d7d7000 r 1 Init Login
> >> to port
> >> [ 1348.638207] host74: rport 706e1: ffff88013d7d7000 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1348.647882] host74: rport 700e1: ffff88012c8cf200 r 1 Init Login
> >> to port
> >> [ 1348.654244] host74: rport 706e1: ffff88013d7d7000 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1348.654247] host74: rport 706e1: ffff88013d7d7000 r 2 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1348.660184] host74: rport 706e1: ffff88013d7d7000 r 2 PRLI
> >> Received a PRLI accept
> >> [ 1348.660187] host74: rport 706e1: ffff88013d7d7000 r 2 PRLI Port
> >> entered RTV state from PRLI state
> >> [ 1348.690498] host74: rport 700e1: ffff88012c8cf200 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1348.700188] host74: lport 6a0000: Discovery succeeded
> >> [ 1348.705366] host74: rport 706e1: ffff88013d7d7000 r 2 RTV
> >> Received a RTV reject
> >> [ 1348.706162] host74: rport 700e1: ffff88012c8cf200 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1348.706164] host74: rport 700e1: ffff88012c8cf200 r 2 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1348.707576] scsi 74:0:0:0: Sequential-Access IBM ULTRIUM-TD3
> >> 54K1 PQ: 0 ANSI: 3
> >> [ 1348.707903] st 74:0:0:0: Attached scsi tape st0
> >> [ 1348.707904] st 74:0:0:0: st0: try direct i/o: yes (alignment 4 B)
> >> [ 1348.707959] st 74:0:0:0: Attached scsi generic sg2 type 1
> >> [ 1348.714198] scsi 74:0:0:1: Medium Changer ADIC Scalar 24
> >> 309A PQ: 0 ANSI: 2
> >> [ 1348.714337] scsi 74:0:0:1: Attached scsi generic sg3 type 8
> >> [ 1348.770027] host74: rport 706e1: ffff88013d7d7000 r 2 Ready Port is
> >> Ready
> >> [ 1348.777311] host74: rport 706e1: ffff88013d7d7000 r 1 Ready work
> >> event 1
> >> [ 1348.777313] host74: rport 700e1: ffff88012c8cf200 r 2 PRLI
> >> Received a PRLI accept
> >> [ 1348.777316] host74: rport 700e1: ffff88012c8cf200 r 2 PRLI Port
> >> entered RTV state from PRLI state
> >> [ 1348.802811] host74: rport 700e1: ffff88012c8cf200 r 2 RTV
> >> Received a RTV reject
> >> [ 1348.811063] host74: rport 700e1: ffff88012c8cf200 r 2 Ready Port is
> >> Ready
> >> [ 1348.824261] host74: rport 700e1: ffff88012c8cf200 r 1 Ready work
> >> event 1
> >> [ 1348.825187] scsi 74:0:1:0: Direct-Access SEAGATE ST318453FC
> >> 0003 PQ: 0 ANSI: 3
> >> [ 1348.825356] sd 74:0:1:0: Attached scsi generic sg4 type 0
> >> [ 1348.847364] sd 74:0:1:0: [sdb] 35843686 512-byte logical blocks: (18.3
> >> GB/17.0 GiB)
> >> [ 1348.856727] sd 74:0:1:0: [sdb] Write Protect is off
> >> [ 1348.861742] sd 74:0:1:0: [sdb] Mode Sense: ab 00 10 08
> >> [ 1348.867036] scsi 74:0:2:0: Direct-Access SEAGATE ST118202FC
> >> 0008 PQ: 0 ANSI: 2
> >> [ 1348.867953] sd 74:0:1:0: [sdb] Write cache: disabled, read cache:
> >> enabled, supports DPO and FUA
> >> [ 1348.872317] __ratelimit: 122 callbacks suppressed
> >> [ 1348.872320] <6>nm-system-setti[22981]: segfault at 0 ip
> >> 00000000004082d9 sp 00007fffdcaa98a0 error 4 in
> >> nm-system-settings[400000+f000]
> >> [ 1348.901961] sd 74:0:2:0: Attached scsi generic sg5 type 0
> >> [ 1348.907533] sdb:
> >> [ 1348.909365] sd 74:0:2:0: [sdc] 35566480 512-byte logical blocks: (18.2
> >> GB/16.9 GiB)
> >> [ 1348.917592] sdb1 sdb2 sdb3 <
> >> [ 1348.923018] sd 74:0:2:0: [sdc] Write Protect is off
> >> [ 1348.928192] sd 74:0:2:0: [sdc] Mode Sense: a7 00 10 08
> >> [ 1348.933458] sdb5 >
> >> [ 1348.936513] sd 74:0:2:0: [sdc] Write cache: disabled, read cache:
> >> enabled, supports DPO and FUA
> >> [ 1348.951088] sdc:
> >> [ 1348.954758] sd 74:0:1:0: [sdb] Attached SCSI disk
> >> [ 1348.960195] unknown partition table
> >> [ 1348.972201] sd 74:0:2:0: [sdc] Attached SCSI disk
> >> [ 1350.410009] host75: lport 0: Received a FLOGI response timeout
> >> [ 1350.416404] host75: lport 0: Error -1 in state FLOGI, retries 1
> >> [ 1350.630010] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI
> >> Received a PLOGI response timeout
> >> [ 1350.639338] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI Error
> >> -1 in state PLOGI, retrying
> >> [ 1350.648402] host74: rport 35000f: ffff88013d7d7a00 r 1 PLOGI Port
> >> entered PLOGI state from PLOGI state
> >> [ 1352.422011] host75: lport 0: Entered FLOGI state from FLOGI state
> >> [ 1352.446174] host75: lport 0: Received a FLOGI accept
> >> [ 1352.451688] libfc: Assigned FID ( 70500) in FLOGI response
> >> [ 1352.457286] host75: lport 70500: Entered DNS state from FLOGI state
> >> [ 1352.463778] host75: rport fffffc: ffff880132542200 r 1 Init Login
> >> to port
> >> [ 1352.471048] host75: rport fffffc: ffff880132542200 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1352.481675] host75: rport fffffc: ffff880132542200 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1352.490089] host75: rport fffffc: ffff880132542200 r 2 Ready Port is
> >> Ready
> >> [ 1352.497371] host75: rport fffffc: ffff880132542200 r 1 Ready work
> >> event 1
> >> [ 1352.504920] host75: rport fffffc: ffff880132542200 r 2 Ready
> >> callback ev 1
> >> [ 1352.512421] host75: lport 70500: Received a 1 event for port (fffffc)
> >> [ 1352.519075] host75: lport 70500: Entered RPN_ID state from dNS state
> >> [ 1352.526679] host75: lport 70500: Received a RPN_ID reject
> >> [ 1352.532316] host75: lport 70500: Entered RFT_ID state from RPN_ID state
> >> [ 1352.540561] host75: lport 70500: Received a RFT_ID reject
> >> [ 1352.546164] host75: lport 70500: Entered SCR state from RFT_ID state
> >> [ 1352.553938] host75: lport 70500: Received a SCR accept
> >> [ 1352.559269] host75: lport 70500: Entered READY from state SCR
> >> [ 1352.565413] host75: disc: Restarting discovery
> >> [ 1352.571972] host75: disc: Received a GPN_FT response
> >> [ 1352.577025] host75: disc: Discovery complete
> >> [ 1352.581389] host75: rport 6a0000: ffff8801399a8600 r 1 Init Login
> >> to port
> >> [ 1352.588650] host75: rport 6a0000: ffff8801399a8600 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1352.598319] host75: rport 706e1: ffff8801399a8400 r 1 Init Login
> >> to port
> >> [ 1352.605600] host75: rport 706e1: ffff8801399a8400 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1352.605607] host75: rport 6a0000: ffff8801399a8600 r 2 PLOGI
> >> Received a PLOGI reject
> >> [ 1352.605610] host75: rport 6a0000: ffff8801399a8600 r 2 PLOGI Error
> >> -131936256018240 in state PLOGI, retrying
> >> [ 1352.633938] host75: rport 700e1: ffff880132542e00 r 1 Init Login
> >> to port
> >> [ 1352.641222] host75: rport 700e1: ffff880132542e00 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1352.650884] host75: rport 706e1: ffff8801399a8400 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1352.650886] host75: lport 70500: Discovery succeeded
> >> [ 1352.658009] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI
> >> Received a PLOGI response timeout
> >> [ 1352.658012] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI Error
> >> -1 in state PLOGI, retrying
> >> [ 1352.658019] host74: rport 35000f: ffff88013d7d7a00 r 1 PLOGI Port
> >> entered PLOGI state from PLOGI state
> >> [ 1352.662658] host75: rport 700e1: ffff880132542e00 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1352.662660] host75: rport 700e1: ffff880132542e00 r 2 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1352.668572] host75: rport 700e1: ffff880132542e00 r 2 PRLI
> >> Received a PRLI accept
> >> [ 1352.668575] host75: rport 700e1: ffff880132542e00 r 2 PRLI Port
> >> entered RTV state from PRLI state
> >> [ 1352.668877] host75: rport 700e1: ffff880132542e00 r 2 RTV
> >> Received a RTV reject
> >> [ 1352.668880] host75: rport 700e1: ffff880132542e00 r 2 Ready Port is
> >> Ready
> >> [ 1352.668886] host75: rport 700e1: ffff880132542e00 r 1 Ready work
> >> event 1
> >> [ 1352.749757] host75: rport 706e1: ffff8801399a8400 r 2 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1352.763304] scsi 75:0:0:0: Direct-Access SEAGATE ST118202FC
> >> 0008 PQ: 0 ANSI: 2
> >> [ 1352.771668] host75: rport 706e1: ffff8801399a8400 r 2 PRLI
> >> Received a PRLI accept
> >> [ 1352.779740] host75: rport 706e1: ffff8801399a8400 r 2 PRLI Port
> >> entered RTV state from PRLI state
> >> [ 1352.789466] sd 75:0:0:0: Attached scsi generic sg6 type 0
> >> [ 1352.789496] host75: rport 706e1: ffff8801399a8400 r 2 RTV
> >> Received a RTV reject
> >> [ 1352.789498] host75: rport 706e1: ffff8801399a8400 r 2 Ready Port is
> >> Ready
> >> [ 1352.789505] host75: rport 706e1: ffff8801399a8400 r 1 Ready work
> >> event 1
> >> [ 1352.818557] sd 75:0:0:0: [sdd] 35566480 512-byte logical blocks: (18.2
> >> GB/16.9 GiB)
> >> [ 1352.828779] scsi 75:0:1:0: Direct-Access SEAGATE ST318453FC
> >> 0003 PQ: 0 ANSI: 3
> >> [ 1352.829902] sd 75:0:0:0: [sdd] Write Protect is off
> >> [ 1352.829904] sd 75:0:0:0: [sdd] Mode Sense: a7 00 10 08
> >> [ 1352.847625] sd 75:0:1:0: Attached scsi generic sg7 type 0
> >> [ 1352.848384] sd 75:0:0:0: [sdd] Write cache: disabled, read cache:
> >> enabled, supports DPO and FUA
> >> [ 1352.862947] sd 75:0:1:0: [sde] 35843686 512-byte logical blocks: (18.3
> >> GB/17.0 GiB)
> >> [ 1352.872587] sd 75:0:1:0: [sde] Write Protect is off
> >> [ 1352.877567] sd 75:0:1:0: [sde] Mode Sense: ab 00 10 08
> >> [ 1352.882857] sdd:
> >> [ 1352.885871] sd 75:0:1:0: [sde] Write cache: disabled, read cache:
> >> enabled, supports DPO and FUA
> >> [ 1352.895690] unknown partition table
> >> [ 1352.902474] sde:
> >> [ 1352.909401] sd 75:0:0:0: [sdd] Attached SCSI disk
> >> [ 1352.914403] sde1 sde2 sde3 < sde5 >
> >> [ 1352.965207] sd 75:0:1:0: [sde] Attached SCSI disk
> >> [ 1354.464657] host74: disc: Received an RSCN event
> >> [ 1354.469392] host74: disc: Port address format for port ( 70500)
> >> [ 1354.475432] host74: disc: RSCN received: not rediscovering. redisc 0
> >> state 6 in_prog 0
> >> [ 1354.483521] host74: rport 70500: ffff88013243a000 r 1 Init Login
> >> to port
> >> [ 1354.490805] host74: rport 70500: ffff88013243a000 r 1 Init Port
> >> entered PLOGI state from Init state
> >> [ 1354.500602] host75: rport 6a0000: ffff8801399a8600 r 1 PLOGI
> >> Received PLOGI request while in state PLOGI
> >> [ 1354.510544] host75: rport 6a0000: ffff8801399a8600 r 1 PLOGI
> >> Received PLOGI in PLOGI state 1
> >> [ 1354.519419] host75: rport 6a0000: ffff8801399a8600 r 1 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1354.529109] host74: rport 70500: ffff88013243a000 r 2 PLOGI
> >> Received a PLOGI accept
> >> [ 1354.537282] host74: rport 70500: ffff88013243a000 r 2 PLOGI Port
> >> entered PRLI state from PLOGI state
> >> [ 1354.546956] host74: rport 70500: ffff88013243a000 r 2 PRLI
> >> Received PRLI request while in state PRLI
> >> [ 1354.547035] host75: rport 6a0000: ffff8801399a8600 r 2 PRLI
> >> Received PRLI request while in state PRLI
> >> [ 1354.547043] host75: rport 6a0000: ffff8801399a8600 r 2 Ready Port is
> >> Ready
> >> [ 1354.547053] host75: rport 6a0000: ffff8801399a8600 r 2 Ready work
> >> event 1
> >> [ 1354.581028] host74: rport 70500: ffff88013243a000 r 2 Ready Port is
> >> Ready
> >> [ 1354.588300] host74: rport 70500: ffff88013243a000 r 2 Ready work
> >> event 1
> >> [ 1354.588304] host75: rport 6a0000: ffff8801399a8600 r 2 Ready
> >> Received a PRLI accept
> >> [ 1354.588306] host75: rport 6a0000: ffff8801399a8600 r 2 Ready
> >> Received a PRLI response, but in state Ready
> >> [ 1354.614370] host74: rport 70500: ffff88013243a000 r 3 Ready
> >> Received a PRLI accept
> >> [ 1354.622545] host74: rport 70500: ffff88013243a000 r 3 Ready
> >> Received a PRLI response, but in state Ready
> >> [ 1354.659016] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI
> >> Received a PLOGI response timeout
> >> [ 1354.668340] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI Error
> >> -1 in state PLOGI, retrying
> >> [ 1354.677750] host74: rport 35000f: ffff88013d7d7a00 r 1 PLOGI Port
> >> entered PLOGI state from PLOGI state
> >> [ 1356.687018] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI
> >> Received a PLOGI response timeout
> >> [ 1356.696363] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI Error 1
> >> in state PLOGI, retries 3
> >> [ 1356.705440] host74: rport 35000f: ffff88013d7d7a00 r 2 PLOGI Delete
> >> port
> >> [ 1356.712547] host74: rport 35000f: ffff88013d7d7a00 r 1 Delete work
> >> event 2
> >> [ 1366.114651] host74: rport 70500: ffff88013243a000 r 1 Ready Remove
> >> port
> >> [ 1366.121756] host74: rport 70500: ffff88013243a000 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.131431] host74: rport 70500: ffff88013243a000 r 2 LOGO Delete
> >> port
> >> [ 1366.138548] host74: rport 70500: ffff88013243a000 r 2 Delete work
> >> event 3
> >> [ 1366.145976] host74: rport d10400: ffff88013d7d7800 r 1 Ready Remove
> >> port
> >> [ 1366.153060] host74: rport d10400: ffff88013d7d7800 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.162751] host74: rport d10400: ffff88013d7d7800 r 2 LOGO Delete
> >> port
> >> [ 1366.169862] host74: rport 706e1: ffff88013d7d7000 r 1 Ready Remove
> >> port
> >> [ 1366.176959] host74: rport 706e1: ffff88013d7d7000 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.186633] host74: rport 706e1: ffff88013d7d7000 r 2 LOGO Delete
> >> port
> >> [ 1366.193738] host74: rport 700e1: ffff88012c8cf200 r 1 Ready Remove
> >> port
> >> [ 1366.200860] host74: rport 700e1: ffff88012c8cf200 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.210543] host74: rport 700e1: ffff88012c8cf200 r 2 LOGO Delete
> >> port
> >> [ 1366.217668] host74: rport 70500: ffff88013243a000 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.226530] host74: rport 70500: ffff88013243a000 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.236649] host74: rport d10400: ffff88013d7d7800 r 2 Delete work
> >> event 3
> >> [ 1366.244064] host74: rport d10400: ffff88013d7d7800 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.252935] host74: rport d10400: ffff88013d7d7800 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.263074] host74: rport 706e1: ffff88013d7d7000 r 2 Delete work
> >> event 3
> >> [ 1366.270279] host74: rport 706e1: ffff88013d7d7000 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.279149] host74: rport 706e1: ffff88013d7d7000 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.289276] host74: rport 700e1: ffff88012c8cf200 r 2 Delete work
> >> event 3
> >> [ 1366.296465] host74: rport 700e1: ffff88012c8cf200 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.305355] host74: rport 700e1: ffff88012c8cf200 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.315481] host74: rport fffffc: ffff88012c8cf000 r 1 Ready Remove
> >> port
> >> [ 1366.322592] host74: rport fffffc: ffff88012c8cf000 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.332259] host74: rport fffffc: ffff88012c8cf000 r 2 LOGO Delete
> >> port
> >> [ 1366.339376] host74: rport fffffc: ffff88012c8cf000 r 2 Delete work
> >> event 3
> >> [ 1366.346561] host74: rport fffffc: ffff88012c8cf000 r 2 Delete
> >> callback ev 3
> >> [ 1366.353824] host74: lport 6a0000: Received a 3 event for port (fffffc)
> >> [ 1366.360473] host74: rport fffffc: ffff88012c8cf000 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.369345] host74: rport fffffc: ffff88012c8cf000 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.379441] host74: lport 6a0000: Entered LOGO state from Ready state
> >> [ 1366.386010] host74: lport 6a0000: Received a LOGO response closed
> >> [ 1366.398840] host75: rport 6a0000: ffff8801399a8600 r 1 Ready Remove
> >> port
> >> [ 1366.406020] host75: rport 6a0000: ffff8801399a8600 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.415767] host75: rport 6a0000: ffff8801399a8600 r 2 LOGO Delete
> >> port
> >> [ 1366.422955] host75: rport 706e1: ffff8801399a8400 r 1 Ready Remove
> >> port
> >> [ 1366.422959] host75: rport 6a0000: ffff8801399a8600 r 2 Delete work
> >> event 3
> >> [ 1366.437221] host75: rport 706e1: ffff8801399a8400 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.446910] host75: rport 706e1: ffff8801399a8400 r 2 LOGO Delete
> >> port
> >> [ 1366.454025] host75: rport 700e1: ffff880132542e00 r 1 Ready Remove
> >> port
> >> [ 1366.461138] host75: rport 700e1: ffff880132542e00 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.470815] host75: rport 700e1: ffff880132542e00 r 2 LOGO Delete
> >> port
> >> [ 1366.477900] host75: rport 6a0000: ffff8801399a8600 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.486811] host75: rport 6a0000: ffff8801399a8600 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.497074] host75: rport 706e1: ffff8801399a8400 r 2 Delete work
> >> event 3
> >> [ 1366.504264] host75: rport 706e1: ffff8801399a8400 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.513153] host75: rport 706e1: ffff8801399a8400 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.523283] host75: rport 700e1: ffff880132542e00 r 2 Delete work
> >> event 3
> >> [ 1366.530471] host75: rport 700e1: ffff880132542e00 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.539339] host75: rport 700e1: ffff880132542e00 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.549453] host75: rport fffffc: ffff880132542200 r 1 Ready Remove
> >> port
> >> [ 1366.556544] host75: rport fffffc: ffff880132542200 r 1 Ready Port
> >> entered LOGO state from Ready state
> >> [ 1366.566245] host75: rport fffffc: ffff880132542200 r 2 LOGO Delete
> >> port
> >> [ 1366.573343] host75: rport fffffc: ffff880132542200 r 2 Delete work
> >> event 3
> >> [ 1366.580538] host75: rport fffffc: ffff880132542200 r 2 Delete
> >> callback ev 3
> >> [ 1366.587820] host75: lport 70500: Received a 3 event for port (fffffc)
> >> [ 1366.594485] host75: rport fffffc: ffff880132542200 r 2 Delete
> >> Received a LOGO response closed
> >> [ 1366.603363] host75: rport fffffc: ffff880132542200 r 2 Delete
> >> Received a LOGO response, but in state Delete
> >> [ 1366.613488] host75: lport 70500: Entered LOGO state from Ready state
> >> [ 1366.620070] host75: lport 70500: Received a LOGO response closed
> >> [ 1366.630261] device eth0 left promiscuous mode
> >> [ 1366.636641] KGDB: Waiting for remote debugger
> >>
> >> KGDB was entered because of debug code I added showing one exchange still
> >> exists
> >> when fc_exch_mgr_destroy is called for host75.
> >>
> >> Thanks for your time on this.
> >>
> >> Joe
> >>
> >> _______________________________________________
> >> devel mailing list
> >> [email protected]
> >> http://www.open-fcoe.org/mailman/listinfo/devel
> >
> > _______________________________________________
> > devel mailing list
> > [email protected]
> > http://www.open-fcoe.org/mailman/listinfo/devel
>
_______________________________________________
devel mailing list
[email protected]
http://www.open-fcoe.org/mailman/listinfo/devel
