[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 Edward Tomasz Napierala changed: What|Removed |Added Status|Open

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #43 from Julien Cigar --- (In reply to Ben RUBSON from comment #41) as soon as I have a little time .. :) -- You are receiving this mail because: You are the assignee for the bug.

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #42 from Julien Cigar --- also, looks similar to bug #215727 -- You are receiving this mail because: You are the assignee for the bug. ___

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #41 from Ben RUBSON --- OK, so same issue as mine (for which I opened this bug report), with 10.3. Which I was not able to reproduce with 11. Any plan to upgrade to 11 to be safe ? -- You are receiving

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #40 from Julien Cigar --- I had to reboot.. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #39 from Ben RUBSON --- Strange, as if mbufs were "consumed" but not released. And when devices are disconnected, can you reconnect to them properly ? Or do you need to change their target name or to

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #38 from Julien Cigar --- (In reply to Ben RUBSON from comment #37) it only delayed the problem, .. but it would be interesting if you could test on your side..! -- You are receiving this mail because:

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #37 from Ben RUBSON --- And no improvements ? -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #36 from Julien Cigar --- (In reply to Ben RUBSON from comment #35) yes, .. but only slightly as the machine has only 8GB of RAM -- You are receiving this mail because: You are the assignee for the

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #35 from Ben RUBSON --- Did you try to increase some sysctls with jumbo frames on, as proposed by Steven ? -- You are receiving this mail because: You are the assignee for the bug.

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #34 from Julien Cigar --- No problems for more than a month now.. I tend to conclude that the MTU 9000 was the root cause of this problem (?) -- You are receiving this mail because: You are the assignee

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #33 from Julien Cigar --- an update on this: I turned off jumbo frames and TSO on the replication interface and the issue hasn't been repeated until now...! filer1% uptime 10:15AM up 18 days, 13:35, 1

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #32 from Julien Cigar --- (In reply to Steven Hartland from comment #30) Yes I already checked the error counters, but it's 0: root@filer1:/var/log # netstat -I bge1 NameMtu Network Address

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #31 from Julien Cigar --- Y(In reply to Steven Hartland from comment #29) Yes it seems so: root@filer1:/var/log # vmstat -z | head -n 1; vmstat -z | grep mbuf ITEM SIZE LIMIT USED

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #30 from Steven Hartland --- If its dropping in the driver due to failure go get a jumbo mbuf you should see this in the output netstat -i. -- You are receiving this mail because: You are the assignee for

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #29 from Steven Hartland --- I'm assuming vmstat agrees with netstat -m e.g. vmstat -z | head -n 1; vmstat -z | grep mbuf It could be that the disconnect is the cause of mbuf issue and not the other way

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #28 from Steven Hartland --- checksums a likely just due to hw checksum offloading. -- You are receiving this mail because: You are the assignee for the bug. ___

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #27 from Steven Hartland --- checksums a likely just due to hw checksum offloading. -- You are receiving this mail because: You are the assignee for the bug. ___

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #26 from Julien Cigar --- I don't know if it's related but I also noticed some incorrect cksum: 00:14:03.380734 IP (tos 0x0, ttl 64, id 19369, offset 0, flags [DF], proto TCP (6), length 100, bad cksum 0

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #25 from Julien Cigar --- (In reply to Steven Hartland from comment #24) I don't have any mention of nmbjumbo9 in /var/log/messages, I haven't raised kern.ipc.nmbjumbo9 yet, currently it's:

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 Steven Hartland changed: What|Removed |Added CC|

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #23 from Julien Cigar --- Ok.. I definitively think it's an issue with jumbo frames, I got another disconnection and again the "requests for jumbo clusters denied" (9k) counter increased: 0/304/0

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #22 from Julien Cigar --- Yep it's strange, I will definitively upgrade to 11 when possible .. Other infos: root@filer1:/home/jcigar # netstat -m 1285/6575/7860 mbufs in use (current/cache/total)

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #21 from Ben RUBSON --- I was also using jumbo frames with 10.3, but unfortunately I did not look at dropped packets when I had the issues. Note that with 11, I still use jumbo frames and never encounter

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #20 from Julien Cigar --- Also I noticed some Idrop packets on the iscsi interface of the initiator side: root@filer1:/home/jcigar # netstat -I bge1 NameMtu Network Address Ipkts

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #19 from Julien Cigar --- this is what I have on the target: root@filer2:/home/jcigar # sysctl -a|grep -i 'iscsi' kern.iscsi.fail_on_shutdown: 1 kern.iscsi.fail_on_disconnection: 1 kern.iscsi.maxtags:

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #18 from Julien Cigar --- Problem appeared again today, after ~15 days of uptime, always on FreeBSD filer1.prod.lan 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #16 from Julien Cigar --- Thank you Ben. Can you confirm (once again) that you haven't had this problem anymore in 11.0-RELEASE? If so I could just upgrade then .. :) -- You are receiving this mail

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #14 from Julien Cigar --- This is also what I thought at first, but there is no error messages related to the network interfaces in the logs, and all Ierrs, Idrop, and Coll counters are zeros. There is a

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #13 from Edward Tomasz Napierala --- The "WARNING: 10.20.30.31 (iqn.1994-09.org.freebsd:filer1.prod.lan): no ping reply (NOP-Out) after 5 seconds; dropping connection" and "WARNING: 10.20.30.32

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #12 from Julien Cigar --- Some more info: on filer2: root@filer2:/home/jcigar # cat /etc/ctl.conf auth-group ag0 { chap xxx xxx } portal-group pg0 { discovery-auth-group no-authentication

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 Ben RUBSON changed: What|Removed |Added Status|New |Closed

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 Ben RUBSON changed: What|Removed |Added Version|11.0-STABLE |10.3-RELEASE

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 Ben RUBSON changed: What|Removed |Added Version|10.3-RELEASE|11.0-STABLE

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #7 from Ben RUBSON --- I tried 11-RC1 and was not able to reproduce the issue. With 10.3, after about 10 NIC toggles, some devices fail. With 11-RC1, more than one thousand toggles later, all my devices

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #6 from Ben RUBSON --- (In reply to Ben RUBSON from comment #5) >And the most important, regarding the bug : >when devices do not want to correctly reconnect, I found that it is because >iscsi is stuck in

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #5 from Ben RUBSON --- I made further troubleshooting. Regarding the number of iscsid processes which increases, I've found timeout settings which permit to always have only one process per target :

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #4 from Ben RUBSON --- One strange thing I noticed. (I put all things that could be interesting from my troubleshooting) As soon as I put the network interface down, I get the following message on target

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #3 from Ben RUBSON --- (In reply to Ben RUBSON from comment #0) >The only workaround I found is to reboot I even have to hard-reboot, because of this never ending : iscsi_terminate_sessions: waiting for

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #2 from Ben RUBSON --- Quite difficult to reproduce, but I managed to do it again. This time the problematic target (only one /17) is seen as disconnected. ### Initiator : # iscsictl -Ra # iscsictl -L

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 --- Comment #1 from Ben RUBSON --- On initiator, netstat also returns the following : tcp4 0 0 192.168.2.1.49715 192.168.2.2.3260 CLOSED tcp4 0 0 192.168.2.1.49708

[Bug 211990] iscsi fails to reconnect and does not release devices

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 Bug ID: 211990 Summary: iscsi fails to reconnect and does not release devices Product: Base System Version: 10.3-RELEASE Hardware: Any OS: Any Status: New