[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, We have started doing focused long-run, performance, stress testing on multiple setups. This is expected to take more than a month time. So based on the internal discussion, we request to keep this bug open until July end. Please let me know if you have any concerns. Regards Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, I am doing good. Thank you. About the bug, I am still trying to reproduce it. Because of some other issues I could not get to the number of test cycles after which this bug reproduce. Now, I am working on two different setups to reproduce this. I suggest to wait until mid next week and then we can decide. Please let me know if you are ok with this. Regards Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, Quick update on this, test is still running and haven't seen kernel panic so far. I will keep it running for few more days. Regards Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, I have configured kdump, enabled "/proc/sys/kernel/hung_task_panic" and started the test. Observed couple of test failures but those are unrelated. Will keep monitoring setup and update you as soon as I hit repro. Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Thanks Guilherme for the details to setup KDUMP. I tried these steps on Ubuntu VM (just to ensure I don't mess up with physical setup) and but landed into some issues. I am still working on to configure KDUMP and repro the issue. Will keep you posted as things progress. Regards Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Thank you so much for the update, Guilherme. I can surely try to reproduce issue again to collect kdump, Please share the configuration setting for kdump. One question, Is there anything more that can be captured from the current repro setup? If not, then I will repurpose the same setup. Thanks Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, Sorry to bother you again. Just thought of checking with you if you have any update on below, 1. Are you able to repro issue at your end? 2. Were you able to confirm if its a lock issue? Thanks Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Thank you Guilherme for the response and taking time to look at this. Thought of checking with you, if there is any further update on reproducing issue at your end? or need more information from the repro setup that I have? Regards Jitendra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
** Attachment added: "Output of dmesg > /root/dmesg.l" https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+attachment/5472103/+files/dmesg.l -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
** Attachment added: "Output of dmesg -c > /root/dmesg.out" https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+attachment/5472100/+files/dmesg.out -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Following command failed root# echo w > /proc/sysrq bash: /proc/sysrq: No such file or directory So tried below command and it worked, root# echo w > /proc/sysrq-trigger ** Attachment added: "Output of dmesg > /root/dmesg.w" https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+attachment/5472102/+files/dmesg.w -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hey Guilherme, Please find below response, (#a) We are using Cisco Hyperflex iSCSI storage. Following link has more details about it. https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-5/m-hxdp-iscsi-manage.html (#b) Please ignore the comment about the "Ubuntu host running in a Virtualized Environment". Ubuntu iSCSI initiator is running on a bare metal. (#c) Yes, the setup is still in repro state. Please refer attached files for output of above commands. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, Please find attachment containing output of commands, "dmesg", "lspci -vvv", "lsblk", "ls -l /sys/block", "mount". Also please note that Ubuntu host is running in a virtualized environment. ** Attachment added: "cmd_output.txt" https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+attachment/5466739/+files/cmd_output.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, Here is the update from my side. 1. sosreport command hung at below message, Setting up archive ... Setting up plugins ... Running plugins. Please wait ... Finishing plugins [Running: block btrfs] Plugin block timed out Plugin btrfs timed out 2. Automation test is long running test that perform below operations in sequence for infinite iterations. a. Create bunch of iSCSI LUNs. b. Discover LUNs through sysfs scan c. Format LUNs d. Perform IO e. Remove LUN f. Delete LUN 3. It takes couple of days to reproduce the issue and its not 100% reproducible. 4. Currently system is in repro state. So I would like to request, If it is possible to get onto WebEx and collect specific information from system? I would like to capture as much info as possible before trying another repro attempt. Thanks for the support. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Hi Guilherme, There is some confidential data present in sosreport and it is scattered all over the place. So team is evaluating if .xz file can be sanitize. Meanwhile, Can you please let me know, specific information that I can extract from the system to share with you? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Thanks for the response Guilherme. I will work with my team to see if there is anything specific that needs to be filter out. Will provide you an update soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] Re: 120 sec kernel timeout is seen during SCSI remove_device.
Just for reference, Following command is used to delete SCSI device. echo "1" >> /sys/block//device/delete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1914456] [NEW] 120 sec kernel timeout is seen during SCSI remove_device.
Private bug reported: Ubuntu Release -- #lsb_release -rd Description:Ubuntu 18.04.5 LTS Release:18.04 #cat /proc/version_signature Ubuntu 4.15.0-122.124-generic 4.15.18 Package version --- #apt-cache policy open-iscsi open-iscsi: Installed: 2.0.874-5ubuntu2.10 Candidate: 2.0.874-5ubuntu2.10 Problem statement and details - During the automation testing of Cisco's iSCSI target using Open-iscsi initiator, Initiator host reported 120 second kernel hang issue. The automation test was doing SCSI remove_device operation when the issue observered. Automation test perform following sequence of operations, 1. Establish iSCSI session 2. Create bunch of iSCSI LUNs. 3. Discover LUNs through sysfs scan 4. Format LUNs 5. Perform IO 6. Remove LUN 7. Delete LUN Observations from initiator host: 1. Already discovered iSCSI LUNs went to offline state. 2. New LUNs are not being discovered. 3. NOP-in/NOP-out PDU exchange works fine from the iSCSI session. Note: Single iSCSI session is present between initiator and target. Expected behavior - SCSI remove_device should succeed and automation test should continue. Issue is observed even with following commit, which has fix for similar issue. https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?id=27dfa4073289ee5737d45b4cfa40b11f5cdeeaa5 Stack trace --- [91832.800739] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [91832.809982] Call Trace: [91832.809994] __schedule+0x24e/0x880 [91832.810002] ? __enqueue_entity+0x5c/0x60 [91832.810006] ? select_task_rq_fair+0x642/0xab0 [91832.810008] schedule+0x2c/0x80 [91832.810010] schedule_preempt_disabled+0xe/0x10 [91832.810012] __mutex_lock.isra.5+0x276/0x4e0 [91832.810017] ? kernfs_name_hash+0x17/0x80 [91832.810020] __mutex_lock_slowpath+0x13/0x20 [91832.810021] ? __mutex_lock_slowpath+0x13/0x20 [91832.810023] mutex_lock+0x2f/0x40 [91832.810030] scsi_remove_device+0x1e/0x40 [91832.810033] sdev_store_delete+0x55/0xa0 [91832.810036] dev_attr_store+0x1b/0x30 [91832.810039] sysfs_kf_write+0x3c/0x50 [91832.810040] kernfs_fop_write+0x125/0x1a0 [91832.810046] __vfs_write+0x1b/0x40 [91832.810048] vfs_write+0xb1/0x1a0 [91832.810050] SyS_write+0x5c/0xe0 [91832.810055] do_syscall_64+0x73/0x130 [91832.810058] entry_SYSCALL_64_after_hwframe+0x41/0xa6 ** Affects: open-iscsi (Ubuntu) Importance: Undecided Status: Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1914456 Title: 120 sec kernel timeout is seen during SCSI remove_device. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1914456/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
