[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
Thanks for coming back rarylson! It seems no one realized this implicitly fixed this issue when the changes were made. Closing the bug as fix released (since artful). ** Changed in: tgt (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
Hi everyone. It seems like the TGT bug (when stopping, reloading or restarting the TGT deamon) is solved in Ubuntu Artful (17.10). I did not make tests, but now TGT uses the upstream init.d script or the TGT systemd config file (both of them address all of the previous described problems). The commands `tgtadm --op update --mode sys --name State -v offline`, `tgt-admin --offline ALL` and `tgt-admin -e -c /etc/tgt/targets.conf`, for example, correctly turns TGT down (the to first commands) and reloads in the correct way (the last command). I think this bug report can be closed now, since it seems to be solved in newer versions. Below, the systemd tgt service config file: ``` [Unit] Description=(i)SCSI target daemon Documentation=man:tgtd(8) After=network.target [Service] Type=notify TasksMax=infinity ExecStart=/usr/sbin/tgtd -f ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline ExecStop=/usr/sbin/tgt-admin --offline ALL ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null -f ExecStop=/usr/sbin/tgtadm --op delete --mode system ExecReload=/usr/sbin/tgt-admin --update ALL -c /etc/tgt/targets.conf Restart=on-failure [Install] WantedBy=multi-user.target ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
Hi Serge Hallyn, I run some new tests and, in short, the problem I found is actually three problems (maybe this bug should be broke in three). I will summarize the problems I found: - Tgtd breaks when it receives a SIGHUP; - When we send a SIGHUP to tgtd (`reload tgt`), the deamon stops (and it logs an fault in syslog), and then upstart respawns the deamon. It is, run `reload tgt` appears to be equivalent to run `kill -9 TGT_ROOT_PID; start tgt;`; - Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command; - When tgtd stops and closes the devices, the system will probably run a `blkid` in this devices. So, when tgtd starts again, some devices may be still opened by the blkid command, and neither tgtd or `tgt-admin -e` will add this device (I'm considering "allow-in-use" is configured to "off" in this moment); - The `stop tgt` command really stops tgtd even if some LUNs still in use; - This can cause data corruption. Debian wheezy (https://packages.debian.org/source/wheezy/tgt) solves these problem using the initd.sample from upstream. In the next explanations, I'm using the source code found in: http://packages.ubuntu.com/source/precise/tgt - Tgtd breaks when it receives a SIGHUP: >From the initd.sample file (tgt-1.0.17/scripts/initd.sample, line 79): reload() { echo "Updating target framework daemon configuration" # Update configuration for targets. Only targets which # are not in use will be updated. tgt-admin --update ALL -c $TGTD_CONFIG &>/dev/null RETVAL=$? if [ "$RETVAL" -eq 107 ] ; then echo "tgtd is not running" exit 1 fi } It is, the `service tgt reload` command will run a `tgt-admin --update ALL` command, and this last command works fine. The `service tgt reload` command DOESN'T send a SIGHUP signal in Debian (and in all Linux distributions that uses the initd.sample file). I don't see how fix this in Ubuntu using upstart, because in Ubuntu 12.04 Upstart always send a SIGHUP signal. In newer versions of Ubuntu we can use the `reload signal` stanza, but we can't run a script (`tgt- admin --update ALL`) when `reload tgt` was called. - Tgtd has an unexpected behavior when we run a `restart tgt` or a `stop tgt; start tgt` command: First, let's read the tgt-admin source code (tgt-1.0.17/scripts/tgt- admin, line 1240): # Check if userspace uses this device my $lsof_check = check_exe("lsof"); if ($lsof_check ne 1) { system("lsof $backing_store &>/dev/null"); my $exit_value = $? >> 8; if ($exit_value eq 0) { execute("# Device $backing_store is used (already tgtd target?)."); execute("# Run 'lsof $backing_store' to see the details."); return 0; } } It is, when `tgt-admin -e` (in the `post-start` stanza) is executed, if some device still opened (in our case, by the `blkid` command), we can add this device as a LUN. In my attachment tgt.override, a hack was proposed to avoid this problem. However, my tests showed that waiting one second between stops and starts tgtd solves the problem. Actually, 0.3 second waiting solved my problem in all tests (but 1 second is a more secure interval). >From the initd.sample file (tgt-1.0.17/scripts/initd.sample, line 125): restart) TASK=restart stop && start ;; And from line 19: # Put tgtd into "offline" state until all the targets are configured. # We don't want initiators to (re)connect and fail the connection # if it's not ready. tgtadm --op update --mode sys --name State -v offline # Configure the targets. tgt-admin -e -c $TGTD_CONFIG # Put tgtd into "ready" state. tgtadm --op update --mode sys --name State -v ready A fix for this problem can be: # Put tgtd into "offline" state until all the targets are configured. # We don't want initiators to (re)connect and fail the connection # if it's not ready. tgtadm --op update --mode sys --name State -v offline + # Wait the system free the backing-stores (a `blkid` command + # may still be running) + # See: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267 + [ "$TASK" == "restart" ] && sleep 1 # Configure the targets. tgt-admin -e -c $TGTD_CONFIG # Put tgtd into "ready" state. tgtadm --op update --mode sys --name State -v ready - The `stop tgt` command really stops tgtd even if some LUNs still in use: Let's see the source code again (tgt-1.0.17/scripts/initd.sample, line 35): # Remove all targets. It only removes targets which are not in use. tgt-admin --update ALL -c /dev/null &>/dev/null # tgtd will exit if all targets were removed tgtadm --op delete --mode sys
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
Thanks for taking the time to report this bug. It doesn't look like it's been addressed upstream at all (no catching of sighup going on in git head). ** Changed in: tgt (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
** Changed in: tgt (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
In the attachment, I implemented a fix for the `restart` and `stop+start` bug. It's a poor fix, but it may help. Unfortunately, I don't found any fix for the reload method, since upstart (in Ubuntu 12.04) always send a SIGHUP signal to the deamon. ** Attachment added: "tgt.override" https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+attachment/4039771/+files/tgt.override -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
I discovered that, although sending a SIGHUG signal to TGT (using the `reload tgt` command) doesn't works, I can correctly "reload" TGT using the following command: tgt-admin --update ALL I know upstart has the option to change the signal send to a deamon. However, if we could change the `reload` behavior to send no signal and exec an script, we could correct the `reload` wrong behavior. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
Sometimes, the restart operation doesn't works also: $ tgt-admin --dump default-driver iscsi backing-store /dev/vg_vmware/vmware_4 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 backing-store /dev/vg_vmware/vmware_3 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 backing-store /dev/vg_vmware/vmware_2 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 backing-store /dev/vg_vmware/vmware_1 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 backing-store /dev/vg_vmware/vmware_5 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 $ restart tgt tgt start/running, process 30315 $ tgt-admin --dump default-driver iscsi initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 backing-store /dev/vg_vmware/vmware_5 initiator-address 192.168.130.0/24 initiator-address 127.0.0.1 In this case, only the /dev/vg_vmware/vmware_5 device block was added as a LUN. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1294267] Re: tgt doesn't export some LUNs after a reload operation (SIGHUP)
I think the problem can be related to this fact: the TGT daemon don't add a block storage to a LUN if some process was accessing it. The following command show another operation in which a similar problem occours: # All block storage added to the targets $ tgt-admin --dump | grep -o "backing-store \(.*\)" | sed -e "s/^backing-store \(.*\)$/\1/" /dev/vg_vmware/vmware_4 /dev/vg_vmware/vmware_3 /dev/vg_vmware/vmware_2 /dev/vg_vmware/vmware_1 /dev/vg_vmware/vmware_5 # Now, we'll run a stop/start operation. Some device block (previously LUNs) still opened by some process # between the `stop` and `start` operations. In our case, the device blocks was opened by the command `blkid`, # running with the 'root' user. $ stop tgt; lsof /dev/mapper/vg_vmware-vmware_1 /dev/mapper/vg_vmware-vmware_2 \ $ /dev/mapper/vg_vmware-vmware_3 /dev/mapper/vg_vmware-vmware_4 \ $ /dev/mapper/vg_vmware-vmware_5; start tgt; tgt-admin --dump | grep -o "backing-store \(.*\)" | \ $ sed -e "s/^backing-store \(.*\)$/\1/" tgt stop/waiting COMMAND PID USER FD TYPE DEVICE SIZE/OFFNODE NAME blkid 29870 root3r BLK 252,2 0t212992 1585695 /dev/mapper/../dm-2 blkid 29871 root3r BLK 252,0 0t143360 1605724 /dev/mapper/../dm-0 blkid 29872 root3r BLK 252,3 0t180224 1585696 /dev/mapper/../dm-3 blkid 29875 root3r BLK 252,4 0x1fd70bce200 1585697 /dev/mapper/../dm-4 tgt start/running, process 29878 /dev/vg_vmware/vmware_5 $ ls -lh /dev/mapper/vg_vmware-vmware_1 /dev/mapper/vg_vmware-vmware_2 /dev/mapper/vg_vmware-vmware_3 /dev/mapper/vg_vmware-vmware_4 /dev/mapper/vg_vmware-vmware_5 lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_1 -> ../dm-0 lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_2 -> ../dm-2 lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_3 -> ../dm-3 lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_4 -> ../dm-4 lrwxrwxrwx 1 root root 7 Mar 18 15:27 /dev/mapper/vg_vmware-vmware_5 -> ../dm-5 In our test, four device blocks (vg_vmware-vmware_1, vg_vmware-vmware_2, vg_vmware-vmware_3 and vg_vmware-vmware_4) were opened by the blkid command, and the `start tgt` command (that runs `/usr/sbin/tgt-admin -e`) only adds the other lun (vg_vmware-vmware_5). I hope this test helps. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1294267 Title: tgt doesn't export some LUNs after a reload operation (SIGHUP) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1294267/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs