It seems clear that the I/O error is because iscsi is stopped before the device is unmounted:
On Tuesday, December 9, 2014 7:28:52 AM UTC-8, awidde...@hotmail.com wrote: > > Setup iSCSI on CentOS7. Mounted a iSCSI disk and am running a small MySQL > instance on the disk. The iSCSI disk and MySQL instance all come online > fine with booting but when shutting down things seem to get very upset and > the drive does not get unmounted cleanly. > > Does not look like I'm the only one having the issue. Another report that > is very similar was posted here: > > https://www.centos.org/forums/viewtopic.php?f=47&t=49337 > > This might be a systemd issue but figured I'd post here first to see if > anyone else has had this issue and has any suggestions. > > Relevant version info: > > CentOS Linux release 7.0.1406 (Core) > systemd-208-11.el7_0.4.x86_64 > iscsi-initiator-utils-6.2.0.873-21.el7.x86_64 > > Systemd unit file for MySQL server: > > [Unit] > Description=MySQL Server > After=nss-lookup.target network.target remote-fs.target time-sync.target > Wants=nss-lookup.target network.target remote-fs.target time-sync.target > > Logs from journalctl: > > Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): > __ext4_get_inode_loc:4039: inode #58989720: block 235930089: comm mysqld: > unable to read itable block > Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1) in > ext4_reserve_inode_write:4962: IO failure > Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logging out of session > [sid: 1, target: example.target, portal: 192.168.1.30,3260] > Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logout of [sid: 1, > target: example.target, portal: 192.168.1.30,3260] successful. > Dec 09 09:27:03 example.server.com systemd[1]: Stopped Login and scanning > of iSCSI devices. > This looks like a race condition between the SQL shutdown and stopping the "iscsi.service" iSCSI login service. > Dec 09 09:27:03 example.server.com systemd[1]: Stopping Open-iSCSI... > Dec 09 09:27:03 example.server.com systemd[1]: Stopped Open-iSCSI. > There should be no more iSCSI from here on, right? But it turns out that "stopping the iscsid service" doesn't actually wait for the service to stop before it continues on to the next systemd task. (See below) > Dec 09 09:27:03 example.server.com systemd[1]: Stopped MySQL database. > But isn't your MySQL database using an iSCSI target? > Dec 09 09:27:03 example.server.com systemd[1]: Stopping System Time > Synchronized. > Dec 09 09:27:03 example.server.com systemd[1]: Stopped target System Time > Synchronized. > Dec 09 09:27:03 example.server.com systemd[1]: Stopping Remote File > Systems. > Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Remote File > Systems. > Dec 09 09:27:03 example.server.com systemd[1]: Unmounting /iscsi-disk... > And now systemd tried to unmount the iscsi-backed device, but iscsid is already trying to stop. That's not going to work. > Dec 09 09:27:03 example.server.com systemd[1]: Stopping Host and Network > Name Lookups. > Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Host and > Network Name Lookups. > Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device > sdb1, logical block 40960 > And, sure enough, I/O to/from the iSCSI device is failing. > Dec 09 09:27:03 example.server.com iscsid[853]: Connection1:0 to [target: > example.target, portal: 192.168.1.30,3260] through [iface: default] > Dec 09 09:27:03 example.server.com iscsid[853]: iscsid shutting down. > And iscsid finally finishes. It actually took less than a second ... > Dec 09 09:27:03 example.server.com mysqld_safe[1694]: 141209 09:27:03 > mysqld_safe mysqld from pid file /backup/mysql-bacula/data/mysqld.pid ended > Dec 09 09:27:03 example.server.com kernel: EXT4-fs warning (device sdb1): > ext4_end_bio:287: I/O error writing to inode 58989720 (offset 0 size 4096 > starting block 40960) > Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): > ext4_find_entry:1309: inode #58982448: comm mysqld_safe: reading directory > lblock 0 > Dec 09 09:27:03 example.server.com kernel: Aborting journal on device > sdb1-8. > Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device > sdb1, logical block 133726208 > Dec 09 09:27:03 example.server.com kernel: lost page write due to I/O > error on sdb1 > Dec 09 09:27:03 example.server.com kernel: JBD2: Error -5 detected when > updating journal superblock for sdb1-8. > Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): > ext4_put_super:789: Couldn't clean up the journal > Dec 09 09:27:03 example.server.com kernel: EXT4-fs (sdb1): Remounting > filesystem read-only > Dec 09 09:27:03 example.server.com systemd[1]: Unmounted /iscsi-disk. > This is all the filesystem trying and retrying to finish it's I/O before it gives up and lets you unmount the now-unreachable device. > Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network is Online. > Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network is > Online. > Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network. > Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network. > > Any help would be greatly appreciated. > One more question: does all of this just work if you do not use MySQL? -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
