Re: [CentOS] mount bind problem
Dear Robert, Thank you. I think that the problem was caused by anti virus GravityZone. After stopping GravityZone demon, reboot makes normal file systems. I'll ask to GravityZone support about this problem. Tadao 2016-04-15 22:09 GMT+09:00 Robert Nichols: > On 04/15/2016 01:05 AM, 望月忠雄 wrote: > >> Dear Robert, >> >> Before sending 'grep -r /home /etc' data, I want tell you what happned >> this >> morning. >> >> In order to solve the /home/home problem, 'umount /home' had been done, >> system had been running in a normal file system. >> But suddenly /home has been lost. >> >> >> Key information of that time is as in the lines. >> And I found in the lines messages log. >> How do you think the reason of trouble. >> > > Since you had two different things mounted on /home, it is unclear how > the system would interpret "umount /home". That is why I did not suggest > doing that. > > From the error, it seems that some checks got bypassed and a busy > filesystem was disconnected. It is likely that there is now some > filesystem corruption on /dev/vdb, and you should reboot with a forced > fsck to clean it up. You really should track down the cause of the > extra mount of /dev/vdb on /home first, or the system will just come > up wrong again. > > > -- > Bob Nichols "NOSPAM" is really part of my email address. > Do NOT delete it. > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos > ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
On 04/15/2016 01:05 AM, 望月忠雄 wrote: Dear Robert, Before sending 'grep -r /home /etc' data, I want tell you what happned this morning. In order to solve the /home/home problem, 'umount /home' had been done, system had been running in a normal file system. But suddenly /home has been lost. Key information of that time is as in the lines. And I found in the lines messages log. How do you think the reason of trouble. Since you had two different things mounted on /home, it is unclear how the system would interpret "umount /home". That is why I did not suggest doing that. From the error, it seems that some checks got bypassed and a busy filesystem was disconnected. It is likely that there is now some filesystem corruption on /dev/vdb, and you should reboot with a forced fsck to clean it up. You really should track down the cause of the extra mount of /dev/vdb on /home first, or the system will just come up wrong again. -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
Dear Robert, Before sending 'grep -r /home /etc' data, I want tell you what happned this morning. In order to solve the /home/home problem, 'umount /home' had been done, system had been running in a normal file system. But suddenly /home has been lost. Key information of that time is as in the lines. And I found in the lines messages log. How do you think the reason of trouble. # ls -l /home total 0 # ls -l / total 3614 -rw---1 root root 12288 Apr 15 05:00 aquota.group -rw---1 root root 10240 Apr 15 05:00 aquota.user dr-xr-xr-x. 2 root root4096 Nov 15 18:35 bin dr-xr-xr-x. 5 root root5120 Nov 15 18:36 boot drwxr-xr-x2 root root4096 Jul 24 2015 cgroup drwxr-xr-x 17 root root3820 Apr 14 10:24 dev drwxr-xr-x. 75 root root4096 Apr 14 10:31 etc drwxr-xr-x2 root root4096 Nov 15 19:04 home dr-xr-xr-x. 8 root root4096 Aug 8 2015 lib dr-xr-xr-x. 8 root root 12288 Nov 15 18:35 lib64 # ls -l /mnt/extradiskA total 48 -rw--- 1 root root 7168 Apr 15 02:14 aquota.group -rw--- 1 root root 7168 Apr 14 10:22 aquota.user drwxr-x--- 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 15 01:10 log drwx-- 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Apr 14 10:24 mysql # ls -l /mnt/extradiskA/home total 8 drwxr-xr-x. 7 admin admin 4096 Sep 21 2015 admin drwxr-xr-x 4 kavfile kavfile 4096 Jan 1 12:00 kavfile # mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw,usrquota,grpquota) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) /dev/vdb on /mnt/extradiskA type ext4 (rw,usrquota,grpquota) /mnt/extradiskA/home on /home type none (rw,bind) /mnt/extradiskA/log on /var/log type none (rw,bind) /mnt/extradiskA/mysql on /var/lib/mysql type none (rw,bind) /mnt/extradiskA/.backup on /.backup type none (rw,bind) /mnt/extradiskA/.daily_backup on /.daily_backup type none (rw,bind) /mnt/extradiskA/backups on /var/backups type none (rw,bind) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) / on /tmp/tmproot type none (rw,bind) # df Filesystem 1K-blocksUsed Available Use% Mounted on /dev/mapper/VolGroup-lv_root 12778200 3149592 8972840 26% / tmpfs 961176 0961176 0% /dev/shm /dev/vda1 487652 205564256488 45% /boot /dev/vdb 41153856 542076 38514628 2% /mnt/extradiskA # cat /etc/fstab # # /etc/fstab # Created by anaconda on Mon Jul 28 08:21:01 2014 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/VolGroup-lv_root / ext4 defaults,usrquota,grpquota1 1 UUID=9a855356-ee00-4ae7-a185-226566ce1dba /boot ext4 defaults1 2 UUID=093c9b95-8336-4682-9927-132dbdf2f082 /mnt/extradiskA ext4 defaults,usrquota,grpquota 1 2 /dev/mapper/VolGroup-lv_swap swapswapdefaults 0 0 tmpfs /dev/shmtmpfs defaults0 0 devpts /dev/ptsdevpts gid=5,mode=620 0 0 sysfs /syssysfs defaults0 0 proc/proc procdefaults0 0 /mnt/extradiskA/home/home nonebind0 0 /mnt/extradiskA/log /var/lognonebind0 0 /mnt/extradiskA/mysql/var/lib/mysql nonebind0 0 /mnt/extradiskA/.backup /.backupnonebind0 0 /mnt/extradiskA/.daily_backup /.daily_backup nonebind0 0 /mnt/extradiskA/backups /var/backups nonebind0 0 Apr 15 00:00:03 teisui2 kernel: [ cut here ] Apr 15 00:00:03 teisui2 kernel: WARNING: at fs/ext4/inode.c:3945 ext4_flush_unwritten_io+0x74/0x80 [ext4]() (Not tainted) Apr 15 00:00:03 teisui2 kernel: Hardware name: KVM Apr 15 00:00:03 teisui2 kernel: Modules linked in: dazukofs(U) ipv6 xt_multiport iptable_filter ip_tables joydev sg virtio_balloon serio_raw virtio_net i2c_piix4 i2 c_core ext4 jbd2 mbcache virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Apr 15 00:00:03 teisui2 kernel: Pid: 1707, comm: spamd Not tainted 2.6.32-573.1.1.el6.x86_64 #1 Apr 15 00:00:03 teisui2 kernel: Call Trace: Apr 15 00:00:03 teisui2 kernel: [] ? warn_slowpath_common+0x91/0xe0 Apr 15 00:00:03 teisui2 kernel: [] ? warn_slowpath_null+0x1a/0x20 Apr 15 00:00:03 teisui2 kernel: [] ? ext4_flush_unwritten_io+0x74/0x80 [ext4]
Re: [CentOS] mount bind problem
On 04/13/2016 08:44 PM, 望月忠雄 wrote: # mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw,usrquota,grpquota) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) /dev/vdb on /mnt/extradiskA type ext4 (rw,usrquota,grpquota) /mnt/extradiskA/home on /home type none (rw,bind) /mnt/extradiskA/log on /var/log type none (rw,bind) /mnt/extradiskA/mysql on /var/lib/mysql type none (rw,bind) /mnt/extradiskA/.backup on /.backup type none (rw,bind) /mnt/extradiskA/.daily_backup on /.daily_backup type none (rw,bind) /mnt/extradiskA/backups on /var/backups type none (rw,bind) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/vdb on /home type ext4 ^^^ (rw,relatime,barrier=1,data=ordered,usrquota,grpquota) Well, there it is, the extra mount of /dev/vdb on /home. It's not apparent how it got that way. Since it appears to be at the bottom of /etc/mtab, it apparently happened _after_ all the other mounts. Some script must have done that. If it's not in /etc/rc.d/rc.local, then I'd do a progressively wider search for references to /home, starting with grep -r /home /etc -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
Dear Robert, Thank you. The state after reboot are as follows. # LANG=C lsblk NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:01 1024M 0 rom vda 252:00 15G 0 disk |-vda1 252:10 500M 0 part /boot `-vda2 252:20 14.5G 0 part |-VolGroup-lv_root (dm-0) 253:00 12.5G 0 lvm / `-VolGroup-lv_swap (dm-1) 253:102G 0 lvm [SWAP] vdb 252:16 0 40G 0 disk /mnt/extradiskA # mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw,usrquota,grpquota) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) /dev/vdb on /mnt/extradiskA type ext4 (rw,usrquota,grpquota) /mnt/extradiskA/home on /home type none (rw,bind) /mnt/extradiskA/log on /var/log type none (rw,bind) /mnt/extradiskA/mysql on /var/lib/mysql type none (rw,bind) /mnt/extradiskA/.backup on /.backup type none (rw,bind) /mnt/extradiskA/.daily_backup on /.daily_backup type none (rw,bind) /mnt/extradiskA/backups on /var/backups type none (rw,bind) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/vdb on /home type ext4 (rw,relatime,barrier=1,data=ordered,usrquota,grpquota) # ls -l /home total 48 -rw--- 1 root root 7168 Nov 15 19:10 aquota.group -rw--- 1 root root 7168 Apr 14 10:22 aquota.user drwxr-x--- 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 14 10:23 log drwx-- 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Apr 14 10:24 mysql # ls -al /mnt/extradiskA total 72 drwxr-xr-x 9 root root 4096 Nov 15 19:10 . drwxr-xr-x. 3 root root 4096 Nov 15 19:00 .. -rw--- 1 root root 7168 Nov 15 19:10 aquota.group -rw--- 1 root root 7168 Apr 14 10:22 aquota.user drwxr-xr-x 2 root root 12288 Apr 14 00:00 .backup drwxr-x--- 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 2 root root 4096 Nov 16 02:30 .daily_backup drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 14 10:23 log drwx-- 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Apr 14 10:24 mysql # ls -al /mnt/extradiskA/home total 16 drwxr-xr-x 4 rootroot4096 Jan 1 11:59 . drwxr-xr-x 9 rootroot4096 Nov 15 19:10 .. drwxr-xr-x. 7 admin admin 4096 Sep 21 2015 admin drwxr-xr-x 4 kavfile kavfile 4096 Jan 1 12:00 kavfile # mkdir /tmp/tmproot # mount --bind / /tmp/tmproot # ls -a /tmp/tmproot/mnt/extradiskA . .. # cat /etc/fstab # # /etc/fstab # Created by anaconda on Mon Jul 28 08:21:01 2014 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/VolGroup-lv_root / ext4 defaults,usrquota,grpquota1 1 UUID=9a855356-ee00-4ae7-a185-226566ce1dba /boot ext4 defaults1 2 UUID=093c9b95-8336-4682-9927-132dbdf2f082 /mnt/extradiskA ext4 defaults,usrquota,grpquota 1 2 /dev/mapper/VolGroup-lv_swap swapswapdefaults 0 0 tmpfs /dev/shmtmpfs defaults0 0 devpts /dev/ptsdevpts gid=5,mode=620 0 0 sysfs /syssysfs defaults0 0 proc/proc procdefaults0 0 /mnt/extradiskA/home/home nonebind0 0 /mnt/extradiskA/log /var/lognonebind0 0 /mnt/extradiskA/mysql/var/lib/mysql nonebind0 0 /mnt/extradiskA/.backup /.backupnonebind0 0 /mnt/extradiskA/.daily_backup /.daily_backup nonebind0 0 /mnt/extradiskA/backups /var/backups nonebind0 0 2016-04-13 23:32 GMT+09:00 Robert Nichols: > On 04/13/2016 12:02 AM, 望月忠雄 wrote: > >> In /etc/rc.d/rc.sysinit, >> there's "mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O >> no_netdev". >> >> >> I have other same type servers. And on other servers with same /etc/fstsb, >> file system is correct. >> There's no difference between normal server's rc.sysinit and this problem >> server's rc.sysinit too. >> > > It's not apparent how /dev/vdb (the whole filesystem) came to be mounted > on both /mnt/extradiskA and /home. Let's see the output > from "lsblk" and "mount". Did you look in /mnt/extradiskA/home > and verify that just the expected user directories are there > (no "/mnt/extradiskA/home/home")? > > It might also be worthwhile to verify that the mount point > directory /mnt/extradiskA in the root filesystem is empty. If > there happens to be a /mnt/extradiskA/home
Re: [CentOS] mount bind problem
On 04/13/2016 12:02 AM, 望月忠雄 wrote: In /etc/rc.d/rc.sysinit, there's "mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O no_netdev". I have other same type servers. And on other servers with same /etc/fstsb, file system is correct. There's no difference between normal server's rc.sysinit and this problem server's rc.sysinit too. It's not apparent how /dev/vdb (the whole filesystem) came to be mounted on both /mnt/extradiskA and /home. Let's see the output from "lsblk" and "mount". Did you look in /mnt/extradiskA/home and verify that just the expected user directories are there (no "/mnt/extradiskA/home/home")? It might also be worthwhile to verify that the mount point directory /mnt/extradiskA in the root filesystem is empty. If there happens to be a /mnt/extradiskA/home directory present before /dev/vdb gets mounted, that could allow the mounts to be performed out of order, though I don't really see how that could lead the to result you have. mkdir /tmp/tmproot mount --bind / /tmp/tmproot ls -a /tmp/tmproot/mnt/extradiskA umount /tmp/tmproot -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
In /etc/rc.d/rc.sysinit, there's "mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O no_netdev". I have other same type servers. And on other servers with same /etc/fstsb, file system is correct. There's no difference between normal server's rc.sysinit and this problem server's rc.sysinit too. Please tell me other possibility which I have to check. 2016-04-12 23:58 GMT+09:00 Robert Nichols: > On 04/12/2016 05:37 AM, Sylvain CANOINE wrote: > >> >> >> - Mail original - >> >>> De: "望月忠雄" À: "centos" >>> Envoyé: Mardi 12 Avril 2016 08:17:13 Objet: >>> [CentOS] mount bind problem >>> >> >> I have set on fstab /home with 'mount bind' but it seems like bind >>> is not effective. >>> >> Indeed. By default, nothing mounts "bind" filesystems. The init >> scripts don't read the hole fstab file, there are filters to mount >> only some filesystems types. If you want to mount the "bind" FS on >> boot, you need to add the related commands into /etc/rc.d/rc.local. >> > > Nonsense! I'm also running CentOS 6.7 and have several bind-mounts > in my /etc/fstab: > >/var/home/home none bind 0 0 >/var/lib/libvirt/etc /etc/libvirtnone bind 0 0 >/srv/news/var/spool/news none bind 0 0 > > All of them occur automatically when the system boots. The > "mount -a" in /etc/rc.d/rc.sysinit takes care of them just fine. > The filter on that command is _excluding_ certain types. Note > that the first two letters are "no", which applies to all the > listed types: > > mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O > no_netdev > > [rkn] ~ $ df > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/mapper/vg_omega3g-rootvol > 11963960 5995348 5337828 53% / > tmpfs 8194160 340 8193820 1% /dev/shm > /dev/sda1 689128130200508752 21% /boot > /dev/mapper/vg_omega3g-var > 38166008 8282340 27921892 23% /var > /dev/mapper/vg_omega3g-srvlv > 564310312 360619492 175002488 68% /srv > /dev/mapper/vg_omega3g-virt > 194274580 119917252 64617656 65% /var/lib/libvirt > tmpfs 2097152 652 2096500 1% /tmp > [rkn] ~ $ df /home /etc/libvirt /var/spool/news > Filesystem 1K-blocks Used Available Use% Mounted on > /var/home 38166008 8282340 27921892 23% /home > /var/lib/libvirt/etc 194274580 119917252 64617656 65% /etc/libvirt > /srv/news564310312 360619508 175002472 68% /var/spool/news > > -- > Bob Nichols "NOSPAM" is really part of my email address. > Do NOT delete it. > > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos > ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
On 04/12/2016 05:37 AM, Sylvain CANOINE wrote: - Mail original - De: "望月忠雄"À: "centos" Envoyé: Mardi 12 Avril 2016 08:17:13 Objet: [CentOS] mount bind problem I have set on fstab /home with 'mount bind' but it seems like bind is not effective. Indeed. By default, nothing mounts "bind" filesystems. The init scripts don't read the hole fstab file, there are filters to mount only some filesystems types. If you want to mount the "bind" FS on boot, you need to add the related commands into /etc/rc.d/rc.local. Nonsense! I'm also running CentOS 6.7 and have several bind-mounts in my /etc/fstab: /var/home/home none bind 0 0 /var/lib/libvirt/etc /etc/libvirtnone bind 0 0 /srv/news/var/spool/news none bind 0 0 All of them occur automatically when the system boots. The "mount -a" in /etc/rc.d/rc.sysinit takes care of them just fine. The filter on that command is _excluding_ certain types. Note that the first two letters are "no", which applies to all the listed types: mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O no_netdev [rkn] ~ $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_omega3g-rootvol 11963960 5995348 5337828 53% / tmpfs 8194160 340 8193820 1% /dev/shm /dev/sda1 689128130200508752 21% /boot /dev/mapper/vg_omega3g-var 38166008 8282340 27921892 23% /var /dev/mapper/vg_omega3g-srvlv 564310312 360619492 175002488 68% /srv /dev/mapper/vg_omega3g-virt 194274580 119917252 64617656 65% /var/lib/libvirt tmpfs 2097152 652 2096500 1% /tmp [rkn] ~ $ df /home /etc/libvirt /var/spool/news Filesystem 1K-blocks Used Available Use% Mounted on /var/home 38166008 8282340 27921892 23% /home /var/lib/libvirt/etc 194274580 119917252 64617656 65% /etc/libvirt /srv/news564310312 360619508 175002472 68% /var/spool/news -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
Thank you Sylvain. I want try to use /etc/rc.d/rc.local. Are there any document about writing into /etc/rc.d/rc.local? Tadao 2016-04-12 19:37 GMT+09:00 Sylvain CANOINE: > > > - Mail original - > > De: "望月忠雄" > > À: "centos" > > Envoyé: Mardi 12 Avril 2016 08:17:13 > > Objet: [CentOS] mount bind problem > > > I have set on fstab /home with 'mount bind' but it seems like bind is not > > effective. > Indeed. By default, nothing mounts "bind" filesystems. The init scripts > don't read the hole fstab file, there are filters to mount only some > filesystems types. If you want to mount the "bind" FS on boot, you need to > add the related commands into /etc/rc.d/rc.local. > > Sylvain. > Pensez ENVIRONNEMENT : n'imprimer que si ncessaire > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos > ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] mount bind problem
- Mail original - > De: "望月忠雄"> À: "centos" > Envoyé: Mardi 12 Avril 2016 08:17:13 > Objet: [CentOS] mount bind problem > I have set on fstab /home with 'mount bind' but it seems like bind is not > effective. Indeed. By default, nothing mounts "bind" filesystems. The init scripts don't read the hole fstab file, there are filters to mount only some filesystems types. If you want to mount the "bind" FS on boot, you need to add the related commands into /etc/rc.d/rc.local. Sylvain. Pensez ENVIRONNEMENT : n'imprimer que si ncessaire ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos