Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
On Tue 03 Jan 2012 at 05:51PM, John D Groenveld wrote: > In message <20120103220311.go24...@ultra24.us.oracle.com>, Mike Gerdts writes: > >It kinda sounds like something from the global zone had stepped into > >some filesystems that were temporarily mounted during an attach process. > >This is backed up by the evil in the attach log: > > > > Lots of evil in attach log: > > [Sun Jan 1 21:11:30 EST 2012] Mounting > > rpool/var/zones/search-1/rpool/export/home at /tmp/tmp.7kayqJ/export/home > > with ZFS temporary mount > > cannot unmount '/tmp/tmp.7kayqJ/export/home': Device busy > > cannot unmount '/tmp/tmp.7kayqJ/export': Device busy > > rmdir: directory "/tmp/tmp.7kayqJ": Directory not empty > > > >Do you by any chance have a /tmp cleaner (or something else that does a > >find or du) running at roughly the same time? If so, the -mount option > >to find or the -d option to du may be a help to prevent recurrence. > >/tmp/tmp.7kayqJ should have been created rwx by root only. > > Besides my backup cron, I don't run any custom bits in global. > Nothing jumps out among the stock services that might be willy nilly > performing IO in /tmp. > > Why shouldn't zoneadm's migration update umount -f these mounts > once the migration has been performed? > I think that's preferred to skipping the attach checks and balances > with attach -F. In most cases, the use of umount -f has been avoided in this code as it is more likely to be hide some other problem that exists. I think I may see the other problem that exists, but it would require a bit of investigation to know for sure. By any chance are either of the following true? - The zone's /export/home file system has more files in it than it used to. In particular, are there now enough files in it that find will now generate more than 5120 bytes of output whereas before that wasn't the case? - The disk is busy doing other things such that these reads from the zone's /export/home are pretty slow to return? In any case, please let me know if you start to see this problem more regularly. I've opened a somewhat low priority bug: 7126819 migrate_export can get EBUSY while unmounting zone's rpool/export/home dataset If it repeats for you I'll bump the priority up. If a fix is important to you, please open a service request and ask for an escalation to be filed. -- Mike Gerdts Solaris Core OS / Zones http://blogs.oracle.com/zoneszone/ ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
In message <20120103220311.go24...@ultra24.us.oracle.com>, Mike Gerdts writes: >It kinda sounds like something from the global zone had stepped into >some filesystems that were temporarily mounted during an attach process. >This is backed up by the evil in the attach log: > > Lots of evil in attach log: > [Sun Jan 1 21:11:30 EST 2012] Mounting > rpool/var/zones/search-1/rpool/export/home at /tmp/tmp.7kayqJ/export/home > with ZFS temporary mount > cannot unmount '/tmp/tmp.7kayqJ/export/home': Device busy > cannot unmount '/tmp/tmp.7kayqJ/export': Device busy > rmdir: directory "/tmp/tmp.7kayqJ": Directory not empty > >Do you by any chance have a /tmp cleaner (or something else that does a >find or du) running at roughly the same time? If so, the -mount option >to find or the -d option to du may be a help to prevent recurrence. >/tmp/tmp.7kayqJ should have been created rwx by root only. Besides my backup cron, I don't run any custom bits in global. Nothing jumps out among the stock services that might be willy nilly performing IO in /tmp. Why shouldn't zoneadm's migration update umount -f these mounts once the migration has been performed? I think that's preferred to skipping the attach checks and balances with attach -F. John groenv...@acm.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
On Tue 03 Jan 2012 at 04:42PM, John D Groenveld wrote: > In message <201201031723.q03hnbfr001...@elvis.arl.psu.edu>, John D Groenveld > writes: > >Lots of evil in attach log: > >[Sun Jan 1 21:11:30 EST 2012] Mounting > >rpool/var/zones/search-1/rpool/export/home at /tmp/tmp.7kayqJ/export/home > >with ZFS temporary mount > >cannot unmount '/tmp/tmp.7kayqJ/export/home': Device busy > >cannot unmount '/tmp/tmp.7kayqJ/export': Device busy > >rmdir: directory "/tmp/tmp.7kayqJ": Directory not empty > > Backgrounded processes performing IO in those directories? > What is the purpose of these temporary mounts? The attach process needs to cope with multiple different dataset layouts that it can be fed. See /usr/lib/brand/shared/README.dsconvert. The zoneadm attach script mounts up the dataset hierarchy at a temporary location to do any required conversion. Obviously, since you just detached the zone from this same host a few seconds earlier, you probably don't need to check for the need to do the conversion. This is part of why I suggested that 'zoneadm attach -F' may be more appropriate for your use. It will assume everying is A-OK and mark the zone as installed. -- Mike Gerdts Solaris Core OS / Zones http://blogs.oracle.com/zoneszone/ ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
On Tue 03 Jan 2012 at 04:02PM, John D Groenveld wrote:
> In message <20120103203031.gl24...@ultra24.us.oracle.com>, Mike Gerdts writes:
> >Can you provide the following:
> >
> >%<---
> >zfs list -o name,mountpoint,canmount,mounted -r rpool/var/zones/search-1
>
> # zfs list -o name,mountpoint,canmount,mounted -r rpool/var/zones/search-1
> NAMEMOUNTPOINT
> CANMOUNT MOUNTED
> rpool/var/zones/search-1/var/opt/zones/search-1
> on yes
> rpool/var/zones/search-1/rpool
> /var/opt/zones/search-1/root/rpool on yes
> rpool/var/zones/search-1/rpool/ROOT legacy
> noauto no
> rpool/var/zones/search-1/rpool/ROOT/zbe-3 /var/opt/zones/search-1/root
> noauto yes
> rpool/var/zones/search-1/rpool/export
> /var/opt/zones/search-1/root/export on yes
> rpool/var/zones/search-1/rpool/export/home
> /var/opt/zones/search-1/root/export/homeon yes
>
> I couldn't figure why from within the zone zfs mount was complaining
> that the export and export/home datasets were busy.
> Then from global I noticed rpool/var/zones/search-1/rpool/export and
> export/home had the temporary mountpoint which was completely
> unexpected.
> After I halt'd and detach'd my zone, umount'd the datasets and
> attach'd the zone the mountpoints corrected themselves.
It kinda sounds like something from the global zone had stepped into
some filesystems that were temporarily mounted during an attach process.
This is backed up by the evil in the attach log:
Lots of evil in attach log:
[Sun Jan 1 21:11:30 EST 2012] Mounting
rpool/var/zones/search-1/rpool/export/home at /tmp/tmp.7kayqJ/export/home with
ZFS temporary mount
cannot unmount '/tmp/tmp.7kayqJ/export/home': Device busy
cannot unmount '/tmp/tmp.7kayqJ/export': Device busy
rmdir: directory "/tmp/tmp.7kayqJ": Directory not empty
Do you by any chance have a /tmp cleaner (or something else that does a
find or du) running at roughly the same time? If so, the -mount option
to find or the -d option to du may be a help to prevent recurrence.
/tmp/tmp.7kayqJ should have been created rwx by root only.
>
> >zonecfg -z search1 info dataset
> >for ds in $(zonecfg -z z1 info dataset | nawk '$1 == "name:" {print $2}')
> >do
> > echo Dataset: $ds
> > zfs list -o name,mountpoint,canmount,mounted,zone $ds
> >done
> >zonecfg -z search1 info fs
Going back to the beginning of the thread I see you had already given
this info. Sorry 'bout that.
>
> # zonecfg -z search-1 info
> zonename: search-1
> zonepath: /var/opt/zones/search-1
> brand: solaris
> autoboot: true
> bootargs: -m verbose
> file-mac-profile:
> pool:
> limitpriv:
> scheduling-class:
> ip-type: exclusive
> hostid:
> fs-allowed:
> fs:
> dir: /ematrix
> special: tank/ematrix
> raw not specified
> type: zfs
> options: []
> net:
> address not specified
> allowed-address not specified
> configure-allowed-address: true
> physical: vnic3
> defrouter not specified
> capped-memory:
> physical: 3G
>
> >Also, any details about changes in the zone configuration and/or package
> >updates since the previous successful backup would be helpful.
>
> I made no changes.
> The other zones on the system had no issues.
It's starting to look like a race with something else on the system.
If there is something beyond your control that likes to walk through
/tmp as root, you could probably add the following to the cron job.
--%<--
mkdir /var/attachtmp
mount -F tmpfs - /var/attachtmp
chmod 1777 /var/attachtmp
export TMPDIR=/var/attachtmp
# Do the stuff you normally do here
unset TMPDIR
umount /var/attachtmp
rmdir /var/attachtmp
--%<--
Adjust as your environment requires.
--
Mike Gerdts
Solaris Core OS / Zones http://blogs.oracle.com/zoneszone/
___
zones-discuss mailing list
zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
In message <201201031723.q03hnbfr001...@elvis.arl.psu.edu>, John D Groenveld writes: >Lots of evil in attach log: >[Sun Jan 1 21:11:30 EST 2012] Mounting >rpool/var/zones/search-1/rpool/export/home at /tmp/tmp.7kayqJ/export/home with >ZFS temporary mount >cannot unmount '/tmp/tmp.7kayqJ/export/home': Device busy >cannot unmount '/tmp/tmp.7kayqJ/export': Device busy >rmdir: directory "/tmp/tmp.7kayqJ": Directory not empty Backgrounded processes performing IO in those directories? What is the purpose of these temporary mounts? John groenv...@acm.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
In message <20120103203031.gl24...@ultra24.us.oracle.com>, Mike Gerdts writes:
>Can you provide the following:
>
>%<---
>zfs list -o name,mountpoint,canmount,mounted -r rpool/var/zones/search-1
# zfs list -o name,mountpoint,canmount,mounted -r rpool/var/zones/search-1
NAMEMOUNTPOINT
CANMOUNT MOUNTED
rpool/var/zones/search-1/var/opt/zones/search-1
on yes
rpool/var/zones/search-1/rpool /var/opt/zones/search-1/root/rpool
on yes
rpool/var/zones/search-1/rpool/ROOT legacy
noauto no
rpool/var/zones/search-1/rpool/ROOT/zbe-3 /var/opt/zones/search-1/root
noauto yes
rpool/var/zones/search-1/rpool/export /var/opt/zones/search-1/root/export
on yes
rpool/var/zones/search-1/rpool/export/home
/var/opt/zones/search-1/root/export/homeon yes
I couldn't figure why from within the zone zfs mount was complaining
that the export and export/home datasets were busy.
Then from global I noticed rpool/var/zones/search-1/rpool/export and
export/home had the temporary mountpoint which was completely
unexpected.
After I halt'd and detach'd my zone, umount'd the datasets and
attach'd the zone the mountpoints corrected themselves.
>zonecfg -z search1 info dataset
>for ds in $(zonecfg -z z1 info dataset | nawk '$1 == "name:" {print $2}')
>do
> echo Dataset: $ds
> zfs list -o name,mountpoint,canmount,mounted,zone $ds
>done
>zonecfg -z search1 info fs
# zonecfg -z search-1 info
zonename: search-1
zonepath: /var/opt/zones/search-1
brand: solaris
autoboot: true
bootargs: -m verbose
file-mac-profile:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
fs:
dir: /ematrix
special: tank/ematrix
raw not specified
type: zfs
options: []
net:
address not specified
allowed-address not specified
configure-allowed-address: true
physical: vnic3
defrouter not specified
capped-memory:
physical: 3G
>Also, any details about changes in the zone configuration and/or package
>updates since the previous successful backup would be helpful.
I made no changes.
The other zones on the system had no issues.
John
groenv...@acm.org
___
zones-discuss mailing list
zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
On Tue 03 Jan 2012 at 12:23PM, John D Groenveld wrote:
> In message <201201031705.q03h5uwi000...@elvis.arl.psu.edu>, John D Groenveld
> wr
> ites:
> >My nightly backup consist of zone shutdown, detach, snapshot,
> >attach, boot.
FWIW, this is one of those cases where 'zoneadm attach -F' would
probably be reasonable.
>
> Here's the output from that cron:
> Progress being logged to
> /var/log/zones/zoneadm.20120101T021243Z.search-1.attach
> Attaching...
> Installing: Using existing zone boot environment
> Zone BE root dataset: rpool/var/zones/search-1/rpool/ROOT/zbe-3
> Cache: Using /var/pkg/publisher.
> Updating image format
> Updating non-global zone: Linking to image /.
> Updating non-global zone: Auditing packages.
> No updates necessary for this image.
>
> Updating non-global zone: Zone updated.
> Result: Attach Succeeded.
> Log saved in non-global zone as
> /var/opt/zones/search-1/root/var/log/zones/zoneadm.20120101T021243Z.search-1.attach
>
Above was your last successful attach. The failed attach starts here:
> Progress being logged to
> /var/log/zones/zoneadm.20120102T021110Z.search-1.attach
> Attaching...
> Installing: Using existing zone boot environment
> Manual migration of export required. Potential conflicts in
> /var/opt/zones/search-1/root/export and rpool/var/zones/search-1/rpool/export.
This error message is saying that it found two things that are supposed
to be mounted at /export. Without understanding your zone configuration
and dataset layout, it is kind hard to know exactly what is going on.
Can you provide the following:
%<---
zfs list -o name,mountpoint,canmount,mounted -r rpool/var/zones/search-1
zonecfg -z search1 info dataset
for ds in $(zonecfg -z z1 info dataset | nawk '$1 == "name:" {print $2}')
do
echo Dataset: $ds
zfs list -o name,mountpoint,canmount,mounted,zone $ds
done
zonecfg -z search1 info fs
%<---
Also, any details about changes in the zone configuration and/or package
updates since the previous successful backup would be helpful.
--
Mike Gerdts
Solaris Core OS / Zones http://blogs.oracle.com/zoneszone/
___
zones-discuss mailing list
zones-discuss@opensolaris.org
Re: [zones-discuss] S11 zone bug with migrated rpool/export ZFS
In message <201201031705.q03h5uwi000...@elvis.arl.psu.edu>, John D Groenveld wr ites: >My nightly backup consist of zone shutdown, detach, snapshot, >attach, boot. Here's the output from that cron: Progress being logged to /var/log/zones/zoneadm.20120101T021243Z.search-1.attach Attaching... Installing: Using existing zone boot environment Zone BE root dataset: rpool/var/zones/search-1/rpool/ROOT/zbe-3 Cache: Using /var/pkg/publisher. Updating image format Updating non-global zone: Linking to image /. Updating non-global zone: Auditing packages. No updates necessary for this image. Updating non-global zone: Zone updated. Result: Attach Succeeded. Log saved in non-global zone as /var/opt/zones/search-1/root/var/log/zones/zoneadm.20120101T021243Z.search-1.attach Progress being logged to /var/log/zones/zoneadm.20120102T021110Z.search-1.attach Attaching... Installing: Using existing zone boot environment Manual migration of export required. Potential conflicts in /var/opt/zones/search-1/root/export and rpool/var/zones/search-1/rpool/export. Zone BE root dataset: rpool/var/zones/search-1/rpool/ROOT/zbe-3 Cache: Using /var/pkg/publisher. Updating image format Updating non-global zone: Linking to image /. Updating non-global zone: Auditing packages. No updates necessary for this image. Updating non-global zone: Zone updated. Result: Attach Succeeded. Log saved in non-global zone as /var/opt/zones/search-1/root/var/log/zones/zoneadm.20120102T021110Z.search-1.attach Progress being logged to /var/log/zones/zoneadm.20120103T022057Z.search-1.attach Attaching... Installing: Using existing zone boot environment ERROR: Error: Command exited with status 1 ERROR: ZFS temporary mount of rpool/var/zones/search-1/rpool/export on /tmp/tmp.TcaOah/export failed. ERROR: Error: migration of /export from active boot environment to the zone's rpool/export dataset failed. Manual cleanup required. Zone BE root dataset: rpool/var/zones/search-1/rpool/ROOT/zbe-3 Cache: Using /var/pkg/publisher. Updating image format Updating non-global zone: Linking to image /. Updating non-global zone: Auditing packages. No updates necessary for this image. Updating non-global zone: Zone updated. Result: Attach Succeeded. Log saved in non-global zone as /var/opt/zones/search-1/root/var/log/zones/zoneadm.20120103T022057Z.search-1.attach Lots of evil in attach log: [Sun Jan 1 21:11:30 EST 2012] Mounting rpool/var/zones/search-1/rpool/export/home at /tmp/tmp.7kayqJ/export/home with ZFS temporary mount cannot unmount '/tmp/tmp.7kayqJ/export/home': Device busy cannot unmount '/tmp/tmp.7kayqJ/export': Device busy rmdir: directory "/tmp/tmp.7kayqJ": Directory not empty [Mon Jan 2 21:21:37 EST 2012] Mounting rpool/var/zones/search-1/rpool/export at /tmp/tmp.TcaOah/export with ZFS temporary mount cannot mount 'rpool/var/zones/search-1/rpool/export': filesystem already mounted [Mon Jan 2 21:21:37 EST 2012] ERROR: Error: Command exited wit h status 1 [Mon Jan 2 21:21:37 EST 2012] ERROR: ZFS temporary mount of rpool/var/zones/sea rch-1/rpool/export on /tmp/tmp.TcaOah/export failed. rmdir: directory "/tmp/tmp.TcaOah": Directory not empty [Mon Jan 2 21:21:37 EST 2012] ERROR: Error: migration of /export from active bo ot environment to the zone's rpool/export dataset failed. Manual cleanup required. John groenv...@acm.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
[zones-discuss] S11 zone bug with migrated rpool/export ZFS
I migrated my S11x system to S11 11/11 a few weeks ago and things have been running fine. I'm at S11 update 2a. However, I seem to have tripped over a bug on New Years. My nightly backup consist of zone shutdown, detach, snapshot, attach, boot. On Sunday the zone's filesystem-local svc could no longer mount /export/home. What are these mount's in /tmp/tmp.7kayqJ/export? My hack was halt the zone, detach, umount these weird mounts, attach and boot. Thanks, John groenv...@acm.org # zfs list -r rpool/var/zones/search-1 NAME USED AVAIL REFER MOUNTPOINT rpool/var/zones/search-1 26.5G 14.9G 34K /var/opt/zones/search-1 rpool/var/zones/search-1/rpool 26.5G 14.9G 31K /rpool rpool/var/zones/search-1/rpool/ROOT 26.4G 14.9G 31K legacy rpool/var/zones/search-1/rpool/ROOT/zbe-3 26.4G 14.9G 24.5G /var/opt/zones/search-1/root rpool/var/zones/search-1/rpool/export 43.0M 14.9G 32K /tmp/tmp.7kayqJ/export rpool/var/zones/search-1/rpool/export/home 42.6M 14.9G 42.1M /tmp/tmp.7kayqJ/export/home # zonecfg -z search-1 export create -b set zonepath=/var/opt/zones/search-1 set brand=solaris set autoboot=true set bootargs=-m verbose set ip-type=exclusive add fs set dir=/ematrix set special=tank/ematrix set type=zfs end add net set configure-allowed-address=true set physical=vnic3 end add capped-memory set physical=3G end # ls -l /tmp/ total 40 drwx-- 3 root buclient 180 Jan 1 21:11 tmp.7kayqJ drwx-- 3 root root 180 Dec 11 20:09 tmp.Iva4Sr drwx-- 3 root root 180 Dec 11 20:14 tmp.KoaiRs drwx-- 3 root root 180 Dec 11 20:05 tmp.QNaiqq drwx-- 3 root buclient 180 Jan 2 21:21 tmp.TcaOah # ls -l /tmp/tmp.7kayqJ total 3 drwxr-xr-x 3 root sys 3 Oct 23 2009 export # ls -l /tmp/tmp.7kayqJ/export total 3 drwxr-xr-x 8 root root 9 Oct 26 2010 home # ls -l /tmp/tmp.7kayqJ/export/home total 21 drwxr-xr-x 8 102 other 14 Nov 13 2009 bat107 -rw-r--r-- 1 root root 0 Dec 11 22:30 foo drwxr-xr-x 2 101 other 7 Oct 23 2009 john drwxr-xr-x 2 100 other 9 Oct 23 2009 klj158 drwxr-xr-x 3 103 other 10 Dec 10 2009 mab615 drwxr-xr-x 20 104 100 42 Dec 18 2009 matrix drwxr-xr-x 2 ryan other 7 Oct 26 2010 ryan # zfs set mountpoint=/var/opt/zones/search-1/root/export rpool/var/zones/search-1/rpool/export cannot set property for 'rpool/var/zones/search-1/rpool/export': 'mountpoint' cannot be set on a dataset mounted in a non-global zone ___ zones-discuss mailing list zones-discuss@opensolaris.org
