Related comments from bug below (X'ed out some paths):

The zone in question clearly has too many references

> 0000030004a09680::print zone_t zone_ref  
zone_ref = 0t11

Ten too many, to be precise.  So what's holding onto the zone?  Well the rnode
cache has 5 entries

> ::kmem_cache ! grep rnode
0000030003a1e988 rnode_cache               0000 000000      640   572988
0000030003a20988 rnode4_cache              0000 000000      984        0
> 0000030003a1e988::walk kmem | ::print rnode_t r_vnode | ::vnode2path
/opt/zones/z1/root/XXXX
/opt/zones/z1/root/XXXX
/opt/zones/z1/root/XXXX
/opt/zones/z1/root/XXXX
/opt/zones/z1/root/XXXX

even though no nfs filesystems are mounted

> ::fsinfo
            VFSP FS              MOUNT
000000000187f420 ufs             /
000000000187f508 devfs           /devices
0000030000315780 ctfs            /system/contract
00000300003156c0 proc            /proc
0000030000315600 mntfs           /etc/mnttab
0000030000315480 tmpfs           /etc/svc/volatile
00000300003153c0 objfs           /system/object
00000300039987c0 namefs          /etc/svc/volatile/repository_door
00000300039984c0 fd              /dev/fd
0000030003a99e00 ufs             /var
0000030003998400 tmpfs           /tmp
0000030003a99680 tmpfs           /var/run
0000030003a98f00 namefs          /var/run/name_service_door
0000030003a98b40 namefs          /var/run/sysevent_channels/syseventd_channel...
0000030003a989c0 namefs          /etc/sysevent/sysevent_door
0000030003a98780 namefs          /etc/sysevent/devfsadm_event_channel/1
0000030003a98540 namefs          /dev/.zone_reg_door
0000030003a983c0 namefs          /dev/.devfsadm_synch_door
0000030003a99380 namefs          /etc/sysevent/piclevent_door
00000300044b1d80 namefs          /var/run/picld_door
0000030003a99200 ufs             /opt
00000300044b0700 namefs          /var/run/zones/z1.zoneadmd_door

And as apparent from the path, all of those rnodes refer to zone z1 through
their mntinfo structure

> 0000030003a1e988::walk kmem | ::print rnode_t r_vnode->v_vfsp->vfs_data | 
> ::print mntinfo_t mi_zone | ::zone
            ADDR     ID NAME                 PATH
0000030004a09680      1 z1                   /opt/zones/z1/root/
0000030004a09680      1 z1                   /opt/zones/z1/root/
0000030004a09680      1 z1                   /opt/zones/z1/root/
0000030004a09680      1 z1                   /opt/zones/z1/root/
0000030004a09680      1 z1                   /opt/zones/z1/root/

So if each of those rnodes has two holds on the zone, then that accounts for
all of the extra holds exactly.


On Wed, May 06, 2009 at 09:04:54PM -0500, Derek McEachern wrote:
>    I don't believe that I can see the comments since they are not public.
> 
>    Is that something you can pass along?
> 
>    On Wed, May 6, 2009 at 5:27 PM, Steve Lawrence
>    <[1]stephen.lawre...@sun.com> wrote:
> 
>      > * *I already tried killing the zoneadmd process and issuing the halt
>      and all
>      > * *it does is start back up the zoneadmd process and hang.* I can't
>      force a
>      > * *crashdump on the system since I can't take the box down.
>      >
>      > * *Bug 6272846 makes reference to nfs version 3, (which is the version
>      we are
>      > * *using), and the client apparently leaking rnodes. Is there any way
>      to
>      > * *verify this other then a forced crashdump? I might take a live core
>      of the
>      > * *system and open a case to see if that yields anything.
> 
>      The zone_ref > 1 means that something in the kernel is holding the zone.
>      You should be able to use "mdb -k" on the live system, and issue dcmds
>      similar
>      to the comments of 6272846. *No need to force a crashdump or take a live
>      crashdump.
> 
>      -Steve L.
>      >
>      > * *Derek
>      >
>      > * *On Wed, May 6, 2009 at 4:08 PM, Steve Lawrence
>      > * *<[1][2]stephen.lawre...@sun.com> wrote:
>      >
>      > * * *zsched is always unkillable. *It will only exit when instructed
>      to by
>      > * * *zoneadmd.
>      >
>      > * * *Is the remaining zone "shutting down", or "down"? *(zoneadm list
>      -v).
>      >
>      > * * *What is the ref_count on the zone?
>      >
>      > * * *# mdb -k
>      > * * *> ::walk zone | ::print zone_t zone_name zone_ref
>      >
>      > * * *If the refcount is greater than 0x1, it could be:
>      > * * ** * * *6272846 User orders zone death; NFS client thumbs nose
>      >
>      > * * *No workaround for this one. *A crashdump would help investigate a
>      > * * *zone_ref
>      > * * *greater than 1.
>      >
>      > * * *Is there a zoneadmd process for the given zone?
>      > * * *# pgrep -lf zoneadmd
>      >
>      > * * *If so, please provide *truss -p <pid>" of this process. *You may
>      also
>      > * * *attempt
>      > * * *killing this zoneadmd process (which lives in the global zone),
>      and then
>      > * * *re-attempting "zoneadm -z <zonename> halt".
>      >
>      > * * *Thanks,
>      >
>      > * * *-Steve L.
>      >
>      > References
>      >
>      > * *Visible links
>      > * *1. mailto:[3]stephen.lawre...@sun.com
> 
> References
> 
>    Visible links
>    1. mailto:stephen.lawre...@sun.com
>    2. mailto:stephen.lawre...@sun.com
>    3. mailto:stephen.lawre...@sun.com
_______________________________________________
zones-discuss mailing list
zones-discuss@opensolaris.org

Reply via email to