When deploying zones in Trusted Extensions some customers want to boot many
zones (hundreds) into the ready state. Since each zone has a unique label and
zfs dataset their filesystems be used by label-aware services in the global
zone such as NFS and sendmail, even when no processes (except zsched) are
running. When a user requests an operation that requires starting processes in
one of these ready zones, it is booted into the running state.
I have been doing some analysis of performance bottlenecks when large numbers
of zones are installed on a system. I can quickly clone hundreds of zones in a
few minutes. Since all these zones share their network interfaces (and IP
addresses) with the global zone, the only difference between the various zone
config files is the root path. However, there is considerable overhead in the
zones SMF service when booting the system even though they are all set to
autoboot=false. I'm running under Parallels on a MacBook Pro. The zones service
goes into maintenance because it takes too more than a minute to determine that
it has nothing to do.
I took a look at the service method, /lib/svc/method/svc-zones. In Solaris 10
the zones are booted using ctrun, but that it not done in Nevada. It seems to
me that the ctrun command isn't really needed. The more interesting issue is
that the zones are booted in background (with an &), instead of sequentially. I
made a copy of this script and changed it so that it would boot zones into the
ready state, and would run the zoneadm commands sequentially, instead of in
background. I found that I go much better system performance, and was able to
make 500 zones ready in about 25 minutes. When I ran the zoneadm commands in
background, however, it never finished. The system just got bogged down in
devfsadm which accumulated 500 threads. I ran this on both Solaris 10 and
Nevada, and got different results, but the general story was the same. In
Nevada, the HAL daeom, hald, was kept very busy because it wakes up on system
event every time an entry is added to the global zone mnttab. Sinc
e all the zone mnttab entries also show up in the global zones, it was
continuously re-reading mnttab.
When I ran the 500 instances of zoneadm sequentially, I only saw 8 threads for
devfsadm instead of 500, and the system remained usable (for example the
pointer tracked smoothly) while all the zones were made ready.
Another issue that affects performance is that every zone is verified against
every other zone for conflicts such as root paths every time they are started.
I would like to see something like a timestamp on the the /etc/zones/index file
that indicates that nothing has changed since the last time all the zones were
installed and verified. I tested this by bypassing some of these checks in my
own copy of devfsadm and zoneadm. It seemed to help by about 10% once the
number of installed zones exceeded 300.
This message posted from opensolaris.org
zones-discuss mailing list