When deploying zones in Trusted Extensions some customers want to boot many 
zones (hundreds) into the ready state. Since each zone has a unique label and 
zfs dataset their filesystems be used by label-aware services in the global 
zone such as NFS and sendmail, even when no processes (except zsched) are 
running. When a user requests an operation that requires starting processes in 
one of these ready zones, it is booted into the running state.

I have been doing some analysis of performance bottlenecks when large numbers 
of zones are installed on a system. I can quickly clone hundreds of zones in a 
few minutes. Since all these zones share their network interfaces (and IP 
addresses) with the global zone, the only difference between the various zone 
config files is the root path. However, there is considerable overhead in the 
zones SMF service when booting the system even though they are all set to 
autoboot=false. I'm running under Parallels on a MacBook Pro. The zones service 
goes into maintenance because it takes too more than a minute to determine that 
it has nothing to do.

I took a look at the service method, /lib/svc/method/svc-zones. In Solaris 10 
the zones are booted using ctrun, but that it not done in Nevada. It seems to 
me that the ctrun command isn't really needed. The more interesting issue is 
that the zones are booted in background (with an &), instead of sequentially. I 
made a copy of this script and changed it so that it would boot zones into the 
ready state, and would run the zoneadm commands sequentially, instead of in 
background. I found that I go much better system performance, and was able to 
make 500 zones ready in about 25 minutes. When I ran the zoneadm commands in 
background, however, it never finished. The system just got bogged down in 
devfsadm which accumulated 500 threads. I ran this on both Solaris 10 and 
Nevada, and got different results, but the general story was the same. In 
Nevada, the HAL daeom, hald, was kept very busy because it wakes up on system 
event every time an entry is added to the global zone mnttab. Sinc
 e all the zone mnttab entries also show up in the global zones, it was 
continuously re-reading mnttab.

When I ran the 500 instances of zoneadm sequentially, I only saw 8 threads for 
devfsadm instead of 500, and the system remained usable (for example the 
pointer tracked smoothly) while all the zones were made ready.

Another issue that affects performance is that every zone is verified against 
every other zone for conflicts such as root paths every time they are started. 
I would like to see something like a timestamp on the the /etc/zones/index file 
that indicates that nothing has changed since the last time all the zones were 
installed and verified. I tested this by bypassing some of these checks in my 
own copy of devfsadm and zoneadm. It seemed to help by about 10% once the 
number of installed zones exceeded 300.
This message posted from opensolaris.org
zones-discuss mailing list

Reply via email to