When deploying zones in Trusted Extensions some customers want to boot many zones (hundreds) into the ready state. Since each zone has a unique label and zfs dataset their filesystems be used by label-aware services in the global zone such as NFS and sendmail, even when no processes (except zsched) are running. When a user requests an operation that requires starting processes in one of these ready zones, it is booted into the running state.
I have been doing some analysis of performance bottlenecks when large numbers of zones are installed on a system. I can quickly clone hundreds of zones in a few minutes. Since all these zones share their network interfaces (and IP addresses) with the global zone, the only difference between the various zone config files is the root path. However, there is considerable overhead in the zones SMF service when booting the system even though they are all set to autoboot=false. I'm running under Parallels on a MacBook Pro. The zones service goes into maintenance because it takes too more than a minute to determine that it has nothing to do. I took a look at the service method, /lib/svc/method/svc-zones. In Solaris 10 the zones are booted using ctrun, but that it not done in Nevada. It seems to me that the ctrun command isn't really needed. The more interesting issue is that the zones are booted in background (with an &), instead of sequentially. I made a copy of this script and changed it so that it would boot zones into the ready state, and would run the zoneadm commands sequentially, instead of in background. I found that I go much better system performance, and was able to make 500 zones ready in about 25 minutes. When I ran the zoneadm commands in background, however, it never finished. The system just got bogged down in devfsadm which accumulated 500 threads. I ran this on both Solaris 10 and Nevada, and got different results, but the general story was the same. In Nevada, the HAL daeom, hald, was kept very busy because it wakes up on system event every time an entry is added to the global zone mnttab. Sinc e all the zone mnttab entries also show up in the global zones, it was continuously re-reading mnttab. When I ran the 500 instances of zoneadm sequentially, I only saw 8 threads for devfsadm instead of 500, and the system remained usable (for example the pointer tracked smoothly) while all the zones were made ready. Another issue that affects performance is that every zone is verified against every other zone for conflicts such as root paths every time they are started. I would like to see something like a timestamp on the the /etc/zones/index file that indicates that nothing has changed since the last time all the zones were installed and verified. I tested this by bypassing some of these checks in my own copy of devfsadm and zoneadm. It seemed to help by about 10% once the number of installed zones exceeded 300. This message posted from opensolaris.org _______________________________________________ zones-discuss mailing list zones-discuss@opensolaris.org
