Tony and Liane determined that svcadm enable -ts wasn't working because Clearview's datalink service didn't have a restarter property group. During investigation, they discovered a way to work around this with some shell scripting.
Ordinarily, all valid service instances have restarter property groups because svc.startd creates them as it determines the instance's properties are valid. At boot, however, svc.startd evaluates service instances as it initially walks the repository, which triggers restarter events as it goes. The events are processed in parallel with the walk, and apparently this results in network/physical (and its pre-Clearview dependencies) being started before svc.startd has finished walking the repository. (I believe services are walked in the order they were added to the repository, so this probably doesn't occur in initial install of Clearview bits, where the datalink service is presumably in the seed repository, near the top of the table. Only on upgrade does datalink end up near the end.) This root cause presents one solution: Modify svc.startd to create restarter property groups on all services before starting any of them. This would provide the same SCF environment to early services as to late ones. That is, service instances only lack restarter property groups when they're not configured properly or something else is wrong. I don't think this would cause a significant performance problem because creating restarter property groups shouldn't require disk I/O and should be relatively fast. It's a nontrivial change to a crucial component, though, and certainly won't be done within the next few days. For now, Clearview could use the shell script workaround Tony and Liane found. An alternative solution, however, is to modify svcadm enable -ts to stop interpreting the restarter property group's absence as an error and instead just wait for it to appear. Since it should be interpreted as an error after boot, but we don't have a good way to tell when that's the case, Tony and I think that for this situation svcadm should inform the user that the restarter property group is missing, that that's usually an error, and that it is waiting in case it isn't. This would be a change in behavior for a relatively rare case, and could in theory be problematic if a program is invoking svcadm enable -ts without passing the message on to the user. This fix should be pretty simple and might be doable with or before the Clearview putback. I don't like the wishy-washy semantics, though. So it seems that our choice is between an easy svcadm special change and an ugly Clearview workaround plus a more complicated startd fix later. I like the startd fix better in the long-term, but I'm not sure it's a slam-dunk. Please opine. David
