Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Steve Lawrence wrote: > Call this requirement (no login prompt) out in your use case. I assume > the patch service will patch, set the boot milestone, and reboot before > the patch milestone is actually met, avoiding the maint prompt. Yes. > Definately get some console messages out of the patch-service so folks don't > think their boot is hung and freak out. :) I believe they're already there. >> There are approximately two tricky parts to this puzzle: >> a) How does the patch tool reboot the system to milestone/patching? It >> could use "reboot -- -m milestone/patching", but that would mean that >> the patching work wouldn't get done if the reboot was done through other >> mechanisms. It could set the system default milestone, but then how >> would it determine the milestone to set the system back to when patching >> was complete? Neither answer is pretty, but either is workable. > > I'm sure you could write the old boot milestone down somewhere. Hmm. I was concerned about that, but also about the possibility that we'd overwrite the admin's subsequent selection with the value that we remembered. I guess that can't really happen, since we won't do the overwriting if the milestone isn't set to our magic one. (Well, except for truly perverse sequences like "we remember A, set it to P, user sets it to B, user boots to P, we reset it to A".) > If the admin modifies the milestone after the patch tool sets it > to milestone-patching, the patch-on-next-boot will just get clobbered. > I suppose the patch-service could then re-instate it on the next boot, and > hope to get it on the subsequent boot. I like that plan. > I suppose being "inside the bounds of SMF" also makes the implementation > vulnerable to other admins manimpulating SMF. The above issue is basically > by design. Sure. It'd just be nice to have a scheme that was both within the bounds *and* didn't offer the user any ways to break it :-) >> b) How do the patching services *avoid* running when the system is >> coming up normally - even if they have work to do? Probably the best >> answer is for them to check the target milestone and fail (or succeed >> without doing anything) if it's not milestone/patching. > > That makes sense. It would have to do-nothing-succeed to work with > milestone "all". Do-nothing-fail would also work - it would just result in a message and putting the service into Maintenance, just like any other miscellaneous failing service. If the patching service has work to do, and isn't able to do it because the system isn't booting the right way, that might be the rightest answer. >> I understand your reluctance to add system/filesystem/local to >> milestone/single-user. If there's consensus that that's an unacceptable >> change, then instead: >> >> 1) Have patchadd mount zone roots if they are not already mounted. >> 2-5) As above. >> >> Hmm. My initial thought was that patchadd could not use >> system/filesystem/local to do these mounts because of deadlock issues. >> However, since we've moved automated patching to slightly *after* >> milestone/single-user, perhaps those deadlock issues do not exist. > > Right. Just make the patch-service depend on filesystem-local. If > then patchadds issued by patch-service enable filesystem-local, it will > be a no-op. No need. The enable done by patchadd will be sufficient. We should represent the dependency on filesystem/local in exactly one place, either in patchadd itself or in the dependency list for milestone/single-user. > Another approach is to just modify patchadd to exec > /lib/svc/method/fs-local if called from SMF context. patchadd also stops > filesystem-local when done, but the stop method for that service is :true, > so stopping it is a no-op. patchadd should really verify fs/local's > dependencies are online before doing this.Patchadd is already parsing > the output of the svcprop command as it is. It could certainly read > start/exec property and whatever else. This would fix both smpatch and > UCE, and whoever else may depend on patchadd working from rcS.d. rcS.d doesn't really work, because of the SMF deadlock issue. (Some SMF operations deadlock when run from rcS.d scripts. Some patches use those SMF operations.) This is kind of unfortunate, because rcS.d is the only way (other than a new milestone) to catch the system in the right state. rcS.d scripts are not the Way Of The Future. It's also just gross :-) However, if we can make it work and everybody can agree that it's THE way to solve the problem, it's OK with me. I like the "new milestone" answer above, plus either moving filesystem/local or enabling it from patchadd, better. --- I know this has looked like wandering in circles, but I think it's been quite helpful - I think we're getting pretty close to a reasonable answer. Thanks. ___ zones-discuss mailing list zones-discuss@opensolari
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
On Thu, Aug 21, 2008 at 04:01:43PM -0700, Jordan Brown wrote: > Steve Lawrence wrote: > > A. Make patchadd verify that the system is in single user milestone when > > installing a single-user patch. > > That's a non-starter. *Many* of our customers ignore our recommendation > to install patches in single-user mode, and will revolt if we attempt to > enforce it. > > In addition, for many patches the single-user mode recommendation is > only the first approximation, primarily intended for automata. If a > human is installing the patch, it may be acceptable to install it after > manually shutting down the affected services. That seems completely unsupportable, but ok. Admins are used to getting away with not following the patch README or suggested procedure. (A) could be dropped without much impact to the solution. > > > B. If patchadd discovers that it needs to patch a zone, patchadd should > > first > > make sure the zone's zonepath is properly mounted. An overkill for this > > could be to issue a "svcadm enable -srt fileystem/local" IF patchadd is > > not being run from the context of an SMF service, otherwise, fail. > > (sorry, no patchadd from smf services or rc*.d scripts). > > "No patchadd from smf services or rc*.d scripts" means "no automated > installation of single-user patches". That's a non-starter. Your final comment addresses this. Post filesystem/local, it is safe for SMF services to call patchadd. > > > An alternate solution is to fail patchadd with a message stating that > > filesystem/local must be enabled to install the patch due to the > > installed zones. The admin could then do as instructed. > > Also a killer for automated installation. > > > C. (2) above will need to somehow set the milestone to single-user, wait > > until single user is reached, and then do the patchadd, which will do > > A and B. This automated tool could also do the: > > > > "svcadm enable -rt fileystem/local" > > > > If B fails do to the alternate solution. The automation tool could also > > enable filesystem/local in cases where the patchadd version the system > > does > > not have this functionality. For simplicity, perhaps just always > > enable filesystem/local in the automation tool after single-user is > > reached. > > > > I think to implement (2), at some point you are going to need to fork off > > some asyncronous process which changes the milestone, waits, and then > > addes the patch, potentially also enabling filesystem/local before patching > > if needed (or just always). > > I'm not happy with doing this stuff outside the bounds of SMF, or with > approaches where the user is offered a single-user login while the > automated tools are installing patches in the background and will > asynchronously reboot the system. I don't think either is necessary. Call this requirement (no login prompt) out in your use case. I assume the patch service will patch, set the boot milestone, and reboot before the patch milestone is actually met, avoiding the maint prompt. Definately get some console messages out of the patch-service so folks don't think their boot is hung and freak out. :) > > My favorite approach is, approximately: > > 1) Move system/filesystem/local into milestone/single-user. > > Note that this alone addresses the issues for interactive > patchadd. > > 2) Define milestone/patching, dependent on milestone/single-user. > 3) Define new a new patching service (or services), dependent on > milestone/single-user and depended on by milestone/patching. > 4) When patch automation needs to install a single-user patch, have it > boot the system to milestone/patching. Or just set the boot milestone if "patching deferred to next reboot". > 5) When the patch services are done with their work, have them let the > system come up to its default milestone, or reboot it to its default > milestone, as required. > > There are approximately two tricky parts to this puzzle: > a) How does the patch tool reboot the system to milestone/patching? It > could use "reboot -- -m milestone/patching", but that would mean that > the patching work wouldn't get done if the reboot was done through other > mechanisms. It could set the system default milestone, but then how > would it determine the milestone to set the system back to when patching > was complete? Neither answer is pretty, but either is workable. I'm sure you could write the old boot milestone down somewhere. If the admin modifies the milestone after the patch tool sets it to milestone-patching, the patch-on-next-boot will just get clobbered. I suppose the patch-service could then re-instate it on the next boot, and hope to get it on the subsequent boot. I suppose being "inside the bounds of SMF" also makes the implementation vulnerable to other admins manimpulating SMF. The above issue is basically by design. > b) How do th
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Steve Lawrence wrote: > A. Make patchadd verify that the system is in single user milestone when > installing a single-user patch. That's a non-starter. *Many* of our customers ignore our recommendation to install patches in single-user mode, and will revolt if we attempt to enforce it. In addition, for many patches the single-user mode recommendation is only the first approximation, primarily intended for automata. If a human is installing the patch, it may be acceptable to install it after manually shutting down the affected services. > B. If patchadd discovers that it needs to patch a zone, patchadd should first > make sure the zone's zonepath is properly mounted. An overkill for this > could be to issue a "svcadm enable -srt fileystem/local" IF patchadd is > not being run from the context of an SMF service, otherwise, fail. > (sorry, no patchadd from smf services or rc*.d scripts). "No patchadd from smf services or rc*.d scripts" means "no automated installation of single-user patches". That's a non-starter. > An alternate solution is to fail patchadd with a message stating that > filesystem/local must be enabled to install the patch due to the > installed zones. The admin could then do as instructed. Also a killer for automated installation. > C. (2) above will need to somehow set the milestone to single-user, wait > until single user is reached, and then do the patchadd, which will do > A and B. This automated tool could also do the: > > "svcadm enable -rt fileystem/local" > > If B fails do to the alternate solution. The automation tool could also > enable filesystem/local in cases where the patchadd version the system > does > not have this functionality. For simplicity, perhaps just always > enable filesystem/local in the automation tool after single-user is > reached. > > I think to implement (2), at some point you are going to need to fork off > some asyncronous process which changes the milestone, waits, and then > addes the patch, potentially also enabling filesystem/local before patching > if needed (or just always). I'm not happy with doing this stuff outside the bounds of SMF, or with approaches where the user is offered a single-user login while the automated tools are installing patches in the background and will asynchronously reboot the system. I don't think either is necessary. My favorite approach is, approximately: 1) Move system/filesystem/local into milestone/single-user. Note that this alone addresses the issues for interactive patchadd. 2) Define milestone/patching, dependent on milestone/single-user. 3) Define new a new patching service (or services), dependent on milestone/single-user and depended on by milestone/patching. 4) When patch automation needs to install a single-user patch, have it boot the system to milestone/patching. 5) When the patch services are done with their work, have them let the system come up to its default milestone, or reboot it to its default milestone, as required. There are approximately two tricky parts to this puzzle: a) How does the patch tool reboot the system to milestone/patching? It could use "reboot -- -m milestone/patching", but that would mean that the patching work wouldn't get done if the reboot was done through other mechanisms. It could set the system default milestone, but then how would it determine the milestone to set the system back to when patching was complete? Neither answer is pretty, but either is workable. b) How do the patching services *avoid* running when the system is coming up normally - even if they have work to do? Probably the best answer is for them to check the target milestone and fail (or succeed without doing anything) if it's not milestone/patching. I understand your reluctance to add system/filesystem/local to milestone/single-user. If there's consensus that that's an unacceptable change, then instead: 1) Have patchadd mount zone roots if they are not already mounted. 2-5) As above. Hmm. My initial thought was that patchadd could not use system/filesystem/local to do these mounts because of deadlock issues. However, since we've moved automated patching to slightly *after* milestone/single-user, perhaps those deadlock issues do not exist. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
> The list of use cases is really pretty simple: > > 1) Administrator has in hand a patch that says "install in single user > mode". What does this administrator do? The answer seems self-evident: > take the system to single-user mode (either by booting the system in > single-user mode using boot -s or boot -m milestone/single-user, or > dropping the system to single-user mode using "init s" or "svcadm > milestone milestone/single-user") and install the patch using patchadd. > > 2) An automated tool has in hand a patch that says "install in single > user mode". What does it do? How about: A. Make patchadd verify that the system is in single user milestone when installing a single-user patch. B. If patchadd discovers that it needs to patch a zone, patchadd should first make sure the zone's zonepath is properly mounted. An overkill for this could be to issue a "svcadm enable -srt fileystem/local" IF patchadd is not being run from the context of an SMF service, otherwise, fail. (sorry, no patchadd from smf services or rc*.d scripts). An alternate solution is to fail patchadd with a message stating that filesystem/local must be enabled to install the patch due to the installed zones. The admin could then do as instructed. C. (2) above will need to somehow set the milestone to single-user, wait until single user is reached, and then do the patchadd, which will do A and B. This automated tool could also do the: "svcadm enable -rt fileystem/local" If B fails do to the alternate solution. The automation tool could also enable filesystem/local in cases where the patchadd version the system does not have this functionality. For simplicity, perhaps just always enable filesystem/local in the automation tool after single-user is reached. I think to implement (2), at some point you are going to need to fork off some asyncronous process which changes the milestone, waits, and then addes the patch, potentially also enabling filesystem/local before patching if needed (or just always). -Steve L. > > It is when we start to look at solutions that the problem becomes more > difficult. > ___ > zones-discuss mailing list > zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Steve Lawrence wrote: > I assume you are targeting this change for s10. Yes. > The single-user milestone is intended to mimic the traditional unix > run-level 1 (S?) Nit: Run level 1 is slightly different from S. > This is typically where an admin would run stuff like > fsck (on filesystems that are not yet mounted). It is also where we tell people to install many patches that touch key parts of the system. Zones and patching collided to break that use case, by making it (potentially) necessary to mount secondary file systems in order to apply patches. > I don't think it is ok > to change this behavior in a patch. Something's got to give. > I'm not sure I understand all the details of the problem you are trying > to solve. For example, I thought it was desired that the "patch service" > run during a boot to "all", but then I saw following mail stating that > the "patch service" should not run in this case, and something about the > user explicitly booting to single user. I don't think I know what the > use cases are. There's been a lot of variations flying around and it's tough to keep them all straight. One key incorrect assumption that we (the patch automation teams) had been making was that we could "catch" system startup at the right point and do our work, as was possible in pre-SMF systems. That would suggest having a patch service that ran on the way to "all". However, on an SMF system that's not a valid assumption: if the target is "all", services might be started at any point, including before milestone/single-user, if their dependencies are met. It now appears clear that we must limit the system to some appropriate milestone, to limit the services that are started. > You may want to draft a brief ARC fastrack describing the desired behavior(s), > and the issues, and perhaps proposed solutions. Getting it all on one page > will facilitate a solution. I agree that an ARC case is needed, if only to formally record the agreed-upon solution. However, since we don't yet have a complete solution to propose, I think it's premature at the moment. Getting it all on one page would, I agree, be helpful. (Is there an externally-visible wiki where such a writeup could be posted and maintained?) --- The list of use cases is really pretty simple: 1) Administrator has in hand a patch that says "install in single user mode". What does this administrator do? The answer seems self-evident: take the system to single-user mode (either by booting the system in single-user mode using boot -s or boot -m milestone/single-user, or dropping the system to single-user mode using "init s" or "svcadm milestone milestone/single-user") and install the patch using patchadd. 2) An automated tool has in hand a patch that says "install in single user mode". What does it do? It is when we start to look at solutions that the problem becomes more difficult. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
On Thu, Aug 21, 2008 at 12:54:14PM -0700, Jordan Brown wrote: > [ Which brain-dead mail client turns all of the spaces in the Subject > into tabs? ] > > Zones folks: the current proposed answers to this problem involve > moving system/filesystem/local into milestone/single-user. That was > apparently considered and rejected as the answer for the patchadd > problem that resulted in the "fix" that brought us here. Can you offer > any insight into why that change was rejected? I assume you are targeting this change for s10. The single-user milestone is intended to mimic the traditional unix run-level 1 (S?) This is typically where an admin would run stuff like fsck (on filesystems that are not yet mounted). I don't think it is ok to change this behavior in a patch. I'm not sure I understand all the details of the problem you are trying to solve. For example, I thought it was desired that the "patch service" run during a boot to "all", but then I saw following mail stating that the "patch service" should not run in this case, and something about the user explicitly booting to single user. I don't think I know what the use cases are. You may want to draft a brief ARC fastrack describing the desired behavior(s), and the issues, and perhaps proposed solutions. Getting it all on one page will faciliate a solution. -Steve L. > ___ > zones-discuss mailing list > zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
[ Which brain-dead mail client turns all of the spaces in the Subject into tabs? ] Zones folks: the current proposed answers to this problem involve moving system/filesystem/local into milestone/single-user. That was apparently considered and rejected as the answer for the patchadd problem that resulted in the "fix" that brought us here. Can you offer any insight into why that change was rejected? ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Nils Goroll wrote: > I suggest to introduce an additional milestone (e.g. milestone/ready) > with optional dependencies on all "system" services, roughly matching > the time when rc3 is run. That's much later than is desirable for these patches. The goal is to have the system as quiet as possible. Historically, I believe the only things running in single-user mode are init and the shell. There's more than that now, but it's still pretty limited. The reason that the system needs to be quiet is that (for instance) a patch might install new versions of two kernel modules. If one of them was currently loaded and the other was not, activity on the system might cause the new one to be loaded. If it's not compatible with the already-loaded old version, bad things might happen. > By convention, all non-system services would be (strongly) recommended > to depend upon this service. This would also make application specific > services more rubust. I've seen a couple of examples where SMFs would, > depend upon filesystem/local and network/physical, but fail if the user > configured the application to use an NFS mountpoint, for instance. This > would also facilitate the introduction of additional services which are > a requirement for all applications in the general case. > > In the current context, this milestone (and the requirement for all user > apps to depend upon it) would provide a way to make sure patching is > done at the right time. I believe that the point you describe pretty much corresponds to milestone/multi-user. I don't have a strong opinion on whether non-system services should be encouraged to wait for such a milestone. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
> The only way that you can get *that* guarantee is by using the
> milestone mechanism to limit the system to a particular milestone, as
> you suggest.
>
> In fact, argh. This problem affects even your proposed scheme. By the
> time that your patch-test-service is running, there could (in theory) be
> all kinds of services running that didn't happen to depend on anything.
> Maybe in practice we could ignore that possibility, but it's still
> bothersome.
>
> Argh. Not quite back to Square One, but that certainly tosses a wrench
> into most of my theories on how to solve this problem.
Argh again. Currently startd hard codes the allowable milestones. My
proposal would require patching startd :(
static int
dgraph_set_milestone(const char *fmri, scf_handle_t *h, boolean_t norepository)
{
const char *cfmri, *fs;
graph_vertex_t *nm, *v;
int ret = 0, r;
scf_instance_t *inst;
boolean_t isall, isnone, rebound = B_FALSE;
/* Validate fmri */
isall = (strcmp(fmri, "all") == 0);
isnone = (strcmp(fmri, "none") == 0);
if (!isall && !isnone) {
if (fmri_canonify(fmri, (char **)&cfmri, B_FALSE) == EINVAL)
goto reject;
if (strcmp(cfmri, single_user_fmri) != 0 &&
strcmp(cfmri, multi_user_fmri) != 0 &&
strcmp(cfmri, multi_user_svr_fmri) != 0) {
startd_free((void *)cfmri, max_scf_fmri_size);
reject:
log_framework(LOG_WARNING,
"Rejecting request for invalid milestone \"%s\".\n",
fmri);
return (EINVAL);
}
}
-Steve L.
> ___
> zones-discuss mailing list
> zones-discuss@opensolaris.org
___
zones-discuss mailing list
zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Steve Lawrence wrote: > I can't see any straightforward way to interrupt boot without changing the > milestone. You could make lots of services dependent on a patching > service, but that will have a maintenance burden. It also may not play well > with 3rd party services. Yep. Hmm. I just realized a flaw in my mental model of an SMF system. In a traditional rc-based system, everything runs in sequence. When you get to the "single user" point, it doesn't matter whether your target was to get to "single user" mode or to get to "multi user" mode - in either case, the system is in the same state. In an SMF system, on the other hand, you could have a service that is intended for "multi user" mode that doesn't have any dependencies. It could come up instantly, even before the system has reached milestone/single-user. I had previously thought that milestone/single-user was equivalent to the legacy end-of-rcS / beginning-of-rc2 point, but that's not true. Although milestone/single-user guarantees that a particular set of services is running, it *does not* guarantee that other services are *not* running! The only way that you can get *that* guarantee is by using the milestone mechanism to limit the system to a particular milestone, as you suggest. In fact, argh. This problem affects even your proposed scheme. By the time that your patch-test-service is running, there could (in theory) be all kinds of services running that didn't happen to depend on anything. Maybe in practice we could ignore that possibility, but it's still bothersome. Argh. Not quite back to Square One, but that certainly tosses a wrench into most of my theories on how to solve this problem. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
> >It should be ok to issue smf commands from an smf service, as long as they > >do not try to do any synchronous operations (-s). > > Seems a little convoluted, but might be workable. I can't see any straightforward way to interrupt boot without changing the milestone. You could make lots of services dependent on a patching service, but that will have a maintenance burden. It also may not play well with 3rd party services. > > patch-install-service might need to > > ms=`svcprop -p options/milestone svc:/system/svc/restarter:default` > svcadm milestone "$ms" Ok. > > if the patches installed don't need a reboot. > > >This approach is also good because an explicit boot to single user WILL NOT > >attempt to install pending patches. > > That would be very nice, but are you sure? It seems like > patch-test-service would override the milestone specified at boot time, > and the system would continue up to the patch installation milestone. Ugg. Right. I suppose the patch-test service needs to do nothing if the current milestone is single-user. A bit hack-ish in that regard :( -Steve > > >Disabling the patch-test and patch-install services will disable the > >automatic installation of pending patches on reboot. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Steve Lawrence wrote: > So you want to be able to interrupt any boot to any milestone, and instead do > the patch processing if a patch is pending. You basically want to interrupt > the current milestone, and instead just boot to filesystem-local and do the > patching. That would be my initial plan, at least roughly. I wouldn't think of it in terms of going to a different milestone, per se, but rather that there is a point in system startup where we look to see if there are any patches pending. If not, we continue; if so, we apply them. If they require reboot, we reboot, else we let the system continue to come up. That point needs to be early enough so that the system is quiescent, but late enough so that the services needed by the patches tools (e.g. local file systems) and the patches themselves are present. > The question is, can the smf milestone be changed mid-milestone? > > My test shows that it can. How about: > > 1. Create patch-test-service, on which single-user depends. This will >"svcadm milestone patch-install-milestone" if a patch needs to be >installed. This service is always enabled. > > 2. Create patch-install-milestone, which depends on patch-install-service >below. > > 3. Create patch-install service, which depends on: > single-user > filesystem-local > >This service is always enabled. It will install a patch if it is pending, >otherwise, do nothing. If the service fails, it might need to: > > # svcadm milestone single-user > >So that a maintenance prompt will be appear on the console. This might >not be necessary. you might get this anyway, as console-login is not >reached. > > It should be ok to issue smf commands from an smf service, as long as they > do not try to do any synchronous operations (-s). Seems a little convoluted, but might be workable. patch-install-service might need to ms=`svcprop -p options/milestone svc:/system/svc/restarter:default` svcadm milestone "$ms" if the patches installed don't need a reboot. > This approach is also good because an explicit boot to single user WILL NOT > attempt to install pending patches. That would be very nice, but are you sure? It seems like patch-test-service would override the milestone specified at boot time, and the system would continue up to the patch installation milestone. > Disabling the patch-test and patch-install services will disable the > automatic installation of pending patches on reboot. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
> 2. Create patch-install-milestone, which depends on patch-install-service >below. The patch-install-milestone could also depend on single-user and filesystem-local so that it is generally useful for admins manually installing patches as well, even if they don't have the patch-test and patch-install services, but want to safely install patches manually when they have zones on local filesystems. -Steve L. > > On Tue, Aug 19, 2008 at 01:03:55PM -0700, Jordan Brown wrote: > > Bob Netherton wrote: > > > And further > > > refinement would only impact patching rather than the booting process > > > as a whole. > > > > Hmm. I don't know how to have a service that runs when a particular > > milestone is selected, that *doesn't* run when "all" is selected. > > (Other than by dynamically enabling and disabling it.) > > > > > rc scripts doing things with SMF seem a permanent solution to a > > > temporary problem. In my virtual universe there are no rc scripts :) > > > And then the alarm clock goes off and I return to reality. But it does > > > promote rc hackery rather than fixing the problem in SMF where it > > > belongs. > > > > Agreed. Besides, I believe that SMF is locked while rc scripts are > > running, and that any attempt to manipulate it deadlocks. > > > > There are related schemes that could work, but the problem is getting > > them properly sequenced into system startup. > > > > > reboot -- -m milestone=patchinstall seems elegantly simple. > > > > Plausible, though it doesn't exactly fit the current application usage > > model. At the moment, the reboot might or might not be triggered by the > > patching application. The patching application leaves the system set up > > to do the patching at the next shutdown/reboot, whenever that might be. > > (For SunUC-S's shutdown-time processing, it's require that that reboot > > be via the "clean" mechanisms - init, shutdown - so that the processing > > gets done.) > > > > This scheme would require either > > > > 1) having the patch application set the default milestone, and then > > having the startup-time processing set it back, or > > 2) having the patch application do the reboot. > > > > There's still the issue with how to keep this service from running when > > you boot to "all". > > > > Hmm. How does single-user-mode login work? What stops it from running > > on a normal boot? Is it a special case? > > > > --- > > > > BTW: I'm not in a position to commit the patch applications. I'm in > > the middle here because I'm relatively familiar with all of the players > > and the issues, but in my day job I'm not responsible for *any* of them. > > ___ > > zones-discuss mailing list > > zones-discuss@opensolaris.org > ___ > zones-discuss mailing list > zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
So you want to be able to interrupt any boot to any milestone, and instead do the patch processing if a patch is pending. You basically want to interrupt the current milestone, and instead just boot to filesystem-local and do the patching. The question is, can the smf milestone be changed mid-milestone? My test shows that it can. How about: 1. Create patch-test-service, on which single-user depends. This will "svcadm milestone patch-install-milestone" if a patch needs to be installed. This service is always enabled. 2. Create patch-install-milestone, which depends on patch-install-service below. 3. Create patch-install service, which depends on: single-user filesystem-local This service is always enabled. It will install a patch if it is pending, otherwise, do nothing. If the service fails, it might need to: # svcadm milestone single-user So that a maintenance prompt will be appear on the console. This might not be necessary. you might get this anyway, as console-login is not reached. It should be ok to issue smf commands from an smf service, as long as they do not try to do any synchronous operations (-s). This approach is also good because an explicit boot to single user WILL NOT attempt to install pending patches. Disabling the patch-test and patch-install services will disable the automatic installation of pending patches on reboot. Thoughts? -Steve L. On Tue, Aug 19, 2008 at 01:03:55PM -0700, Jordan Brown wrote: > Bob Netherton wrote: > > And further > > refinement would only impact patching rather than the booting process > > as a whole. > > Hmm. I don't know how to have a service that runs when a particular > milestone is selected, that *doesn't* run when "all" is selected. > (Other than by dynamically enabling and disabling it.) > > > rc scripts doing things with SMF seem a permanent solution to a > > temporary problem. In my virtual universe there are no rc scripts :) > > And then the alarm clock goes off and I return to reality. But it does > > promote rc hackery rather than fixing the problem in SMF where it > > belongs. > > Agreed. Besides, I believe that SMF is locked while rc scripts are > running, and that any attempt to manipulate it deadlocks. > > There are related schemes that could work, but the problem is getting > them properly sequenced into system startup. > > > reboot -- -m milestone=patchinstall seems elegantly simple. > > Plausible, though it doesn't exactly fit the current application usage > model. At the moment, the reboot might or might not be triggered by the > patching application. The patching application leaves the system set up > to do the patching at the next shutdown/reboot, whenever that might be. > (For SunUC-S's shutdown-time processing, it's require that that reboot > be via the "clean" mechanisms - init, shutdown - so that the processing > gets done.) > > This scheme would require either > > 1) having the patch application set the default milestone, and then > having the startup-time processing set it back, or > 2) having the patch application do the reboot. > > There's still the issue with how to keep this service from running when > you boot to "all". > > Hmm. How does single-user-mode login work? What stops it from running > on a normal boot? Is it a special case? > > --- > > BTW: I'm not in a position to commit the patch applications. I'm in > the middle here because I'm relatively familiar with all of the players > and the issues, but in my day job I'm not responsible for *any* of them. > ___ > zones-discuss mailing list > zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Bob Netherton wrote: > And further > refinement would only impact patching rather than the booting process > as a whole. Hmm. I don't know how to have a service that runs when a particular milestone is selected, that *doesn't* run when "all" is selected. (Other than by dynamically enabling and disabling it.) > rc scripts doing things with SMF seem a permanent solution to a > temporary problem. In my virtual universe there are no rc scripts :) > And then the alarm clock goes off and I return to reality. But it does > promote rc hackery rather than fixing the problem in SMF where it > belongs. Agreed. Besides, I believe that SMF is locked while rc scripts are running, and that any attempt to manipulate it deadlocks. There are related schemes that could work, but the problem is getting them properly sequenced into system startup. > reboot -- -m milestone=patchinstall seems elegantly simple. Plausible, though it doesn't exactly fit the current application usage model. At the moment, the reboot might or might not be triggered by the patching application. The patching application leaves the system set up to do the patching at the next shutdown/reboot, whenever that might be. (For SunUC-S's shutdown-time processing, it's require that that reboot be via the "clean" mechanisms - init, shutdown - so that the processing gets done.) This scheme would require either 1) having the patch application set the default milestone, and then having the startup-time processing set it back, or 2) having the patch application do the reboot. There's still the issue with how to keep this service from running when you boot to "all". Hmm. How does single-user-mode login work? What stops it from running on a normal boot? Is it a special case? --- BTW: I'm not in a position to commit the patch applications. I'm in the middle here because I'm relatively familiar with all of the players and the issues, but in my day job I'm not responsible for *any* of them. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
On Tue, 2008-08-19 at 11:36 -0700, Steve Lawrence wrote: > Add a new service "do-single-user-patch", make it depend on filesystem-local. > This service is typically disabled. This service will add the patch(es) > and reboot. The same could be done with a custom milestone which might be less confusing. I like the elegance of that solution as it relies less on what works now in favor of a defined environment which presumably would get better coverage for zones and zfs [zone]root. And further refinement would only impact patching rather than the booting process as a whole. rc scripts doing things with SMF seem a permanent solution to a temporary problem. In my virtual universe there are no rc scripts :) And then the alarm clock goes off and I return to reality. But it does promote rc hackery rather than fixing the problem in SMF where it belongs. reboot -- -m milestone=patchinstall seems elegantly simple. Bob ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Add a new service "do-single-user-patch", make it depend on filesystem-local. This service is typically disabled. This service will add the patch(es) and reboot. In rcS.d/Swhatever, do: if (we want to do-single-user-patchs) assert(we are currently booting to single-user milestone) svcadm enable -tr do-single-user-patch fi -Steve L. On Tue, Aug 19, 2008 at 08:42:58AM -0700, Liane Praza wrote: > I believe the original concern about making system/filesystem/local part > of single-user was that it changes the definition of single-user. The > zones team was involved in that discussion, and I've just tried to > re-involve them in the resolution discussions. (And have cc'ed them > here. Apologies for the crosspost.) It may just be morning, but I'm > currently having a hard time finding that argument too compelling on its > own -- it seems there must have been something more specific. > > Creating a patch milestone has previously been my preferred direction > (as it requires actual definition and design of what the patching > frameworks need, rather than just hoping that single user is > appropriate), but I'm no longer certain this is a sensible investment, > as my understanding of IPS is that they're sticking to their guns and > there will be no live patching of the OS that leads us to these types of > scenarios. Thankfully. > > Given that, I wonder if a different option is to confirm that > system/filesystem/local is idempotent (and not already running), and > then have patchadd just run its method directly. It leaves a bad taste > in my mouth, but then again so does the fact that we've got two > different patching systems which require the system to be in different > states when they run. > > But, again, I think the zones team explored many of these options in the > first round and would like to make sure they have a chance to weigh in. > > liane (in theory, on vacation today.) > > Renaud Manus wrote: > > A solution could be to move system/filesystem/local in the single-user > > milestone. > > > > -- Renaud > > > > Jordan Brown wrote: > >> Could some SMFers please weigh in on 6725004 and give some opinions on > >> how best to address it? > >> > >> Here's a summary of the problem: > >> > >> Sun Update Connection Enterprise attempts to install "single-user" > >> patches using an rcS.d script. This has historically been a problem, > >> since zone roots may not have been mounted yet. Patchadd was recently > >> changed to partially work around this problem, by enabling > >> system/filesystem/local, but when that mechanism is triggered from an > >> rcS.d script (or, presumably, from a service on which > >> milestone/single-user depends), it yields a deadlock - > >> system/filesystem/local can't be run, because milestone/single-user > >> hasn't been reached. (This is, I believe, in addition to the technical > >> issues surrounding performing SMF operations from an rcS.d script.) > >> > >> I believe that the most truly correct way to address this problem is to > >> have a deferred patching service that depends on > >> system/filesystem/local, and on which all other later services depend. > >> However, I think the only way to implement such a service would be to > >> modify the dependencies of those later services, and that seems awkward. > >> (One could also rename system/filesystem/local, retain the original > >> name as something of a milestone, and insert the deferred patching > >> service as a new service between the renamed s/f/l and the original > >> s/f/l, but that seems quite ugly.) > >> ___ > ___ > zones-discuss mailing list > zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Liane Praza wrote: > It leaves a bad taste > in my mouth, but then again so does the fact that we've got two > different patching systems which require the system to be in different > states when they run. Three :-) Well, sort of. All of them agree that the system should be "in single user mode". The difference is how you get there, and *exactly* what it means. The legacy is that it means interactive, shell-prompt single-user mode. We're trying to implement (or, rather keep supporting) automatic installation of these patches, and interactive shell prompt single-user mode isn't reasonable for automatic installation. (There are one or two really gross ways to do it - boot the system to single user mode, including a service that runs before single user mode, puts itself in the background, and waits for SMF to reach milestone/single-user, then *while the interactive single-user login is available* do the patch installs in the background and reboot the system.) The two automatic schemes attempt to install patches at a system state that is epsilon different from interactive single-user mode. (Either epsilon earlier than interactive or epsilon later than single-user would be OK, with epsilon earlier being easier to implement and epsilon later being slightly more desirable.) One, UCE, does this by running its automatic mechanism from a high-numbered rcS.d script. The other, SunUC-S (a.k.a. smpatch or Update Manager) does it by running its automatic mechanism from a SMF service that runs during system shutdown at a point intended to be equivalent to single-user mode. (The reason that SunUC-S does its work during shutdown rather than startup is that most of these patches require a subsequent reboot, and doing the patching during shutdown means that there's only one reboot. This is better in theory than in practice, because system shutdown is not as well controlled as system startup and many services are left running until the bitter end.) Anyhow, the goal here is to find at least one strategy for automatic installation of these patches that everybody can agree to support. Of course, my slight preference is that it be installation during shutdown (because that reduces the number of reboots), but my expectation is that it will involve installation epsilon from interactive single-user mode, with a subsequent reboot required. ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Enda O'Connor ( Sun Micro Systems Ireland) writes: > alternate BE ), I have seen issues with compilers failing due to SUNWcsr and > SUNWtoo > getting out of sync, because user updated the live system. I think I understand that problem, and I don't think it has anything to do with a live update. The underlying problem there is that there's a hidden dependency built into the way /usr/bin/ld (in SUNWtoo) is constructed. It depends on a private version of libld.so.4 (in SUNWcsr) -- one that gets incremented occasionally. Thus, if you upgrade SUNWcsr, you must also upgrade SUNWtoo, even if you didn't want to, and even though the package dependency runs the other direction (SUNWtoo depends on SUNWcsr, not vice-versa). Since SUNWcsr is one of the common packages on which everyone depends, it's at high risk of being upgraded frequently. It'll be upgraded when someone installs a new package, or when someone upgrades an existing one, because it's a dependency. But the system doesn't seem to know that SUNWtoo is tightly bound to a special library version and must be upgraded as well. It's really a mutual dependency. I don't think we know how many of these sorts of special dependencies exist in ON. Except for the controlled environment of patches, we've always assumed synchronous delivery of everything built in ON as part of our design. -- James Carlson, Solaris Networking <[EMAIL PROTECTED]> Sun Microsystems / 35 Network Drive71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
Liane Praza wrote: > I believe the original concern about making system/filesystem/local part > of single-user was that it changes the definition of single-user. The > zones team was involved in that discussion, and I've just tried to > re-involve them in the resolution discussions. (And have cc'ed them > here. Apologies for the crosspost.) It may just be morning, but I'm > currently having a hard time finding that argument too compelling on its > own -- it seems there must have been something more specific. > > Creating a patch milestone has previously been my preferred direction > (as it requires actual definition and design of what the patching > frameworks need, rather than just hoping that single user is > appropriate), but I'm no longer certain this is a sensible investment, > as my understanding of IPS is that they're sticking to their guns and > there will be no live patching of the OS that leads us to these types of > scenarios. Thankfully. not clear when this will be included, today IPS allows you to update SUNWcsr for instance in the running system ( ie pkg update as opposed to pkg image-update which updates an alternate BE ), I have seen issues with compilers failing due to SUNWcsr and SUNWtoo getting out of sync, because user updated the live system. Also IPS is not relevant to s10 users, who will be patching their systems for some time to come. ( i.e. many years ) I'd like to hear why fs/local being incorporated into single user milestone was not considered a good idea. The patch milestone does sound interesting, as it would involve an agreed consensus cheers Enda > > Given that, I wonder if a different option is to confirm that > system/filesystem/local is idempotent (and not already running), and > then have patchadd just run its method directly. It leaves a bad taste > in my mouth, but then again so does the fact that we've got two > different patching systems which require the system to be in different > states when they run. > > But, again, I think the zones team explored many of these options in the > first round and would like to make sure they have a chance to weigh in. > > liane (in theory, on vacation today.) > > Renaud Manus wrote: >> A solution could be to move system/filesystem/local in the single-user >> milestone. >> >> -- Renaud >> >> Jordan Brown wrote: >>> Could some SMFers please weigh in on 6725004 and give some opinions on >>> how best to address it? >>> >>> Here's a summary of the problem: >>> >>> Sun Update Connection Enterprise attempts to install "single-user" >>> patches using an rcS.d script. This has historically been a problem, >>> since zone roots may not have been mounted yet. Patchadd was recently >>> changed to partially work around this problem, by enabling >>> system/filesystem/local, but when that mechanism is triggered from an >>> rcS.d script (or, presumably, from a service on which >>> milestone/single-user depends), it yields a deadlock - >>> system/filesystem/local can't be run, because milestone/single-user >>> hasn't been reached. (This is, I believe, in addition to the technical >>> issues surrounding performing SMF operations from an rcS.d script.) >>> >>> I believe that the most truly correct way to address this problem is to >>> have a deferred patching service that depends on >>> system/filesystem/local, and on which all other later services depend. >>> However, I think the only way to implement such a service would be to >>> modify the dependencies of those later services, and that seems awkward. >>> (One could also rename system/filesystem/local, retain the original >>> name as something of a milestone, and insert the deferred patching >>> service as a new service between the renamed s/f/l and the original >>> s/f/l, but that seems quite ugly.) >>> ___ > ___ > zones-discuss mailing list > zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] [smf-discuss] 6725004 - installing single-user-mode patches automatically
I believe the original concern about making system/filesystem/local part of single-user was that it changes the definition of single-user. The zones team was involved in that discussion, and I've just tried to re-involve them in the resolution discussions. (And have cc'ed them here. Apologies for the crosspost.) It may just be morning, but I'm currently having a hard time finding that argument too compelling on its own -- it seems there must have been something more specific. Creating a patch milestone has previously been my preferred direction (as it requires actual definition and design of what the patching frameworks need, rather than just hoping that single user is appropriate), but I'm no longer certain this is a sensible investment, as my understanding of IPS is that they're sticking to their guns and there will be no live patching of the OS that leads us to these types of scenarios. Thankfully. Given that, I wonder if a different option is to confirm that system/filesystem/local is idempotent (and not already running), and then have patchadd just run its method directly. It leaves a bad taste in my mouth, but then again so does the fact that we've got two different patching systems which require the system to be in different states when they run. But, again, I think the zones team explored many of these options in the first round and would like to make sure they have a chance to weigh in. liane (in theory, on vacation today.) Renaud Manus wrote: > A solution could be to move system/filesystem/local in the single-user > milestone. > > -- Renaud > > Jordan Brown wrote: >> Could some SMFers please weigh in on 6725004 and give some opinions on >> how best to address it? >> >> Here's a summary of the problem: >> >> Sun Update Connection Enterprise attempts to install "single-user" >> patches using an rcS.d script. This has historically been a problem, >> since zone roots may not have been mounted yet. Patchadd was recently >> changed to partially work around this problem, by enabling >> system/filesystem/local, but when that mechanism is triggered from an >> rcS.d script (or, presumably, from a service on which >> milestone/single-user depends), it yields a deadlock - >> system/filesystem/local can't be run, because milestone/single-user >> hasn't been reached. (This is, I believe, in addition to the technical >> issues surrounding performing SMF operations from an rcS.d script.) >> >> I believe that the most truly correct way to address this problem is to >> have a deferred patching service that depends on >> system/filesystem/local, and on which all other later services depend. >> However, I think the only way to implement such a service would be to >> modify the dependencies of those later services, and that seems awkward. >> (One could also rename system/filesystem/local, retain the original >> name as something of a milestone, and insert the deferred patching >> service as a new service between the renamed s/f/l and the original >> s/f/l, but that seems quite ugly.) >> ___ ___ zones-discuss mailing list zones-discuss@opensolaris.org
