Hook firing
Hi All, I am still a little confused as to when Juju decides to run hooks. Obviously hooks run when something changes. However, I see juju rujnning config-changed hooks at times when the configuration has not changed recently. So, I'd like to know how Juju decides to run the hooks and when? Any advice would be helpful, Thanks to everyone in advance, -- Best Wishes Darryl Weaver Openstack Solutions Architect, Canonical Desk: +44 (0)207 630 2400 ext 507516 Mobile: +44 (0)7720088049 GPG FPR: EA3F 3805 9347 87EC 9CBB 8C1E DADC 82C9 B16B 0403 -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Hook firing
On Wed, Sep 3, 2014 at 9:57 PM, Darryl Weaver darryl.wea...@canonical.com wrote: Hi All, I am still a little confused as to when Juju decides to run hooks. Obviously hooks run when something changes. However, I see juju rujnning config-changed hooks at times when the configuration has not changed recently. Hi Darryl, we've been seeing this too... So, I'd like to know how Juju decides to run the hooks and when? Simon was asking the same question on #juju the other day, apparently juju runs config-changed after a juju agent restart, but I don't know what's triggering that in our case. More at: http://irclogs.ubuntu.com/2014/09/01/%23juju.html#t15:20 Any advice would be helpful, Thanks to everyone in advance, -- Best Wishes Darryl Weaver Openstack Solutions Architect, Canonical Desk: +44 (0)207 630 2400 ext 507516 Mobile: +44 (0)7720088049 GPG FPR: EA3F 3805 9347 87EC 9CBB 8C1E DADC 82C9 B16B 0403 -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Hook firing
On 3 September 2014 13:12, Michael Nelson michael.nel...@canonical.com wrote: On Wed, Sep 3, 2014 at 9:57 PM, Darryl Weaver darryl.wea...@canonical.com wrote: Hi All, I am still a little confused as to when Juju decides to run hooks. Obviously hooks run when something changes. However, I see juju rujnning config-changed hooks at times when the configuration has not changed recently. Hi Darryl, we've been seeing this too... So, I'd like to know how Juju decides to run the hooks and when? Simon was asking the same question on #juju the other day, apparently juju runs config-changed after a juju agent restart, but I don't know what's triggering that in our case. More at: http://irclogs.ubuntu.com/2014/09/01/%23juju.html#t15:20 Also, at machine restart (which I suspect is due primarily to the agent restart case, of course). I don't know if there is a docs page that lists all this explicitly, but it would be nice is if there was, for reference. -- Simon -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Hook firing
Even if the config-changed hook was run when it was not supposed to, it shouldn't have caused any changed if values were not moved to anything different. If it did, then I believe we're having an idempotency problem there. Still, hooks running when they shouldn't is not what expected. On 09/03/2014 08:03 AM, Andreas Hasenack wrote: On Wed, Sep 3, 2014 at 10:00 AM, Simon Davy bloodearn...@gmail.com mailto:bloodearn...@gmail.com wrote: On 3 September 2014 13:42, Darryl Weaver darryl.wea...@canonical.com mailto:darryl.wea...@canonical.com wrote: Thanks the IRC logs are pretty helpful. We are seeing the same issue here at Sky, config-changed hook runs At random times. This is when there are no user changes to the Juju config and no reboots happening. It is possible the juju agent restarted, but I don't think so, but will have to collect some evidence first. Right, this is what I expect is happening to us, but tracking it is tricky. We only noticed because our charm has an implementation issue in that it tries to reuse an authtoken that has a 24hr expiry. It might be helpful if some one on juju-core could details if there are specific circumstances where the juju agent may restart? Look in /var/log/syslog and /var/log/upstart for evidence of this restart. Maybe it died, and then upstart will start it again to keep it running. -- José Antonio Rey -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Hook firing
Given CAP theorem any message can really only be at most once or at least once. You can never guarantee exactly only once. I'm sure juju is biased to the at least once for hooks (hence why it runs at agent restart because it hasn't been around to know for sure that nothing changed in the interim). So while there could be a bug, hooks should be written with a design that they may get called more than once for the same state. (At most once would mean that changes could occur that you never see, which is usually a far worse failure mode.) John =:- On Sep 3, 2014 6:39 PM, Simon Davy bloodearn...@gmail.com wrote: Arg, resending reply to list On 3 September 2014 14:27, José Antonio Rey j...@ubuntu.com wrote: Even if the config-changed hook was run when it was not supposed to, it shouldn't have caused any changed if values were not moved to anything different. If it did, then I believe we're having an idempotency problem there. I suspect this has been happening for a while, but unnoticed, as most case the hook will run fine. In our cases, an implementation issue with tokens that expire after a fixed period of 24hr were being reused (to fetch build assets from swift), so we noticed the run as it tripped an error state, which we have nagios alerts for. We are fixing our charms to not have this problem. But we raised the questions for 2 reasons: 1) We want to understand why it's happening. Its probably ok that it does happen, but it's the apparent randomness that is confusing. 2) If a charm has relied on juju's idempotency provisions exclusively, then a config-changed hook's implementation may *always* restart whatever server process the charm is managing even though nothing has actually changed (we specifically avoid this in our charms where possible). This could lead to unnecessary service wide restarts, which is undesirable in many circumstances, especially with no current way to control those. So yeah, we'd like to get to the bottom of the cause for 1), for our own understand the system, but perhaps 2) needs some consideration from juju-core. Thanks -- Simon -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
