Re: PROPOSAL: stop recording 'executing update-status hook'

[Ian Booth]

On 23/05/17 06:39, Stuart Bishop wrote:
> On 22 May 2017 at 20:02, roger peppe  wrote:
> 
>> not to show in the status history.  Given that the motivation behind
>> the proposal is to reduce load on the database and on controllers, I
> 
> One of the motivations was to reduce load. Another motivation, that
> I'm more interested in, was to make the status log history readable.
> Currently it is page after page of noise about update-status running
> with occasional bits of information.
> 
> (I've leave it to others to argue if it is better to fix this when
> generating the report or by not logging the noise in the first place)
> 

Since Juju 2.1.1, the juju show-status-log command no longer shows
status-history entries by default. There's a new --include-status-updates flag
which can be used if those entries are required in the output.

There's also squashing of repeated log entries. These enhancements were meant to
address the "I don't want to see it problem".

The idea to not record it was meant to address the load issue (both retrieval
and recording). As part of the ongoing performance tuning and scaling efforts,
some hard numbers are being gathered to measure the impact of keeping
update-status in the database.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Stuart Bishop]

On 22 May 2017 at 20:02, roger peppe  wrote:

> not to show in the status history.  Given that the motivation behind
> the proposal is to reduce load on the database and on controllers, I

One of the motivations was to reduce load. Another motivation, that
I'm more interested in, was to make the status log history readable.
Currently it is page after page of noise about update-status running
with occasional bits of information.

(I've leave it to others to argue if it is better to fix this when
generating the report or by not logging the noise in the first place)

-- 
Stuart Bishop 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[roger peppe]

Another optimisation possibility might be to send less status updates.
For example, the uniter could wait until the hook
has been executing for some period of time (e.g. 100ms) before
sending the "executing" status. If the hook finishes within that
period, it would send a bulk update with two updates (setting the status to
"executing" and back). The API server could see that the second update
doesn't affect the current status of the unit and just append
the two items to the history without needing to update the current unit status,
thus (usually, assuming fast hook execution) saving one transaction
per update and halving the number of API calls.

As an aside, I wonder if it's right that the status as set by set-status
is the same status that's set when executing a hook - ISTM that
it might be better if they were separate items of information
so that the status set by the hook code is always available regardless
of what hooks are executing.



On 22 May 2017 at 14:02, roger peppe  wrote:
> On 22 May 2017 at 09:55, Ian Booth  wrote:
>> On 22/05/17 18:23, roger peppe wrote:
>>> I think it's slightly unfortunate that update-status exists at all -
>>> it doesn't really need to,
>>> AFAICS, as a charm can always do the polling itself if needed; for example:
>>>
>>> while :; do
>>>  sleep 30
>>>  juju-run $UNIT 'status-set current-status "this is what is 
>>> happening"'
>>> done &
>>>
>>> Or (better) use juju-run to set the status when the workload
>>> executable starts and exits, avoiding the need for polling at all.
>>
>> It's not sufficient to just set the status when the workload starts and 
>> exits.
>> One example is a database which periodically goes offline for a short time 
>> for
>> maintenance. The workload executable itself should not have to know how to
>> communicate this to Juju. By the agent running update-status hook 
>> periodically,
>> it allows the charm itself to establish whether the database status should be
>> marked as "maintetance" for example.
>
> Of course there are many ways that a workload can become unavailable,
> and sometimes polling may be the only decent answer. My thought is simply
> that rather than baking in polling fundamentally, the charm can be the
> thing with that knowledge. Even without update-status, a charm could
> poll and update its status appropriately if it wants.
>
>> Using a hook provides a standard way all
>> charms can rely on to communicate workload status in a consistent way.
>
> Isn't that the "status-set" command? :)
>
> More generally, on the original proposal, I think I tend to agree with
> Alex Kavanagh - it would be unexpected for a hook to run but
> not to show in the status history.  Given that the motivation behind
> the proposal is to reduce load on the database and on controllers, I
> wonder if it might be possible to make the status recording mechanism
> more efficient.  For example, the document used to store status updates
> currently takes about 178 bytes. This could be reduced to about 75 bytes
> with some simple changes. It should be possible to reduce the number of
> database operations involved in each of the status update calls too - it
> might be possible to reduce a status update RPC call to a single insert
> operation.
>
> Making the mechanism more efficient would also help other cases too -
> for example where an application is continually running some hook other
> than update-status.
>
>   rog.
>
> PS I suspect that almost no-one knows that "juju-run" can be used to run
> non-hook-initiated charm actions - it's hard to search for, its documentation
> isn't available on the client, and it's easily confused with the
> client-side "juju run"
> command. So my reply was also about raising awareness of this.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[roger peppe]

On 22 May 2017 at 09:55, Ian Booth  wrote:
> On 22/05/17 18:23, roger peppe wrote:
>> I think it's slightly unfortunate that update-status exists at all -
>> it doesn't really need to,
>> AFAICS, as a charm can always do the polling itself if needed; for example:
>>
>> while :; do
>>  sleep 30
>>  juju-run $UNIT 'status-set current-status "this is what is 
>> happening"'
>> done &
>>
>> Or (better) use juju-run to set the status when the workload
>> executable starts and exits, avoiding the need for polling at all.
>
> It's not sufficient to just set the status when the workload starts and exits.
> One example is a database which periodically goes offline for a short time for
> maintenance. The workload executable itself should not have to know how to
> communicate this to Juju. By the agent running update-status hook 
> periodically,
> it allows the charm itself to establish whether the database status should be
> marked as "maintetance" for example.

Of course there are many ways that a workload can become unavailable,
and sometimes polling may be the only decent answer. My thought is simply
that rather than baking in polling fundamentally, the charm can be the
thing with that knowledge. Even without update-status, a charm could
poll and update its status appropriately if it wants.

> Using a hook provides a standard way all
> charms can rely on to communicate workload status in a consistent way.

Isn't that the "status-set" command? :)

More generally, on the original proposal, I think I tend to agree with
Alex Kavanagh - it would be unexpected for a hook to run but
not to show in the status history.  Given that the motivation behind
the proposal is to reduce load on the database and on controllers, I
wonder if it might be possible to make the status recording mechanism
more efficient.  For example, the document used to store status updates
currently takes about 178 bytes. This could be reduced to about 75 bytes
with some simple changes. It should be possible to reduce the number of
database operations involved in each of the status update calls too - it
might be possible to reduce a status update RPC call to a single insert
operation.

Making the mechanism more efficient would also help other cases too -
for example where an application is continually running some hook other
than update-status.

  rog.

PS I suspect that almost no-one knows that "juju-run" can be used to run
non-hook-initiated charm actions - it's hard to search for, its documentation
isn't available on the client, and it's easily confused with the
client-side "juju run"
command. So my reply was also about raising awareness of this.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Alex Kavanagh]

>From a 'principle of least surprise', I think that this proposal is
probably a bad idea.  The main driver is to (I guess) reduce the volume of
'chat' between the controller and 100's or 1000's of charms, and the effect
this has on the status log.  However, my 'guess' is also probably wrong as
it would continue to live in the debug log, which implies it is still being
sent/received.

I think it would be 'odd' for a status to be running, but not show up in
the status or status log - this would be unexpected.

My counter proposal is to have a toggle to disable 'update status' on a
model and/or application level, and/or have the ability to reduce the
frequency of the updates.

However, the downside of reducing/eliminating update-status is that (I
suspect) several charms rely on it to recover from errors (particularly)
racy ones.  Perhaps, this can be offset (during install) by usage of 'juju
run ...' and the fixing the charms with the bugs?

Another 'worry' is that if update-status is not reflected in 'juju-status'
will the workload status be updated (i.e. showing problems, fixing
problems, etc.)?

Alex.

On Mon, May 22, 2017 at 9:56 AM, Stuart Bishop 
wrote:

> On 22 May 2017 at 14:36, Tim Penhey  wrote:
> > On 20/05/17 19:48, Merlijn Sebrechts wrote:
> >>
> >> On May 20, 2017 09:05, "John Meinel"  >> > wrote:
> >>
> >> I would actually prefer if it shows up in 'juju status' but that we
> >> suppress it from 'juju status-log' by default.
> >>
> >>
> >> This is still very strange behavior. Why should this be default? Just
> pipe
> >> the output of juju status through grep and exclude update-status if
> that is
> >> really what you want.
> >>
> >> However, I would even argue that this isn't what you want in most
> >> use-cases.  "update-status" isn't seen as a special hook in
> charms.reactive.
> >> Anything can happen in that hook if the conditions are right. Ignoring
> >> update-status will have unforeseen consequences...
> >
> >
> > Hmm... there are (at least) two problems here.
> >
> > Firstly, update-status *should* be a special case hook, and it shouldn't
> > take long.
> >
> > The purpose of the update-status hook was to provide a regular beat for
> the
> > charm to report on the workload status. Really it shouldn't be doing
> other
> > things.
> >
> > The fact that it is a periodic execution rather than being executed in
> > response to model changes is the reason it isn't fitting so well into the
> > regular status and status history updates.
> >
> > The changes to the workload status would still be shown in the history of
> > the workload status, and the workload status is shown in the status
> output.
> >
> > One way to limit the execution of the update-status hook call would be to
> > put a hard timeout on it enforced by the agent.
> >
> > Thoughts?
>
> Unfortunately update-status got wired into charms.reactive like all
> the other standard hooks, and just means 'do whatever still needs to
> be done'. I think its too late to add timeouts or restrictions. But I
> do think special casing it in the status history is needed. Anything
> important will still end up in there due to workload status changes.
>
> --
> Stuart Bishop 
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>



-- 
Alex Kavanagh - Software Engineer
Cloud Dev Ops - Solutions & Product Engineering - Canonical Ltd
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Stuart Bishop]

On 22 May 2017 at 14:36, Tim Penhey  wrote:
> On 20/05/17 19:48, Merlijn Sebrechts wrote:
>>
>> On May 20, 2017 09:05, "John Meinel" > > wrote:
>>
>> I would actually prefer if it shows up in 'juju status' but that we
>> suppress it from 'juju status-log' by default.
>>
>>
>> This is still very strange behavior. Why should this be default? Just pipe
>> the output of juju status through grep and exclude update-status if that is
>> really what you want.
>>
>> However, I would even argue that this isn't what you want in most
>> use-cases.  "update-status" isn't seen as a special hook in charms.reactive.
>> Anything can happen in that hook if the conditions are right. Ignoring
>> update-status will have unforeseen consequences...
>
>
> Hmm... there are (at least) two problems here.
>
> Firstly, update-status *should* be a special case hook, and it shouldn't
> take long.
>
> The purpose of the update-status hook was to provide a regular beat for the
> charm to report on the workload status. Really it shouldn't be doing other
> things.
>
> The fact that it is a periodic execution rather than being executed in
> response to model changes is the reason it isn't fitting so well into the
> regular status and status history updates.
>
> The changes to the workload status would still be shown in the history of
> the workload status, and the workload status is shown in the status output.
>
> One way to limit the execution of the update-status hook call would be to
> put a hard timeout on it enforced by the agent.
>
> Thoughts?

Unfortunately update-status got wired into charms.reactive like all
the other standard hooks, and just means 'do whatever still needs to
be done'. I think its too late to add timeouts or restrictions. But I
do think special casing it in the status history is needed. Anything
important will still end up in there due to workload status changes.

-- 
Stuart Bishop 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Ian Booth]

On 22/05/17 18:23, roger peppe wrote:
> I think it's slightly unfortunate that update-status exists at all -
> it doesn't really need to,
> AFAICS, as a charm can always do the polling itself if needed; for example:
> 
> while :; do
>  sleep 30
>  juju-run $UNIT 'status-set current-status "this is what is 
> happening"'
> done &
> 
> Or (better) use juju-run to set the status when the workload
> executable starts and exits, avoiding the need for polling at all.
>

It's not sufficient to just set the status when the workload starts and exits.
One example is a database which periodically goes offline for a short time for
maintenance. The workload executable itself should not have to know how to
communicate this to Juju. By the agent running update-status hook periodically,
it allows the charm itself to establish whether the database status should be
marked as "maintetance" for example. Using a hook provides a standard way all
charms can rely on to communicate workload status in a consistent way.


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[roger peppe]

I think it's slightly unfortunate that update-status exists at all -
it doesn't really need to,
AFAICS, as a charm can always do the polling itself if needed; for example:

while :; do
 sleep 30
 juju-run $UNIT 'status-set current-status "this is what is happening"'
done &

Or (better) use juju-run to set the status when the workload
executable starts and exits, avoiding the need for polling at all.

  cheers,
rog.

On 22 May 2017 at 08:36, Tim Penhey  wrote:
> On 20/05/17 19:48, Merlijn Sebrechts wrote:
>>
>> On May 20, 2017 09:05, "John Meinel" > > wrote:
>>
>> I would actually prefer if it shows up in 'juju status' but that we
>> suppress it from 'juju status-log' by default.
>>
>>
>> This is still very strange behavior. Why should this be default? Just pipe
>> the output of juju status through grep and exclude update-status if that is
>> really what you want.
>>
>> However, I would even argue that this isn't what you want in most
>> use-cases.  "update-status" isn't seen as a special hook in charms.reactive.
>> Anything can happen in that hook if the conditions are right. Ignoring
>> update-status will have unforeseen consequences...
>
>
> Hmm... there are (at least) two problems here.
>
> Firstly, update-status *should* be a special case hook, and it shouldn't
> take long.
>
> The purpose of the update-status hook was to provide a regular beat for the
> charm to report on the workload status. Really it shouldn't be doing other
> things.
>
> The fact that it is a periodic execution rather than being executed in
> response to model changes is the reason it isn't fitting so well into the
> regular status and status history updates.
>
> The changes to the workload status would still be shown in the history of
> the workload status, and the workload status is shown in the status output.
>
> One way to limit the execution of the update-status hook call would be to
> put a hard timeout on it enforced by the agent.
>
> Thoughts?
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[roger peppe]

I think it's slightly unfortunate that update-status exists at all -
it doesn't really need to,
AFAICS, as a charm can always do the polling itself if needed; for example:

while :; do
 sleep 30
 juju-run $UNIT 'status-set current-status "this is what is happening"'
done &

Or (better) use juju-run to set the status when the workload
executable starts and exits, avoiding the need for polling at all.

  cheers,
rog.

On 22 May 2017 at 08:36, Tim Penhey  wrote:
> On 20/05/17 19:48, Merlijn Sebrechts wrote:
>>
>> On May 20, 2017 09:05, "John Meinel" > > wrote:
>>
>> I would actually prefer if it shows up in 'juju status' but that we
>> suppress it from 'juju status-log' by default.
>>
>>
>> This is still very strange behavior. Why should this be default? Just pipe
>> the output of juju status through grep and exclude update-status if that is
>> really what you want.
>>
>> However, I would even argue that this isn't what you want in most
>> use-cases.  "update-status" isn't seen as a special hook in charms.reactive.
>> Anything can happen in that hook if the conditions are right. Ignoring
>> update-status will have unforeseen consequences...
>
>
> Hmm... there are (at least) two problems here.
>
> Firstly, update-status *should* be a special case hook, and it shouldn't
> take long.
>
> The purpose of the update-status hook was to provide a regular beat for the
> charm to report on the workload status. Really it shouldn't be doing other
> things.
>
> The fact that it is a periodic execution rather than being executed in
> response to model changes is the reason it isn't fitting so well into the
> regular status and status history updates.
>
> The changes to the workload status would still be shown in the history of
> the workload status, and the workload status is shown in the status output.
>
> One way to limit the execution of the update-status hook call would be to
> put a hard timeout on it enforced by the agent.
>
> Thoughts?
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Tim Penhey]

On 20/05/17 19:48, Merlijn Sebrechts wrote:
On May 20, 2017 09:05, "John Meinel" > wrote:


I would actually prefer if it shows up in 'juju status' but that we
suppress it from 'juju status-log' by default.


This is still very strange behavior. Why should this be default? Just 
pipe the output of juju status through grep and exclude update-status if 
that is really what you want.


However, I would even argue that this isn't what you want in most 
use-cases.  "update-status" isn't seen as a special hook in 
charms.reactive. Anything can happen in that hook if the conditions are 
right. Ignoring update-status will have unforeseen consequences...


Hmm... there are (at least) two problems here.

Firstly, update-status *should* be a special case hook, and it shouldn't 
take long.


The purpose of the update-status hook was to provide a regular beat for 
the charm to report on the workload status. Really it shouldn't be doing 
other things.


The fact that it is a periodic execution rather than being executed in 
response to model changes is the reason it isn't fitting so well into 
the regular status and status history updates.


The changes to the workload status would still be shown in the history 
of the workload status, and the workload status is shown in the status 
output.


One way to limit the execution of the update-status hook call would be 
to put a hard timeout on it enforced by the agent.


Thoughts?

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Tim Penhey]

On 19/05/17 19:21, roger peppe wrote:

On 19 May 2017 at 03:13, Tim Penhey  wrote:

Hi folks,

Currently juju will update the status of any hook execution for any unit to
show that it is busy doing things. This was all well and good until we do
things based on time.

Every five minutes (or so) each unit will have the update-status hook
executed to allow the unit to set or update the workload status based on
what is currently going on with that unit.

Since all hook executions are stored, this means that the show-status-log
will show the unit jumping from executing update-status to ready and back
every five minutes.

The proposal is to special case the update-status hook and show in status
(or the status-log) that the hook is being executed. debug-log will continue
to show the hook executing if you are looking.


Presumably you mean *not* show in status here?


Yes. Sorry.

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Merlijn Sebrechts]

On May 20, 2017 09:05, "John Meinel"  wrote:

I would actually prefer if it shows up in 'juju status' but that we
suppress it from 'juju status-log' by default.


This is still very strange behavior. Why should this be default? Just pipe
the output of juju status through grep and exclude update-status if that is
really what you want.

However, I would even argue that this isn't what you want in most
use-cases.  "update-status" isn't seen as a special hook in
charms.reactive. Anything can happen in that hook if the conditions are
right. Ignoring update-status will have unforeseen consequences...

(*possibly* we could show it with a flag, but I'm not sure that it is worth
tracking or not.)

John
=:->


On Fri, May 19, 2017 at 9:42 PM, Gregory Lutostanski  wrote:

> What would happen if the update-status hook hangs here? I know it only is
> supposed to be for short-duration status messages (but charm bugs are known
> to happen).
>
> In addition, it will also block other hooks/actions from happening until
> completed and it will remain in a stuck state with the status reporting as
> "all good". Is there some middle ground rather than not exposing that the
> unit is working in this situation?
>
> Not to be a nay-sayer just want it more thoroughly looked at from a user's
> least surprise in not-great situations where the deployment is wedged.
>
> We would not know that from the status right? Only from the debug-log.
>
> On May 19, 2017 3:46 AM, "John Meinel"  wrote:
>
>> All agents start up in DEBUG until they can talk to the controller and
>> read what the current logging config is set to. Otherwise you wouldn't be
>> able to debug startup issues.
>> That said, I think there was a request to cache the last-known value in
>> agent.conf which would let restarts be less noisy.
>>
>> John
>> =:->
>>
>>
>> On Fri, May 19, 2017 at 12:23 PM, Samuel Cozannet <
>> samuel.cozan...@canonical.com> wrote:
>>
>>> +1
>>>
>>> Maybe one good thing would also be removing the default --debug flag
>>> from all juju machine startup scripts.
>>> It seems hard coded, and requires edition on most deployment.
>>>
>>> ++
>>> Sam
>>>
>>>
>>> On May 19, 2017 10:12, "Adam Collard" 
>>> wrote:
>>>
>>> On Fri, 19 May 2017 at 03:14 Tim Penhey 
>>> wrote:
>>>
 Hi folks,

 Currently juju will update the status of any hook execution for any unit
 to show that it is busy doing things. This was all well and good until
 we do things based on time.

 Every five minutes (or so) each unit will have the update-status hook
 executed to allow the unit to set or update the workload status based on
 what is currently going on with that unit.

 Since all hook executions are stored, this means that the
 show-status-log will show the unit jumping from executing update-status
 to ready and back every five minutes.

 The proposal is to special case the update-status hook and show in
 status (or the status-log) that the hook is being executed. debug-log
 will continue to show the hook executing if you are looking.

 This will reduce noise in the status-log, simplify some of our code
 around dealing with status-log, and reduce load on controllers looking
 after hundreds or thousands of units.

>>>
>>> +1
>>>
>>>
>>> --
>>> Juju mailing list
>>> j...@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>> an/listinfo/juju
>>>
>>>
>>>
>>> --
>>> Juju mailing list
>>> j...@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>> an/listinfo/juju
>>>
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju-dev
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Merlijn Sebrechts]

Not showing subsequent runs seems very strange behavior as an end user and
will complicate debugging issues with update-status hooks.

Don't remove logs just because they are too verbose, there are use-cases
where this info is needed.


On May 19, 2017 04:14, "Tim Penhey"  wrote:

Hi folks,

Currently juju will update the status of any hook execution for any unit to
show that it is busy doing things. This was all well and good until we do
things based on time.

Every five minutes (or so) each unit will have the update-status hook
executed to allow the unit to set or update the workload status based on
what is currently going on with that unit.

Since all hook executions are stored, this means that the show-status-log
will show the unit jumping from executing update-status to ready and back
every five minutes.

The proposal is to special case the update-status hook and show in status
(or the status-log) that the hook is being executed. debug-log will
continue to show the hook executing if you are looking.

This will reduce noise in the status-log, simplify some of our code around
dealing with status-log, and reduce load on controllers looking after
hundreds or thousands of units.

Is anyone opposed to this change?

Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju-dev
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[John Meinel]

I would actually prefer if it shows up in 'juju status' but that we
suppress it from 'juju status-log' by default. (*possibly* we could show it
with a flag, but I'm not sure that it is worth tracking or not.)

John
=:->


On Fri, May 19, 2017 at 9:42 PM, Gregory Lutostanski <
gregory.lutostan...@canonical.com> wrote:

> What would happen if the update-status hook hangs here? I know it only is
> supposed to be for short-duration status messages (but charm bugs are known
> to happen).
>
> In addition, it will also block other hooks/actions from happening until
> completed and it will remain in a stuck state with the status reporting as
> "all good". Is there some middle ground rather than not exposing that the
> unit is working in this situation?
>
> Not to be a nay-sayer just want it more thoroughly looked at from a user's
> least surprise in not-great situations where the deployment is wedged.
>
> We would not know that from the status right? Only from the debug-log.
>
> On May 19, 2017 3:46 AM, "John Meinel"  wrote:
>
>> All agents start up in DEBUG until they can talk to the controller and
>> read what the current logging config is set to. Otherwise you wouldn't be
>> able to debug startup issues.
>> That said, I think there was a request to cache the last-known value in
>> agent.conf which would let restarts be less noisy.
>>
>> John
>> =:->
>>
>>
>> On Fri, May 19, 2017 at 12:23 PM, Samuel Cozannet <
>> samuel.cozan...@canonical.com> wrote:
>>
>>> +1
>>>
>>> Maybe one good thing would also be removing the default --debug flag
>>> from all juju machine startup scripts.
>>> It seems hard coded, and requires edition on most deployment.
>>>
>>> ++
>>> Sam
>>>
>>>
>>> On May 19, 2017 10:12, "Adam Collard" 
>>> wrote:
>>>
>>> On Fri, 19 May 2017 at 03:14 Tim Penhey 
>>> wrote:
>>>
 Hi folks,

 Currently juju will update the status of any hook execution for any unit
 to show that it is busy doing things. This was all well and good until
 we do things based on time.

 Every five minutes (or so) each unit will have the update-status hook
 executed to allow the unit to set or update the workload status based on
 what is currently going on with that unit.

 Since all hook executions are stored, this means that the
 show-status-log will show the unit jumping from executing update-status
 to ready and back every five minutes.

 The proposal is to special case the update-status hook and show in
 status (or the status-log) that the hook is being executed. debug-log
 will continue to show the hook executing if you are looking.

 This will reduce noise in the status-log, simplify some of our code
 around dealing with status-log, and reduce load on controllers looking
 after hundreds or thousands of units.

>>>
>>> +1
>>>
>>>
>>> --
>>> Juju mailing list
>>> j...@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>> an/listinfo/juju
>>>
>>>
>>>
>>> --
>>> Juju mailing list
>>> j...@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>> an/listinfo/juju
>>>
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Gregory Lutostanski]

What would happen if the update-status hook hangs here? I know it only is
supposed to be for short-duration status messages (but charm bugs are known
to happen).

In addition, it will also block other hooks/actions from happening until
completed and it will remain in a stuck state with the status reporting as
"all good". Is there some middle ground rather than not exposing that the
unit is working in this situation?

Not to be a nay-sayer just want it more thoroughly looked at from a user's
least surprise in not-great situations where the deployment is wedged.

We would not know that from the status right? Only from the debug-log.

On May 19, 2017 3:46 AM, "John Meinel"  wrote:

> All agents start up in DEBUG until they can talk to the controller and
> read what the current logging config is set to. Otherwise you wouldn't be
> able to debug startup issues.
> That said, I think there was a request to cache the last-known value in
> agent.conf which would let restarts be less noisy.
>
> John
> =:->
>
>
> On Fri, May 19, 2017 at 12:23 PM, Samuel Cozannet <
> samuel.cozan...@canonical.com> wrote:
>
>> +1
>>
>> Maybe one good thing would also be removing the default --debug flag from
>> all juju machine startup scripts.
>> It seems hard coded, and requires edition on most deployment.
>>
>> ++
>> Sam
>>
>>
>> On May 19, 2017 10:12, "Adam Collard"  wrote:
>>
>> On Fri, 19 May 2017 at 03:14 Tim Penhey  wrote:
>>
>>> Hi folks,
>>>
>>> Currently juju will update the status of any hook execution for any unit
>>> to show that it is busy doing things. This was all well and good until
>>> we do things based on time.
>>>
>>> Every five minutes (or so) each unit will have the update-status hook
>>> executed to allow the unit to set or update the workload status based on
>>> what is currently going on with that unit.
>>>
>>> Since all hook executions are stored, this means that the
>>> show-status-log will show the unit jumping from executing update-status
>>> to ready and back every five minutes.
>>>
>>> The proposal is to special case the update-status hook and show in
>>> status (or the status-log) that the hook is being executed. debug-log
>>> will continue to show the hook executing if you are looking.
>>>
>>> This will reduce noise in the status-log, simplify some of our code
>>> around dealing with status-log, and reduce load on controllers looking
>>> after hundreds or thousands of units.
>>>
>>
>> +1
>>
>>
>> --
>> Juju mailing list
>> j...@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju
>>
>>
>>
>> --
>> Juju mailing list
>> j...@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju
>>
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[John Meinel]

All agents start up in DEBUG until they can talk to the controller and read
what the current logging config is set to. Otherwise you wouldn't be able
to debug startup issues.
That said, I think there was a request to cache the last-known value in
agent.conf which would let restarts be less noisy.

John
=:->


On Fri, May 19, 2017 at 12:23 PM, Samuel Cozannet <
samuel.cozan...@canonical.com> wrote:

> +1
>
> Maybe one good thing would also be removing the default --debug flag from
> all juju machine startup scripts.
> It seems hard coded, and requires edition on most deployment.
>
> ++
> Sam
>
>
> On May 19, 2017 10:12, "Adam Collard"  wrote:
>
> On Fri, 19 May 2017 at 03:14 Tim Penhey  wrote:
>
>> Hi folks,
>>
>> Currently juju will update the status of any hook execution for any unit
>> to show that it is busy doing things. This was all well and good until
>> we do things based on time.
>>
>> Every five minutes (or so) each unit will have the update-status hook
>> executed to allow the unit to set or update the workload status based on
>> what is currently going on with that unit.
>>
>> Since all hook executions are stored, this means that the
>> show-status-log will show the unit jumping from executing update-status
>> to ready and back every five minutes.
>>
>> The proposal is to special case the update-status hook and show in
>> status (or the status-log) that the hook is being executed. debug-log
>> will continue to show the hook executing if you are looking.
>>
>> This will reduce noise in the status-log, simplify some of our code
>> around dealing with status-log, and reduce load on controllers looking
>> after hundreds or thousands of units.
>>
>
> +1
>
>
> --
> Juju mailing list
> j...@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
> an/listinfo/juju
>
>
>
> --
> Juju mailing list
> j...@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Uros Jovanovic]

+1

On Fri, May 19, 2017 at 4:13 AM, Tim Penhey 
wrote:

> Hi folks,
>
> Currently juju will update the status of any hook execution for any unit
> to show that it is busy doing things. This was all well and good until we
> do things based on time.
>
> Every five minutes (or so) each unit will have the update-status hook
> executed to allow the unit to set or update the workload status based on
> what is currently going on with that unit.
>
> Since all hook executions are stored, this means that the show-status-log
> will show the unit jumping from executing update-status to ready and back
> every five minutes.
>
> The proposal is to special case the update-status hook and show in status
> (or the status-log) that the hook is being executed. debug-log will
> continue to show the hook executing if you are looking.
>
> This will reduce noise in the status-log, simplify some of our code around
> dealing with status-log, and reduce load on controllers looking after
> hundreds or thousands of units.
>
> Is anyone opposed to this change?
>
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
> an/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[Adam Collard]

On Fri, 19 May 2017 at 03:14 Tim Penhey  wrote:

> Hi folks,
>
> Currently juju will update the status of any hook execution for any unit
> to show that it is busy doing things. This was all well and good until
> we do things based on time.
>
> Every five minutes (or so) each unit will have the update-status hook
> executed to allow the unit to set or update the workload status based on
> what is currently going on with that unit.
>
> Since all hook executions are stored, this means that the
> show-status-log will show the unit jumping from executing update-status
> to ready and back every five minutes.
>
> The proposal is to special case the update-status hook and show in
> status (or the status-log) that the hook is being executed. debug-log
> will continue to show the hook executing if you are looking.
>
> This will reduce noise in the status-log, simplify some of our code
> around dealing with status-log, and reduce load on controllers looking
> after hundreds or thousands of units.
>

+1
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[roger peppe]

On 19 May 2017 at 03:13, Tim Penhey  wrote:
> Hi folks,
>
> Currently juju will update the status of any hook execution for any unit to
> show that it is busy doing things. This was all well and good until we do
> things based on time.
>
> Every five minutes (or so) each unit will have the update-status hook
> executed to allow the unit to set or update the workload status based on
> what is currently going on with that unit.
>
> Since all hook executions are stored, this means that the show-status-log
> will show the unit jumping from executing update-status to ready and back
> every five minutes.
>
> The proposal is to special case the update-status hook and show in status
> (or the status-log) that the hook is being executed. debug-log will continue
> to show the hook executing if you are looking.

Presumably you mean *not* show in status here?

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: PROPOSAL: stop recording 'executing update-status hook'

[David Britton]

+1 from me: https://bugs.launchpad.net/juju/+bug/1530840

:)

On Thu, May 18, 2017 at 8:13 PM, Tim Penhey 
wrote:

> Hi folks,
>
> Currently juju will update the status of any hook execution for any unit
> to show that it is busy doing things. This was all well and good until we
> do things based on time.
>
> Every five minutes (or so) each unit will have the update-status hook
> executed to allow the unit to set or update the workload status based on
> what is currently going on with that unit.
>
> Since all hook executions are stored, this means that the show-status-log
> will show the unit jumping from executing update-status to ready and back
> every five minutes.
>
> The proposal is to special case the update-status hook and show in status
> (or the status-log) that the hook is being executed. debug-log will
> continue to show the hook executing if you are looking.
>
> This will reduce noise in the status-log, simplify some of our code around
> dealing with status-log, and reduce load on controllers looking after
> hundreds or thousands of units.
>
> Is anyone opposed to this change?
>
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
> an/listinfo/juju-dev
>



-- 
David Britton 
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev