** Changed in: landscape
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
** Changed in: landscape
Status: Incomplete => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
** Changed in: landscape
Milestone: 17.02 => 17.03
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
Worker multiplier set to 1.0 relieves memory pressure and may have
resolved this issue.
** Changed in: landscape
Status: Confirmed => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Re: comment #21 and gating; if this is fully async, then gating is only
going to tighten the race, but not eliminate it. i.e. code like:
if is_leader():
leader_set(...)
can still error as the leadership could be lost between the two calls.
The only safe way of doing leadership code
So, this is a problem with charm-helpers or the charms themselves? I'm
kinda thinking that 'under load' is the exact condition that you want
features like leadership to work. I'll ask @thedac to review the updates
to this bug since his last comment and see if there are any next steps
from the
Marking this as Invalid for Juju.
The problem occurs under heavy load and as per comment # 17, there is nothing
Juju can do about it. It behaves as expected.
You may find comment # 21 very helpful.
** Changed in: juju
Status: New => Invalid
** Changed in: juju/2.1
Status: New =>
I was finally able to reproduce this issue with logging set to TRACE.
The full logs of the deployment (basically /var/log/* from every unit
and the bootstrap node) are available from https://private-
fileshare.canonical.com/~fginther/juju/lp-1654116/unit-not-leader-with-
trace-logs.tar.gz (it's
You are going to want some better instrumentation. The leader-elected
hook is not run when the unit becomes leader. It is run some time after
the unit has become leader, and as far as I know there is no guarantee
that it is still the leader when it happens. I'd stick an 'assert
is_leader()' in the
** Changed in: juju/2.1
Milestone: 2.1.0 => None
** Changed in: juju
Milestone: 2.2.0-alpha1 => None
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write
** Changed in: landscape
Milestone: 17.01 => 17.02
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
** Changed in: juju
Importance: Critical => High
** Also affects: juju/2.1
Importance: Undecided
Status: New
** Changed in: juju/2.1
Status: New => Incomplete
** Changed in: juju/2.1
Importance: Undecided => High
** Changed in: juju/2.1
Milestone: None => 2.1.0
**
** Changed in: juju
Milestone: 2.1-rc1 => 2.1.0
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
relation-changed
I'll note that 'peer_store' isn't safe to directly call, but looking at the
traceback of the original description it is line ~217 of
hooks/amqp-relation-changed which looks to be:
# If this node is the elected leader then share our secret with other nodes
if
Sorry, I'm on crack, I missed line 70/71 which is exactly the 'check if I'm
leader first':
if not is_leader():
return _leader_get(attribute=attribute)
Forgive my earlier rambling. I missed that line and then dug all over to see if
it was trapped outside of that function.
--
You
I haven't dug particularly deeply. However if I do
charm pull ceilometer
I get: cs:ceilometer-24
And then dig into the contents of:
charmhelpers/contrib/peerstorage/__init__.py
I see that it has a function:
def leader_get():
which looks like it is supposed to be a compatibility function, so
One possible root cause may be that if the machine running the unit is
loaded, the time between checking for leadership and the lease expiring
may have passed before the hook gets to write the leadership settings.
It's very unusual that the unit leader changes so often as seen in the
logs. This
I'm thinking also that a controller log (at debug or even trace level)
may be useful to see how the controller is handling leadership changes.
Just the part of the log around the time of the issue and what leads up
to it.
--
You received this bug notification because you are a member of Ubuntu
** Changed in: juju
Status: Triaged => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
Looking at the rabbitmq logs, it appears to me (I could be
misinterpreting):
At 21:39:23 -> unit 2 becomes the leader (the leader-elected hook is run)
At 21:47:25 -> unit 0 becomes the leader (the leader-elected hook is run)
At 22:01:27 -> unit 1 becomes the leader (the leader-elected hook is
** Changed in: juju
Assignee: Menno Smits (menno.smits) => (unassigned)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader
** Also affects: juju
Importance: Undecided
Status: New
** Changed in: juju
Milestone: None => 2.1-rc1
** Changed in: juju
Status: New => Triaged
** Changed in: juju
Importance: Undecided => Critical
** Changed in: juju
Assignee: (unassigned) => Menno Smits
** Changed in: juju (Ubuntu)
Assignee: (unassigned) => Menno Smits (menno.smits)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the
The actual promise from Juju is that from the time you call 'is-leader'
and get a True value, that you will have 30s before we would possibly
return True to any other unit.
Internally the mechanism is that we obtain a lease (valid for 1 minute)
and attempt to renew that lease every 30s (so the
Switch juju-core task to juju.
** Package changed: juju-core (Ubuntu) => juju (Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not
Switch juju-core task to juju.
** Package changed: juju-core (Ubuntu) => juju (Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju in Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership
** Changed in: landscape
Milestone: 16.12 => 17.01
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
** Tags added: uosci
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
relation-changed hooks
To manage notifications
@thedac, I wonder if bug #1657245 might be related. It also happened
with the rabbit charm and looks like it's a split brain issue (/0 and /1
formed a cluster, /2 is the juju leader and went to lala land).
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is
** Tags added: cdo-qa-blocker
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1654116
Title:
Attempts to write leadership settings when not the leader during
relation-changed hooks
To manage
This is a juju is-leader bug.
I have tipple checked that any call to leader-set is gated by an is-
leader check in our charms. Specifically in rabbitmq-server, percona-
cluster and ceilometer.
With the juju 2.1b3 and rabbitmq you can see that leadership is bouncing
around between the three
31 matches
Mail list logo
