It has returned:
http://reports.vapour.ws/releases/2691/job/run-unit-tests-trusty-amd64/attempt/2554
** Changed in: juju-core
Status: Fix Released = Triaged
** Changed in: juju-core
Milestone: 1.21-alpha1 = 1.25.0
** Information type changed from Public to Private
--
You received
It has returned:
http://reports.vapour.ws/releases/2691/job/run-unit-tests-trusty-amd64/attempt/2554
** Changed in: juju-core
Status: Fix Released = Triaged
** Changed in: juju-core
Milestone: 1.21-alpha1 = 1.25.0
** Information type changed from Public to Private
--
You received
** Changed in: juju-core (Ubuntu)
Status: Triaged = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To
** Changed in: juju-core (Ubuntu)
Status: Triaged = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses
** Changed in: juju-core
Status: Fix Committed = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction
** Changed in: juju-core
Status: Fix Committed = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To manage
** Changed in: juju-core/1.20
Status: Fix Committed = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses
** Changed in: juju-core/1.20
Status: Fix Committed = Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To
Is there a way to run Go processes under a debugger and generate very
high-resolution debugging output? I'm seeing this every second or third
attempt to build a cloud. It might be that debugging overhead makes the
problem vanish (yay Heisenberg) but it might give us a useful picture.
As posted in juju-dev last night:
Okay, I couldn't resist investigating a bit. I've been looking at the
database dump from earlier today and it's smelling like a simpler bug
in the txn package, and I might have found the cause already.
Here is a quick walkthrough while debugging the problem, to
Alright, the guess last night was correct, and the candidate fix as
well. I've managed to reproduce the problem by stressing out the
scenario described with 4 concurrent runners running the following two
operations, meanwhile the chaos mechanism injects random slowdowns in
various critical points:
Thanks Gustavo, this is 50% of the issues I see on cloud builds so am
excited to get a build of the tools with this fix applied. Curtis, think
we can spin a build through CI asap that would show up in the testing
tools bucket on S3?
--
You received this bug notification because you are a member
** Also affects: juju-core/1.20
Importance: Undecided
Status: New
** Changed in: juju-core/1.20
Milestone: None = 1.20.2
** Changed in: juju-core/1.20
Importance: Undecided = High
** Changed in: juju-core/1.20
Status: New = In Progress
** Changed in: juju-core
** Changed in: juju-core/1.20
Status: In Progress = Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses
Is there a way to run Go processes under a debugger and generate very
high-resolution debugging output? I'm seeing this every second or third
attempt to build a cloud. It might be that debugging overhead makes the
problem vanish (yay Heisenberg) but it might give us a useful picture.
As posted in juju-dev last night:
Okay, I couldn't resist investigating a bit. I've been looking at the
database dump from earlier today and it's smelling like a simpler bug
in the txn package, and I might have found the cause already.
Here is a quick walkthrough while debugging the problem, to
Alright, the guess last night was correct, and the candidate fix as
well. I've managed to reproduce the problem by stressing out the
scenario described with 4 concurrent runners running the following two
operations, meanwhile the chaos mechanism injects random slowdowns in
various critical points:
Thanks Gustavo, this is 50% of the issues I see on cloud builds so am
excited to get a build of the tools with this fix applied. Curtis, think
we can spin a build through CI asap that would show up in the testing
tools bucket on S3?
--
You received this bug notification because you are a member
** Also affects: juju-core/1.20
Importance: Undecided
Status: New
** Changed in: juju-core/1.20
Milestone: None = 1.20.2
** Changed in: juju-core/1.20
Importance: Undecided = High
** Changed in: juju-core/1.20
Status: New = In Progress
** Changed in: juju-core
** Changed in: juju-core/1.20
Status: In Progress = Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To
Yes, I saw the same restarting of Mongo, looks like every 10-15 seconds.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses
Attached is a dump of the Juju database in this case.
** Attachment added: dump.tgz
https://bugs.launchpad.net/juju-core/+bug/1318366/+attachment/4164924/+files/dump.tgz
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in
After some experiments in compression options, here are all the Juju
logs you could ever want :)
http://people.canonical.com/~mark/juju-server-crash-logs.tar.xz
68M compressed, about 1.9G uncompressed. That's /var/log/juju/ from
machine 0.
--
You received this bug notification because you are
And the db dump (732k) of the juju db in mongo on mark's state server is at
http://chinstrap.canonical.com/~kapil/bug-13183656-juju-db.dump.tbz2
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
Here is a snippet of syslog showing two cycles of Mongo starts and
restarts. This is happening constantly!
Gustavo and I are wondering whether the numactl advice might be
relevant.
** Attachment added: syslog.mongorestarts.log
On 29 July 2014 20:12, Mark Shuttleworth 1318...@bugs.launchpad.net
wrote:
Gustavo and I are wondering whether the numactl advice might be
relevant.
I humbly suggest that's a red herring. Note this happened on an Orange Box
(see log attached in
Digging in further, it appears that jujud is writing to /etc/init/juju-
db.conf (the Upstart job for its database) every few seconds. I'll file
a separate bug about this because it plausibly is the root cause of the
mongo restarts we're seeing.
--
You received this bug notification because you
Summarising what we know so far and adding a few more details:
1. We understand why mongo was continually restarting. jujud currently
restarts mongo every time it starts so every time jujud panicked (~
every 10s), upstart would restart jujud and jujud would restart mongo.
This explains the
Yes, I saw the same restarting of Mongo, looks like every 10-15 seconds.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To
Attached is a dump of the Juju database in this case.
** Attachment added: dump.tgz
https://bugs.launchpad.net/juju-core/+bug/1318366/+attachment/4164924/+files/dump.tgz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
After some experiments in compression options, here are all the Juju
logs you could ever want :)
http://people.canonical.com/~mark/juju-server-crash-logs.tar.xz
68M compressed, about 1.9G uncompressed. That's /var/log/juju/ from
machine 0.
--
You received this bug notification because you are
And the db dump (732k) of the juju db in mongo on mark's state server is at
http://chinstrap.canonical.com/~kapil/bug-13183656-juju-db.dump.tbz2
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Here is a snippet of syslog showing two cycles of Mongo starts and
restarts. This is happening constantly!
Gustavo and I are wondering whether the numactl advice might be
relevant.
** Attachment added: syslog.mongorestarts.log
On 29 July 2014 20:12, Mark Shuttleworth 1318...@bugs.launchpad.net
wrote:
Gustavo and I are wondering whether the numactl advice might be
relevant.
I humbly suggest that's a red herring. Note this happened on an Orange Box
(see log attached in
Digging in further, it appears that jujud is writing to /etc/init/juju-
db.conf (the Upstart job for its database) every few seconds. I'll file
a separate bug about this because it plausibly is the root cause of the
mongo restarts we're seeing.
--
You received this bug notification because you
Summarising what we know so far and adding a few more details:
1. We understand why mongo was continually restarting. jujud currently
restarts mongo every time it starts so every time jujud panicked (~
every 10s), upstart would restart jujud and jujud would restart mongo.
This explains the
I've got this in a live environment from the cloud-installer too. How do
I know where to point mongodump?
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on
I've got this in a live environment from the cloud-installer too. How do
I know where to point mongodump?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic
** Tags added: orange-box
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To manage notifications about
** Changed in: juju-core
Milestone: next-stable = 1.21-alpha1
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in
I'm looking at this right now.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To manage notifications
Is there a more complete log than the one posted on james-page's link?
That one appears to be cut off at a million lines, and doesn't contain
the full panic output, and nothing before the panic, which would be very
helpful.
--
You received this bug notification because you are a member of Ubuntu
Note: I received full logs privately, as they may have sensitive info.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses
This error should never happen on a healthy database. The only case I've
debugged with such an issue was on a system that had a corrupted
database due to an out-of-space situation.
The reason why this should never happen is clear in the code of the txn
package: before anything is ever done with
@John, it's definitely a bad idea to have transactions in a capped
collection for that sort of reason, but as far as I can see the _txns_
collection, the one holding the transactions themselves, is not capped.
Having missing logs for a transaction would not cause this issue.
--
You received this
If anyone hits this again, you should grab a dump of the Mongo db
(mongodump pointed at juju-db address)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on
Looking at the logs from Adam that Nate forwarded to me, I can see the
database is being terminated and restarted over and over and over, every
few seconds. Looking at logs around it, looks like at least rsyslogd is
also being re-freshed on the same cadence.
By itself, this should not be an
** Tags added: orange-box
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To manage notifications about this bug go to:
** Changed in: juju-core
Milestone: next-stable = 1.21-alpha1
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To manage
I'm looking at this right now.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To manage notifications about this bug go to:
Is there a more complete log than the one posted on james-page's link?
That one appears to be cut off at a million lines, and doesn't contain
the full panic output, and nothing before the panic, which would be very
helpful.
--
You received this bug notification because you are a member of Ubuntu
Note: I received full logs privately, as they may have sensitive info.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic misses transaction in queue
To
This error should never happen on a healthy database. The only case I've
debugged with such an issue was on a system that had a corrupted
database due to an out-of-space situation.
The reason why this should never happen is clear in the code of the txn
package: before anything is ever done with
@John, it's definitely a bad idea to have transactions in a capped
collection for that sort of reason, but as far as I can see the _txns_
collection, the one holding the transactions themselves, is not capped.
Having missing logs for a transaction would not cause this issue.
--
You received this
If anyone hits this again, you should grab a dump of the Mongo db
(mongodump pointed at juju-db address)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server panic
Looking at the logs from Adam that Nate forwarded to me, I can see the
database is being terminated and restarted over and over and over, every
few seconds. Looking at logs around it, looks like at least rsyslogd is
also being re-freshed on the same cadence.
By itself, this should not be an
** Summary changed:
- provisioning large environments with complex relations caused jujud on
bootstrap node to fail
+ jujud on state server panic misses transaction in queue
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core
Note my panic was from Juju 1.20.1, I have the full log, am worried
about leaking sensitive information though.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
Please see attached mongo log, it seems to be dying lots.
** Attachment added: grep for mongod in syslog
https://bugs.launchpad.net/juju-core/+bug/1318366/+attachment/4160060/+files/mongod-syslog
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is
** Summary changed:
- provisioning large environments with complex relations caused jujud on
bootstrap node to fail
+ jujud on state server panic misses transaction in queue
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Note my panic was from Juju 1.20.1, I have the full log, am worried
about leaking sensitive information though.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1318366
Title:
jujud on state server
Please see attached mongo log, it seems to be dying lots.
** Attachment added: grep for mongod in syslog
https://bugs.launchpad.net/juju-core/+bug/1318366/+attachment/4160060/+files/mongod-syslog
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is
62 matches
Mail list logo
