Re: How to make juju aware of IP address changes?

2013-12-09 Thread roger peppe
On 5 December 2013 18:52, Mark Shuttleworth m...@ubuntu.com wrote: On 04/12/13 17:34, Peter Waller wrote: This situation is now resolved with thanks to Roger, Gustavo and others in real time. There is no way we could have resolved it ourselves since there was corruption of the juju database

Re: Fwd: How to make juju aware of IP address changes?

2013-12-08 Thread Tim Penhey
On 07/12/13 05:04, roger peppe wrote: I think that there are a few things that could help here, most important points first: - We should limit agent restarting in some way (exponential backoff or retry limits or both) I think this is a key one. If the agent is repeatedly failing, we should

Fwd: How to make juju aware of IP address changes?

2013-12-06 Thread roger peppe
On 5 December 2013 18:52, Mark Shuttleworth m...@ubuntu.com wrote: On 04/12/13 17:34, Peter Waller wrote: This situation is now resolved with thanks to Roger, Gustavo and others in real time. There is no way we could have resolved it ourselves since there was corruption of the juju database

Re: How to make juju aware of IP address changes?

2013-11-29 Thread Peter Waller
On 28 November 2013 17:44, Peter Waller pe...@scraperwiki.com wrote: I'm still having the problem of it spinning, every few seconds all of the machines are still spewing into the logs, despite my attempt at asking it to upgrade to a different version. Last night I left it spinning absent any

Re: How to make juju aware of IP address changes?

2013-11-29 Thread Peter Waller
The pids appear to be constant since I last reported them. Your theory about the machine being out of disk is correct. Indeed the log files are 1.4 and 1.6 GB for all-machines and machine-0.log. I'll try xz'ing them and then sending them along to you. Is it okay if I e-mail them directly to you

Re: How to make juju aware of IP address changes?

2013-11-29 Thread roger peppe
On 29 November 2013 10:26, Peter Waller pe...@scraperwiki.com wrote: On 28 November 2013 17:44, Peter Waller pe...@scraperwiki.com wrote: I'm still having the problem of it spinning, every few seconds all of the machines are still spewing into the logs, despite my attempt at asking it to

Fwd: How to make juju aware of IP address changes?

2013-11-29 Thread roger peppe
On 29 November 2013 11:44, Peter Waller pe...@scraperwiki.com wrote: The pids appear to be constant since I last reported them. Your theory about the machine being out of disk is correct. Indeed the log files are 1.4 and 1.6 GB for all-machines and machine-0.log. I'll try xz'ing them and then

Fwd: How to make juju aware of IP address changes?

2013-11-29 Thread roger peppe
On 29 November 2013 14:42, Peter Waller pe...@scraperwiki.com wrote: Hi Roger et al, I've restarted juju-machine-0 and rsyslog. juju status hangs (seemingly indefinitely) and I'm seeing rapid log growth still. In machine-0.log I'm seeing the below. In all-machines.log I'm seeing what looks

Fwd: How to make juju aware of IP address changes?

2013-11-29 Thread roger peppe
On 29 November 2013 15:35, Peter Waller pe...@scraperwiki.com wrote: On 29 November 2013 14:59, roger peppe rogpe...@gmail.com wrote: Have you verified that disk space has actually been freed up? Yup. Assuming so, have you tried restarting juju-db ? Nope. I had managed to miss that

Re: How to make juju aware of IP address changes?

2013-11-29 Thread roger peppe
On 29 November 2013 15:35, Peter Waller pe...@scraperwiki.com wrote: On 29 November 2013 14:59, roger peppe rogpe...@gmail.com wrote: Have you verified that disk space has actually been freed up? Yup. Assuming so, have you tried restarting juju-db ? Nope. I had managed to miss that

Re: How to make juju aware of IP address changes?

2013-11-29 Thread Peter Waller
I've not supplied the full output because it contains private keys, but here is the bit that I think is relevant. Please advise if you need more. 2013-11-29 16:01:20 DEBUG juju.environs.simplestreams simplestreams.go:577 candidate matches for products [com.ubuntu.juju:12.04:amd64

Re: How to make juju aware of IP address changes?

2013-11-29 Thread Peter Waller
I'm still experiencing rapid log growth so any advice on how to prevent it would be appreciated. I guess as a quick short term solution over the weekend I could stop the juju daemons but this doesn't seem ideal. On 29 November 2013 16:13, Peter Waller pe...@scraperwiki.com wrote: For the

Re: How to make juju aware of IP address changes?

2013-11-29 Thread Peter Waller
It appears that over the last few hours the logs grew a bit but the /var/lib/{juju,mongodb} directories grew and are now occupying a lot of disk space (5.5GB), which is now causing problems. In particular /var/lib/mongodb/journal/* is taking 3.1GB in three prealloc.N files and /var/lib/juju/db is

How to make juju aware of IP address changes?

2013-11-28 Thread Peter Waller
Hi All, We needed to attach an elastic IP to some of our machines, and it seems juju hasn't recognized that they have new addresses. How can I force juju to notice? Thanks, - Peter -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at:

Re: How to make juju aware of IP address changes?

2013-11-28 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-11-28 17:54, Peter Waller wrote: Hi All, We needed to attach an elastic IP to some of our machines, and it seems juju hasn't recognized that they have new addresses. How can I force juju to notice? Thanks, - Peter I know in the

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Peter Waller
Our clients are 1.16. How do I check the version of the thing which does the addressupdatering and how do I update it? If it's the bootstrap node, how do I find the juju binary? It doesn't seem to be in the $PATH. On 28 November 2013 14:48, John Arbash Meinel j...@arbash-meinel.comwrote:

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Mark Shuttleworth
On 28/11/13 14:58, Peter Waller wrote: I know in the 1.16 series we have an addressupdater worker that polls loop that polls every 15 minutes. I'm not sure what version you're using. Hmm... is there not an inotify or similar equivalent for network interfaces? Mark signature.asc

Re: How to make juju aware of IP address changes?

2013-11-28 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-11-28 18:58, Peter Waller wrote: Our clients are 1.16. How do I check the version of the thing which does the addressupdatering and how do I update it? If it's the bootstrap node, how do I find the juju binary? It doesn't seem to be in

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Peter Waller
Okay, our juju agents are ancient. How do I update them? Is this documented somewhere? I can't find it very easily. Thanks, - Peter On 28 November 2013 16:47, John Arbash Meinel j...@arbash-meinel.comwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-11-28 18:58, Peter Waller

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Kapil Thangavelu
$ juju upgrade-juju is used for updating the agents within the environment. What version of juju is on the agents in the environment? On Thu, Nov 28, 2013 at 11:52 AM, Peter Waller pe...@scraperwiki.comwrote: Okay, our juju agents are ancient. How do I update them? Is this documented

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Kapil Thangavelu
There is.. $ ip monitor will do it.. its using netlink to watch for changes. -k On Thu, Nov 28, 2013 at 10:13 AM, Mark Shuttleworth m...@ubuntu.com wrote: On 28/11/13 14:58, Peter Waller wrote: I know in the 1.16 series we have an addressupdater worker that polls loop that polls every 15

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Peter Waller
I've done `juju upgrade-juju` and it seems to not be working correctly. For all of my machines I'm seeing something like this in my logs every few seconds. What now? machine-44:2013-11-28 17:09:59 INFO juju runner.go:253 worker: start machiner machine-44:2013-11-28 17:09:59 INFO juju

Re: How to make juju aware of IP address changes?

2013-11-28 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-11-28 21:03, Peter Waller wrote: On 28 November 2013 16:54, Kapil Thangavelu kapil.thangav...@canonical.com mailto:kapil.thangav...@canonical.com wrote: $ juju upgrade-juju is used for updating the agents within the environment. What

Re: How to make juju aware of IP address changes?

2013-11-28 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You should be able to force a version with juju upgrade-juju - --version=$XYZ, but the client should have detected what versions were available before setting the target version. It is possible you are running into the problem of jumping too many

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Mark Shuttleworth
On 28/11/13 17:08, Kapil Thangavelu wrote: There is.. $ ip monitor will do it.. its using netlink to watch for changes. In that case, event-driven with a sweeper process for platforms where the monitor is unavailable or unreliable would be tighter. -- Juju mailing list Juju@lists.ubuntu.com

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Peter Waller
On 28 November 2013 17:15, John Arbash Meinel j...@arbash-meinel.comwrote: 0.13.2. 0.13? or 1.13.2 ? - From 1.13.2 you need to: juju upgrade-juju --version 1.14.1 and then juju upgrade-juju --version 1.16.0 There were some changes from 1.13 that need to be applied by the 1.14

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Mark Shuttleworth
On 28/11/13 17:25, Peter Waller wrote: Is it safe to run that whilst my agents appear to be spinning as I described in my previous e-mail? A very good question. If updates are required to be applied in a sequence, surely Juju would know that better than any given user of Juju, and either

Re: How to make juju aware of IP address changes?

2013-11-28 Thread Peter Waller
I'm still having the problem of it spinning, every few seconds all of the machines are still spewing into the logs, despite my attempt at asking it to upgrade to a different version. Interestingly, there is this tidbit at the end of the versions from the log (these messages are from the bootstrap

Re: How to make juju aware of IP address changes?

2013-11-28 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-11-28 21:28, Mark Shuttleworth wrote: On 28/11/13 17:25, Peter Waller wrote: Is it safe to run that whilst my agents appear to be spinning as I described in my previous e-mail? A very good question. If updates are required to be