Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Paul Belanger
On Tue, Mar 07, 2017 at 07:39:13PM +1100, Ian Wienand wrote: > On 03/07/2017 07:20 PM, Gene Kuo wrote: > > These errors do line up to the time where it's down. > > However, I have no idea what cause apache to seg fault. > > Something disappearing underneath it would be my suspicion > > Anyway, I

Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Ian Wienand
On 03/07/2017 07:20 PM, Gene Kuo wrote: These errors do line up to the time where it's down. However, I have no idea what cause apache to seg fault. Something disappearing underneath it would be my suspicion Anyway, I added "CoreDumpDirectory /var/cache/apache2" to /etc/apache2/apache2.conf

Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Gene Kuo
;Tom Fifield" <t...@openstack.org> Cc: openstack-infra@lists.openstack.org Subject: Re: [OpenStack-Infra] Ask.o.o down On 03/07/2017 06:46 PM, Gene Kuo wrote: > I found that ask.o.o is down again. I restarted apache --- root@ask:/var/log/apache2# date Tue Mar 7 07:54:26 UTC 2017 [Tue Ma

Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Ian Wienand
On 03/07/2017 06:46 PM, Gene Kuo wrote: > I found that ask.o.o is down again. I restarted apache --- root@ask:/var/log/apache2# date Tue Mar 7 07:54:26 UTC 2017 [Tue Mar 07 06:01:38.469993 2017] [core:notice] [pid 19511:tid 140460060575616] AH00052: child pid 19517 exit signal Segmentation

Re: [OpenStack-Infra] Ask.o.o down

2017-03-06 Thread Gene Kuo
d" <t...@openstack.org> Sent: Tuesday, February 21, 2017 2:15am To: openstack-infra@lists.openstack.org Subject: Re: [OpenStack-Infra] Ask.o.o down On 廿十七年二月廿一日 暮 03:11, Tom Fifield wrote: > > > On 廿十七年二月十四日 暮 04:19, Joshua Hesketh wrote: >> >> >> On T

Re: [OpenStack-Infra] Ask.o.o down

2017-02-20 Thread Tom Fifield
On 廿十七年二月廿一日 暮 03:11, Tom Fifield wrote: On 廿十七年二月十四日 暮 04:19, Joshua Hesketh wrote: On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield > wrote: On 14/02/17 16:11, Joshua Hesketh wrote: Hey Tom, Where is that script being

Re: [OpenStack-Infra] Ask.o.o down

2017-02-20 Thread Tom Fifield
On 廿十七年二月十四日 暮 04:19, Joshua Hesketh wrote: On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield > wrote: On 14/02/17 16:11, Joshua Hesketh wrote: Hey Tom, Where is that script being fired from (a quick grep doesn't find

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Joshua Hesketh
On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield wrote: > On 14/02/17 16:11, Joshua Hesketh wrote: > >> Hey Tom, >> >> Where is that script being fired from (a quick grep doesn't find it), or >> is it a tool people are using? >> >> If it's a tool we'd need to make sure whoever is

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Tom Fifield
On 14/02/17 16:11, Joshua Hesketh wrote: Hey Tom, Where is that script being fired from (a quick grep doesn't find it), or is it a tool people are using? If it's a tool we'd need to make sure whoever is using it gets a new version to rule it out. Indeed. It's fired from a PHP service on

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Joshua Hesketh
Hey Tom, Where is that script being fired from (a quick grep doesn't find it), or is it a tool people are using? If it's a tool we'd need to make sure whoever is using it gets a new version to rule it out. - Josh On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield wrote: > On

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Tom Fifield
On 14/02/17 16:06, Joshua Hesketh wrote: Hey, I've brought the service back up, but have no new clues as to why. Cheers. Going to try: https://review.openstack.org/#/c/433478/ to see if this script is culprit. - Josh On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Joshua Hesketh
Hey, I've brought the service back up, but have no new clues as to why. - Josh On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield wrote: > On 10/02/17 22:39, Jeremy Stanley wrote: > >> On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote: >> [...] >> >>> Down again, this time

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Jeremy Stanley
On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote: [...] > Down again, this time with "Network is unreachable". [...] I'm not finding any obvious errors on the server nor relevant maintenance notices/trouble tickets from the service provider to explain this. I do see conspicuous gaps in

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Tom Fifield
On 10/02/17 16:41, Marton Kiss wrote: Hi Tom Still not working from your end? Works perfectly from here. It is back now. Marton On Fri, Feb 10, 2017 at 9:14 AM Tom Fifield > wrote: On 10/02/17 16:08, Tom Fifield wrote: > On 13/01/17

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Marton Kiss
Hi Tom Still not working from your end? Works perfectly from here. Marton On Fri, Feb 10, 2017 at 9:14 AM Tom Fifield wrote: > On 10/02/17 16:08, Tom Fifield wrote: > > On 13/01/17 14:41, Tom Fifield wrote: > >> As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Tom Fifield
On 13/01/17 14:41, Tom Fifield wrote: As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/ Resolving ask.openstack.org

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Tom Fifield
On 10/02/17 16:08, Tom Fifield wrote: On 13/01/17 14:41, Tom Fifield wrote: As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/

Re: [OpenStack-Infra] Ask.o.o down

2017-01-13 Thread Jeremy Stanley
On 2017-01-13 10:33:24 + (+), Marton Kiss wrote: > You can find more details about the host here: > http://cacti.openstack.org/cacti/graph_view.php?action=tree_id=1_id=156 > It had a network outage somewhere, if you check the eth0, the > traffic was zero. Unfortunately I find no

Re: [OpenStack-Infra] Ask.o.o down

2017-01-13 Thread Marton Kiss
Tom, You can find more details about the host here: http://cacti.openstack.org/cacti/graph_view.php?action=tree_id=1_id=156 It had a network outage somewhere, if you check the eth0, the traffic was zero. Marton On Fri, Jan 13, 2017 at 8:05 AM Tom Fifield wrote: > ... and

Re: [OpenStack-Infra] Ask.o.o down

2017-01-12 Thread Tom Fifield
... and at 0700Z it is back ... On 13/01/17 14:41, Tom Fifield wrote: As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/ Resolving