Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[William Siegrist]

On Aug 21, 2012, at 6:48 AM, Thiago Marcos P. Santos  
wrote:

> On Tue, Aug 21, 2012 at 4:42 PM, William Siegrist  wrote:
>> 
>> On Aug 21, 2012, at 6:33 AM, William Siegrist  wrote:
>> 
>>> On Aug 21, 2012, at 3:36 AM, Mark Rowe  wrote:
>>> 
 
 On 2012-08-21, at 02:25, Osztrogonac Csaba  wrote:
 
> and again ...
 
 I poked the server to work around the issue. I may have also tracked down 
 the root cause of these issues too. I've sent Bill details of my theory, 
 so hopefully we can come up with a permanent solution to it tomorrow!
 
>>> 
>>> Mark is awesome. But we need to reboot the database server to implement the 
>>> fix. Since its still early for a lot of webkit folks, I'm going to see if 
>>> anyone minds in IRC and then just do it. The usual websites will go down 
>>> for about 30 seconds.
>>> 
>> 
>> Fix is implemented.
>> 
> 
> Looks like we have a problem with commit queue.

I don't think the commit queue relies on the database directly and I don't own 
the hardware it runs on, so you should start a new thread and be more specific 
about the problem you're seeing. 

-Bill

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Thiago Marcos P. Santos]

On Tue, Aug 21, 2012 at 4:42 PM, William Siegrist  wrote:
>
> On Aug 21, 2012, at 6:33 AM, William Siegrist  wrote:
>
>> On Aug 21, 2012, at 3:36 AM, Mark Rowe  wrote:
>>
>>>
>>> On 2012-08-21, at 02:25, Osztrogonac Csaba  wrote:
>>>
 and again ...
>>>
>>> I poked the server to work around the issue. I may have also tracked down 
>>> the root cause of these issues too. I've sent Bill details of my theory, so 
>>> hopefully we can come up with a permanent solution to it tomorrow!
>>>
>>
>> Mark is awesome. But we need to reboot the database server to implement the 
>> fix. Since its still early for a lot of webkit folks, I'm going to see if 
>> anyone minds in IRC and then just do it. The usual websites will go down for 
>> about 30 seconds.
>>
>
> Fix is implemented.
>

Looks like we have a problem with commit queue.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[William Siegrist]

On Aug 21, 2012, at 6:33 AM, William Siegrist  wrote:

> On Aug 21, 2012, at 3:36 AM, Mark Rowe  wrote:
> 
>> 
>> On 2012-08-21, at 02:25, Osztrogonac Csaba  wrote:
>> 
>>> and again ...
>> 
>> I poked the server to work around the issue. I may have also tracked down 
>> the root cause of these issues too. I've sent Bill details of my theory, so 
>> hopefully we can come up with a permanent solution to it tomorrow!
>> 
> 
> Mark is awesome. But we need to reboot the database server to implement the 
> fix. Since its still early for a lot of webkit folks, I'm going to see if 
> anyone minds in IRC and then just do it. The usual websites will go down for 
> about 30 seconds. 
> 

Fix is implemented.

-Bill



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[William Siegrist]

On Aug 21, 2012, at 3:36 AM, Mark Rowe  wrote:

> 
> On 2012-08-21, at 02:25, Osztrogonac Csaba  wrote:
> 
>> and again ...
> 
> I poked the server to work around the issue. I may have also tracked down the 
> root cause of these issues too. I've sent Bill details of my theory, so 
> hopefully we can come up with a permanent solution to it tomorrow!
> 

Mark is awesome. But we need to reboot the database server to implement the 
fix. Since its still early for a lot of webkit folks, I'm going to see if 
anyone minds in IRC and then just do it. The usual websites will go down for 
about 30 seconds. 

-Bill


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Mark Rowe]

On 2012-08-21, at 02:25, Osztrogonac Csaba  wrote:

> and again ...

I poked the server to work around the issue. I may have also tracked down the 
root cause of these issues too. I've sent Bill details of my theory, so 
hopefully we can come up with a permanent solution to it tomorrow!

- Mark

> 
> Osztrogonac Csaba írta:
>> Hi,
>> ... and again and again ... Bill, Mark or Lucas, could you check
>> it, please? Have you got any idea why does it happen regularly?
>> br,
>> Ossy
>> Osztrogonac Csaba írta:
>>> bugs.webkit.org and trac.webkit.org is unavailable again. :(
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Osztrogonac Csaba]

and again ...

Osztrogonac Csaba írta:

Hi,

... and again and again ... Bill, Mark or Lucas, could you check
it, please? Have you got any idea why does it happen regularly?

br,
Ossy

Osztrogonac Csaba írta:

bugs.webkit.org and trac.webkit.org is unavailable again. :(

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[William Siegrist]

On Aug 9, 2012, at 3:41 AM, Mark Rowe  wrote:

> On 2012-08-09, at 03:14, Peter Beverloo  wrote:
> 
>> On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:
>> On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:
>> 
>> > Hi,
>> >
>> > bugs.webkit.org and trac.webkit.org is unavailable again. :(
>> > Could you check it, please?
>> 
>> This is caused by a problem on a host that I don't have sufficient 
>> privileges on to be able to address the issue myself. I've pinged people 
>> that should be able to resolve the issue, but given that it's currently 3am 
>> in California it may be a few hours before they're awake and fixing it.
>> 
>> Thank you. Is there any way an on-call or monitoring system could be set up? 
>> While it fortunately occurs much less frequently nowadays than it did 
>> earlier in the year, it --please excuse my directness-- is unacceptable for 
>> a project the size of WebKit to have critical infrastructure unavailable 
>> like this for several hours. Even though it's 3am in California, there are 
>> many contributors in Asia and Europe who are severely impacted by this.
> 
> We have people online virtually 24x7 that are capable of investigating and 
> addressing issues with the webkit.org infrastructure. I'm one such person. 
> This particular case is an unfortunate combination of events: a configuration 
> error on a subset of the new hardware that webkit.org was recently migrated 
> to has unintentionally limited the number of people that can access the host 
> that is currently experiencing problems, and the person that such issues are 
> escalated to is currently on vacation. It's an unfortunate combination, but 
> also one that is unlikely to repeat.
> 
> One thing we should look in to is improving the process for reporting issues 
> with webkit.org infrastructure. webkit-dev isn't an ideal way to report 
> issues that our monitoring system hasn't noticed as there's no separation 
> between the regular discussion and more urgent issues, so the emails can be 
> overlooked.
> 

I am back from vacation and just wanted to let everyone know that I am sorry 
that so many things went wrong and combined to create significant downtime. 
Like Mark said, we do have systems in place for monitoring, and we usually 
catch and fix things quickly, even at 3am.  The new hardware is a big change 
for us, and our monitoring system is also changing to accommodate the new 
environment, so I will be spending time this week getting to the bottom of the 
database issue and making sure we're covering everything we need to with our 
monitoring systems. 

As for a better notification system, we can create a webkit-sysadmin list that 
contains the people with shell access and send admin@webkit mail there. 
Currently, I'm the only recipient of admin@webkit mail, so we should fix that 
single point of failure. Does that seem reasonable to everyone?

Thanks,
-Bill


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Mark Rowe]

On 2012-08-18, at 07:54, Osztrogonac Csaba  wrote:

> Hi,
> 
> ... and again and again ... Bill, Mark or Lucas, could you check
> it, please? Have you got any idea why does it happen regularly?

Bill is back from vacation next week and will look in to the underlying cause 
of the problem.

- Mark

> 
> br,
> Ossy
> 
> Osztrogonac Csaba írta:
>> bugs.webkit.org and trac.webkit.org is unavailable again. :(

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Osztrogonac Csaba]

Hi,

... and again and again ... Bill, Mark or Lucas, could you check
it, please? Have you got any idea why does it happen regularly?

br,
Ossy

Osztrogonac Csaba írta:

bugs.webkit.org and trac.webkit.org is unavailable again. :(

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Dirk Schulze]

On Aug 12, 2012, at 6:26 PM, Florin Malita  wrote:

> And down it goes again…
It is down for me as well…

Dirk

PS: Sorry, I always wanted to that :D

> 
> On Fri, Aug 10, 2012 at 11:14 AM, Levi Weintraub  wrote:
> We're back!
> 
> 
> On Fri, Aug 10, 2012 at 8:07 AM, Peter Beverloo  wrote:
> It's been down for the past hour, again.  Just a nudge in case it slipped 
> through.
> 
> Peter
> 
> 
> On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba  
> wrote:
> Hi,
> 
> bugs.webkit.org and trac.webkit.org is unavailable again. :(
> Could you check it, please?
> 
> br,
> Ossy
> 
> Osztrogonac Csaba írta:
> 
> It seems bugs.webkit.org and trac.webkit.org is unavailable
> now. (at least from Hungary) Have you got any idea what happened?
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Florin Malita]

And down it goes again...

On Fri, Aug 10, 2012 at 11:14 AM, Levi Weintraub  wrote:

> We're back!
>
>
> On Fri, Aug 10, 2012 at 8:07 AM, Peter Beverloo wrote:
>
>> It's been down for the past hour, again.  Just a nudge in case it slipped
>> through.
>>
>> Peter
>>
>>
>> On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba 
>> wrote:
>>
>>> Hi,
>>>
>>> bugs.webkit.org and trac.webkit.org is unavailable again. :(
>>> Could you check it, please?
>>>
>>> br,
>>> Ossy
>>>
>>> Osztrogonac Csaba írta:
>>>
>>>  It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?

>>> __**_
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> http://lists.webkit.org/**mailman/listinfo/webkit-dev
>>>
>>
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Levi Weintraub]

We're back!

On Fri, Aug 10, 2012 at 8:07 AM, Peter Beverloo  wrote:

> It's been down for the past hour, again.  Just a nudge in case it slipped
> through.
>
> Peter
>
>
> On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba 
> wrote:
>
>> Hi,
>>
>> bugs.webkit.org and trac.webkit.org is unavailable again. :(
>> Could you check it, please?
>>
>> br,
>> Ossy
>>
>> Osztrogonac Csaba írta:
>>
>>  It seems bugs.webkit.org and trac.webkit.org is unavailable
>>> now. (at least from Hungary) Have you got any idea what happened?
>>>
>> __**_
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> http://lists.webkit.org/**mailman/listinfo/webkit-dev
>>
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Peter Beverloo]

It's been down for the past hour, again.  Just a nudge in case it slipped
through.

Peter

On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba wrote:

> Hi,
>
> bugs.webkit.org and trac.webkit.org is unavailable again. :(
> Could you check it, please?
>
> br,
> Ossy
>
> Osztrogonac Csaba írta:
>
>  It seems bugs.webkit.org and trac.webkit.org is unavailable
>> now. (at least from Hungary) Have you got any idea what happened?
>>
> __**_
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/**mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[KwangYul Seo]

Down again?

I can't access bugs.webkit.org again from Korea.


On Thu, Aug 9, 2012 at 9:51 PM, Peter Beverloo  wrote:

> On Thu, Aug 9, 2012 at 12:49 PM, Mark Rowe  wrote:
>
>>
>> On 2012-08-09, at 04:22, Peter Beverloo  wrote:
>>
>> On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe  wrote:
>>
>>>
>>> On 2012-08-09, at 03:14, Peter Beverloo  wrote:
>>>
>>> On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:
>>>
 On 2012-08-09, at 02:41, Osztrogonac Csaba 
 wrote:

 > Hi,
 >
 > bugs.webkit.org and trac.webkit.org is unavailable again. :(
 > Could you check it, please?

 This is caused by a problem on a host that I don't have sufficient
 privileges on to be able to address the issue myself. I've pinged people
 that should be able to resolve the issue, but given that it's currently 3am
 in California it may be a few hours before they're awake and fixing it.

>>>
>>> Thank you. Is there any way an on-call or monitoring system could be set
>>> up? While it fortunately occurs much less frequently nowadays than it did
>>> earlier in the year, it --please excuse my directness-- is unacceptable for
>>> a project the size of WebKit to have critical infrastructure unavailable
>>> like this for several hours. Even though it's 3am in California, there are
>>> many contributors in Asia and Europe who are severely impacted by this.
>>>
>>>
>>> We have people online virtually 24x7 that are capable of investigating
>>> and addressing issues with the webkit.org infrastructure. I'm one such
>>> person. This particular case is an unfortunate combination of events: a
>>> configuration error on a subset of the new hardware that webkit.org was
>>> recently migrated to has unintentionally limited the number of people that
>>> can access the host that is currently experiencing problems, and the person
>>> that such issues are escalated to is currently on vacation. It's an
>>> unfortunate combination, but also one that is unlikely to repeat.
>>>
>>> One thing we should look in to is improving the process for reporting
>>> issues with webkit.org infrastructure. webkit-dev isn't an ideal way to
>>> report issues that our monitoring system hasn't noticed as there's no
>>> separation between the regular discussion and more urgent issues, so the
>>> emails can be overlooked.
>>>
>>
>> I see. Thank you for the insight, and I hope the access issues get sorted
>> out. I presume this is something Bill or Lucas will be able to pick up?
>> Hopefully the remaining downtime won't be more than three or four hours,
>> then.
>>
>>
>> Bill is the one that's on vacation, and Lucas is in the same boat that I
>> am as far as access to the host in question.  I managed to track down
>> someone else with sufficient access though, and they've fixed the issue
>> that was impacting bugs.webkit.org and trac.webkit.org.
>>
>
> Awesome, cheers!
>
> Peter
>
> - Mark
>>
>>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Peter Beverloo]

On Thu, Aug 9, 2012 at 12:49 PM, Mark Rowe  wrote:

>
> On 2012-08-09, at 04:22, Peter Beverloo  wrote:
>
> On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe  wrote:
>
>>
>> On 2012-08-09, at 03:14, Peter Beverloo  wrote:
>>
>> On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:
>>
>>> On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:
>>>
>>> > Hi,
>>> >
>>> > bugs.webkit.org and trac.webkit.org is unavailable again. :(
>>> > Could you check it, please?
>>>
>>> This is caused by a problem on a host that I don't have sufficient
>>> privileges on to be able to address the issue myself. I've pinged people
>>> that should be able to resolve the issue, but given that it's currently 3am
>>> in California it may be a few hours before they're awake and fixing it.
>>>
>>
>> Thank you. Is there any way an on-call or monitoring system could be set
>> up? While it fortunately occurs much less frequently nowadays than it did
>> earlier in the year, it --please excuse my directness-- is unacceptable for
>> a project the size of WebKit to have critical infrastructure unavailable
>> like this for several hours. Even though it's 3am in California, there are
>> many contributors in Asia and Europe who are severely impacted by this.
>>
>>
>> We have people online virtually 24x7 that are capable of investigating
>> and addressing issues with the webkit.org infrastructure. I'm one such
>> person. This particular case is an unfortunate combination of events: a
>> configuration error on a subset of the new hardware that webkit.org was
>> recently migrated to has unintentionally limited the number of people that
>> can access the host that is currently experiencing problems, and the person
>> that such issues are escalated to is currently on vacation. It's an
>> unfortunate combination, but also one that is unlikely to repeat.
>>
>> One thing we should look in to is improving the process for reporting
>> issues with webkit.org infrastructure. webkit-dev isn't an ideal way to
>> report issues that our monitoring system hasn't noticed as there's no
>> separation between the regular discussion and more urgent issues, so the
>> emails can be overlooked.
>>
>
> I see. Thank you for the insight, and I hope the access issues get sorted
> out. I presume this is something Bill or Lucas will be able to pick up?
> Hopefully the remaining downtime won't be more than three or four hours,
> then.
>
>
> Bill is the one that's on vacation, and Lucas is in the same boat that I
> am as far as access to the host in question.  I managed to track down
> someone else with sufficient access though, and they've fixed the issue
> that was impacting bugs.webkit.org and trac.webkit.org.
>

Awesome, cheers!

Peter

- Mark
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Mark Rowe]

On 2012-08-09, at 04:22, Peter Beverloo  wrote:

> On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe  wrote:
> 
> On 2012-08-09, at 03:14, Peter Beverloo  wrote:
> 
>> On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:
>> On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:
>> 
>> > Hi,
>> >
>> > bugs.webkit.org and trac.webkit.org is unavailable again. :(
>> > Could you check it, please?
>> 
>> This is caused by a problem on a host that I don't have sufficient 
>> privileges on to be able to address the issue myself. I've pinged people 
>> that should be able to resolve the issue, but given that it's currently 3am 
>> in California it may be a few hours before they're awake and fixing it.
>> 
>> Thank you. Is there any way an on-call or monitoring system could be set up? 
>> While it fortunately occurs much less frequently nowadays than it did 
>> earlier in the year, it --please excuse my directness-- is unacceptable for 
>> a project the size of WebKit to have critical infrastructure unavailable 
>> like this for several hours. Even though it's 3am in California, there are 
>> many contributors in Asia and Europe who are severely impacted by this.
> 
> We have people online virtually 24x7 that are capable of investigating and 
> addressing issues with the webkit.org infrastructure. I'm one such person. 
> This particular case is an unfortunate combination of events: a configuration 
> error on a subset of the new hardware that webkit.org was recently migrated 
> to has unintentionally limited the number of people that can access the host 
> that is currently experiencing problems, and the person that such issues are 
> escalated to is currently on vacation. It's an unfortunate combination, but 
> also one that is unlikely to repeat.
> 
> One thing we should look in to is improving the process for reporting issues 
> with webkit.org infrastructure. webkit-dev isn't an ideal way to report 
> issues that our monitoring system hasn't noticed as there's no separation 
> between the regular discussion and more urgent issues, so the emails can be 
> overlooked.
> 
> I see. Thank you for the insight, and I hope the access issues get sorted 
> out. I presume this is something Bill or Lucas will be able to pick up? 
> Hopefully the remaining downtime won't be more than three or four hours, then.

Bill is the one that's on vacation, and Lucas is in the same boat that I am as 
far as access to the host in question.  I managed to track down someone else 
with sufficient access though, and they've fixed the issue that was impacting 
bugs.webkit.org and trac.webkit.org.

- Mark

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Peter Beverloo]

On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe  wrote:

>
> On 2012-08-09, at 03:14, Peter Beverloo  wrote:
>
> On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:
>
>> On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:
>>
>> > Hi,
>> >
>> > bugs.webkit.org and trac.webkit.org is unavailable again. :(
>> > Could you check it, please?
>>
>> This is caused by a problem on a host that I don't have sufficient
>> privileges on to be able to address the issue myself. I've pinged people
>> that should be able to resolve the issue, but given that it's currently 3am
>> in California it may be a few hours before they're awake and fixing it.
>>
>
> Thank you. Is there any way an on-call or monitoring system could be set
> up? While it fortunately occurs much less frequently nowadays than it did
> earlier in the year, it --please excuse my directness-- is unacceptable for
> a project the size of WebKit to have critical infrastructure unavailable
> like this for several hours. Even though it's 3am in California, there are
> many contributors in Asia and Europe who are severely impacted by this.
>
>
> We have people online virtually 24x7 that are capable of investigating and
> addressing issues with the webkit.org infrastructure. I'm one such
> person. This particular case is an unfortunate combination of events: a
> configuration error on a subset of the new hardware that webkit.org was
> recently migrated to has unintentionally limited the number of people that
> can access the host that is currently experiencing problems, and the person
> that such issues are escalated to is currently on vacation. It's an
> unfortunate combination, but also one that is unlikely to repeat.
>
> One thing we should look in to is improving the process for reporting
> issues with webkit.org infrastructure. webkit-dev isn't an ideal way to
> report issues that our monitoring system hasn't noticed as there's no
> separation between the regular discussion and more urgent issues, so the
> emails can be overlooked.
>

I see. Thank you for the insight, and I hope the access issues get sorted
out. I presume this is something Bill or Lucas will be able to pick up?
Hopefully the remaining downtime won't be more than three or four hours,
then.

Peter

- Mark
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Mark Rowe]

On 2012-08-09, at 03:14, Peter Beverloo  wrote:

> On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:
> On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:
> 
> > Hi,
> >
> > bugs.webkit.org and trac.webkit.org is unavailable again. :(
> > Could you check it, please?
> 
> This is caused by a problem on a host that I don't have sufficient privileges 
> on to be able to address the issue myself. I've pinged people that should be 
> able to resolve the issue, but given that it's currently 3am in California it 
> may be a few hours before they're awake and fixing it.
> 
> Thank you. Is there any way an on-call or monitoring system could be set up? 
> While it fortunately occurs much less frequently nowadays than it did earlier 
> in the year, it --please excuse my directness-- is unacceptable for a project 
> the size of WebKit to have critical infrastructure unavailable like this for 
> several hours. Even though it's 3am in California, there are many 
> contributors in Asia and Europe who are severely impacted by this.

We have people online virtually 24x7 that are capable of investigating and 
addressing issues with the webkit.org infrastructure. I'm one such person. This 
particular case is an unfortunate combination of events: a configuration error 
on a subset of the new hardware that webkit.org was recently migrated to has 
unintentionally limited the number of people that can access the host that is 
currently experiencing problems, and the person that such issues are escalated 
to is currently on vacation. It's an unfortunate combination, but also one that 
is unlikely to repeat.

One thing we should look in to is improving the process for reporting issues 
with webkit.org infrastructure. webkit-dev isn't an ideal way to report issues 
that our monitoring system hasn't noticed as there's no separation between the 
regular discussion and more urgent issues, so the emails can be overlooked.

- Mark

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Peter Beverloo]

On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe  wrote:

> On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:
>
> > Hi,
> >
> > bugs.webkit.org and trac.webkit.org is unavailable again. :(
> > Could you check it, please?
>
> This is caused by a problem on a host that I don't have sufficient
> privileges on to be able to address the issue myself. I've pinged people
> that should be able to resolve the issue, but given that it's currently 3am
> in California it may be a few hours before they're awake and fixing it.
>

Thank you. Is there any way an on-call or monitoring system could be set
up? While it fortunately occurs much less frequently nowadays than it did
earlier in the year, it --please excuse my directness-- is unacceptable for
a project the size of WebKit to have critical infrastructure unavailable
like this for several hours. Even though it's 3am in California, there are
many contributors in Asia and Europe who are severely impacted by this.

Peter


>
> - Mark
>
> >
> > br,
> > Ossy
> >
> > Osztrogonac Csaba írta:
> >> It seems bugs.webkit.org and trac.webkit.org is unavailable
> >> now. (at least from Hungary) Have you got any idea what happened?
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo/webkit-dev
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Mark Rowe]

On 2012-08-09, at 02:41, Osztrogonac Csaba  wrote:

> Hi,
> 
> bugs.webkit.org and trac.webkit.org is unavailable again. :(
> Could you check it, please?

This is caused by a problem on a host that I don't have sufficient privileges 
on to be able to address the issue myself. I've pinged people that should be 
able to resolve the issue, but given that it's currently 3am in California it 
may be a few hours before they're awake and fixing it.

- Mark

> 
> br,
> Ossy
> 
> Osztrogonac Csaba írta:
>> It seems bugs.webkit.org and trac.webkit.org is unavailable
>> now. (at least from Hungary) Have you got any idea what happened?
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Osztrogonac Csaba]

Hi,

bugs.webkit.org and trac.webkit.org is unavailable again. :(
Could you check it, please?

br,
Ossy

Osztrogonac Csaba írta:

It seems bugs.webkit.org and trac.webkit.org is unavailable
now. (at least from Hungary) Have you got any idea what happened?

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Mark Rowe]

On 2012-08-07, at 10:55, Sravan  wrote:

> I am still seeing the problem(ofcourse, Sunnyvale, CA) ? any one, when this 
> will be up?

This should be fixed now. Part of the database infrastructure (pgpool) was 
timing out all new connections. This caused all of the services that depend on 
the database to stop working.

- Mark

> On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba  
> wrote:
> Hi,
> 
> It seems bugs.webkit.org and trac.webkit.org is unavailable
> now. (at least from Hungary) Have you got any idea what happened?
> 
> br,
> Ossy
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Sravan]

I am still seeing the problem(ofcourse, Sunnyvale, CA) ? any one, when this
will be up?

-Sravan.

On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba wrote:

> Hi,
>
> It seems bugs.webkit.org and trac.webkit.org is unavailable
> now. (at least from Hungary) Have you got any idea what happened?
>
> br,
> Ossy
> __**_
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/**mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Lucas Forschler]

Hi everyone.  We are working on getting things back online.
Thanks!
Lucas

On Aug 7, 2012, at 9:29 AM, Konrad Piascik wrote:

> It's down in Waterloo, ON as well, so I guess it's just down :( 
> Konrad 
> Sent from my BlackBerry on the Rogers Wireless Network
>  
> From: Shawn Singh [mailto:shawnsi...@chromium.org] 
> Sent: Tuesday, August 07, 2012 12:27 PM
> To: Osztrogonac Csaba  
> Cc: WebKit Development  
> Subject: Re: [webkit-dev] bugs.webkit.org and trac.webkit org down? 
>  
> It also seems to be down for me here in Mountain View, CA USA, right next to 
> Apple...
> 
> 
> On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba  
> wrote:
> Hi,
> 
> It seems bugs.webkit.org and trac.webkit.org is unavailable
> now. (at least from Hungary) Have you got any idea what happened?
> 
> br,
> Ossy
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> - 
> This transmission (including any attachments) may contain confidential 
> information, privileged material (including material protected by the 
> solicitor-client or other applicable privileges), or constitute non-public 
> information. Any use of this information by anyone other than the intended 
> recipient is prohibited. If you have received this transmission in error, 
> please immediately reply to the sender and delete this information from your 
> system. Use, dissemination, distribution, or reproduction of this 
> transmission by unintended recipients is not authorized and may be unlawful.
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Konrad Piascik]

It's down in Waterloo, ON as well, so I guess it's just down :(
Konrad
Sent from my BlackBerry on the Rogers Wireless Network

From: Shawn Singh [mailto:shawnsi...@chromium.org]
Sent: Tuesday, August 07, 2012 12:27 PM
To: Osztrogonac Csaba 
Cc: WebKit Development 
Subject: Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

It also seems to be down for me here in Mountain View, CA USA, right next to 
Apple...


On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba 
mailto:o...@inf.u-szeged.hu>> wrote:
Hi,

It seems bugs.webkit.org<http://bugs.webkit.org> and 
trac.webkit.org<http://trac.webkit.org> is unavailable
now. (at least from Hungary) Have you got any idea what happened?

br,
Ossy
___
webkit-dev mailing list
webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>
http://lists.webkit.org/mailman/listinfo/webkit-dev


-
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

[Shawn Singh]

It also seems to be down for me here in Mountain View, CA USA, right next
to Apple...


On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba wrote:

> Hi,
>
> It seems bugs.webkit.org and trac.webkit.org is unavailable
> now. (at least from Hungary) Have you got any idea what happened?
>
> br,
> Ossy
> __**_
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/**mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] bugs.webkit.org and trac.webkit org down?

[Osztrogonac Csaba]

Hi,

It seems bugs.webkit.org and trac.webkit.org is unavailable
now. (at least from Hungary) Have you got any idea what happened?

br,
Ossy
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev