Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-21 Thread Osztrogonac Csaba

and again ...

Osztrogonac Csaba írta:

Hi,

... and again and again ... Bill, Mark or Lucas, could you check
it, please? Have you got any idea why does it happen regularly?

br,
Ossy

Osztrogonac Csaba írta:

bugs.webkit.org and trac.webkit.org is unavailable again. :(

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-21 Thread Mark Rowe

On 2012-08-21, at 02:25, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

 and again ...

I poked the server to work around the issue. I may have also tracked down the 
root cause of these issues too. I've sent Bill details of my theory, so 
hopefully we can come up with a permanent solution to it tomorrow!

- Mark

 
 Osztrogonac Csaba írta:
 Hi,
 ... and again and again ... Bill, Mark or Lucas, could you check
 it, please? Have you got any idea why does it happen regularly?
 br,
 Ossy
 Osztrogonac Csaba írta:
 bugs.webkit.org and trac.webkit.org is unavailable again. :(
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-21 Thread William Siegrist
On Aug 21, 2012, at 3:36 AM, Mark Rowe mr...@apple.com wrote:

 
 On 2012-08-21, at 02:25, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:
 
 and again ...
 
 I poked the server to work around the issue. I may have also tracked down the 
 root cause of these issues too. I've sent Bill details of my theory, so 
 hopefully we can come up with a permanent solution to it tomorrow!
 

Mark is awesome. But we need to reboot the database server to implement the 
fix. Since its still early for a lot of webkit folks, I'm going to see if 
anyone minds in IRC and then just do it. The usual websites will go down for 
about 30 seconds. 

-Bill


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-21 Thread William Siegrist

On Aug 21, 2012, at 6:33 AM, William Siegrist wsiegr...@apple.com wrote:

 On Aug 21, 2012, at 3:36 AM, Mark Rowe mr...@apple.com wrote:
 
 
 On 2012-08-21, at 02:25, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:
 
 and again ...
 
 I poked the server to work around the issue. I may have also tracked down 
 the root cause of these issues too. I've sent Bill details of my theory, so 
 hopefully we can come up with a permanent solution to it tomorrow!
 
 
 Mark is awesome. But we need to reboot the database server to implement the 
 fix. Since its still early for a lot of webkit folks, I'm going to see if 
 anyone minds in IRC and then just do it. The usual websites will go down for 
 about 30 seconds. 
 

Fix is implemented.

-Bill



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-21 Thread Thiago Marcos P. Santos
On Tue, Aug 21, 2012 at 4:42 PM, William Siegrist wsiegr...@apple.com wrote:

 On Aug 21, 2012, at 6:33 AM, William Siegrist wsiegr...@apple.com wrote:

 On Aug 21, 2012, at 3:36 AM, Mark Rowe mr...@apple.com wrote:


 On 2012-08-21, at 02:25, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

 and again ...

 I poked the server to work around the issue. I may have also tracked down 
 the root cause of these issues too. I've sent Bill details of my theory, so 
 hopefully we can come up with a permanent solution to it tomorrow!


 Mark is awesome. But we need to reboot the database server to implement the 
 fix. Since its still early for a lot of webkit folks, I'm going to see if 
 anyone minds in IRC and then just do it. The usual websites will go down for 
 about 30 seconds.


 Fix is implemented.


Looks like we have a problem with commit queue.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-21 Thread William Siegrist
On Aug 21, 2012, at 6:48 AM, Thiago Marcos P. Santos tmpsan...@gmail.com 
wrote:

 On Tue, Aug 21, 2012 at 4:42 PM, William Siegrist wsiegr...@apple.com wrote:
 
 On Aug 21, 2012, at 6:33 AM, William Siegrist wsiegr...@apple.com wrote:
 
 On Aug 21, 2012, at 3:36 AM, Mark Rowe mr...@apple.com wrote:
 
 
 On 2012-08-21, at 02:25, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:
 
 and again ...
 
 I poked the server to work around the issue. I may have also tracked down 
 the root cause of these issues too. I've sent Bill details of my theory, 
 so hopefully we can come up with a permanent solution to it tomorrow!
 
 
 Mark is awesome. But we need to reboot the database server to implement the 
 fix. Since its still early for a lot of webkit folks, I'm going to see if 
 anyone minds in IRC and then just do it. The usual websites will go down 
 for about 30 seconds.
 
 
 Fix is implemented.
 
 
 Looks like we have a problem with commit queue.

I don't think the commit queue relies on the database directly and I don't own 
the hardware it runs on, so you should start a new thread and be more specific 
about the problem you're seeing. 

-Bill

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-20 Thread William Siegrist
On Aug 9, 2012, at 3:41 AM, Mark Rowe mr...@apple.com wrote:

 On 2012-08-09, at 03:14, Peter Beverloo pe...@chromium.org wrote:
 
 On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe mr...@apple.com wrote:
 On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:
 
  Hi,
 
  bugs.webkit.org and trac.webkit.org is unavailable again. :(
  Could you check it, please?
 
 This is caused by a problem on a host that I don't have sufficient 
 privileges on to be able to address the issue myself. I've pinged people 
 that should be able to resolve the issue, but given that it's currently 3am 
 in California it may be a few hours before they're awake and fixing it.
 
 Thank you. Is there any way an on-call or monitoring system could be set up? 
 While it fortunately occurs much less frequently nowadays than it did 
 earlier in the year, it --please excuse my directness-- is unacceptable for 
 a project the size of WebKit to have critical infrastructure unavailable 
 like this for several hours. Even though it's 3am in California, there are 
 many contributors in Asia and Europe who are severely impacted by this.
 
 We have people online virtually 24x7 that are capable of investigating and 
 addressing issues with the webkit.org infrastructure. I'm one such person. 
 This particular case is an unfortunate combination of events: a configuration 
 error on a subset of the new hardware that webkit.org was recently migrated 
 to has unintentionally limited the number of people that can access the host 
 that is currently experiencing problems, and the person that such issues are 
 escalated to is currently on vacation. It's an unfortunate combination, but 
 also one that is unlikely to repeat.
 
 One thing we should look in to is improving the process for reporting issues 
 with webkit.org infrastructure. webkit-dev isn't an ideal way to report 
 issues that our monitoring system hasn't noticed as there's no separation 
 between the regular discussion and more urgent issues, so the emails can be 
 overlooked.
 

I am back from vacation and just wanted to let everyone know that I am sorry 
that so many things went wrong and combined to create significant downtime. 
Like Mark said, we do have systems in place for monitoring, and we usually 
catch and fix things quickly, even at 3am.  The new hardware is a big change 
for us, and our monitoring system is also changing to accommodate the new 
environment, so I will be spending time this week getting to the bottom of the 
database issue and making sure we're covering everything we need to with our 
monitoring systems. 

As for a better notification system, we can create a webkit-sysadmin list that 
contains the people with shell access and send admin@webkit mail there. 
Currently, I'm the only recipient of admin@webkit mail, so we should fix that 
single point of failure. Does that seem reasonable to everyone?

Thanks,
-Bill


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-18 Thread Osztrogonac Csaba

Hi,

... and again and again ... Bill, Mark or Lucas, could you check
it, please? Have you got any idea why does it happen regularly?

br,
Ossy

Osztrogonac Csaba írta:

bugs.webkit.org and trac.webkit.org is unavailable again. :(

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-18 Thread Mark Rowe

On 2012-08-18, at 07:54, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

 Hi,
 
 ... and again and again ... Bill, Mark or Lucas, could you check
 it, please? Have you got any idea why does it happen regularly?

Bill is back from vacation next week and will look in to the underlying cause 
of the problem.

- Mark

 
 br,
 Ossy
 
 Osztrogonac Csaba írta:
 bugs.webkit.org and trac.webkit.org is unavailable again. :(

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-12 Thread Florin Malita
And down it goes again...

On Fri, Aug 10, 2012 at 11:14 AM, Levi Weintraub le...@google.com wrote:

 We're back!


 On Fri, Aug 10, 2012 at 8:07 AM, Peter Beverloo pe...@chromium.orgwrote:

 It's been down for the past hour, again.  Just a nudge in case it slipped
 through.

 Peter


 On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba 
 o...@inf.u-szeged.huwrote:

 Hi,

 bugs.webkit.org and trac.webkit.org is unavailable again. :(
 Could you check it, please?

 br,
 Ossy

 Osztrogonac Csaba írta:

  It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?

 __**_
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/**mailman/listinfo/webkit-devhttp://lists.webkit.org/mailman/listinfo/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-12 Thread Dirk Schulze

On Aug 12, 2012, at 6:26 PM, Florin Malita fmal...@google.com wrote:

 And down it goes again…
It is down for me as well…

Dirk

PS: Sorry, I always wanted to that :D

 
 On Fri, Aug 10, 2012 at 11:14 AM, Levi Weintraub le...@google.com wrote:
 We're back!
 
 
 On Fri, Aug 10, 2012 at 8:07 AM, Peter Beverloo pe...@chromium.org wrote:
 It's been down for the past hour, again.  Just a nudge in case it slipped 
 through.
 
 Peter
 
 
 On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba o...@inf.u-szeged.hu 
 wrote:
 Hi,
 
 bugs.webkit.org and trac.webkit.org is unavailable again. :(
 Could you check it, please?
 
 br,
 Ossy
 
 Osztrogonac Csaba írta:
 
 It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev
 
 
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev
 
 
 
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev
 
 
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-10 Thread Levi Weintraub
We're back!

On Fri, Aug 10, 2012 at 8:07 AM, Peter Beverloo pe...@chromium.org wrote:

 It's been down for the past hour, again.  Just a nudge in case it slipped
 through.

 Peter


 On Thu, Aug 9, 2012 at 10:41 AM, Osztrogonac Csaba 
 o...@inf.u-szeged.huwrote:

 Hi,

 bugs.webkit.org and trac.webkit.org is unavailable again. :(
 Could you check it, please?

 br,
 Ossy

 Osztrogonac Csaba írta:

  It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?

 __**_
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/**mailman/listinfo/webkit-devhttp://lists.webkit.org/mailman/listinfo/webkit-dev



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread Osztrogonac Csaba

Hi,

bugs.webkit.org and trac.webkit.org is unavailable again. :(
Could you check it, please?

br,
Ossy

Osztrogonac Csaba írta:

It seems bugs.webkit.org and trac.webkit.org is unavailable
now. (at least from Hungary) Have you got any idea what happened?

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread Mark Rowe
On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

 Hi,
 
 bugs.webkit.org and trac.webkit.org is unavailable again. :(
 Could you check it, please?

This is caused by a problem on a host that I don't have sufficient privileges 
on to be able to address the issue myself. I've pinged people that should be 
able to resolve the issue, but given that it's currently 3am in California it 
may be a few hours before they're awake and fixing it.

- Mark

 
 br,
 Ossy
 
 Osztrogonac Csaba írta:
 It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread Peter Beverloo
On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe mr...@apple.com wrote:

 On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

  Hi,
 
  bugs.webkit.org and trac.webkit.org is unavailable again. :(
  Could you check it, please?

 This is caused by a problem on a host that I don't have sufficient
 privileges on to be able to address the issue myself. I've pinged people
 that should be able to resolve the issue, but given that it's currently 3am
 in California it may be a few hours before they're awake and fixing it.


Thank you. Is there any way an on-call or monitoring system could be set
up? While it fortunately occurs much less frequently nowadays than it did
earlier in the year, it --please excuse my directness-- is unacceptable for
a project the size of WebKit to have critical infrastructure unavailable
like this for several hours. Even though it's 3am in California, there are
many contributors in Asia and Europe who are severely impacted by this.

Peter



 - Mark

 
  br,
  Ossy
 
  Osztrogonac Csaba írta:
  It seems bugs.webkit.org and trac.webkit.org is unavailable
  now. (at least from Hungary) Have you got any idea what happened?
  ___
  webkit-dev mailing list
  webkit-dev@lists.webkit.org
  http://lists.webkit.org/mailman/listinfo/webkit-dev

 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread Peter Beverloo
On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe mr...@apple.com wrote:


 On 2012-08-09, at 03:14, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe mr...@apple.com wrote:

 On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

  Hi,
 
  bugs.webkit.org and trac.webkit.org is unavailable again. :(
  Could you check it, please?

 This is caused by a problem on a host that I don't have sufficient
 privileges on to be able to address the issue myself. I've pinged people
 that should be able to resolve the issue, but given that it's currently 3am
 in California it may be a few hours before they're awake and fixing it.


 Thank you. Is there any way an on-call or monitoring system could be set
 up? While it fortunately occurs much less frequently nowadays than it did
 earlier in the year, it --please excuse my directness-- is unacceptable for
 a project the size of WebKit to have critical infrastructure unavailable
 like this for several hours. Even though it's 3am in California, there are
 many contributors in Asia and Europe who are severely impacted by this.


 We have people online virtually 24x7 that are capable of investigating and
 addressing issues with the webkit.org infrastructure. I'm one such
 person. This particular case is an unfortunate combination of events: a
 configuration error on a subset of the new hardware that webkit.org was
 recently migrated to has unintentionally limited the number of people that
 can access the host that is currently experiencing problems, and the person
 that such issues are escalated to is currently on vacation. It's an
 unfortunate combination, but also one that is unlikely to repeat.

 One thing we should look in to is improving the process for reporting
 issues with webkit.org infrastructure. webkit-dev isn't an ideal way to
 report issues that our monitoring system hasn't noticed as there's no
 separation between the regular discussion and more urgent issues, so the
 emails can be overlooked.


I see. Thank you for the insight, and I hope the access issues get sorted
out. I presume this is something Bill or Lucas will be able to pick up?
Hopefully the remaining downtime won't be more than three or four hours,
then.

Peter

- Mark


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread Mark Rowe

On 2012-08-09, at 04:22, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe mr...@apple.com wrote:
 
 On 2012-08-09, at 03:14, Peter Beverloo pe...@chromium.org wrote:
 
 On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe mr...@apple.com wrote:
 On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:
 
  Hi,
 
  bugs.webkit.org and trac.webkit.org is unavailable again. :(
  Could you check it, please?
 
 This is caused by a problem on a host that I don't have sufficient 
 privileges on to be able to address the issue myself. I've pinged people 
 that should be able to resolve the issue, but given that it's currently 3am 
 in California it may be a few hours before they're awake and fixing it.
 
 Thank you. Is there any way an on-call or monitoring system could be set up? 
 While it fortunately occurs much less frequently nowadays than it did 
 earlier in the year, it --please excuse my directness-- is unacceptable for 
 a project the size of WebKit to have critical infrastructure unavailable 
 like this for several hours. Even though it's 3am in California, there are 
 many contributors in Asia and Europe who are severely impacted by this.
 
 We have people online virtually 24x7 that are capable of investigating and 
 addressing issues with the webkit.org infrastructure. I'm one such person. 
 This particular case is an unfortunate combination of events: a configuration 
 error on a subset of the new hardware that webkit.org was recently migrated 
 to has unintentionally limited the number of people that can access the host 
 that is currently experiencing problems, and the person that such issues are 
 escalated to is currently on vacation. It's an unfortunate combination, but 
 also one that is unlikely to repeat.
 
 One thing we should look in to is improving the process for reporting issues 
 with webkit.org infrastructure. webkit-dev isn't an ideal way to report 
 issues that our monitoring system hasn't noticed as there's no separation 
 between the regular discussion and more urgent issues, so the emails can be 
 overlooked.
 
 I see. Thank you for the insight, and I hope the access issues get sorted 
 out. I presume this is something Bill or Lucas will be able to pick up? 
 Hopefully the remaining downtime won't be more than three or four hours, then.

Bill is the one that's on vacation, and Lucas is in the same boat that I am as 
far as access to the host in question.  I managed to track down someone else 
with sufficient access though, and they've fixed the issue that was impacting 
bugs.webkit.org and trac.webkit.org.

- Mark

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread Peter Beverloo
On Thu, Aug 9, 2012 at 12:49 PM, Mark Rowe mr...@apple.com wrote:


 On 2012-08-09, at 04:22, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe mr...@apple.com wrote:


 On 2012-08-09, at 03:14, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe mr...@apple.com wrote:

 On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu wrote:

  Hi,
 
  bugs.webkit.org and trac.webkit.org is unavailable again. :(
  Could you check it, please?

 This is caused by a problem on a host that I don't have sufficient
 privileges on to be able to address the issue myself. I've pinged people
 that should be able to resolve the issue, but given that it's currently 3am
 in California it may be a few hours before they're awake and fixing it.


 Thank you. Is there any way an on-call or monitoring system could be set
 up? While it fortunately occurs much less frequently nowadays than it did
 earlier in the year, it --please excuse my directness-- is unacceptable for
 a project the size of WebKit to have critical infrastructure unavailable
 like this for several hours. Even though it's 3am in California, there are
 many contributors in Asia and Europe who are severely impacted by this.


 We have people online virtually 24x7 that are capable of investigating
 and addressing issues with the webkit.org infrastructure. I'm one such
 person. This particular case is an unfortunate combination of events: a
 configuration error on a subset of the new hardware that webkit.org was
 recently migrated to has unintentionally limited the number of people that
 can access the host that is currently experiencing problems, and the person
 that such issues are escalated to is currently on vacation. It's an
 unfortunate combination, but also one that is unlikely to repeat.

 One thing we should look in to is improving the process for reporting
 issues with webkit.org infrastructure. webkit-dev isn't an ideal way to
 report issues that our monitoring system hasn't noticed as there's no
 separation between the regular discussion and more urgent issues, so the
 emails can be overlooked.


 I see. Thank you for the insight, and I hope the access issues get sorted
 out. I presume this is something Bill or Lucas will be able to pick up?
 Hopefully the remaining downtime won't be more than three or four hours,
 then.


 Bill is the one that's on vacation, and Lucas is in the same boat that I
 am as far as access to the host in question.  I managed to track down
 someone else with sufficient access though, and they've fixed the issue
 that was impacting bugs.webkit.org and trac.webkit.org.


Awesome, cheers!

Peter

- Mark


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-09 Thread KwangYul Seo
Down again?

I can't access bugs.webkit.org again from Korea.


On Thu, Aug 9, 2012 at 9:51 PM, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 12:49 PM, Mark Rowe mr...@apple.com wrote:


 On 2012-08-09, at 04:22, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 11:41 AM, Mark Rowe mr...@apple.com wrote:


 On 2012-08-09, at 03:14, Peter Beverloo pe...@chromium.org wrote:

 On Thu, Aug 9, 2012 at 11:09 AM, Mark Rowe mr...@apple.com wrote:

 On 2012-08-09, at 02:41, Osztrogonac Csaba o...@inf.u-szeged.hu
 wrote:

  Hi,
 
  bugs.webkit.org and trac.webkit.org is unavailable again. :(
  Could you check it, please?

 This is caused by a problem on a host that I don't have sufficient
 privileges on to be able to address the issue myself. I've pinged people
 that should be able to resolve the issue, but given that it's currently 3am
 in California it may be a few hours before they're awake and fixing it.


 Thank you. Is there any way an on-call or monitoring system could be set
 up? While it fortunately occurs much less frequently nowadays than it did
 earlier in the year, it --please excuse my directness-- is unacceptable for
 a project the size of WebKit to have critical infrastructure unavailable
 like this for several hours. Even though it's 3am in California, there are
 many contributors in Asia and Europe who are severely impacted by this.


 We have people online virtually 24x7 that are capable of investigating
 and addressing issues with the webkit.org infrastructure. I'm one such
 person. This particular case is an unfortunate combination of events: a
 configuration error on a subset of the new hardware that webkit.org was
 recently migrated to has unintentionally limited the number of people that
 can access the host that is currently experiencing problems, and the person
 that such issues are escalated to is currently on vacation. It's an
 unfortunate combination, but also one that is unlikely to repeat.

 One thing we should look in to is improving the process for reporting
 issues with webkit.org infrastructure. webkit-dev isn't an ideal way to
 report issues that our monitoring system hasn't noticed as there's no
 separation between the regular discussion and more urgent issues, so the
 emails can be overlooked.


 I see. Thank you for the insight, and I hope the access issues get sorted
 out. I presume this is something Bill or Lucas will be able to pick up?
 Hopefully the remaining downtime won't be more than three or four hours,
 then.


 Bill is the one that's on vacation, and Lucas is in the same boat that I
 am as far as access to the host in question.  I managed to track down
 someone else with sufficient access though, and they've fixed the issue
 that was impacting bugs.webkit.org and trac.webkit.org.


 Awesome, cheers!

 Peter

 - Mark



 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-07 Thread Shawn Singh
It also seems to be down for me here in Mountain View, CA USA, right next
to Apple...


On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba o...@inf.u-szeged.huwrote:

 Hi,

 It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?

 br,
 Ossy
 __**_
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/**mailman/listinfo/webkit-devhttp://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-07 Thread Konrad Piascik
It's down in Waterloo, ON as well, so I guess it's just down :(
Konrad
Sent from my BlackBerry on the Rogers Wireless Network

From: Shawn Singh [mailto:shawnsi...@chromium.org]
Sent: Tuesday, August 07, 2012 12:27 PM
To: Osztrogonac Csaba o...@inf.u-szeged.hu
Cc: WebKit Development webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

It also seems to be down for me here in Mountain View, CA USA, right next to 
Apple...


On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba 
o...@inf.u-szeged.humailto:o...@inf.u-szeged.hu wrote:
Hi,

It seems bugs.webkit.orghttp://bugs.webkit.org and 
trac.webkit.orghttp://trac.webkit.org is unavailable
now. (at least from Hungary) Have you got any idea what happened?

br,
Ossy
___
webkit-dev mailing list
webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


-
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-07 Thread Lucas Forschler
Hi everyone.  We are working on getting things back online.
Thanks!
Lucas

On Aug 7, 2012, at 9:29 AM, Konrad Piascik wrote:

 It's down in Waterloo, ON as well, so I guess it's just down :( 
 Konrad 
 Sent from my BlackBerry on the Rogers Wireless Network
  
 From: Shawn Singh [mailto:shawnsi...@chromium.org] 
 Sent: Tuesday, August 07, 2012 12:27 PM
 To: Osztrogonac Csaba o...@inf.u-szeged.hu 
 Cc: WebKit Development webkit-dev@lists.webkit.org 
 Subject: Re: [webkit-dev] bugs.webkit.org and trac.webkit org down? 
  
 It also seems to be down for me here in Mountain View, CA USA, right next to 
 Apple...
 
 
 On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba o...@inf.u-szeged.hu 
 wrote:
 Hi,
 
 It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?
 
 br,
 Ossy
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev
 
 - 
 This transmission (including any attachments) may contain confidential 
 information, privileged material (including material protected by the 
 solicitor-client or other applicable privileges), or constitute non-public 
 information. Any use of this information by anyone other than the intended 
 recipient is prohibited. If you have received this transmission in error, 
 please immediately reply to the sender and delete this information from your 
 system. Use, dissemination, distribution, or reproduction of this 
 transmission by unintended recipients is not authorized and may be unlawful.
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-07 Thread Sravan
I am still seeing the problem(ofcourse, Sunnyvale, CA) ? any one, when this
will be up?

-Sravan.

On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba o...@inf.u-szeged.huwrote:

 Hi,

 It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?

 br,
 Ossy
 __**_
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/**mailman/listinfo/webkit-devhttp://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] bugs.webkit.org and trac.webkit org down?

2012-08-07 Thread Mark Rowe

On 2012-08-07, at 10:55, Sravan sra1sand...@gmail.com wrote:

 I am still seeing the problem(ofcourse, Sunnyvale, CA) ? any one, when this 
 will be up?

This should be fixed now. Part of the database infrastructure (pgpool) was 
timing out all new connections. This caused all of the services that depend on 
the database to stop working.

- Mark

 On Tue, Aug 7, 2012 at 9:20 AM, Osztrogonac Csaba o...@inf.u-szeged.hu 
 wrote:
 Hi,
 
 It seems bugs.webkit.org and trac.webkit.org is unavailable
 now. (at least from Hungary) Have you got any idea what happened?
 
 br,
 Ossy
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev
 
 ___
 webkit-dev mailing list
 webkit-dev@lists.webkit.org
 http://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo/webkit-dev