Re: Mass Reboot: Sunday 20th, 22:00 - 23:30 PM GMT+2
Great job, guys! -- Sriram Ramkrishna (sriram.ramkrishna_@@_...@.gmail.com (remove _@@_) On Mon, Oct 21, 2013 at 12:31 PM, Andrea Veri a...@gnome.org wrote: Status update: 1. bugzilla and clipboard are back up again and the services were restored just fine. 2. the manegement cards have been finally configured. (this allows us to reboot, power cycle, check the status of the hardware remotely) 3. mysqld's replication has been configured and will be finalized in the next few hours. 4. a bugzilla-backup node will be set up to avoid any downtime in the case vbox (the machine hosting bugzilla) will get in trouble. The GNOME Sysadmin team worked as much hard as it could to restore the services but most of all we would like to thank everyone for the usual patience and help you gave us reporting issues and contacting us when a problem occurred. Have an awesome day, 2013/10/21 Andrea Veri a...@gnome.org Hi, all the servers rebooted successfully except two machines (seems they are stuck at Rebooting machine) that are actually going to be power-cycled as soon as the on-site personnel will phisically reach the machines given that (unfortunately) the PDUs for the two machines appear to be non-responsive to our logins. (a ticket has been opened to have this fixed asap as well) The machines affected are: 1. bugzilla 2. clipboard (ns-master.gnome.org and ldap.gnome.org, all the DNS requests are currently being served by our slave while SSSD is luckily caching all the logins to git.gnome.org, you might be experiecing slow git pushes though given sssd tries to ping ldap.gnome.org anyway) We'll be making sure that the machines will be power cycled as soon as possible to restore all the services. Thanks for your patience and as always keep an eye on [1]. [1] https://status.gnome.org 2013/10/16 Andrea Veri a...@gnome.org Hi, we'll be performing a mass reboot of all our machines this upcoming sunday to apply all the kernel and security updates. All the services will be affected for a few minutes until the machines reboot, please stay tuned for more updates from the Sysadmin Team and most of all keep an eye at [1], we'll make sure to update it as things move forward. Have an awesome day, [1] https://status.gnome.org -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av ___ desktop-devel-list mailing list desktop-devel-l...@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list ___ foundation-list mailing list foundation-list@gnome.org https://mail.gnome.org/mailman/listinfo/foundation-list
Re: Mass Reboot: Sunday 20th, 22:00 - 23:30 PM GMT+2
Status update: 1. bugzilla and clipboard are back up again and the services were restored just fine. 2. the manegement cards have been finally configured. (this allows us to reboot, power cycle, check the status of the hardware remotely) 3. mysqld's replication has been configured and will be finalized in the next few hours. 4. a bugzilla-backup node will be set up to avoid any downtime in the case vbox (the machine hosting bugzilla) will get in trouble. The GNOME Sysadmin team worked as much hard as it could to restore the services but most of all we would like to thank everyone for the usual patience and help you gave us reporting issues and contacting us when a problem occurred. Have an awesome day, 2013/10/21 Andrea Veri a...@gnome.org Hi, all the servers rebooted successfully except two machines (seems they are stuck at Rebooting machine) that are actually going to be power-cycled as soon as the on-site personnel will phisically reach the machines given that (unfortunately) the PDUs for the two machines appear to be non-responsive to our logins. (a ticket has been opened to have this fixed asap as well) The machines affected are: 1. bugzilla 2. clipboard (ns-master.gnome.org and ldap.gnome.org, all the DNS requests are currently being served by our slave while SSSD is luckily caching all the logins to git.gnome.org, you might be experiecing slow git pushes though given sssd tries to ping ldap.gnome.org anyway) We'll be making sure that the machines will be power cycled as soon as possible to restore all the services. Thanks for your patience and as always keep an eye on [1]. [1] https://status.gnome.org 2013/10/16 Andrea Veri a...@gnome.org Hi, we'll be performing a mass reboot of all our machines this upcoming sunday to apply all the kernel and security updates. All the services will be affected for a few minutes until the machines reboot, please stay tuned for more updates from the Sysadmin Team and most of all keep an eye at [1], we'll make sure to update it as things move forward. Have an awesome day, [1] https://status.gnome.org -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av ___ foundation-list mailing list foundation-list@gnome.org https://mail.gnome.org/mailman/listinfo/foundation-list
Re: Mass Reboot: Sunday 20th, 22:00 - 23:30 PM GMT+2
Hi, all the servers rebooted successfully except two machines (seems they are stuck at Rebooting machine) that are actually going to be power-cycled as soon as the on-site personnel will phisically reach the machines given that (unfortunately) the PDUs for the two machines appear to be non-responsive to our logins. (a ticket has been opened to have this fixed asap as well) The machines affected are: 1. bugzilla 2. clipboard (ns-master.gnome.org and ldap.gnome.org, all the DNS requests are currently being served by our slave while SSSD is luckily caching all the logins to git.gnome.org, you might be experiecing slow git pushes though given sssd tries to ping ldap.gnome.org anyway) We'll be making sure that the machines will be power cycled as soon as possible to restore all the services. Thanks for your patience and as always keep an eye on [1]. [1] https://status.gnome.org 2013/10/16 Andrea Veri a...@gnome.org Hi, we'll be performing a mass reboot of all our machines this upcoming sunday to apply all the kernel and security updates. All the services will be affected for a few minutes until the machines reboot, please stay tuned for more updates from the Sysadmin Team and most of all keep an eye at [1], we'll make sure to update it as things move forward. Have an awesome day, [1] https://status.gnome.org -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av ___ foundation-list mailing list foundation-list@gnome.org https://mail.gnome.org/mailman/listinfo/foundation-list
Re: Mass Reboot: Sunday 20th, 22:00 - 23:30 PM GMT+2
Thanks for the status, Andrea. Much appreciated. On Sun, Oct 20, 2013 at 5:24 PM, Andrea Veri a...@gnome.org wrote: Hi, all the servers rebooted successfully except two machines (seems they are stuck at Rebooting machine) that are actually going to be power-cycled as soon as the on-site personnel will phisically reach the machines given that (unfortunately) the PDUs for the two machines appear to be non-responsive to our logins. (a ticket has been opened to have this fixed asap as well) The machines affected are: 1. bugzilla 2. clipboard (ns-master.gnome.org and ldap.gnome.org, all the DNS requests are currently being served by our slave while SSSD is luckily caching all the logins to git.gnome.org, you might be experiecing slow git pushes though given sssd tries to ping ldap.gnome.org anyway) We'll be making sure that the machines will be power cycled as soon as possible to restore all the services. Thanks for your patience and as always keep an eye on [1]. [1] https://status.gnome.org 2013/10/16 Andrea Veri a...@gnome.org Hi, we'll be performing a mass reboot of all our machines this upcoming sunday to apply all the kernel and security updates. All the services will be affected for a few minutes until the machines reboot, please stay tuned for more updates from the Sysadmin Team and most of all keep an eye at [1], we'll make sure to update it as things move forward. Have an awesome day, [1] https://status.gnome.org -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av ___ desktop-devel-list mailing list desktop-devel-l...@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list ___ foundation-list mailing list foundation-list@gnome.org https://mail.gnome.org/mailman/listinfo/foundation-list
