[Users] Outage window :: resources.ovirt.org / Mailman / etc. :: 2013-06-07 03:00 - 11:00 UTC
There is going to be an outage of resources.ovirt.org for up to 2 hours sometime during an 8 hour service window. The outage will occur 2013-06-07 between 03:00 and 11:00 UTC. To view in your local time: date -d '2013-06-07 03:00 UTC' date -d '2013-06-07 11:00 UTC' == Details == From the Linode service bulletin/reminder: This is just a friendly reminder that your maintenance window starts at 8PM PDT tonight and lasts until 4AM PDT. Downtime from this maintenance is expected to be less than 120 minutes, however please note that the entire maintenance window may be used if required. Your Linode will be gracefully powered down and rebooted during the maintenance. Services not configured to start on a reboot will need to be manually started. Additional information regarding this maintenance event can be found here: https://blog.linode.com/2013/05/17/fremont-upgrades/ == Affected services == * resources.ovirt.org * Mailman (lists.ovirt.org) * some backup services === Not-affected services == * www.ovirt.org / wiki.ovirt.org * gerrit.ovirt.org * jenkins.ovirt.org * Anything at AlterWay and RackSpace facilities ** Foreman ** Puppet ** Jenkins slaves ** ... == Future plans == We'll be migrating all services from this host and decommissioning it ASAP. signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Unplanned Outage :: Mailman and resources :: 2013-04-24 09:00 UTC
There was an unplanned outage of lists.ovirt.org/resources.ovirt.org for 30 minutes. == Details == We began receiving disk full alerts from Postfix on linode01.ovirt.org, that houses lists.ovirt.org and resources.ovirt.org. I looked around the server to find more disk space, but there was nothing obvious that I could drop. Regardless, it's clear from existing needs that more disk space was needed. I shutdown the server, added 12 GB more to the drive image, and restarted the host. == Affected services == * Mailman (lists.ovirt.org) * Package repo (resources.ovirt.org) * ovirtbot === Not-affected services == * www.ovirt.org * wiki.ovirt.org * jenkins.ovirt.org * gerrit.ovirt.org == Future plans == Migrate the entire host to a set of VMs running in one of the new hosting environments. ASAP. -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] [JENKINS][ANN] jenkins.ovirt.org new look and infra
On 03/06/2013 01:43 PM, Eyal Edri wrote: - Original Message - From: Itamar Heim ih...@redhat.com can we shutdown the ec2 instance for now? not yet, me quaid should change the dns 1st (tomorrow)? and then we can do it. If the host thinks it's already jenkins.ovirt.org, then we can just do the DNS switch soonest. Do we need to coordinate more closely on timing? Otherwise I can just file the ticket. (I'll wait for your word before picking a time.) do we have more horsepower to start running say engine findbugs on gerrit patches? not really, since we're still using the same ec2 slaves. (unless we'll run it on the master, but that's not recommended in terms of security) once we'll have ovirt instance running with vms, i imagine we can. hopefully we'll have it running soon (either on alterway02 or on the rackspace servers) I was supposed to be working RackSpace servers today, but I got caught up in being a bit sick and post-travel. But the plan is to load F18 + the oVirt all-in-one on rax01. - Karsten -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Planned outage :: resources.ovirt.org/lists.ovirt.org :: 2013-01-21 01:00 UTC
There will be an outage of www.ovirt.org for approximately 45 minutes.
The outage will occur at 2013-01-21 01:00 UTC. To view in your local time:
date -d '2013-01-21 01:00 UTC'
I may start part of the outage 15 minutes before that. If you anticipate
needing services until the outage window, reply back to me with details
ASAP.
== Details ==
We need to resize the Linode instance to get another 15 GB for storage
until we can move services off the Linode permanently, as planned. This
resizing should give us some breathing room.
The account resize is estimated to take 30 minutes, during which time
the Linode VM will be offline. After that, there will be a few minutes
for reboot and restart of services, including the manual starting of the
IRC bot 'ovirtbot'.
The time window chosen coincides with the lowest CPU usage typically
seen on any given day - 01:00 UTC tends to be very quiet for about an
hour. Hopefully no one will even notice the downtime.
If you have any services, such as Jenkins or Gerrit backup, that may go
off during that window, you may want to retime it or be prepared for an
error.
== Affected services ==
* resources.ovirt.org
** meeting logs
** packages
* lists.ovirt.org (MailMan)
* ovirtbot
* Gerrit backup (anacron may pick this up)
* Other cronjobs (anacron may pick this up)
== Not-affected services ==
* www.ovirt.org (MediaWiki)
* jenkins.ovirt.org
* gerrit.ovirt.org
* alterway{01,02}.ovirt.org
--
Karsten 'quaid' Wade, Sr. Analyst - Community Growth
http://TheOpenSourceWay.org .^\ http://community.redhat.com
@quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
[Users] Missing archives
On 21 Dec I fixed the archive for this mailing list at the request of a user. When I did that, I failed to set the permissions and ownership for the changed mailbox file. It remained owned by root, so Mailman was unable to write to the archives. :( :( I think we can fix this if anyone has a copy of all the email sent to this list since 21 Dec. in an .mbox format. Anyone have everything from 21 Dec. to this message I just sent (which should hit the archives)? - Karsten -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Missing archives
(Sorry for the broken threading by replying to myself, I didn't get Joop's reply directly, not sure why, but at least it appeared in the archives!) Joop wrote: I have an archive of Ovirt-Users starting from 2-jun til now in Thunderbird format. Can prune a copy if you want, just let me know. That would be great, thanks. Is it small enough to gzip and email to me? I think the Thunderbird mbox format is the same. (I have a copy of all the messages as well, but I have all my oVirt project email going in to one folder, so the users@ email is mixed in with arch@ etc.) Thanks - Karsten -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Maintenance outage :: lists.ovirt.org :: 2012-12-20 01:00 UTC
There is an outage of lists.ovirt.org for ten minutes to increase disk space on the host. The outage will occur at 2012-12-20 01:00 UTC. To view in your local time: date -d '2012-12-20 01:00 UTC' == Details == We are constantly nearly out of disk space on linode01.ovirt.org, which hosts the Mailman and yum/download repositories. As a stop-gap help, I've got 5 GB more of space I can put in to the host VM. We plan to migrate this host to one of the Alter Way instances, probably a VM running on one of the two physical machines. If possible, we'll make this move soonest for both the Mailman and download repositories. == Affected services == * lists.ovirt.org * resources.ovirt.org * httpd redirects of old URLs * IRC bot === Not-affected services == * www.ovirt.org * jenkins.ovirt.org * gerrit.ovirt.org * git.ovirt.org == Future plans == Move services to a host with more space and capability, as part of the Infra team general hosting plan. -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] What's going on with images on the wiki
== Summary ==
Since the migration to OpenShift, due to a misconfiguration of
MediaWiki, we were writing images uploaded to the wiki to a
non-persistent data store location.
This means that most images uploaded since the migration may have been
destroyed, including archives of past versions of images.
== What's next ==
Jason Brooks and I diagnosed the problem and fixed it last night. We
didn't completely figure out which images were removed, but it's likely
to be most or all the images since the migration.
For any images you still have -- possibly including older version of the
images -- we can likely reinsert them in to their (new) location in the
filesystem and MediaWiki will pick them up. (Older versions are stored
in 'images/archives/'. Depending on the trickiness of the problem, we
may be able to reinsert the archived images.)
Email me with details of what images you have, page names, and so forth.
I'll work to get each one back where it belongs. Feel free to file a
ticket on Trac at http://fedorahosted.org/ovirt to help keep track of
the details. Or you can reply with details (but not images)
If you've lost any work as a result of this mistake, please accept my
apology. It was a mistake that didn't show itself in testing
post-migration (which points to failings in the testing.)
== Questions and comments ==
It's possible we haven't optimized our solution from here. Feel free to
chime in with your thoughts and questions. (Please read the 'Details'
section below first.)
== Details ==
When I configured the MediaWiki instance on OpenShift, I made the
writeable 'images' directory a symlink to what I thought was the
persistent data store location. This appears as 'wiki/data', where the
MediaWiki code and content live in 'wiki/php'.
cd wiki
ls -hal | grep images
images = ../data
That is, 'wiki/php/images' was a symlink to 'wiki/data'.
(Un)fortunately, 'wiki' is all entirely in a git repository. That means
I copied the images from the old MediaWiki instance in to a git
repository that OpenShift uses to completely rewrite the folder
structure on the server every time the code is checked in ('git push'.)
As it happens, the true persistent data store is '../../data', that is,
one directory *above* the 'wiki' directory. When you put content in
there, it is not overwritten with each 'git push'.
What happened this week was I did an outage window on Monday to push a
change to the app to make it hot deployable. (I actually did the same
thing last week, but I don't know if anyone had uploaded any new images.)
When I did a 'git push' of my changes, I apparently overwrote the
'wiki/data/' directory with my older copy of the content. I did a 'git
pull' before pushing but nothing came down; I'm not sure why but I
believe it has to do with the way OpenShift is configured.
Since Monday night, people began noticing images broken, complaints on
making thumbnails, and so forth. Jason and I worked on the problem on
Tuesday and figured out the solution. He moved all the content to the
persistent data store (also, got our quota doubled to 2 GB as we were
almost at 1 GB), and removed the data from the git repo.
In the end, Jason added a build hook that i) makes sure the
'../../data/images' directory is present, then ii) creates the necessary
symlink in the 'wiki/' structure after the code is checked in.
The same was done for 'compiled_templates', which was previously a
symlink for the MediaWiki Widgets extension. (That symlink was also
incorrectly written, so it may have been part of problems with the new
Widgets extensions?)
Thank you for your understanding, and I apologize for the inconvenience.
- Karsten
--
Karsten 'quaid' Wade, Sr. Analyst - Community Growth
http://TheOpenSourceWay.org .^\ http://community.redhat.com
@quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
[Users] Maintenance outage :: www.ovirt.org :: 2012-12-11 01:00 UTC
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 There is going to be a brief planned outage at 5 pm PST (0100 UTC) today. The outage window is ten minutes. No downtime is expected during the outage, but it may occur. == Details == I want to setup hot_deploy for our OpenShift instance running MediaWiki. When I did this the other day, it failed to work properly. Refer to: http://lists.ovirt.org/pipermail/infra/2012-December/001612.html This outage is another attempt to set the app to hot deploy. I intend to do exactly this: * 'touch wiki/.openshift/markers/hot_deploy' * 'git add wiki/.openshift/markers/hot_deploy' * 'git commit wiki/.openshift/markers/hot_deploy' * 'git push' If all works correctly, the app will be set to hot deploy without doing a restart. If it goes like last time, I may attempt to troubleshoot for a few minutes but no longer than 10 minutes. At that time I will force restart the app. == Affected services == www.ovirt.org (wiki) === Not-affected services == lists.ovirt.org resources.ovirt.org gerrit.ovirt.org jenkins.ovirt.org == Future plans == Hopefully this hot deploy trick will allow us to make minor configuration changes to the website without it having to restart the app. - -- Karsten 'quaid' Wade . http://iquaid.org . gpg key: AD0E0C41 http://Fairy-TaleFarm.com .. Urban homestead http://MicahForCouncil.org Your advocate on council http://SantaCruzPedicab.com .. Sensible local transportation -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iD8DBQFQxm/N2ZIOBq0ODEERAkF2AKCG4kT4U8IP82dzHmj3ZjIWszYBiwCfayJI BNq6othQiZCoy6Eg49lC/jU= =UmMF -END PGP SIGNATURE- ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] WARNING: Short outage of www.ovirt.org in 1h (15:30 UTC)
On 12/03/2012 07:42 AM, Dave Neary wrote: Hi all, To resolve a critical issue with the new website (http://ovirt.org/pagename links are not getting correctly redirected to http://www.ovirt.org/pagename) we will be committing a change in the server configuration, which will restart the service. This problem is now resolved. ovirt.org/$pagename links should now redirect to www.ovirt.org/$pagename. Let us know if you see anything still not working. This will result in a short (~30s) outage. To help avoid any loss of work, we will be announcing critical changes such as this with one hour's notice. We ended up resolving this without an outage, as the correction needed to be made where the site did not need to be restarted. As Dave pointed out, when we have a critical need to restart services that may affect any of you, we'll try to give an hour notice. Rarely there won't be time for a notice, for the most part we can give one hour notice for critical and 24-hour notice for non-critical service restarts. The Infra team will work on a policy for this and publish it for review. Thanks - Karsten -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Outage :: wiki and www :: 2100 to 0100 UTC Thu 29 Nov/Fri 30 Nov
This outage is now over. The wiki is writeable again, redirects of old content to new locations are working, and you should be able to go about reading, writing, creating, and collaborating. Enjoy the new lookfeel: http://www.ovirt.org There are a lot of small components making *.ovirt.org run, it's fairly certain we've missed something. If you see anything wrong, broken, or confusing a, you can drop by IRC, email in...@ovirt.org, or file a ticket here: http://fedorahosted.org/ovirt Thanks to everyone for making this migration happen. It's going to be nice not having to manage the platform for these non-core-technology components. The Infra team will be able to focus a bit more on the deeper parts of the oVirt infrastructure of participation. In particular, thanks to: * Garrett Lesage lead the effort to improve not only the lookfeel of the ovirt.org website, but to reconsider the user experience. He worked on making the top-level of the site show what the technology is, what the project is, and what you can download and do here. He took on the challenge of making MediaWiki look good for all the user classes - casual browser through to regular contributor. (And thanks especially for saving the front page this week when we realized we didn't have the desired video content and feed widgets available.) * Dave Neary focused on taking the framework of the new site and breathing life in to it with content pages, redirects, MediaWiki extensions, and so forth. He worked on troubleshooting our transfer details, tracking project details and progress, and generally beating on parts until he hammered them in to shape. Thanks also to Jason Brooks, who took advantage of the tradition of IRC to just get involved in what was happening. Jason helped troubleshoot and fix some of the transfer details, such as http rewrite rules. Overall, I've been pleased with the transparency and accessibility of the process. I hope others feel the same, but regardless of how you feel, your feedback is very appreciated. Thanks - Karsten On 11/29/2012 01:02 PM, Karsten 'quaid' Wade wrote: This outage is now starting. We'll be freezing the wiki at wiki.ovirt.org for the duration of the outage window. If you want to watch, participate, or have urgent questions, we're on #ovirt on irc.oftc.net. On 11/28/2012 10:48 PM, Karsten 'quaid' Wade wrote: We are having a four-hour freeze and short outage of the wiki and www portions of ovirt.org to migrate to OpenShift. 2100 UTC Thu 29 Nov to 0100 UTC Fri 30 Nov date -d '2100 UTC Thu Nov 29' We are targeting the DNS switch to happen at 2359 UTC Thu 29 Nov. This migration includes updating to a new theme for the site. We expect there to be no downtime for access to content on the website. During the four-hour window we'll be working to make sure URLs linking in to the old content forward directly to the new website. The freeze on the wiki is expected to complete on or before the four-hour mark. The new theme and information layout have been presented and discussed, this is just the final portion where we turn the lights on in a new location, and dim the lights in the old. Any questions, concerns, ideas, etc. just let us know. We'll do a final minute reminder via email and IRC when the outage window starts. Status updates will also by via email and IRC, as arisen. == Affected services == www.ovirt.org wiki.ovirt.org ovirt.org/releases ovirt.org/meetings ovirt.org/stats == Future considerations == If we had a strict infrastructure change management process, we would have coordinated better with the rest of the contributor teams to be sure this work didn't interrupt their workflow. ___ Arch mailing list a...@ovirt.org http://lists.ovirt.org/mailman/listinfo/arch ___ Arch mailing list a...@ovirt.org http://lists.ovirt.org/mailman/listinfo/arch -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Fwd: Outage :: wiki and www :: 2100 to 0100 UTC Thu 29 Nov/Fri 30 Nov
Hi: I should have sent these to the announce@ and users@ mailing lists, just in case there are wiki editors on those lists who are not reading arch@ (which is ideally the central all-contributors mailing list.) We'll adjust the Infra team methods in the future to send outage announcements to all these lists for maximum coverage. Below are details of this outage, which began 30 minutes ago. The wiki is in readonly mode until it is migrated fully to OpenShift. Original Message Subject: Outage :: wiki and www :: 2100 to 0100 UTC Thu 29 Nov/Fri 30 Nov Date: Wed, 28 Nov 2012 22:48:03 -0800 From: Karsten 'quaid' Wade kw...@redhat.com Organization: Red Hat To: a...@ovirt.org a...@ovirt.org CC: infra in...@ovirt.org We are having a four-hour freeze and short outage of the wiki and www portions of ovirt.org to migrate to OpenShift. 2100 UTC Thu 29 Nov to 0100 UTC Fri 30 Nov date -d '2100 UTC Thu Nov 29' We are targeting the DNS switch to happen at 2359 UTC Thu 29 Nov. This migration includes updating to a new theme for the site. We expect there to be no downtime for access to content on the website. During the four-hour window we'll be working to make sure URLs linking in to the old content forward directly to the new website. The freeze on the wiki is expected to complete on or before the four-hour mark. The new theme and information layout have been presented and discussed, this is just the final portion where we turn the lights on in a new location, and dim the lights in the old. Any questions, concerns, ideas, etc. just let us know. We'll do a final minute reminder via email and IRC when the outage window starts. Status updates will also by via email and IRC, as arisen. == Affected services == www.ovirt.org wiki.ovirt.org ovirt.org/releases ovirt.org/meetings ovirt.org/stats == Future considerations == If we had a strict infrastructure change management process, we would have coordinated better with the rest of the contributor teams to be sure this work didn't interrupt their workflow. -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 ___ Infra mailing list in...@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] What are you looking for from oVirt?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/13/2012 07:28 AM, Dave Neary wrote: My name is Dave Neary, and I recently started working for a group which is being formed inside Red Hat called OSAS (open source and standards). The role of my group is to help make any open source projects that Red Hat works in successful, and I would really like to help take oVirt to the next level in terms of adoption and contribution. Just some context for folks - OSAS is the team that I work on with other people you've seen around oVirt: Leslie Hawthorn, Carl Trieloff, Jim Jagielski, and Jason Brooks. (You'll likely see others of us, such as Garrett LeSage, as we all work on upstream projects together.) Helping with the open sourcing and community growth of oVirt is really the first project the OSAS team worked on together. To quote Dave: 07:43 dneary We're basically the A-Team for community stuff 07:44 dneary Except our theme music isn't as cool Cheers - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth Red Hat Open Source and Standards (OSAS) http://TheOpenSourceWay.org @quaid (identi.ca/twitter/IRC) | gpg: AD0E0C41 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFP2Kg92ZIOBq0ODEERAkqZAKDC6RwtSiboq14hb1T7KrHm7YOFTwCglzzf OvLipZfZGgfgNabGCW+zkEk= =pY6R -END PGP SIGNATURE- ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
