[Labs-l] Labs project migration *Important, deadlines approaching*
We've had a few hiccups, but project migration is going reasonably well. You can check on the current migration status of a project here: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress About 20 or so projects are still in the 'pending' section up top. If your project is among them, please finish up and file your project entry in the appropriate section (presumably, 'Finished migration'). I don't have any open migration bugs in bugzilla, so if you are waiting for me to help with something, I don't know about it. Please contact me immediately if you still need migration help from me. Friday Deadline On Friday I'll be shutting down all pmtpa instances. They won't be immediately deleted, but they will become inaccessible, and wikitech will no longer display an interface for pmtpa. DNS for the old bastions (e.g. bastion.wikimedia.org) will be changed to point to bastions in eqiad. Shared Storage copies from pmtpa I've copied the shared data from /home and /data/project in every project to eqiad. Copies of this data can be found filed away in each project's new shared directories. If you browse to /data/project or /home in your new eqiad instance you will see two directories, glustercopy and pmtpa_nfs_copy. These dirs contain backups of the shared gluster and nfs data from pmtpa. Note, however, that these copies were made at unpredictable times during the last couple of weeks. If you require a live, up-to-date copy of your data, please file a bug so that I know to re-sync the data after pmtpa instances are fully shut off. Next steps, coming weeks After pmtpa instances are shut down, I'll wait a few days for the dust to settle and to listen for surprised shouts of alarm from folks who haven't been following this list. I will also migrate wikitech from virt0 (the pmtpa web host) to virt1000 (the equivalent host in eqiad.) This will entail a small amount of downtime, to be announced in a future email. Sometime after the first of the month (that's next Tuesday) we will start shutting down pmtpa in earnest -- powering off machines, wiping drives, packing servers into crates, etc. Anything that's left in pmtpa by then will be truly lost for good. Next steps, coming months The mothballed projects are currently on long-term probation. In a few months I'll review the list and determine which projects remain entirely untouched. Projects that seem fully abandoned will be wiped from eqiad as well, and their resources returned to the pool for use by new projects. I will, of course, provide a further series of announcements and warnings before anything of this nature happens. Thank you all for your patience and assistance during this difficult process! -Andrew ___ Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
[Labs-l] Changes to cron on tools
Hey all, We've had difficulty in the past with cron jobs not running for a number of reasons (the primary being an overloaded bastion because of bots running on it), so I made a few changes to how cron is to be used in the future to vastly improve reliability: a) cron for tools will now be executed from a different, dedicated server that serves only that purpose. This will insulate the bastions from running cron jobs, and vice-versa. b) the 'crontab' executable will edit the crontabs /on that host/ directly (so there will be no need to change how you're doing thing); regardless of which bastion you run it from. (So, from tools-login or tools-dev, crontab will access the same schedule). c) crontabs will be automatically edited to add a prefix to send your jobs to the grid with reasonable default values unless the entry already invokes jsub, jstart or qsub. If the crontab program does that, it will tell you so that you can inspect and edit the entries and adapt them to your needs. Those three measures will result in considerably more reliable execution of scheduled jobs, as well as improved performance on the bastion hosts. FOR THE NEXT WEEK OR SO: both systems will be available at the same time. You can use the new system on the submit host by using 'xcrontab' rather than 'crontab' to set or edit your crontab. Be aware that /both/ schedules will be executed unless you also remove your bastion crontab. AT THE END OF THAT WEEK: any crontabs you may have had on tools-login or tools-bastion will be moved to the new server, and 'crontab' will be made to use the new system exclusively. You have no intervention to make; this will be done automatically. -- Marc ___ Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Re: [Labs-l] Changes to cron on tools
I assume that means it should make tools-login more stable as well, since the crontab will auto adapt crontab entries to use jsub and not login. On Mar 24, 2014, at 4:43 PM, Marc A. Pelletier m...@uberbox.org wrote: Hey all, We've had difficulty in the past with cron jobs not running for a number of reasons (the primary being an overloaded bastion because of bots running on it), so I made a few changes to how cron is to be used in the future to vastly improve reliability: a) cron for tools will now be executed from a different, dedicated server that serves only that purpose. This will insulate the bastions from running cron jobs, and vice-versa. b) the 'crontab' executable will edit the crontabs /on that host/ directly (so there will be no need to change how you're doing thing); regardless of which bastion you run it from. (So, from tools-login or tools-dev, crontab will access the same schedule). c) crontabs will be automatically edited to add a prefix to send your jobs to the grid with reasonable default values unless the entry already invokes jsub, jstart or qsub. If the crontab program does that, it will tell you so that you can inspect and edit the entries and adapt them to your needs. Those three measures will result in considerably more reliable execution of scheduled jobs, as well as improved performance on the bastion hosts. FOR THE NEXT WEEK OR SO: both systems will be available at the same time. You can use the new system on the submit host by using 'xcrontab' rather than 'crontab' to set or edit your crontab. Be aware that /both/ schedules will be executed unless you also remove your bastion crontab. AT THE END OF THAT WEEK: any crontabs you may have had on tools-login or tools-bastion will be moved to the new server, and 'crontab' will be made to use the new system exclusively. You have no intervention to make; this will be done automatically. -- Marc ___ Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l ___ Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Re: [Labs-l] Changes to cron on tools
On 03/24/2014 04:51 PM, Mr. Maximilian Doerr wrote: I assume that means it should make tools-login more stable as well It will, by offloading that work to tools-sumbit. -- Marc ___ Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
