Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
Now, should the technician not be able to resurrect ries, our backup plan extends to have the disks shipped over and replace the ones currently in rietz. I'm wondering if Debian has the resources (DSA, local admins and hardware) to have a hot-swappable backup machine for ftpmaster, since it does go down occasionally and when it does the downtime is fairly disruptive to Debian. Well, this would mean: a) double ftpmaster. Just as a rough number, the new machine that is currently in progress has an estimated cost of 2 Dollar (if you take prices from HP Website. This is not what it will cost in the end, but it shows what category of stuff we have to get) b) Have the double space at our hoster. c) Run it with heartbeat, drbd and all that. Point c) is actually easy enough, even though im not DSA and can't decide for them to do it. But technically it would be a working setup, provided b) works out, as you really want a *FAST* connection between the two. Which means local. The only trouble this setup has is that you have a pretty huge expensive machine always on and running, but not actually doing stuff for 99.% of the time. And usually the support pack we ordered provides a service that means getting machine back faster than this, so the question in the end is: Do we want the added complexity (and costs), or can we just stand a day or three of downtime? Its annoying, but world doesnt really die. (And note, the support has done great jobs replacing harddisks multiple times in the past already, for example) -- bye, Joerg I read the DUMP and agree to it. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87pr2l6izk@gkar.ganneff.de
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
On Wed, Mar 31, 2010 at 3:35 PM, Joerg Jaspert jo...@debian.org wrote: The only trouble this setup has is that you have a pretty huge expensive machine always on and running, but not actually doing stuff for 99.% of the time. And usually the support pack we ordered provides a service that means getting machine back faster than this, so the question in the end is: Do we want the added complexity (and costs), or can we just stand a day or three of downtime? Its annoying, but world doesnt really die. Hmmm, OK. There don't seem to be any other machines hosted in the same place so having another machine that is used for other stuff most of the time and also as a backup ftp-master during outages wouldn't work, for now at least. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/u2ne13a36b31003310113q11c0b00dw34e2a55971b2b...@mail.gmail.com
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
Joerg Jaspert jo...@debian.org writes: Now, should the technician not be able to resurrect ries, our backup plan extends to have the disks shipped over and replace the ones currently in rietz. I'm wondering if Debian has the resources (DSA, local admins and hardware) to have a hot-swappable backup machine for ftpmaster, since it does go down occasionally and when it does the downtime is fairly disruptive to Debian. Well, this would mean: a) double ftpmaster. Just as a rough number, the new machine that is currently in progress has an estimated cost of 2 Dollar (if you take prices from HP Website. This is not what it will cost in the end, but it shows what category of stuff we have to get) b) Have the double space at our hoster. c) Run it with heartbeat, drbd and all that. Point c) is actually easy enough, even though im not DSA and can't decide for them to do it. But technically it would be a working setup, provided b) works out, as you really want a *FAST* connection between the two. Which means local. The only trouble this setup has is that you have a pretty huge expensive machine always on and running, but not actually doing stuff for 99.% of the time. And usually the support pack we ordered provides a service that means getting machine back faster than this, so the question in the end is: Do we want the added complexity (and costs), or can we just stand a day or three of downtime? Its annoying, but world doesnt really die. Is there any way to build a distributed service instead of relying on one central machine (or two machines sitting next to each other)? I don't know exactly what services are involved, but typically and generally, when I deploy a server infrastructure, I try to setup (at least) two machines at different geographical places that each can provide the entire service. The complexity in getting a heartbeat, drbd, etc solution to work can easily eat up any downtime savings you plan to get. /Simon -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fx3gg3dn@mocca.josefsson.org
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
On Wed, Mar 31, 2010 at 09:35:59AM +0200, Joerg Jaspert wrote: I'm wondering if Debian has the resources (DSA, local admins and hardware) to have a hot-swappable backup machine for ftpmaster, since it does go down occasionally and when it does the downtime is fairly disruptive to Debian. Well, this would mean: [...] c) Run it with heartbeat, drbd and all that. Point c) is actually easy enough, even though im not DSA and can't decide for them to do it. But technically it would be a working setup, provided b) works out, as you really want a *FAST* connection between the two. Which means local. What is it that ftp-master does that can only run on one computer at a time? If most of the services it provides could be distributed, you could spread the load to multiple machines, and get redundancy at the same time. -- Met vriendelijke groet / with kind regards, Guus Sliepen g...@debian.org signature.asc Description: Digital signature
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
Replying to three in one. Well, this would mean: a) double ftpmaster. Just as a rough number, the new machine that is currently in progress has an estimated cost of 2 Dollar (if you take prices from HP Website. This is not what it will cost in the end, but it shows what category of stuff we have to get) b) Have the double space at our hoster. c) Run it with heartbeat, drbd and all that. Point c) is actually easy enough, even though im not DSA and can't decide for them to do it. But technically it would be a working setup, provided b) works out, as you really want a *FAST* connection between the two. Which means local. Is there any way to build a distributed service instead of relying on one central machine (or two machines sitting next to each other)? Not currently, no. The actions in the archive can not split in a way this would make sense to place on various machines. That is, not those that are the important ones here (upload processing, mirrortree updating, that parts all around it). I don't know exactly what services are involved, but typically and generally, when I deploy a server infrastructure, I try to setup (at least) two machines at different geographical places that each can provide the entire service. ... The complexity in getting a heartbeat, drbd, etc solution to work can easily eat up any downtime savings you plan to get. It is actually a very easy setup and I am running multiple of them. The complexity is mainly in the double rackspace, double internet connection, *FAST* inter-machine connection, not in the bit drbd stuff. On 12071 March 1977, Guus Sliepen wrote: What is it that ftp-master does that can only run on one computer at a time? If most of the services it provides could be distributed, you could spread the load to multiple machines, and get redundancy at the same time. Thank you, we never thought of that... On 12071 March 1977, Thomas Koch wrote: Joerg Jaspert: SNIP The only trouble this setup has is that you have a pretty huge expensive machine always on and running, but not actually doing stuff for 99.% of the time. /SNIP Hadoop is now in Debian: http://packages.qa.debian.org/h/hadoop.htmlHadoop is an Open Source implementation of Google's File System, MapReduce and BigTable (HBase, not yet packaged). The idea behind Google's infrastructure and therefor Hadoop is: Have many cheap comodity servers that together form a powerful cluster. Each node of the cluster is redundant and can be replaced without downtime. I believe, but can't know for sure, that everything what FTP-Master does, could be implemented on top of hadoop. However it means for sure a lot of work and many hardcore sysadmins will feel very uncomfortable to use Java, the language Hadoop is written in. I'm planning to give a presentation of hadoop at the DebConf in Bosnia and maybe then we may discuss, if hadoop should have a place in Debian's infrastructure. - For now I'm happy, if somebody became curious. :-) The idea behind hadoop and stuff is certainly a nice one. Yet, I do believe it has an entirely different target and can not easily be adopted to the tasks ftp-master does. There simply are things you can not, in a sensible way, make distributed. On 12071 March 1977, Obey Arthur Liu wrote: Isn't there /some/ python/jython support ? Would you co-mentor such a project as part of a Summer of Code project ? Do you know someone who would ? It need not be ftpmaster. There are probably other critical debian infrastructure which could use this. I do not think changing dak and all it does to hadoop and distributed is possible within gsoc. Nor within ten of it. I wont block any who tries though, so have fun. :) -- bye, Joerg andreasj Also diese neuen Spam-Mails muten an wie Blog-Posts von Clint Adams andreasj irgendwie ist es eine Geschichte, aber ich versteh sie nicht -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87bpe471t0@gkar.ganneff.de
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
On 12065 March 1977, Joerg Jaspert wrote: ries.debian.org, the host behind ftp-master.debian.org, has hardware trouble, a failed memory module keeps resetting the machine at random intervals. Will send a notice when service is back to normal. And another update for you all out there, waiting: After several ping-pongs the support finally understood that no round of firmware updates, changing slots of the DIMMs and trying whatever does not help, so we are now waiting for a new mainboard to arrive, as well as a technician. The current timeline lets us assume this is done, latest, day after tomorrow, please be patient. Will keep you updated when status changes. The backup plan still is moving this service to another machine. This hasn't been done yet for various reasons. One of them is simply that it took a little longer to move the remaining services away from it, but also the amount of work included. Now, should the technician not be able to resurrect ries, our backup plan extends to have the disks shipped over and replace the ones currently in rietz. -- bye, Joerg I'm in no condition to drive...wait! I shouldn't listen to myself, I'm drunk! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fx3hdbus@gkar.ganneff.de
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
On Wed, Mar 31, 2010 at 12:15 AM, Joerg Jaspert jo...@debian.org wrote: On 12065 March 1977, Joerg Jaspert wrote: ries.debian.org, the host behind ftp-master.debian.org, has hardware trouble, a failed memory module keeps resetting the machine at random intervals. Will send a notice when service is back to normal. And another update for you all out there, waiting: After several ping-pongs the support finally understood that no round of firmware updates, changing slots of the DIMMs and trying whatever does not help, so we are now waiting for a new mainboard to arrive, as well as a technician. The current timeline lets us assume this is done, latest, day after tomorrow, please be patient. Will keep you updated when status changes. The backup plan still is moving this service to another machine. This hasn't been done yet for various reasons. One of them is simply that it took a little longer to move the remaining services away from it, but also the amount of work included. Now, should the technician not be able to resurrect ries, our backup plan extends to have the disks shipped over and replace the ones currently in rietz. I'm wondering if Debian has the resources (DSA, local admins and hardware) to have a hot-swappable backup machine for ftpmaster, since it does go down occasionally and when it does the downtime is fairly disruptive to Debian. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/i2oe13a36b31003302254h2ef20b6ey61f4922cee6ae...@mail.gmail.com
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
ries.debian.org, the host behind ftp-master.debian.org, has hardware trouble, a failed memory module keeps resetting the machine at random intervals. We seem to have more than one broken module, as we already asked local admins to take DIMMs out, and then got another random reboot. The support tools also reporting more broken RAM. We didn't find that very helpful, nor do we think running the services on it with that problem below is a good idea. There are currently two options for us to go on: - move ftp-master and release services elsewhere. We already think about it, and a machine for it that has about similar spec to current ftp-master is about ready (pending the final move of another service to yet another box, which is in progress). - deal more with HP support and let them change more hardware (and maybe even the mainboard). While the latter will definitely take a little longer (beginning of next week we should know more on that front), the first option creates a huge amount of work, which we would like to avoid, if possible. So, please bear with us, people are working on the issue, and we will keep you informed about it. -- bye, Joerg Oh, Marge, cartoons don't have any deep meaning. They're just stupid drawings that give you a cheap laugh. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87d3ypl4xa@gkar.ganneff.de
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
On Sun, Mar 28, 2010 at 12:29:21AM +0100, Joerg Jaspert wrote:
So, please bear with us, people are working on the issue, and we will
keep you informed about it.
Thanks a lot for the steady info flow and for this emergency work.
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
signature.asc
Description: Digital signature
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
Hi! Joerg Jaspert schrieb: just a short notice for everyone out there who wants to upload or waits for a package migration to testing: ries.debian.org, the host behind ftp-master.debian.org, has hardware trouble, a failed memory module keeps resetting the machine at random intervals. A small update: Thanks to our DSAs ries seems to be working again, but we noticed some other broken files in the archive. We are therefore comparing checksums over the entire archive, which might take same time (as it is ~500GB). Best regards, Alexander -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4bacde46.5030...@debian.org
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
Hi, On Fri Mar 26, 2010 at 17:18:14 +0100, Alexander Reichle-Schmehl wrote: Hi! Joerg Jaspert schrieb: just a short notice for everyone out there who wants to upload or waits for a package migration to testing: ries.debian.org, the host behind ftp-master.debian.org, has hardware trouble, a failed memory module keeps resetting the machine at random intervals. A small update: Thanks to our DSAs ries seems to be working again, but *cough* well, 2 DIMMs are now taken out of the machine, which we hope have caused that problem, but we will see There will be a further downtime when the replacement DIMMs arrive. Greetings Martin -- Martin Zobel-Helas zo...@debian.org | Debian System Administrator Debian GNU/Linux Developer | Debian Listmaster Public key http://zobel.ftbfs.de/5d64f870.asc - KeyID: 5D64 F870 GPG Fingerprint: 5DB3 1301 375A A50F 07E7 302F 493E FB8E 5D64 F870 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100326162721.ga18...@ftbfs.de
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
On 12065 March 1977, Joerg Jaspert wrote: ries.debian.org, the host behind ftp-master.debian.org, has hardware trouble, a failed memory module keeps resetting the machine at random intervals. No, its not fixed, just thought of giving a little update: I did an initial check of stuff that recently happened in the ftpmaster part of ries. Right now we have a set of packages that are broken beyond repair (on ries and also pushed to mirrors): gparted 0.5.2-2 alpha python-oss 0.0.0.20010624-6.1+b1 armel python-pam 0.4.2-12.1+b1 armel fluxbox 1.1.1-7 hppa fqterm 0.9.6.8-1 hppa gparted 0.5.2-2 i386 gss 0.1.5-1 mipsel Those are fixable by a binnmu for the broken architecture. The binnmus are already scheduled (thanks aba), NO need for any maintainer action! Additionally we have one package in the archive that we can not help with a binnmu, a full source upload is required. The maintainer got a seperate mail asking for the upload, but for reference, its fatsort, the latest version. And then we had 3 armel binary-only uploads getting trashed before they could be accepted. Those just need a reupload: deets 0.0.1-1 armel dpkg 1.15.6.1 armel clutter-1.0 1.2.4-1 armel And there was one source-ful upload which ries also managed to destroy, libmodule-signature-perl 0.62-1, and I also informed the maintainer here. Besides that we have some other files suddenly containing crap, but none that affect the archive or need any work from elsewhere. Note that I did not yet run a full check over all files. This will follow, but not right now. The above is merely from looking at recent activity and checking the usual places in our archive. -- bye, Joerg (23:02) liw I should take a photograph of my stapler, the maker of which is RAPESCO -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87ocickp9c@gkar.ganneff.de
Re: Hardware trouble ries.debian.org - ftpmaster.debian.org / release.d.o services disabled
Am Donnerstag, den 25.03.2010, 23:30 +0100 schrieb Joerg Jaspert: Additionally we have one package in the archive that we can not help with a binnmu, a full source upload is required. The maintainer got a seperate mail asking for the upload, but for reference, its fatsort, the latest version. That explains why fatsort is gone. Thanks for the info. -- Benjamin Drung Ubuntu Developer (www.ubuntu.com) | Debian Maintainer (www.debian.org) signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
