Re: explanations of the recent download problems
Hello, On Jan 16, 2008 7:18 PM, Marius Gedminas [EMAIL PROTECTED] wrote: As a techie, I'd like to know more about how the content distribution network happened to cache HTML error pages for many of the packages on repository.maemo.org. Did the error pages come with long cache expiration headers by mistake, or does the CDN tend to assume URLs ending in .deb never change? No, there is no special expiration header set for the index page of tablets-dev.nokia.com. It seems that the CDN never realized that the file size has changed, although I have reverted the normal index page for a long time. I requested the CDN admins to fix this issue. About the debs: no, CDN should not assume that a deb never changes. The file names are always unique for newly uploaded packages. I can't tell you why new debs are not picked up by the CDN properly, if this was also part of your question. Thanks Marius for the questions. Please shoot more and I try to answer as much as I know. Marius Gedminas -- Colorless green ideas sleep furiously. Cheers, ferenc ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
RE: explanations of the recent download problems
Ferenc wrote: When apt-get update talks to our repository it actually fetches a package database [1] first. From that file apt-get knows that it supposed to receive 12380 bytes for the pidgin-extprefs_0.7-0nix2_armel.deb package. If you go to the pool's index [2] you can see that the file is actually 12382 bytes long. This could explain the size mismatch error. Wget will not match the file size with a package db, so it will just fetch it without a problem. Question: why the Packages have the wrong size. Well, frankly I don't know. It can only be because our repo manager tool had a hiccup, though it has been working quite well in the past years. Anyways, I try to rebuild that package database and see if that will help. Fwiw there are a number of mismatched files but most of them are pidgin related, I found them while trying to mirror and trouble shooting people's complaints. I ended up writing a script that compared my repo against my index. I think I decided part of the problem might relate to multiple uploads (of the same exact filename) or multiple arches, but that's pure speculation. I can run the script this weekend, but I'm sure ferenc can make an equivalent script... ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: explanations of the recent download problems
Hello Thomas, If you don't mind I cc: the dev list. You might not be the only one experiencing this problem. On Jan 16, 2008 11:25 PM, Thomas J. Baker [EMAIL PROTECTED] wrote: Hello Ferenc, I still regularly have problems with Maemo Extras. Here's an example: apt-get install pidgin-extprefs snip Get:1 http://repository.maemo.org chinook/free pidgin-extprefs 0.7-0nix2 [12.4kB] Fetched 12.4kB in 0s (25.4kB/s) Failed to fetch http://repository.maemo.org/extras/pool/chinook/free/p/pidgin-extprefs/pidgin-extprefs_0.7-0nix2_armel.deb Size mismatch snip The strange thing is that just using wget to get the package seems to work. Could be a metadata problem? When apt-get update talks to our repository it actually fetches a package database [1] first. From that file apt-get knows that it supposed to receive 12380 bytes for the pidgin-extprefs_0.7-0nix2_armel.deb package. If you go to the pool's index [2] you can see that the file is actually 12382 bytes long. This could explain the size mismatch error. Wget will not match the file size with a package db, so it will just fetch it without a problem. Question: why the Packages have the wrong size. Well, frankly I don't know. It can only be because our repo manager tool had a hiccup, though it has been working quite well in the past years. Anyways, I try to rebuild that package database and see if that will help. Thanks, tjb -- Thanks to you. -ferenc [1] http://repository.maemo.org/extras/dists/chinook/free/binary-armel/Packages [2] http://repository.maemo.org/extras/pool/chinook/free/p/pidgin-extprefs/ === | Thomas Baker email: [EMAIL PROTECTED]| | Systems Programmer | | Research Computing Center voice: (603) 862-4490 | | University of New Hampshire fax: (603) 862-1761 | | 332 Morse Hall | | Durham, NH 03824 USA http://wintermute.sr.unh.edu/~tjb | === ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: explanations of the recent download problems
Hello again, Question: why the Packages have the wrong size. Well, frankly I don't know. It can only be because our repo manager tool had a hiccup, though it has been working quite well in the past years. Anyways, I try to rebuild that package database and see if that will help. Rebuilding the Packages file helped. I suspect it is our tool which made the mistake. We will have to look into that. In the meantime please report all size-mismatch errors, so I can collect test cases. Cheers, ferenc [1] http://repository.maemo.org/extras/dists/chinook/free/binary-armel/Packages [2] http://repository.maemo.org/extras/pool/chinook/free/p/pidgin-extprefs/ ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
RE: explanations of the recent download problems
Hey, list! On Thu, 2008-01-17 at 14:08 +0200, ext [EMAIL PROTECTED] wrote: Ferenc wrote: [...] Fwiw there are a number of mismatched files but most of them are pidgin related, I found them while trying to mirror and trouble shooting people's complaints. I have tried to install all of Pidgin on Gregale, Bora, and Chinook, and it looks like only pidgin-extprefs on Bora and Gregale remains broken. All other packages work. This may well be the fault of the package. Thanks for your work, Gabriel ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
explanations of the recent download problems
Hello, I owe you an explanation about the problems we faced with the firmware downloads, repository.maemo.org and the windows update wizard server backend since Xmas till the beginning of Jan. The infrastructure is built so that the content from repository.maemo.org is served thru a huge caching network. We were supposed to use the same network for the firmware downloads as well, but due to a misconfiguration and _my negligence_ all the requests ended up at our origin server (stage.maemo.org). Soon after the N810 launch the traffic dramatically increased and the server could not handle it anymore. It was actually never meant to handle such load. We first realized the problem with tablets-dev.nokia.com where it took ages to get any firmware. This was the time when somebody started a torrent for the new OS 2008 image which helped a lot. We could talk about why Nokia did not offer this, but I am not the person who will tell you the reasons. So the firmware images from tablets-dev could have never been served from the caching network properly, because our backend scripts did not send the correct HTTP headers back to the caches and the caches were not configured either to fulfill our authentication requirements. I take all the blame for this, so this is why I spoke about negligence above. I must admit that fixing this was on my todo list for ages. Almost at the same when tablets-dev broke down we started to receive complaints about the update wizard on windows failing to fetch images from our server. Well, what a coincidence... Making this backend work properly with the caching network required a one liner change in a script. This was done pretty quickly by the developers and the patch was applied a week ago. Right now I am confident that the requests coming form windows users end up at the caching network. Beside that I am pretty sure we also got some DOS attacks at the same time, which we clearly see at garage.maemo.org. I never went and checked logs at stage.maemo.org, but I suspect that we were hit on all fronts. We have taken the necessary measures to avoid these things in the future. I must thank my fellow Nokia colleagues (the file delivery platform and our team here), our ISP crew and our cache providers to help and find solutions for the problems. I believe if it was not a holiday season we could have solved these much faster. Never the less the problems should be over by now. If you still experience long delays, or weird HTTP responses then please mail me with the exact URL you're trying to access. Thanks for the patience, and apologies for the inconvenience we caused. Best regards, ferenc ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: explanations of the recent download problems
On Wed, Jan 16, 2008 at 02:29:01PM +0200, Ferenc Szekely wrote: I owe you an explanation about the problems we faced with the firmware downloads, repository.maemo.org and the windows update wizard server backend since Xmas till the beginning of Jan. Thank you for the open and informative reply. As a techie, I'd like to know more about how the content distribution network happened to cache HTML error pages for many of the packages on repository.maemo.org. Did the error pages come with long cache expiration headers by mistake, or does the CDN tend to assume URLs ending in .deb never change? Marius Gedminas -- Colorless green ideas sleep furiously. signature.asc Description: Digital signature ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: explanations of the recent download problems
Ferenc Szekely wrote: Hello, I owe you an explanation about the problems we faced with the firmware downloads, repository.maemo.org and the windows update wizard server backend since Xmas till the beginning of Jan. Hi Ferenc, don't beat yourself up as mistakes happen and nobody is looking for a scapegoat. Your honest appraisal of the problems is very much appreciated. The biggest concern however is that this was the second failure within a few weeks - a torrent had to be setup for the N800/OS 2008 Beta and original N810 OS 2008 releases as well... alarm bells should have been ringing following this Beta release and not a few weeks later on 15 December when everyone had gone on their Christmas vacation. If there is one lesson to be learned from the most recent release it surely has to be that maemo.org should never again release firmware immediately before a holiday season. maemo.org must always have developers on site for at least one week following release to handle the immediate problems that arise, be they serious firmware issues or, more likely, infrastructure/download problems. If necessary, delay the firmware release until after the holiday season rather than rush it out and have the community and new users endure the ghastly experience that befell us all in December. Nobody should be blamed for the past problems if the right lessons are learned for the future. Problems will undoubtedly happen again, but maemo.org must ensure that people are around to fix the problems sooner rather than later. Another 3+ weeks of broken servers and maemo.org in complete radio silence - this is the first time this problem has been publicly acknowledged by anyone from maemo - cannot be permitted to happen again. Thanks again for the post mortem. Neil ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
