Bug#636292: Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
Le 2011-09-17 11:34, Henrique de Moraes Holschuh a écrit : retitle 636292 dak/apt: deficiencies at handling out-of-sync metadata summary 636292 87 thanks If anyone disagrees with the above triage, please change the summary and/or title. Thank you. On Sat, 17 Sep 2011, Filipus Klutiero wrote: #636292 is labelled as being about round-robin mirrors and problems APT has with those, even though that is probably not the actual cause of the reporter's problem. You're expected to read the entire thing when refered to a bug report in a thread you're replying to. I was unaware of that. [...] But indeed, there would be room for a general bug unreliable package indices fetching protocol :-S That misses the point, IMO. To me, it looks like what's broken is that the repository format _and_ the front-ends have deficiencies at handling metadata which is unsyncronized either in-mirror or across mirrors. And these deficiencies are a lot more important nowadays than they once were, as we have now many dinstall runs per day, lots of users tracking testing and unstable, a larger set of metadata files, a larger and more diverse set of mirrors... I.e: a lot more chances to hit unsyncronized metadata windows. I don't think increasing dinstall frequency worsens these issues significantly if dinstalls get shorter (unless previous dinstalls ran during the night). I also think archive size growth should have been compensated by performance increases. I think the time spent synchronizing a mirror must not have increased a lot. What did change here (dramatically) is the proportion of that time where APT indices updates fail. Round-robin mirrors might also have worsened. Anyway, the repository format is not a problem per se, it's the combination of what's on a mirror and how APT fetches it that's a problem. If you assume the communication protocol is HTTP-like, then indeed there should be mechanisms to cope with race conditions - i.e. file versioning and/or having APT retry or report desynchronizations. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
Le 2011-09-17 05:18, David Kalnischkies a écrit :
On Fri, Sep 16, 2011 at 19:44, Filipus Klutierochea...@gmail.com wrote:
If you decrease the time between the dinstall (or equivalent) runs to lets
say
an hour as e.g. ubuntu does you get far more likely in situations in which
Release and Release.gpg doesn't belong together. Thats specifically one of
the reasons ftpmasters implemented it in 2009.
So, you are currently experience mirror-problems because we want to
protect you from mirror problems in the future. Crazy, heh? ;)
Could you clarify what incoherence scenario InRelease avoids? It is true
that Release and Release.gpg need to be either versioned or updated
simultaneously, but this problem applies to all files used to fetch indices,
which are all updated during stage 1 (except for InRelease). Therefore the
duration of incoherence should be insignificant in comparison to the problem
with InRelease.
Be careful with your guesses. If it would be that insignificant, why was it
implemented by ftpmasters?
I did specify in comparison to the problem with InRelease, and as
previously discussed, eliminating an HTTP request improves performance.
This might also have more long-term advantages.
Also you are just looking at mirror sync,
the world consists of more than just mirror syncs! We have the creation
of these files for example which is far from being atomic at the moment.
So, could you show a scenario where non-atomatic creation of these files
constitutes a problem?
Also, if you want to follow this route, from a security point is an InRelease
file better as a man-in-the-middle could drop the request for Release.gpg -
the archive has no signature then and APT will cry for an extra-yes, but
many users are pretty fast in pressing yes -- especially as they are
used to having these questions as not every thirdparty archive provides
a good signature… But for a real attack he would properly send modified
Release and co and doesn't respond to Release.gpg. Try that with
InRelease - sure, as we need to support the old way for a while these
kind of attacks can still be executed and dropping all requests for files
related to Release is a possibility, too, but one step at the time.
The problem of users getting used to this kind of prompts can be tackled
by either {encouraging,making it easy for} third-party repositories to
provide signatures, or by preventing users from needing third-party
repositories.
If you still think administrators are too dumb for these prompts to be
worth the security risk they pose, you could either request APT to
simply fail when no signature is provided (at least for official
sources), or to have prompts which get more attention (red text, Yes,
/do as I say/!, etc.).
Anyway, this issue is hopefully a short-term one.
So with disabling InRelease we would just hide the underlying problem,
this can't be a solution, especially not that early in the releasecycle.
After all, we are talking about unstable-only here.
I'm not sure what problem we would be hiding here. Mirrors using ftpsync
versions that don't handle InRelease correctly is only a problem insofar as
APT uses InRelease with problematic ftpsync versions...
That's a pretty short view. Forget InRelease for a moment: Are you really
willing to use a mirror there the admins didn't care? I am not.
If they are that sloppy to use old update-scripts, can i be sure that they
are not going to silently stop running their old scripts all together?
I am not using any specific mirror. Before I had to debug this issue, I
was using my country's official mirrors round-robin. It may be a sad
state of affairs that official mirrors do not have better
administration, but that is the way it currently is.
I am not sure that my mirror won't stop updating, but that is the case
for [almost] all Debian systems. APT is still in development and does
not yet behave appropriately with mirrors.
As I said, there is not even a definition of what is an old ftpsync
version. To a mirror administrator, using ftpsync 80286 may not appear
worst than using apt 0.8.10.
You are later talking about versioning and co, how should it be possible
to use that if we are unable to use a single file after three years - or if
we want to be a bit more fair after eight months?
Not every damn feature should need six years before it can be used,
at least the last time i checked the file isn't called MultiRelease…
6 months is nothing in Debian development. Even if InRelease usage is
reintroduced in this release cycle, it won't be generally used before at
least another year.
I am not arguing that any specific time should be waited. I'm hoping
that mirrors are updated quickly and that InRelease can be used quickly,
but this needs works. I don't know how long it can take.
It is easy to add versioned indices even with non-cooperative mirrors.
If several versions are kept, APT can just use the latest index version
which has a complete set of index files.
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
On Fri, Sep 16, 2011 at 19:44, Filipus Klutiero chea...@gmail.com wrote: If you decrease the time between the dinstall (or equivalent) runs to lets say an hour as e.g. ubuntu does you get far more likely in situations in which Release and Release.gpg doesn't belong together. Thats specifically one of the reasons ftpmasters implemented it in 2009. So, you are currently experience mirror-problems because we want to protect you from mirror problems in the future. Crazy, heh? ;) Could you clarify what incoherence scenario InRelease avoids? It is true that Release and Release.gpg need to be either versioned or updated simultaneously, but this problem applies to all files used to fetch indices, which are all updated during stage 1 (except for InRelease). Therefore the duration of incoherence should be insignificant in comparison to the problem with InRelease. Be careful with your guesses. If it would be that insignificant, why was it implemented by ftpmasters? Also you are just looking at mirror sync, the world consists of more than just mirror syncs! We have the creation of these files for example which is far from being atomic at the moment. Also, if you want to follow this route, from a security point is an InRelease file better as a man-in-the-middle could drop the request for Release.gpg - the archive has no signature then and APT will cry for an extra-yes, but many users are pretty fast in pressing yes -- especially as they are used to having these questions as not every thirdparty archive provides a good signature… But for a real attack he would properly send modified Release and co and doesn't respond to Release.gpg. Try that with InRelease - sure, as we need to support the old way for a while these kind of attacks can still be executed and dropping all requests for files related to Release is a possibility, too, but one step at the time. So with disabling InRelease we would just hide the underlying problem, this can't be a solution, especially not that early in the releasecycle. After all, we are talking about unstable-only here. I'm not sure what problem we would be hiding here. Mirrors using ftpsync versions that don't handle InRelease correctly is only a problem insofar as APT uses InRelease with problematic ftpsync versions... That's a pretty short view. Forget InRelease for a moment: Are you really willing to use a mirror there the admins didn't care? I am not. If they are that sloppy to use old update-scripts, can i be sure that they are not going to silently stop running their old scripts all together? You are later talking about versioning and co, how should it be possible to use that if we are unable to use a single file after three years - or if we want to be a bit more fair after eight months? Not every damn feature should need six years before it can be used, at least the last time i checked the file isn't called MultiRelease… As I said, this problem does not just affect unstable, it already affects testing and would probably affect stable, and eventually all releases if nothing is done. stable doesn't have an InRelease file and apt/stable doesn't support it, so there is no possibility for this to effect stable. Given that InRelease is used by others already without any problems it would be a shame to revert them just because a single archive and (parts of) its mirror network has problems with them - even if this single archive is debian itself. Uh, Debian is an actual GNU/Linux distribution, not just a draft. If Uh, InRelease is an actual file, not just an inode waster. I don't understand what you want to tell me with that line as i am not a draft as well as various archives and complete debian-derivatives are not a draft. And you seem to completely ignore that i said that e.g. for me and the mirrors I use InRelease was never a problem, and i am reasonable sure that i was the first users of it back in January after implementing it, so this problem effects only parts of the mirror network - and as i have no knowledge about mirrors i leave it up to the mirror team to decide how big this part is and what should be done about that - and as long as this team doesn't say David, you bloody idiot, revert it or we will tell mirror-admins where you live to take care of you. i assume i can be of no help. (but i am sure the mirror-team would have worded it differently ;) ) If offering a patch is not enough, feel free to leave code to ease having InRelease support, perhaps a compile-time option. Or even to make the APT source support InRelease by default and to only disable it in Debian. All I'm saying is that something needs to be done for Debian proper. Wait, where is this mysterious patch you are talking about? I already said that i would try my best to merge a patch, but my time is limited and i prefer to do in my free-time stuff i want to do and not what others think i have to, so i am not going to write that patch myself or for that matter any
Bug#636292: Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
retitle 636292 dak/apt: deficiencies at handling out-of-sync metadata summary 636292 87 thanks If anyone disagrees with the above triage, please change the summary and/or title. Thank you. On Sat, 17 Sep 2011, Filipus Klutiero wrote: #636292 is labelled as being about round-robin mirrors and problems APT has with those, even though that is probably not the actual cause of the reporter's problem. You're expected to read the entire thing when refered to a bug report in a thread you're replying to. Anyway, triaged. I didn't want to do it because it is not my bug, it is not a package I work on, and I have no idea wether the apt developers agree with my anaylsis of #636292 or not. Please feel free to improve the title or chose a new summary. But indeed, there would be room for a general bug unreliable package indices fetching protocol :-S That misses the point, IMO. To me, it looks like what's broken is that the repository format _and_ the front-ends have deficiencies at handling metadata which is unsyncronized either in-mirror or across mirrors. And these deficiencies are a lot more important nowadays than they once were, as we have now many dinstall runs per day, lots of users tracking testing and unstable, a larger set of metadata files, a larger and more diverse set of mirrors... I.e: a lot more chances to hit unsyncronized metadata windows. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
As I said, this problem does not just affect unstable, it already affects testing and would probably affect stable, and eventually all releases if nothing is done. stable doesn't have an InRelease file and apt/stable doesn't support it, so there is no possibility for this to effect stable. And wont gain one. what should be done about that - and as long as this team doesn't say David, you bloody idiot, revert it or we will tell mirror-admins where you live to take care of you. i assume i can be of no help. (but i am sure the mirror-team would have worded it differently ;) ) Dont think that happens. apt is the wrong place. -- bye, Joerg [2005-10-09] I want you to be a maintainer before christmas. (AM finished report August 2006, DAM approval was next christmas then) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
Hi Filipus, Thanks for your detailed mail. On Fri, Sep 16, 2011 at 01:21, Filipus Klutiero chea...@gmail.com wrote: Since 0.8.11, APT fetches InRelease in preference to the classic Release file. InRelease is an Inline signed Release file announced in http://lists.debian.org/debian-devel-announce/2009/11/msg1.html If I understand correctly, the only real noteworthy advantage of InRelease at the moment is that it avoids one HTTP request, improving performance. It avoids more than just a http request - in fact it doesn't save a request at all as multiple requests are combined by APT (http request pipelining). If you decrease the time between the dinstall (or equivalent) runs to lets say an hour as e.g. ubuntu does you get far more likely in situations in which Release and Release.gpg doesn't belong together. Thats specifically one of the reasons ftpmasters implemented it in 2009. So, you are currently experience mirror-problems because we want to protect you from mirror problems in the future. Crazy, heh? ;) retry indices updates about 1 day on 2. If everyone was using my mirror, that would have affected quite a lot of people. […] The problem with ftpsync is that the mirrors team doesn't seem to have much infrastructure to ensure it stays up-to-date. There is no package for […] be considered outdated. Overall, coordination between the mirrors team and mirror administrators appears to have little structure. I have never seen this in real life. Maybe i am just incredible lucky, but more like is that others have hit this problem and contacted mirror-admins before i could. Sure, we could disable InRelease now or add an option or whatever, but this means that again nobody will notice that mirror-admins are using outdated scripts for updating. It's not unlikely that in future we will have other files we need to take care of in one or the other way, so we are better getting all this in order now before we have to later. So with disabling InRelease we would just hide the underlying problem, this can't be a solution, especially not that early in the releasecycle. After all, we are talking about unstable-only here. The other solution, which I came to the conclusion should be implemented after polling #debian-mirrors, is to make APT avoid InRelease. Josh Triplett filed an RFE asking APT to provide an option to disable use of InRelease files: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626026 Feel free to implement it and provide a patch. I personally don't believe in hiding problems, so i will not invest time into that (even if the patch shouldn't be damn hard), but commiting a patch with good documentation would be in order, i guess. The question is then when should InRelease use be re-enabled [by default]? I suggest to file an issue report against the mirrors pseudo-package to track this issue, and to request the mirrors team to close it when they'll consider that APT can safely use InRelease. Given that InRelease is used by others already without any problems it would be a shame to revert them just because a single archive and (parts of) its mirror network has problems with them - even if this single archive is debian itself. If the mirror-team really recommends to revert it - and i haven't heard that so far - it might be a better idea to remove the file from the debian archive instead for the foreseeable future as it is not really a bug in APT… But either way this is something the mirror-team has to decide and ask for as they have the overview over the status. I can only speak for the very limited world of de and de2 and these seem to behave fine. I don't know what happens on the rest of the world (mirror-wise) and i assume every other normal user has a similar limited view. Best regards David Kalnischkies -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
[Adding the mirrors team in the loop] Hi David, Le 2011-09-16 05:22, David Kalnischkies a écrit : Hi Filipus, Thanks for your detailed mail. On Fri, Sep 16, 2011 at 01:21, Filipus Klutierochea...@gmail.com wrote: Since 0.8.11, APT fetches InRelease in preference to the classic Release file. InRelease is an Inline signed Release file announced in http://lists.debian.org/debian-devel-announce/2009/11/msg1.html If I understand correctly, the only real noteworthy advantage of InRelease at the moment is that it avoids one HTTP request, improving performance. It avoids more than just a http request - in fact it doesn't save a request at all as multiple requests are combined by APT (http request pipelining). FWIW, HTTP pipelining does not reduce the number of requests, it just changes the way they're sent. What it could save is TCP segments. If you decrease the time between the dinstall (or equivalent) runs to lets say an hour as e.g. ubuntu does you get far more likely in situations in which Release and Release.gpg doesn't belong together. Thats specifically one of the reasons ftpmasters implemented it in 2009. So, you are currently experience mirror-problems because we want to protect you from mirror problems in the future. Crazy, heh? ;) Could you clarify what incoherence scenario InRelease avoids? It is true that Release and Release.gpg need to be either versioned or updated simultaneously, but this problem applies to all files used to fetch indices, which are all updated during stage 1 (except for InRelease). Therefore the duration of incoherence should be insignificant in comparison to the problem with InRelease. retry indices updates about 1 day on 2. If everyone was using my mirror, that would have affected quite a lot of people. […] The problem with ftpsync is that the mirrors team doesn't seem to have much infrastructure to ensure it stays up-to-date. There is no package for […] be considered outdated. Overall, coordination between the mirrors team and mirror administrators appears to have little structure. I have never seen this in real life. Maybe i am just incredible lucky, but more like is that others have hit this problem and contacted mirror-admins before i could. Sure, we could disable InRelease now or add an option or whatever, but this means that again nobody will notice that mirror-admins are using outdated scripts for updating. It's not unlikely that in future we will have other files we need to take care of in one or the other way, so we are better getting all this in order now before we have to later. I absolutely agree with that, and would encourage anyone to get involved in the mirrors team and get mirrors up-to-date. However, until someone finds the time to do that, APT needs to take the situation into account. So with disabling InRelease we would just hide the underlying problem, this can't be a solution, especially not that early in the releasecycle. After all, we are talking about unstable-only here. I'm not sure what problem we would be hiding here. Mirrors using ftpsync versions that don't handle InRelease correctly is only a problem insofar as APT uses InRelease with problematic ftpsync versions... As I said, this problem does not just affect unstable, it already affects testing and would probably affect stable, and eventually all releases if nothing is done. The other solution, which I came to the conclusion should be implemented after polling #debian-mirrors, is to make APT avoid InRelease. Josh Triplett filed an RFE asking APT to provide an option to disable use of InRelease files: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626026 Feel free to implement it and provide a patch. I personally don't believe in hiding problems, so i will not invest time into that (even if the patch shouldn't be damn hard), but commiting a patch with good documentation would be in order, i guess. As I said after, I'm not convinced that #626026 is worth the effort - if I had the time, I would work on the general issue with the mirrors team, I certainly won't be sending a patch for #626026. Just implementing that enhancement wouldn't really hide problems - users would still hit the bug, until they would configure APT to workaround it. The question is then when should InRelease use be re-enabled [by default]? I suggest to file an issue report against the mirrors pseudo-package to track this issue, and to request the mirrors team to close it when they'll consider that APT can safely use InRelease. Given that InRelease is used by others already without any problems it would be a shame to revert them just because a single archive and (parts of) its mirror network has problems with them - even if this single archive is debian itself. Uh, Debian is an actual GNU/Linux distribution, not just a draft. If offering a patch is not enough, feel free to leave code to ease having InRelease support, perhaps a compile-time option. Or even to make the
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
On Fri, Sep 16, 2011 at 01:44:10PM -0400, Filipus Klutiero wrote: [Adding the mirrors team in the loop] You also might want to look at #636292. Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
Le 2011-09-16 15:54, Kurt Roeckx a écrit : On Fri, Sep 16, 2011 at 01:44:10PM -0400, Filipus Klutiero wrote: [Adding the mirrors team in the loop] You also might want to look at #636292. Kurt Thanks. I was thinking about you when I talked about those who believe that #641769 is actually what's under #636292: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641769#5 I pointed #636292 readers to #641769. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
On Fri, 16 Sep 2011, Filipus Klutiero wrote: Thanks. I was thinking about you when I talked about those who believe that #641769 is actually what's under #636292: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641769#5 I pointed #636292 readers to #641769. To me it looks like #636292 is the general case, and #641769 is just one specific (and quite nasty) way to hit #636292. IMHO, the root cause is that repository metadata is unversioned, and therefore there is no good way to detect all cases of in-mirror and across-mirrors inconsistencies, so that the front-ends like apt can try to work around it. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
Le 2011-09-16 22:54, Henrique de Moraes Holschuh a écrit : On Fri, 16 Sep 2011, Filipus Klutiero wrote: Thanks. I was thinking about you when I talked about those who believe that #641769 is actually what's under #636292: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=641769#5 I pointed #636292 readers to #641769. To me it looks like #636292 is the general case, and #641769 is just one specific (and quite nasty) way to hit #636292. IMHO, the root cause is that repository metadata is unversioned, and therefore there is no good way to detect all cases of in-mirror and across-mirrors inconsistencies, so that the front-ends like apt can try to work around it. #636292 is labelled as being about round-robin mirrors and problems APT has with those, even though that is probably not the actual cause of the reporter's problem. But indeed, there would be room for a general bug unreliable package indices fetching protocol :-S -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#641769: [apt] fetches InRelease file, problematic on several mirrors (aka Packages Hash Sum mismatch)
Package: apt Version: 0.8.15.6 Severity: important Since 0.8.11, APT fetches InRelease in preference to the classic Release file. InRelease is an Inline signed Release file announced in http://lists.debian.org/debian-devel-announce/2009/11/msg1.html If I understand correctly, the only real noteworthy advantage of InRelease at the moment is that it avoids one HTTP request, improving performance. Although InRelease is available since 2009, the mirrors team reportedly only became aware of its existence in February 2011, when it changed ftpsync to account for the InRelease file, excluding it from the files synchronized in the first mirror synchronization stage. http://lists.debian.org/debian-devel/2011/03/msg00598.html gives some background. The first ftpsync version to consider InRelease was 80387, which was released on 2011-02-23: http://lists.debian.org/debian-mirrors-announce/2011/02/msg1.html When a mirror uses a problematic ftpsync version, its InRelease files are incoherent with its Packages files from a moment in the first stage to a moment in the second stage. This could (and I think basically does) mean the files are incoherent for the duration of the synchronization. When Packages and Release are incoherent, updating APT indices fails, for example: # LANG=C apt-get update Hit http://ftp.ca.debian.org sid InRelease Get:1 http://ftp.ca.debian.org sid/main i386 Packages [7703 kB] Hit http://ftp.ca.debian.org sid/main TranslationIndex Fetched 1 B in 3s (0 B/s) W: Failed to fetch bzip2:/var/lib/apt/lists/partial/ftp.ca.debian.org_debian_dists_sid_main_binary-i386_Packages Hash Sum mismatch E: Some index files failed to download. They have been ignored, or old ones used instead. root@vinci:/etc/apt# ftp.ca.debian.org is a round-robin that used until yesterday 2 mirrors, mirror.mountaincable.net aka debian.mirror.rafal.ca and debian.mirror.iweb.ca. Both of these mirrors used problematic ftpsync versions. Both of these are children of the primary mirror syncproxy.wna.debian.org, running on rietz.debian.org. rietz is the north american archive sync proxy and appears to have 25 children (see http://mirror-master.debian.org/status.html ). If the archive has something in the order of magnitude of 1 GB of new packages daily, that means rietz must be uploading about 25 GB a day just for synchronization. It's quite possible that rietz's uplink is saturated for a good proportion of time. There are currently 4 dinstall-s a day. The mirror I'm using, mirror.mountaincable.net, stayed incoherent for more than 2 hours the only time I was able to measure it somewhat precisely. With a dinstall each 6 hours, that's a lot of time. Mirror synchronization is explained on http://www.debian.org/mirror/ftpmirror#how This issue only affects modified Packages/Sources files. This is typically the case of sid main, surely most dinstall-s. This therefore affects sid systems, and in a less important way testing systems. Having a testing/unstable mix which I usually upgrade every day, I would have to retry indices updates about 1 day on 2. If everyone was using my mirror, that would have affected quite a lot of people. The problem with ftpsync is that the mirrors team doesn't seem to have much infrastructure to ensure it stays up-to-date. There is no package for ftpsync. When I reported my problem to #debian-mirrors, Simon Paillard contacted my mirror's administration and they upgraded their ftpsync in less than a day, fixing a problem which was affecting me since half a year. Each mirror reports its ftpsync version in debian/project/trace/ but this information is apparently not centrally collected, allowing to see which mirrors are affected at a glance and to contact all administrators quickly. If your mirror is affected, see http://anonscm.debian.org/viewvc/webwml/webwml/english/mirror/Mirrors.masterlist to find the contact address. There are now 2 synchronization methods that don't cause this problem that I'm aware of: using ftpsync 80387, and using ftpsync-pushrsync, which runs on the parent server rather than pulling updates. Unfortunately the latter is reportedly used only on very few mirrors. There are hundreds of official mirrors. Furthermore, there is no definition of what is a supported/acceptable ftpsync version and what should be considered outdated. Overall, coordination between the mirrors team and mirror administrators appears to have little structure. I don't know what proportion of mirrors are affected. As I said, as of 2 days ago, both mirrors behind ftp.ca.debian.org were affected. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=625160 has some more reports, citing 4/4 ftp.us.debian.org mirrors affected as of May. Some believe http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636292 stems from this issue too. The proportion looks high enough to do something. I'm setting severity based on my feeling, but could be very wrong. A
