On 9/3/17 11:40 AM, Philipp Kern wrote: > On 2017-09-02 23:48, Holger Levsen wrote: >> On Mon, Jul 03, 2017 at 07:23:29PM +0200, Philipp Kern wrote: >>> > Not yet. We people from the reproducible team couldn't find a way to >>> > usefully talk to ftp-masters people, whom never replied to any of the >>> > questions in the thread at #763822 (they only did some quick >>> comments on >>> > IRC, and we have been left on guessing what they would like…). >>> > >>> > Anyhow, .buildinfo files are stored in ftp-master, just not >>> exported to >>> > the mirrors, you can find them in >>> > coccia.debian.org:/srv/ftp-master.debian.org/<something>. >>> >>> So I suppose we talk about 13 GB of static content in about 1.7M >>> files. Is that something that could be distributed through >>> static.debian.org if there are concerns around inodes for the main >>> mirrors? Given that they would be accessed mostly rarely? >>> >>>  7.7kB (75%ile as mentioned in the referenced bug) * 55000 binary >>> packages * 10 architectures * 3 versions - so quite conservatively >>>  So supposedly a CDN wouldn't bring a lot of benefit as individual >>> files aren't likely to be hit frequently. >> >> using static.debian.org seems to be a good idea to me, what would be >> needed to make >> this happen? >> >> or, we could put them in a git repo instead, and use git.debian.org… > > Git is an interesting thought for incremental mirroring. But then it > also seems to be a poor choice for something that is an only growing > repository of data. > > What I think should be a requirement is that the data is pushed out > before the mirror pulse. Otherwise you end up with a race where you try > to mirror the data including the buildinfo but can't access it. (It's a > little unfortunate that we don't simply put them onto the mirrors.
So what would be needed to make at least a simple export of the data happen? I think the requirements I'd have are these: * Data is sufficiently fresh and optimally accessible before the mirror pulse happens so that you can always fetch the corresponding buildinfo for a newly pushed package. * Some way of actually deducing the path to the buildinfo file, either through some sort of redirector or by naming the files in a consistent fashion. Right now the second point does not work with the date-based farm that is used to archive the buildinfo files. It would work if we were to just apply the same splitting as in the regular pool. For the former just pushing the content through static.d.o should work and dak could push the content before pushing the mirrors? Intuitively I would not care about cryptographic authentication of the data. After all it can be verified by rebuilding if the package is reproducible. Kind regards and thanks Philipp Kern
Description: OpenPGP digital signature