Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
On Mon, Apr 29, 2019 at 10:43:11AM -0700, Vagrant Cascadian wrote: > On 2019-04-29, Vagrant Cascadian wrote: > > It seems to be missing the .buildinfo.N, which in some cases are the > > actual .buildinfo files built by the buildd's and the corresponding .deb > > files shipped in the archive. > > > > The .buildinfo without a numbered increment is frequently provided by > > developers who follow best practices and do source-only uploads that > > include a signed .buildinfo file. I'll take a look at the code and see > > if I can't propose a simple fix. > It seems like you did in fact catch these, and named them as > ARCH-source.buildinfo. Nice! in the pool structure the filenames are changed so that the original filename $package_$version_$arch.buildinfo is replaced with one where the $arch part is replaced like this: ARCHITECTURE=$(grep ^Architecture: $FILE | cut -d ' ' -f2-|sed 's# #-#g') or in plain English: with the Architecture: field from inside the file, where the architectures are concated with hyphons, so that Architecture: all source amd64 results in a $package_$version_all-source-amd64.buildinfo file. And then strangely there are a few hundred cases of identically named $package_$version_$arch.buildinfo files (out of allmost a million) in the /mm/dd structure, these get a .0 suffix. And then there are 4 cases with 3 identically named files, which get a .1 suffix. (This needs to be investigated why this happens...) -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C we'll all die. make a difference while you can. disobey. smile. signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
Hi, please note that https://buildinfos.debian.net/buildinfo-pool/ is currently being recreated and thus not have all data again yet... (I've added support for 3 (and more) .buildinfo files with the same name and noticed that the numbered links were created in the wrong order, thus the recreation.) On Mon, Apr 29, 2019 at 10:39:30AM -0700, Vagrant Cascadian wrote: > I really like that it provides a view in a "pool" style, e.g.: > https://buildinfos.debian.net/buildinfo-pool/u/u-boot/ thanks. (and me too) > I almost wonder if we shouldn't try to coordinate archiving this data > with: > https://www.softwareheritage.org/ thats an interesting idea! > It might be a slight stretch of their mission to call .buildinfo files > "source code" ... but I wouldn't mind making the case that .buildinfo > files should be considered source code. indeed. > > buildinfos.debian.net has all .buildinfo files since December 2016. > Very cool! It is definitely a much simpler approach and catches many > corner cases (unsigned, signatures, etc.) that my method doesn't! yup. > It seems to be missing the .buildinfo.N, which in some cases are the > actual .buildinfo files built by the buildd's and the corresponding .deb > files shipped in the archive. it has them. see eg dpkg (once the pool structure is back). > The .buildinfo without a numbered increment is frequently provided by > developers who follow best practices and do source-only uploads that > include a signed .buildinfo file. I'll take a look at the code and see > if I can't propose a simple fix. please do. > The presence of multiple .buildinfo* files does make it harder to know > which .buildinfo to use to reproduce a build from the archive if they > differ, unfortunately. the one with the correct hash for the .deb > Unsigned .buildinfo files are of limited usefulness, if we're really > trying to establish a chain of verification... though perhaps it's still > better than no .buildinfo at all, since the archive verifies the > .changes file before including it... though obviously a compromised > archive could inject malicious unsigned .buildinfo files more easily and > requires some trust needed in specific parties. yes. unsigned .buildinfo files should not exist. for this to happen, as a first step, I plan to record this state in the db (in a new table...) -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C Our civilization is being sacrificed for the opportunity of a very small number of people to continue making enormous amounts of money... It is the sufferings of the many which pay for the luxuries of the few... You say you love your children above all else, and yet you are stealing their future in front of their very eyes... signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
On 2019-04-29, Vagrant Cascadian wrote: > It seems to be missing the .buildinfo.N, which in some cases are the > actual .buildinfo files built by the buildd's and the corresponding .deb > files shipped in the archive. > > The .buildinfo without a numbered increment is frequently provided by > developers who follow best practices and do source-only uploads that > include a signed .buildinfo file. I'll take a look at the code and see > if I can't propose a simple fix. It seems like you did in fact catch these, and named them as ARCH-source.buildinfo. Nice! live well, vagrant signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
On 2019-04-29, Holger Levsen wrote: > On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote: >> On 2019-02-15, Holger Levsen wrote: > we now have two similar implementations of a buildinfo server for Debian > .buildinfo files: > > - https://buildinfo.debian.net - this service exists since Oct. 2016 > - https://buildinfos.debian.net - this service exist since March 2019 I really like that it provides a view in a "pool" style, e.g.: https://buildinfos.debian.net/buildinfo-pool/u/u-boot/ I almost wonder if we shouldn't try to coordinate archiving this data with: https://www.softwareheritage.org/ It might be a slight stretch of their mission to call .buildinfo files "source code" ... but I wouldn't mind making the case that .buildinfo files should be considered source code. >> The vast majority of buildinfo files uploaded to the archive should be >> present in buildinfo.debian.net since November 2018. I also "manually" >> uploaded all the available buildinfo files from 2017-2018 (most of the >> very small number from 2016 failed for one reason or another). > > buildinfos.debian.net has all .buildinfo files since December 2016. Very cool! It is definitely a much simpler approach and catches many corner cases (unsigned, signatures, etc.) that my method doesn't! It seems to be missing the .buildinfo.N, which in some cases are the actual .buildinfo files built by the buildd's and the corresponding .deb files shipped in the archive. The .buildinfo without a numbered increment is frequently provided by developers who follow best practices and do source-only uploads that include a signed .buildinfo file. I'll take a look at the code and see if I can't propose a simple fix. The presence of multiple .buildinfo* files does make it harder to know which .buildinfo to use to reproduce a build from the archive if they differ, unfortunately. >> There are a few individual developers uploading unsigned .buildinfo >> files, as well as a few buildds for non-release architectures >> (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a >> legitimate use for the technique described in: >> https://xkcd.com/1181/ >> Which basically means I don't even bother attempting to upload unsigned >> buildinfo files. > > see above :) (=buildinfos.d.n has those.) Unsigned .buildinfo files are of limited usefulness, if we're really trying to establish a chain of verification... though perhaps it's still better than no .buildinfo at all, since the archive verifies the .changes file before including it... though obviously a compromised archive could inject malicious unsigned .buildinfo files more easily and requires some trust needed in specific parties. live well, vagrant signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
On Mon, Apr 29, 2019 at 04:53:16PM +, Holger Levsen wrote: > - https://buildinfos.debian.net - this service exist since March 2019 to expand on this: provides two views: https://buildinfos.debian.net/ftp-master.debian.org/buildinfo/ - this has the same contents as /srv/ftp-master.debian.org/buildinfo/ on (ftp-master|coccia).debian.org. https://buildinfos.debian.net/buildinfo-pool/ - this is a pool view on the same data. (implemented using links to the former.) -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
Hi, On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote: > On 2019-02-15, Holger Levsen wrote: > > I've just been re-reading this old and joyful thread... :) > I come back to it now and then myself... :) and here we go again... I'll also kind of top-post now, because I want to emphasize one important bit of information: we now have two similar implementations of a buildinfo server for Debian .buildinfo files: - https://buildinfo.debian.net - this service exists since Oct. 2016 - https://buildinfos.debian.net - this service exist since March 2019 I've set up the latter because the former doesnt provide an easy means to only get .buildinfo files for .debs from ftp.debian.org and because it was easy to set it up. (The code lives in jenkins.debian.net.git) I'm not really happy about the duplication of services and the very similar names, but I couldnt come up with a better name and this is all prototyping land anyway, so - for now - I just went ahead. I'm also not sure how to proceed with those two services in future. For now I think it's ok to have both and let us experiment with both. > > On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote: > >> I started the process of uploading all the .buildinfo files available on > >> ftp.debian.org to buildinfo.debian.net. > >> > >> Then I hope to set up a cron job to do uploads at least daily with a > >> little better error-handling Would be more ideal to have something > >> more formally integrated into infrastructure, but maybe I can work out a > >> proof-of-concept implementation as a basis for something that can be > >> integrated. > > > > did you manage to setup this cron job? > > I had thought I left more detail about the current status, but > apparently not! Thanks for the nudge. > > I have a cron job running on coccia.debian.org since November, as my own > "vagrant" user: I have a similar cronjob on coccia... > coccia.debian.org:~vagrant/rb-buildinfos/upload-buildinfos > > Logs for various upload passes in are the same directory, which should > probably be migrated to sqlite or some real database. The script is > checked into it's own git repository, but not properly pushed > anywhere. > > The cron job runs several times per day, checking the queues for > buildinfos uploaded both the current day yesterday to make sure we don't > miss a .buildinfo file uploaded in the middle of a processing run. If > coccia were down for longer than 24 hours, it might need to manually be > run to check for missing ones. > > The vast majority of buildinfo files uploaded to the archive should be > present in buildinfo.debian.net since November 2018. I also "manually" > uploaded all the available buildinfo files from 2017-2018 (most of the > very small number from 2016 failed for one reason or another). buildinfos.debian.net has all .buildinfo files since December 2016. > There are a small number of buildinfo uploads that buildinfo.debian.net > rejects for some reason probably related to ed25519 signing keys: > https://github.com/lamby/buildinfo.debian.net/issues/51 see above :) (=buildinfos.d.n has those.) > There are a few individual developers uploading unsigned .buildinfo > files, as well as a few buildds for non-release architectures > (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a > legitimate use for the technique described in: > https://xkcd.com/1181/ > Which basically means I don't even bother attempting to upload unsigned > buildinfo files. see above :) (=buildinfos.d.n has those.) > So, it's working, but we probably would need a little more work on it to > integrate into debian's infrastructure. this is also true for buildinfos.d.n. -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C In Europe there are people prosecuted by courts because they saved other people from drowning in the Mediterranean Sea. That is almost as absurd as if there were people being prosecuted because they save humans from drowning in the sea. signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote: > There are a few individual developers uploading unsigned .buildinfo > files, as well as a few buildds for non-release architectures > (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a > legitimate use for the technique described in: > > https://xkcd.com/1181/ Do you think you could provide more info about the kbsd and hurd buildinfo that are unsigned? James, who manages the current kbsd buildd, reads ftp.d.o bugs so should be receiving this email (and I can otherwise nudge him), and I could poke youpi for the hurd buildds. But I would expect both of them to be running unstable, and so have up-to-date software. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
On 2019-02-15, Holger Levsen wrote: > I've just been re-reading this old and joyful thread... :) I come back to it now and then myself... :) > On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote: >> I started the process of uploading all the .buildinfo files available on >> ftp.debian.org to buildinfo.debian.net. >> >> Then I hope to set up a cron job to do uploads at least daily with a >> little better error-handling Would be more ideal to have something >> more formally integrated into infrastructure, but maybe I can work out a >> proof-of-concept implementation as a basis for something that can be >> integrated. > > did you manage to setup this cron job? I had thought I left more detail about the current status, but apparently not! Thanks for the nudge. I have a cron job running on coccia.debian.org since November, as my own "vagrant" user: coccia.debian.org:~vagrant/rb-buildinfos/upload-buildinfos Logs for various upload passes in are the same directory, which should probably be migrated to sqlite or some real database. The script is checked into it's own git repository, but not properly pushed anywhere. The cron job runs several times per day, checking the queues for buildinfos uploaded both the current day yesterday to make sure we don't miss a .buildinfo file uploaded in the middle of a processing run. If coccia were down for longer than 24 hours, it might need to manually be run to check for missing ones. The vast majority of buildinfo files uploaded to the archive should be present in buildinfo.debian.net since November 2018. I also "manually" uploaded all the available buildinfo files from 2017-2018 (most of the very small number from 2016 failed for one reason or another). There are a small number of buildinfo uploads that buildinfo.debian.net rejects for some reason probably related to ed25519 signing keys: https://github.com/lamby/buildinfo.debian.net/issues/51 There are a few individual developers uploading unsigned .buildinfo files, as well as a few buildds for non-release architectures (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a legitimate use for the technique described in: https://xkcd.com/1181/ Which basically means I don't even bother attempting to upload unsigned buildinfo files. So, it's working, but we probably would need a little more work on it to integrate into debian's infrastructure. live well, vagrant signature.asc Description: PGP signature
Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net
Hi Vagrant, I've just been re-reading this old and joyful thread... :) On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote: > I started the process of uploading all the .buildinfo files available on > ftp.debian.org to buildinfo.debian.net. > > Then I hope to set up a cron job to do uploads at least daily with a > little better error-handling Would be more ideal to have something > more formally integrated into infrastructure, but maybe I can work out a > proof-of-concept implementation as a basis for something that can be > integrated. did you manage to setup this cron job? -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
