Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-15 Thread Frédéric Pierret
Hi, Le 2/27/21 à 3:09 PM, Holger Levsen a écrit : Hi, snapshot.debian.org is an awesome service for the wider free software community and especially for those working on reproducible builds. Sadly accessing *many* packages from it is limited and troublesome (see below for bug numbers), thus we

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-05 Thread Paul Wise
On Fri, Mar 5, 2021 at 8:26 AM Frédéric Pierret wrote: > Is it possible to have an exported version somewhere for starting working on > the mirroring? To not bothering you currently, just a one shot export until > we are sure that would be absolutely necessary to have this daily export. Please

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-05 Thread Frédéric Pierret
Hi, Le 3/5/21 à 4:19 AM, Paul Wise a écrit : On Thu, Mar 4, 2021 at 8:57 AM Frédéric Pierret wrote: Following discussion on #debian-reproducible, would this be feasible from the snapshot team to allow something like a daily export of the database on a public exposed server? Looking on the

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-04 Thread Paul Wise
On Thu, Mar 4, 2021 at 8:57 AM Frédéric Pierret wrote: > Following discussion on #debian-reproducible, would this be feasible from the > snapshot team to allow something like a daily export of the database on a > public exposed server? Looking on the database server, the raw PostgreSQL tables

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-04 Thread Frédéric Pierret
Hi, Le 3/2/21 à 10:58 AM, Lucas Nussbaum a écrit : On 01/03/21 at 22:41 +, Paul Wise wrote: On Mon, Mar 1, 2021 at 5:25 PM Holger Levsen wrote: For https://trends.debian.net/, I have a local mirror of snapshot.d.o (with sources only, and only for specific versions). The code used to

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-02 Thread Lucas Nussbaum
On 02/03/21 at 18:06 +0800, Paul Wise wrote: > On Tue, 2021-03-02 at 10:58 +0100, Lucas Nussbaum wrote: > > > - queries the snapshot DB to identify the files and hashes for each > > source package > > - fetches and analyses Sources files to identify (source,version) of > > interest, and thus

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-02 Thread Paul Wise
On Tue, 2021-03-02 at 10:58 +0100, Lucas Nussbaum wrote: > - queries the snapshot DB to identify the files and hashes for each > source package > - fetches and analyses Sources files to identify (source,version) of > interest, and thus hashes to transfer You can probably replace this with

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-02 Thread Lucas Nussbaum
On 01/03/21 at 22:41 +, Paul Wise wrote: > On Mon, Mar 1, 2021 at 5:25 PM Holger Levsen wrote: > > > > How would the mirroring work? > > > > to be discussed, but my raw idea would be to use rsync with excluding the > > years > > before 2015 or 2017. or can't this work? 8-) > > That won't

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-01 Thread Paul Wise
On Mon, Mar 1, 2021 at 5:25 PM Holger Levsen wrote: > > How would the mirroring work? > > to be discussed, but my raw idea would be to use rsync with excluding the > years > before 2015 or 2017. or can't this work? 8-) That won't work, since the filesystem storing the data is hash (SHA1) based,

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-01 Thread Frédéric Pierret
Le 3/1/21 à 6:25 PM, Holger Levsen a écrit : On Mon, Mar 01, 2021 at 06:12:39PM +0100, Julien Cristau wrote: On Sat, Feb 27, 2021 at 02:09:08PM +, Holger Levsen wrote: It's possible to work around them partly/sometimes, but it seems clear by now that using snapshot.d.o *as it is* doesn't

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-01 Thread Holger Levsen
On Mon, Mar 01, 2021 at 06:12:39PM +0100, Julien Cristau wrote: > On Sat, Feb 27, 2021 at 02:09:08PM +, Holger Levsen wrote: > > It's possible to work around them partly/sometimes, but it seems clear by > > now that using snapshot.d.o *as it is* doesn't scale for our use cases. > > Hence > >

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-03-01 Thread Julien Cristau
On Sat, Feb 27, 2021 at 02:09:08PM +, Holger Levsen wrote: > It's possible to work around them partly/sometimes, but it seems clear by > now that using snapshot.d.o *as it is* doesn't scale for our use cases. Hence > the idea of a partial snapshot mirror which is only suitable to rebuild >

Re: partial snapshot mirror amd64/bullseye/bookworm

2021-02-28 Thread Vagrant Cascadian
On 2021-02-27, Holger Levsen wrote: > snapshot.debian.org is an awesome service for the wider free software > community > and especially for those working on reproducible builds. Sadly accessing > *many* > packages from it is limited and troublesome (see below for bug numbers), thus > we (mostly

partial snapshot mirror amd64/bullseye/bookworm

2021-02-27 Thread Holger Levsen
Hi, snapshot.debian.org is an awesome service for the wider free software community and especially for those working on reproducible builds. Sadly accessing *many* packages from it is limited and troublesome (see below for bug numbers), thus we (mostly Frédéric Pierret and myself) came up with