Re: New tool to (quickly) check for available package upgrades

2020-06-18 Thread Marc Espie
On Wed, Jun 17, 2020 at 09:12:08PM -, Stuart Henderson wrote: > On 2020-06-17, Marc Espie wrote: > > The only way you end up with broken installations is when porters don't do > > their jobs, that is they fail to bump a shared library or something like > > that. > > They do still break in

Re: New tool to (quickly) check for available package upgrades

2020-06-18 Thread Marc Espie
On Wed, Jun 17, 2020 at 09:12:08PM -, Stuart Henderson wrote: > This is already a problem when pkg_add fetches the directory listing > (though a smaller one because the filenames don't change as often). > > Firstly the contents of the mirror can change during the pkg_add run > so the listing

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Stuart Henderson
On 2020-06-17, Jeremy O'Brien wrote: >> What if a new batch of amd64/i386 files appears while one of the ongoing >> syncs run, do you restart over and hope yet another new one doesn't appear >> while that one is running? > > Is this something that actually happens? These 4 arches all take <3

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Stuart Henderson
On 2020-06-17, Marc Espie wrote: > The only way you end up with broken installations is when porters don't do > their jobs, that is they fail to bump a shared library or something like > that. They do still break in some cases: libA depends on libB someapp depends on libA, libB libB has a

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Theo de Raadt
Jeremy O'Brien wrote: > > What if a new batch of amd64/i386 files appears while one of the > > ongoing syncs run, do you restart over and hope yet another new one > > doesn't appear while that one is running? > > Is this something that actually happens? Yes. Apparently you don't like reading.

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Jeremy O'Brien
On Wed, Jun 17, 2020, at 13:45, Janne Johansson wrote: > Do think of what you call "the index file" in terms of "I check/replace some > 100+G of snapshots and packages every 24h", at which point do you replace > that single file, before, under or after none,most,all packages for your arch > are

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Stuart Henderson
On 2020-06-17, Jeremy O'Brien wrote: > On Wed, Jun 17, 2020, at 13:45, Janne Johansson wrote: >> >> Now if someone invents a decent piece of code to use http connection >> pooling, quic/http3/rsync or whatever to speed up getting the required info, >> I'm sure we mirror admins would be happy

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Jeremy O'Brien
On Wed, Jun 17, 2020, at 13:45, Janne Johansson wrote: > > Now if someone invents a decent piece of code to use http connection pooling, > quic/http3/rsync or whatever to speed up getting the required info, I'm sure > we mirror admins would be happy to add/edit our server programs to serve it.

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Janne Johansson
Den ons 17 juni 2020 kl 17:04 skrev Marc Espie : > > > > > > The concept you need to understand is snapshot shearing. > > > > > A full package snapshot is large enough that it's hard to > guarantee that > > > > > you will have a full snapshot on a mirror at any point in time. > > > > > In fact,

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Marc Espie
On Wed, Jun 17, 2020 at 09:44:32AM -0400, Jeremy O'Brien wrote: > On Wed, Jun 17, 2020, at 08:47, Marc Espie wrote: > > On Wed, Jun 17, 2020 at 08:28:02AM -0400, Jeremy O'Brien wrote: > > > On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > > > > > > > The concept you need to understand is

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Jeremy O'Brien
On Wed, Jun 17, 2020, at 08:47, Marc Espie wrote: > On Wed, Jun 17, 2020 at 08:28:02AM -0400, Jeremy O'Brien wrote: > > On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > > > > > The concept you need to understand is snapshot shearing. > > > > > > A full package snapshot is large enough that

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Marc Espie
On Wed, Jun 17, 2020 at 08:28:02AM -0400, Jeremy O'Brien wrote: > On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > > > The concept you need to understand is snapshot shearing. > > > > A full package snapshot is large enough that it's hard to guarantee that > > you will have a full snapshot

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Jeremy O'Brien
On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > The concept you need to understand is snapshot shearing. > > A full package snapshot is large enough that it's hard to guarantee that > you will have a full snapshot on a mirror at any point in time. > > In fact, you will sometimes encounter

Re: New tool to (quickly) check for available package upgrades

2020-06-17 Thread Jeremy O'Brien
On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > The concept you need to understand is snapshot shearing. > > A full package snapshot is large enough that it's hard to guarantee that > you will have a full snapshot on a mirror at any point in time. > > In fact, you will sometimes encounter

Re: New tool to (quickly) check for available package upgrades

2020-06-16 Thread Marc Espie
On Tue, Jun 16, 2020 at 04:59:07PM -0400, Jeremy O'Brien wrote: > Hey misc@, > > I wrote a quick little tool here: > https://github.com/neutralinsomniac/obsdpkgup in Go to show available package > upgrades from your configured mirror. > > It takes no more than a few seconds (the time it takes

Re: New tool to (quickly) check for available package upgrades

2020-06-16 Thread Stuart Henderson
On 2020-06-16, Jeremy O'Brien wrote: > On Tue, Jun 16, 2020, at 17:19, Daniel Jakots wrote: >> I think if I wanted to compare packages between a machine of mine and a >> mirror, I would compare the quirks package signature timestamps. On >> your machine you can find it with >> $ grep

Re: New tool to (quickly) check for available package upgrades

2020-06-16 Thread Jeremy O'Brien
On Tue, Jun 16, 2020, at 17:19, Daniel Jakots wrote: > I think if I wanted to compare packages between a machine of mine and a > mirror, I would compare the quirks package signature timestamps. On > your machine you can find it with > $ grep digital-signature /var/db/pkg/quirks*/+CONTENTS > and on

Re: New tool to (quickly) check for available package upgrades

2020-06-16 Thread Daniel Jakots
On Tue, 16 Jun 2020 16:59:07 -0400, "Jeremy O'Brien" wrote: > I wrote a quick little tool here: > https://github.com/neutralinsomniac/obsdpkgup in Go to show available > package upgrades from your configured mirror. > > It takes no more than a few seconds (the time it takes to download >

New tool to (quickly) check for available package upgrades

2020-06-16 Thread Jeremy O'Brien
Hey misc@, I wrote a quick little tool here: https://github.com/neutralinsomniac/obsdpkgup in Go to show available package upgrades from your configured mirror. It takes no more than a few seconds (the time it takes to download index.txt from the package repo) to show you all packages that