On Fri, Jan 8, 2021, at 19:05, Stuart Henderson wrote: > > For the place where I think it makes most sense to generate index files, > the requirements are pretty much the opposite - OpenBSD tools and simple > scripts are ok but Go (or indeed anything from ports) is not wanted. > So I've bodged together a shell script to generate a new style index > using pkg_info -qPS from local tgz files and given it a run on the brand > new amd64 pkg snap (at the time I generated it wasn't on cdn/mirrors, > it may be on some now but may need a bit longer to feed out). > I don't know how that will go for pkg_add over the network but with > local files it's not too bad; about 6 minutes. > > Could do with comparing with the results of your genpkgup with mine > when the files are on the mirrors, copy of my generated file at > junkpile.org/index.pkgup.amd64-20210108.gz (it is deliberately missing > entries for all the packages which use "@option always-update" i.e. > quirks/sqlports/etc because pkg_info -S is not quite working as expected > for them ;) >
I have updated genpkgup/obsdpkgup to match exactly with your format (and by extension, pkg_info -S's format). > The following two paragraphs might relate to my different genpkgup but > I don't _think_ they do; > > I ran the pkg_add command generated by obsdpkgup followed by a normal > pkg_add -u; the normal one picked up a few more packages (some of them > definitely in need of an update because the installed packages still > linked against old libc++). > > I haven't got proper logs from the run (I used pkg_add -vv but forgot > to run it in script and ran out of scrollback..) but the packages picked > up by pkg_add -u included ansible+deps, audacity, boehm-gc, jack, > distcc/distcc-gtk, graphviz, pulseaudio, powerdns. > > I would expect any discrepancy in how the signature is generated to > result in a possibly unneeded update, rather than missing one, which is > why I think this is probably not on the gen side. > I experienced this same issue. I'm not sure what's going on here. After an obsdpkgup update, every package that 'pkg_add -u' reported as still needing updates had *identical* signatures between the installed version and the remote version according to 'pkg_info -S' and the PKG_DBDIR=/var/empty trick. I have to assume this is an overzealous update on pkg_add's part, but of course this is just conjecture. > Another minor thing I spotted; the displayed pkg_add command has full > paths after %, pkg_add accepts this but the canonical format is just the > last part i.e. openssl%1.0.2 rather than openssl%security/openssl/1.0.2. > Fixed as well. > pkg_add tries to make as good a job as it can from an partially updated > pkg mirror and has some overhead due to that (it intentionally doesn't > use an index file because it doesn't want to trust that it reflects > the actual contents of the mirror). I'm not convinced this flexibility > is useful in many cases - certainly if I notice that a mirror is not > fully synced my usual course of action is to look for another mirror or > postpone the update because continuing will often result in a mess of > installed packages built against conflicting library versions. > I do the same thing when encountering a seemingly out-of-sync mirror. > So I think there is use for a tool that takes a different approach on > this and I'm definitely interested to see how update times compare > when using this to pre-calculate things, as long as it's able to find > all the packages that do need an update :) > I'm glad I'm not the only one that sees potential merit in this. Right now though, 'pkg_add -u' will still check the headers of every specified package's dependencies (even though we know which of those dependencies don't actually need an update), which is another source of substantial speed improvement. I did some experimenting with 'pkg_add -r -Dupdatedepends' to see how much faster upgrades could be with the dependency check removed (because we *know* everything that actually needs to be updated already) and the speed improvement was even more pronounced. The issue with using pkg_add -r though is that because it bypasses dependency checking/package ordering for every package that gets installed, pkg_add will refuse to install a package if a dependent library version isn't installed yet. For example, if we want to upgrade pkgA, and it depends on upgraded pkgB, and we call 'pkg_add -r pkgA pkgB', pkg_add will first try to install pkgA, notice that pkgB's library versions aren't on the disk yet, and refuse to install pkgA. Then it installs pkgB, and a subsequent re-run of pkg_add -r will successfully install pkgA. This doesn't touch on circular dependencies either though, which I understand that 'pkg_add -u' figures out as well. Anyway, if we can agree that perhaps being resilient to out-of-sync mirrors isn't necessarily a desired goal for the pkg tools, I think there's enormous performance + bandwidth improvements to be made.
