[moved misc@ to ports@] > > Between those two it could cause problems because the user may try to > > update a too-small subset of packages. The first problem is obvious. > > The second problem, if a library is bumped after the index is generated, > > the required updates won't show up. For both if people use it and then > > run into problems it's likely the bug reports will end up with openbsd > > rather than pkgup. This makes me not want to add it to packages yet > > (adding it could easily be seen as an endorsement of using it). > > This would be less of a problem if it at least tries to detect outdated > > caches and prints a clear warning. > > > > I hope that my above two fixes rectify this situation in your mind.
This definitely helps a lot. > I would like this as well. The problem is that pkg_info -qS is slow. It takes > orders of magnitude more time to run than my current signature generation > code. > I can currently build a complete index from a remote mirror in less than ten > minutes. If I switched to using pkg_info, it would take several hours by my > math. In addition, I would like to keep genpkgup able to be run on any OS that > Go supports instead of only OpenBSD machines. I went ahead and > sorta-implemented your suggestion though by matching OpenBSD's current > signature format. No more hashes. I was torn on this before, but I actually > like your approach better because A: it's easier to debug when things go wrong > and B: it's much less CPU-intensive not having to do sha256 stuff. So again, > thank you for the recommendation. If the signature format changes in the > future, I will gladly update my code to match, or revisit the problem if > necessary. For the place where I think it makes most sense to generate index files, the requirements are pretty much the opposite - OpenBSD tools and simple scripts are ok but Go (or indeed anything from ports) is not wanted. So I've bodged together a shell script to generate a new style index using pkg_info -qPS from local tgz files and given it a run on the brand new amd64 pkg snap (at the time I generated it wasn't on cdn/mirrors, it may be on some now but may need a bit longer to feed out). I don't know how that will go for pkg_add over the network but with local files it's not too bad; about 6 minutes. Could do with comparing with the results of your genpkgup with mine when the files are on the mirrors, copy of my generated file at junkpile.org/index.pkgup.amd64-20210108.gz (it is deliberately missing entries for all the packages which use "@option always-update" i.e. quirks/sqlports/etc because pkg_info -S is not quite working as expected for them ;) The following two paragraphs might relate to my different genpkgup but I don't _think_ they do; I ran the pkg_add command generated by obsdpkgup followed by a normal pkg_add -u; the normal one picked up a few more packages (some of them definitely in need of an update because the installed packages still linked against old libc++). I haven't got proper logs from the run (I used pkg_add -vv but forgot to run it in script and ran out of scrollback..) but the packages picked up by pkg_add -u included ansible+deps, audacity, boehm-gc, jack, distcc/distcc-gtk, graphviz, pulseaudio, powerdns. I would expect any discrepancy in how the signature is generated to result in a possibly unneeded update, rather than missing one, which is why I think this is probably not on the gen side. Another minor thing I spotted; the displayed pkg_add command has full paths after %, pkg_add accepts this but the canonical format is just the last part i.e. openssl%1.0.2 rather than openssl%security/openssl/1.0.2. https://github.com/neutralinsomniac/obsdpkgup/commit/36ff10dd2f67b666b219b72de80bf5efd69a4027 ouch! I didn't realise they had an hex sha256 of each 64K block. I suppose overall it's not _huge_ in modern disk space terms (about 50MB for the full amd64 package build) but more than I expected. > Again, thank you for taking the time to look at my tool! pkg_add tries to make as good a job as it can from an partially updated pkg mirror and has some overhead due to that (it intentionally doesn't use an index file because it doesn't want to trust that it reflects the actual contents of the mirror). I'm not convinced this flexibility is useful in many cases - certainly if I notice that a mirror is not fully synced my usual course of action is to look for another mirror or postpone the update because continuing will often result in a mess of installed packages built against conflicting library versions. So I think there is use for a tool that takes a different approach on this and I'm definitely interested to see how update times compare when using this to pre-calculate things, as long as it's able to find all the packages that do need an update :)
