On Tue, May 23, 2017 at 9:55 AM, Michael Stapelberg <stapelb...@debian.org> wrote:
> > > On Tue, May 23, 2017 at 3:34 AM, Holger Levsen <hol...@layer-acht.org> > wrote: > >> Hi Michael, >> >> are you subscribed to this bug or do you need cc:s? >> > > I’m not subscribed, so please keep me cc'ed :). > > >> >> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote: >> > This is in relation to https://bugs.debian.org/850917, where we added >> > code to log update-alternatives calls. >> > >> > Now that the calls are logged, we need to somehow make Debiman aware >> > of the calls. To that end, I wrote a program which will read all >> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON >> > file with the links created by the various binary packages. >> >> thanks for the nice summary and proposal! >> >> > Find the source of that program at >> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effc >> d85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go >> >> looks simple and clean enough! >> >> > I then ran the program like so on piuparts.debian.org: >> >> /me really likes that pejacevic is developer accessable! ;) >> >> > % debiman-piuparts-distill \ >> > -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \ >> > -output=debiman/testing.json.gz >> > >> > % debiman-piuparts-distill \ >> > -logs_dir=/srv/piuparts.debian.org/htdocs/sid \ >> > -output=debiman/unstable.json.gz >> > >> > % debiman-piuparts-distill \ >> > -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \ >> > -output=debiman/experimental.json.gz >> > >> > % debiman-piuparts-distill \ >> > -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \ >> > -output=debiman/stable.json.gz >> >> ok, so only parsing 4 suites… how long did these commands run? >> > > 294,21s user 212,64s system 30% cpu 27:36,09 total > > >> >> > I then copied debiman/*.json.gz over to manziarly.debian.org, >> > re-generated the entire site, and slave alternative manpages now work >> > as expected :). >> >> hehe, nice! got an example? >> > > Sure: > https://manpages.debian.org/unstable/postgresql-client-9. > 6/pg_dump.1.en.html > https://manpages.debian.org/unstable/vis/vi.1.en.html > > >> >> > Obviously, the above was a one-off manual effort, so now we should >> > discuss how we’d like to set things up for a more permanent export. >> >> sure! >> >> > Here’s a strawman proposal. We could: >> > >> > 1. Add a crontab entry, which triggers a shell script, which will >> > update debiman-piuparts-distill, then run it on the logs >> > directories. >> >> sounds good, how often would you like to run it? >> > > Once a day would be sufficient, I think. > > >> >> > 2. Store the resulting *.json.gz files in a directory called >> > /srv/piuparts.debian.org/htdocs/for-manpages.d.o >> >> I don't really like the "for-manpages.d.o" directory name, but cannot >> think >> of a better one either, so… >> >> > 3. Download the files via HTTPS on manziarly.d.o. >> >> great, so only step 1+2 are on the piuparts side :) >> >> > Step ① requires installing the Go compiler >> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We >> > could avoid this if we’re all okay with just copying the >> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to >> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within >> > the piuparts git repository feels somewhat unclean. >> >> nah, installing golang-go from jessie or jessie-backports is easy, it >> just needs a patch for git.debian.org/git/mirror/debian.org.git for the >> files >> debian/control and debian/changelog and a mail to admin@rt.d.o with the >> subject "Debian RT: .*" to generate a ticket… which golang-go version do >> you want? >> > > Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7 > from jessie-backports, please :) > > >> >> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare >> > a patch. >> >> Yup, seems very fine to me! Please do! >> > > Great. Find a patch attached. Note that it’s untested, as golang-go is not > yet installed and I lack permission to run as piupartsm. I’m not sure what > your development process is for changes like this, but feel free to fix up > the patch as necessary. > > >> >> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in >> main() >> and explaining the purpose of the various functions ;-) I was just giving >> it >> a 2nd look after wanting to ask you to explain it/go-basics to me the next >> time we met but then thought I should first try to understand it myself >> ;-) >> And then I noticed some more comments explaining/introducing the purpose >> of the following code lines would be nice… >> > > Sure. I’ll add some commentary, and I’m also happy to explain any details > — just ask. > Commentary added: https://github.com/Debian/debiman/commit/8306cf79924d18db55a1bc0123d63ac98ad2f2c4 Let me know if anything else deserves an explanation. > > >> >> I suppose cmd/debiman-piuparts-distill/piuparts.go should live in >> master-bin >> in piuparts.git or what would you propose? I don't think we wanna compile >> it >> when building the piuparts-master package (for the Debian archive) but >> rather >> only build it when installing on pejacevic… >> > > It could continue living in github.com/Debian/debiman, or we could move > it to piuparts. The advantage of keeping it in github.com/Debian/debiman > is that we don’t need to duplicate code between different git repositories. > Installing/updating the command is as simple as running “go get -u > github.com/Debian/debiman/cmd/debiman-piuparts-distill”. > > -- > Best regards, > Michael > -- Best regards, Michael