Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

Alright, then. Find attached a patch against the piuparts git to add
debiman-piuparts-distill. You can build it by running “go build” in the
debiman-piuparts-distill subdirectory.

On Wed, May 31, 2017 at 11:47 AM, Holger Levsen 
wrote:

> On Wed, May 31, 2017 at 11:32:46AM +0200, Michael Stapelberg wrote:
> > The fact that these manpages are included at all is what the
> > slave-alternative handling got us :)
>
> ah!
>
> > > ok, I'll request this once we got closer… (see below)
> > Thanks :).
>
> done (as you know, but for the record of this bug…)
>
> > The code is authenticated via HTTPS. If you don’t trust GitHub and/or the
> > TLS certificate infrastructure, then we should find an alternative.
>
> I dont trust GitHub, or rather, it's bad enough trusting alioth, I dont
> want
> to also trust Github… (and TLS, no…)
>
> (we should probably rather trust signed tags, but anyway, as we dont have
> that I would prefer to only trust one repo…)
>
> > > Also, I don't expect debiman-piuparts-distill to change very often, do
> you?
> > Agreed, most likely it will not change very often.
>
> ok, cool.
>
> > > does the code need to live in github.com/Debian/debiman at all? One
> copy
> > > in the piuparts repo should be enough, or?
> > That’s correct; I was referring to code that is shared between different
> > debiman components. Specifically,
> > https://github.com/Debian/debiman/blob/54dd6050a6ce8a454c14e172a8687d
> 93d0fd241b/internal/write/atomically.go
>
> ic
>
> > But, if the conclusion is that debiman-piuparts-distill should live in
> the
> > piuparts repo, we can duplicate that code. I don’t expect it to change
> > either.
>
> I think I would really prefer the code to live in the piuparts repo.
>
> Or alternativly, life in your repo and then we deploy it on demand, but not
> everyday. Having it in the piuparts repo would be easier for deployments,
> but
> I can see how its a bit more hassle for you to maintain the code in another
> repo. But then, I know how to be very liberal to accept your code changes
> to
> your code in our piuparts repo :)
>
> Patches welcome! :-D
>
>
> --
> cheers,
> Holger
>



-- 
Best regards,
Michael
From 7001c015ef65f548a0baf5f5488a05d91f2a3a64 Mon Sep 17 00:00:00 2001
From: Michael Stapelberg 
Date: Thu, 1 Jun 2017 21:28:15 +0200
Subject: [PATCH] add debiman-piuparts-distill

see https://bugs.debian.org/863089
---
 debiman-piuparts-distill/atomically.go |  75 +
 debiman-piuparts-distill/piuparts.go   | 145 +
 2 files changed, 220 insertions(+)
 create mode 100644 debiman-piuparts-distill/atomically.go
 create mode 100644 debiman-piuparts-distill/piuparts.go

diff --git a/debiman-piuparts-distill/atomically.go b/debiman-piuparts-distill/atomically.go
new file mode 100644
index ..c1b2d2db
--- /dev/null
+++ b/debiman-piuparts-distill/atomically.go
@@ -0,0 +1,75 @@
+package main
+
+import (
+	"bufio"
+	"compress/gzip"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+)
+
+func tempDir(dest string) string {
+	tempdir := os.Getenv("TMPDIR")
+	if tempdir == "" {
+		// Convenient for development: decreases the chance that we
+		// cannot move files due to /tmp being on a different file
+		// system.
+		tempdir = filepath.Dir(dest)
+	}
+	return tempdir
+}
+
+func writeAtomically(dest string, compress bool, write func(w io.Writer) error) (err error) {
+	f, err := ioutil.TempFile(tempDir(dest), "debiman-")
+	if err != nil {
+		return err
+	}
+	defer func() {
+		// Remove the tempfile if an error occurred
+		if err != nil {
+			os.Remove(f.Name())
+		}
+	}()
+	defer f.Close()
+
+	bufw := bufio.NewWriter(f)
+
+	w := io.Writer(bufw)
+	var gzipw *gzip.Writer
+	if compress {
+		// NOTE(stapelberg): gzip’s decompression phase takes the same
+		// time, regardless of compression level. Hence, we invest the
+		// maximum CPU time once to achieve the best compression.
+		gzipw, err = gzip.NewWriterLevel(bufw, gzip.BestCompression)
+		if err != nil {
+			return err
+		}
+		defer gzipw.Close()
+		w = gzipw
+	}
+
+	if err := write(w); err != nil {
+		return err
+	}
+
+	if compress {
+		if err := gzipw.Close(); err != nil {
+			return err
+		}
+	}
+
+	if err := bufw.Flush(); err != nil {
+		return err
+	}
+
+	if err := f.Chmod(0644); err != nil {
+		return err
+	}
+
+	if err := f.Close(); err != nil {
+		return err
+	}
+
+	return os.Rename(f.Name(), dest)
+}
diff --git a/debiman-piuparts-distill/piuparts.go b/debiman-piuparts-distill/piuparts.go
new file mode 100644
index ..73f773dc
--- /dev/null
+++ b/debiman-piuparts-distill/piuparts.go
@@ -0,0 +1,145 @@
+// debiman-piuparts-distill extracts slave alternative links from
+// LOG-ALTERNATIVES lines found in piuparts logs.
+//
+// See https://github.com/Debian/debiman/issues/12 for more details.
+package main
+
+import (
+	"bufio"
+	"encoding/json"
+	"flag"
+	"io"
+	"log"
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"sync"
+)

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Wed, May 31, 2017 at 11:32:46AM +0200, Michael Stapelberg wrote:
> The fact that these manpages are included at all is what the
> slave-alternative handling got us :)
 
ah!

> > ok, I'll request this once we got closer… (see below)
> Thanks :).
 
done (as you know, but for the record of this bug…)

> The code is authenticated via HTTPS. If you don’t trust GitHub and/or the
> TLS certificate infrastructure, then we should find an alternative.
 
I dont trust GitHub, or rather, it's bad enough trusting alioth, I dont want
to also trust Github… (and TLS, no…) 

(we should probably rather trust signed tags, but anyway, as we dont have
that I would prefer to only trust one repo…)

> > Also, I don't expect debiman-piuparts-distill to change very often, do you?
> Agreed, most likely it will not change very often.
 
ok, cool.

> > does the code need to live in github.com/Debian/debiman at all? One copy
> > in the piuparts repo should be enough, or?
> That’s correct; I was referring to code that is shared between different
> debiman components. Specifically,
> https://github.com/Debian/debiman/blob/54dd6050a6ce8a454c14e172a8687d93d0fd241b/internal/write/atomically.go

ic

> But, if the conclusion is that debiman-piuparts-distill should live in the
> piuparts repo, we can duplicate that code. I don’t expect it to change
> either.

I think I would really prefer the code to live in the piuparts repo.

Or alternativly, life in your repo and then we deploy it on demand, but not
everyday. Having it in the piuparts repo would be easier for deployments, but
I can see how its a bit more hassle for you to maintain the code in another
repo. But then, I know how to be very liberal to accept your code changes to
your code in our piuparts repo :)

Patches welcome! :-D


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Tue, May 30, 2017 at 4:43 PM, Holger Levsen 
wrote:

> On Tue, May 23, 2017 at 09:55:11AM +0200, Michael Stapelberg wrote:
> > I’m not subscribed, so please keep me cc'ed :).
>
> ok!
>
> > > ok, so only parsing 4 suites… how long did these commands run?
> > 294,21s user 212,64s system 30% cpu 27:36,09 total
>
> ok, so minutes rather than hours…
>
> > > hehe, nice! got an example?
> > Sure:
> > https://manpages.debian.org/unstable/postgresql-client-9.
> 6/pg_dump.1.en.html
> > https://manpages.debian.org/unstable/vis/vi.1.en.html
>
> those are the links to other version, languages and conflicting commands?
>

The fact that these manpages are included at all is what the
slave-alternative handling got us :)


>
> > > sounds good, how often would you like to run it?
> > Once a day would be sufficient, I think.
>
> cool.
>
> > Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> > from jessie-backports, please :)
>
> ok, I'll request this once we got closer… (see below)
>

Thanks :).


>
> > Great. Find a patch attached. Note that it’s untested, as golang-go is
> not
> > yet installed and I lack permission to run as piupartsm. I’m not sure
> what
> > your development process is for changes like this, but feel free to fix
> up
> > the patch as necessary.
>
> process wise it's fine to just install it and then fix it up…
>
> but…
>
> +export GOPATH=$(mktemp -d)
> +trap 'rm -rf "${GOPATH}"' TERM INT EXIT QUIT
> +go get -u github.com/Debian/debiman/cmd/debiman-piuparts-distill
> +export PATH=${GOPATH}/bin:$PATH
>
> from distill_alternatives_log from your patch makes me uneasy… because,
> for a
> start, how is the code authenticated?
>

The code is authenticated via HTTPS. If you don’t trust GitHub and/or the
TLS certificate infrastructure, then we should find an alternative.


>
> Also, I don't expect debiman-piuparts-distill to change very often, do you?
>

Agreed, most likely it will not change very often.


>
> > Sure. I’ll add some commentary, and I’m also happy to explain any
> details —
> > just ask.
>
> thanks, the code is much nicer to read now! :)
>
> > It could continue living in github.com/Debian/debiman, or we could move
> it
> > to piuparts. The advantage of keeping it in github.com/Debian/debiman is
> > that we don’t need to duplicate code between different git repositories.
>
> does the code need to live in github.com/Debian/debiman at all? One copy
> in the piuparts repo should be enough, or?


That’s correct; I was referring to code that is shared between different
debiman components. Specifically,
https://github.com/Debian/debiman/blob/54dd6050a6ce8a454c14e172a8687d93d0fd241b/internal/write/atomically.go
.

But, if the conclusion is that debiman-piuparts-distill should live in the
piuparts repo, we can duplicate that code. I don’t expect it to change
either.

-- 
Best regards,
Michael

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Tue, May 23, 2017 at 09:55:11AM +0200, Michael Stapelberg wrote:
> I’m not subscribed, so please keep me cc'ed :).
 
ok!

> > ok, so only parsing 4 suites… how long did these commands run?
> 294,21s user 212,64s system 30% cpu 27:36,09 total
 
ok, so minutes rather than hours… 

> > hehe, nice! got an example?
> Sure:
> https://manpages.debian.org/unstable/postgresql-client-9.6/pg_dump.1.en.html
> https://manpages.debian.org/unstable/vis/vi.1.en.html
 
those are the links to other version, languages and conflicting commands?

> > sounds good, how often would you like to run it?
> Once a day would be sufficient, I think.
 
cool.

> Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> from jessie-backports, please :)
 
ok, I'll request this once we got closer… (see below)

> Great. Find a patch attached. Note that it’s untested, as golang-go is not
> yet installed and I lack permission to run as piupartsm. I’m not sure what
> your development process is for changes like this, but feel free to fix up
> the patch as necessary.
 
process wise it's fine to just install it and then fix it up…

but… 

+export GOPATH=$(mktemp -d)
+trap 'rm -rf "${GOPATH}"' TERM INT EXIT QUIT
+go get -u github.com/Debian/debiman/cmd/debiman-piuparts-distill
+export PATH=${GOPATH}/bin:$PATH

from distill_alternatives_log from your patch makes me uneasy… because, for a
start, how is the code authenticated?

Also, I don't expect debiman-piuparts-distill to change very often, do you?

> Sure. I’ll add some commentary, and I’m also happy to explain any details —
> just ask.
 
thanks, the code is much nicer to read now! :)

> It could continue living in github.com/Debian/debiman, or we could move it
> to piuparts. The advantage of keeping it in github.com/Debian/debiman is
> that we don’t need to duplicate code between different git repositories.

does the code need to live in github.com/Debian/debiman at all? One copy
in the piuparts repo should be enough, or?


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Tue, May 23, 2017 at 9:55 AM, Michael Stapelberg 
wrote:

>
>
> On Tue, May 23, 2017 at 3:34 AM, Holger Levsen 
> wrote:
>
>> Hi Michael,
>>
>> are you subscribed to this bug or do you need cc:s?
>>
>
> I’m not subscribed, so please keep me cc'ed :).
>
>
>>
>> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
>> > This is in relation to https://bugs.debian.org/850917, where we added
>> > code to log update-alternatives calls.
>> >
>> > Now that the calls are logged, we need to somehow make Debiman aware
>> > of the calls. To that end, I wrote a program which will read all
>> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
>> > file with the links created by the various binary packages.
>>
>> thanks for the nice summary and proposal!
>>
>> > Find the source of that program at
>> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effc
>> d85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go
>>
>> looks simple and clean enough!
>>
>> > I then ran the program like so on piuparts.debian.org:
>>
>> /me really likes that pejacevic is developer accessable! ;)
>>
>> > % debiman-piuparts-distill \
>> > -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
>> > -output=debiman/testing.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
>> >   -output=debiman/unstable.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
>> >   -output=debiman/experimental.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
>> >   -output=debiman/stable.json.gz
>>
>> ok, so only parsing 4 suites… how long did these commands run?
>>
>
> 294,21s user 212,64s system 30% cpu 27:36,09 total
>
>
>>
>> > I then copied debiman/*.json.gz over to manziarly.debian.org,
>> > re-generated the entire site, and slave alternative manpages now work
>> > as expected :).
>>
>> hehe, nice! got an example?
>>
>
> Sure:
> https://manpages.debian.org/unstable/postgresql-client-9.
> 6/pg_dump.1.en.html
> https://manpages.debian.org/unstable/vis/vi.1.en.html
>
>
>>
>> > Obviously, the above was a one-off manual effort, so now we should
>> > discuss how we’d like to set things up for a more permanent export.
>>
>> sure!
>>
>> > Here’s a strawman proposal. We could:
>> >
>> > 1. Add a crontab entry, which triggers a shell script, which will
>> >update debiman-piuparts-distill, then run it on the logs
>> >directories.
>>
>> sounds good, how often would you like to run it?
>>
>
> Once a day would be sufficient, I think.
>
>
>>
>> > 2. Store the resulting *.json.gz files in a directory called
>> >/srv/piuparts.debian.org/htdocs/for-manpages.d.o
>>
>> I don't really like the "for-manpages.d.o" directory name, but cannot
>> think
>> of a better one either, so…
>>
>> > 3. Download the files via HTTPS on manziarly.d.o.
>>
>> great, so only step 1+2 are on the piuparts side :)
>>
>> > Step ① requires installing the Go compiler
>> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
>> > could avoid this if we’re all okay with just copying the
>> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to
>> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
>> > the piuparts git repository feels somewhat unclean.
>>
>> nah, installing golang-go from jessie or jessie-backports is easy, it
>> just needs a patch for git.debian.org/git/mirror/debian.org.git for the
>> files
>> debian/control and debian/changelog and a mail to admin@rt.d.o with the
>> subject "Debian RT: .*" to generate a ticket… which golang-go version do
>> you want?
>>
>
> Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> from jessie-backports, please :)
>
>
>>
>> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
>> > a patch.
>>
>> Yup, seems very fine to me! Please do!
>>
>
> Great. Find a patch attached. Note that it’s untested, as golang-go is not
> yet installed and I lack permission to run as piupartsm. I’m not sure what
> your development process is for changes like this, but feel free to fix up
> the patch as necessary.
>
>
>>
>> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in
>> main()
>> and explaining the purpose of the various functions ;-) I was just giving
>> it
>> a 2nd look after wanting to ask you to explain it/go-basics to me the next
>> time we met but then thought I should first try to understand it myself
>> ;-)
>> And then I noticed some more comments explaining/introducing the purpose
>> of the following code lines would be nice…
>>
>
> Sure. I’ll add some commentary, and I’m also happy to explain any details
> — just ask.
>

Commentary added:
https://github.com/Debian/debiman/commit/8306cf79924d18db55a1bc0123d63ac98ad2f2c4

Let me know if anything else deserves an explanation.


>
>
>>
>> 

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Tue, May 23, 2017 at 3:34 AM, Holger Levsen 
wrote:

> Hi Michael,
>
> are you subscribed to this bug or do you need cc:s?
>

I’m not subscribed, so please keep me cc'ed :).


>
> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
> > This is in relation to https://bugs.debian.org/850917, where we added
> > code to log update-alternatives calls.
> >
> > Now that the calls are logged, we need to somehow make Debiman aware
> > of the calls. To that end, I wrote a program which will read all
> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
> > file with the links created by the various binary packages.
>
> thanks for the nice summary and proposal!
>
> > Find the source of that program at
> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541
> ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go
>
> looks simple and clean enough!
>
> > I then ran the program like so on piuparts.debian.org:
>
> /me really likes that pejacevic is developer accessable! ;)
>
> > % debiman-piuparts-distill \
> > -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
> > -output=debiman/testing.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
> >   -output=debiman/unstable.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
> >   -output=debiman/experimental.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
> >   -output=debiman/stable.json.gz
>
> ok, so only parsing 4 suites… how long did these commands run?
>

294,21s user 212,64s system 30% cpu 27:36,09 total


>
> > I then copied debiman/*.json.gz over to manziarly.debian.org,
> > re-generated the entire site, and slave alternative manpages now work
> > as expected :).
>
> hehe, nice! got an example?
>

Sure:
https://manpages.debian.org/unstable/postgresql-client-9.6/pg_dump.1.en.html
https://manpages.debian.org/unstable/vis/vi.1.en.html


>
> > Obviously, the above was a one-off manual effort, so now we should
> > discuss how we’d like to set things up for a more permanent export.
>
> sure!
>
> > Here’s a strawman proposal. We could:
> >
> > 1. Add a crontab entry, which triggers a shell script, which will
> >update debiman-piuparts-distill, then run it on the logs
> >directories.
>
> sounds good, how often would you like to run it?
>

Once a day would be sufficient, I think.


>
> > 2. Store the resulting *.json.gz files in a directory called
> >/srv/piuparts.debian.org/htdocs/for-manpages.d.o
>
> I don't really like the "for-manpages.d.o" directory name, but cannot think
> of a better one either, so…
>
> > 3. Download the files via HTTPS on manziarly.d.o.
>
> great, so only step 1+2 are on the piuparts side :)
>
> > Step ① requires installing the Go compiler
> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
> > could avoid this if we’re all okay with just copying the
> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to
> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
> > the piuparts git repository feels somewhat unclean.
>
> nah, installing golang-go from jessie or jessie-backports is easy, it
> just needs a patch for git.debian.org/git/mirror/debian.org.git for the
> files
> debian/control and debian/changelog and a mail to admin@rt.d.o with the
> subject "Debian RT: .*" to generate a ticket… which golang-go version do
> you want?
>

Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
from jessie-backports, please :)


>
> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
> > a patch.
>
> Yup, seems very fine to me! Please do!
>

Great. Find a patch attached. Note that it’s untested, as golang-go is not
yet installed and I lack permission to run as piupartsm. I’m not sure what
your development process is for changes like this, but feel free to fix up
the patch as necessary.


>
> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in
> main()
> and explaining the purpose of the various functions ;-) I was just giving
> it
> a 2nd look after wanting to ask you to explain it/go-basics to me the next
> time we met but then thought I should first try to understand it myself ;-)
> And then I noticed some more comments explaining/introducing the purpose
> of the following code lines would be nice…
>

Sure. I’ll add some commentary, and I’m also happy to explain any details —
just ask.


>
> I suppose cmd/debiman-piuparts-distill/piuparts.go should live in
> master-bin
> in piuparts.git or what would you propose? I don't think we wanna compile
> it
> when building the piuparts-master package (for the Debian archive) but
> rather
> only build it when installing on pejacevic…
>

It could continue living in github.com/Debian/debiman, or we could move it
to piuparts. The advantage of keeping it in 

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

Hi Michael,

are you subscribed to this bug or do you need cc:s?

On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
> This is in relation to https://bugs.debian.org/850917, where we added
> code to log update-alternatives calls.
> 
> Now that the calls are logged, we need to somehow make Debiman aware
> of the calls. To that end, I wrote a program which will read all
> available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
> file with the links created by the various binary packages.
 
thanks for the nice summary and proposal!

> Find the source of that program at
> https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go

looks simple and clean enough!
 
> I then ran the program like so on piuparts.debian.org:

/me really likes that pejacevic is developer accessable! ;)

> % debiman-piuparts-distill \
> -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
> -output=debiman/testing.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
>   -output=debiman/unstable.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
>   -output=debiman/experimental.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
>   -output=debiman/stable.json.gz

ok, so only parsing 4 suites… how long did these commands run?

> I then copied debiman/*.json.gz over to manziarly.debian.org,
> re-generated the entire site, and slave alternative manpages now work
> as expected :).

hehe, nice! got an example?

> Obviously, the above was a one-off manual effort, so now we should
> discuss how we’d like to set things up for a more permanent export.

sure!

> Here’s a strawman proposal. We could:
> 
> 1. Add a crontab entry, which triggers a shell script, which will
>update debiman-piuparts-distill, then run it on the logs
>directories.

sounds good, how often would you like to run it?

> 2. Store the resulting *.json.gz files in a directory called
>/srv/piuparts.debian.org/htdocs/for-manpages.d.o

I don't really like the "for-manpages.d.o" directory name, but cannot think
of a better one either, so…

> 3. Download the files via HTTPS on manziarly.d.o.

great, so only step 1+2 are on the piuparts side :)

> Step ① requires installing the Go compiler
> (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
> could avoid this if we’re all okay with just copying the
> debiman-piuparts-distill binary onto piuparts.d.o, but referring to
> e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
> the piuparts git repository feels somewhat unclean.

nah, installing golang-go from jessie or jessie-backports is easy, it
just needs a patch for git.debian.org/git/mirror/debian.org.git for the files
debian/control and debian/changelog and a mail to admin@rt.d.o with the 
subject "Debian RT: .*" to generate a ticket… which golang-go version do
you want?

> Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
> a patch.

Yup, seems very fine to me! Please do!

cmd/debiman-piuparts-distill/piuparts.go also could use a comments in main()
and explaining the purpose of the various functions ;-) I was just giving it 
a 2nd look after wanting to ask you to explain it/go-basics to me the next
time we met but then thought I should first try to understand it myself ;-) 
And then I noticed some more comments explaining/introducing the purpose
of the following code lines would be nice…

I suppose cmd/debiman-piuparts-distill/piuparts.go should live in master-bin
in piuparts.git or what would you propose? I don't think we wanna compile it
when building the piuparts-master package (for the Debian archive) but rather
only build it when installing on pejacevic…


-- 
cheers,
Holger


signature.asc
Description: Digital signature