Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Sat, Aug 05, 2017 at 08:17:26PM +0200, Michael Stapelberg wrote:
> All done. Find attached the updated patch series (I hope that was what we
> needed, I’m losing track over what is already merged and what isn’t).

those all seems to be merged.

still missing: a fix for the Makefile so that your golang stuff aint tried
to be build on the slaves…


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

All done. Find attached the updated patch series (I hope that was what we
needed, I’m losing track over what is already merged and what isn’t).

Thanks!

On Thu, Aug 3, 2017 at 7:09 PM, Holger Levsen  wrote:

> On Thu, Aug 03, 2017 at 03:35:48PM +0200, Andreas Beckmann wrote:
> > > Hmm, so the issue is that /srv/piuparts.debian.org/sbin is not in the
> PATH,
> > > hence it doesn’t find the binary. Looking at
> /etc/piuparts/piuparts.conf, I
> > > can’t find a config variable which points to sbin. Do we need to
> introduce
> > > one, or should we use some other approach entirely?
> >
> > Just install the binary into the master binary directory
> > $(DESTDIR)$(sharedir)/piuparts/master/
> > That is in the path for master's cronjob.
>
> yup, indeed. Michael, please provice a patch which does that.
>
> > BTW, why sbin?
>
> good catch! should be bin indeed.
>
> > >> Finally, please do provide a debian/changelog entry for your work, you
> > >> can much better describe it than I and I'm lazy ;-)
> > Copyright+License statements are missing as well in the new files.
>
> indeed. Michael, please! :)
>
> Thanks!
>
>
> --
> cheers,
> Holger
>



-- 
Best regards,
Michael
From 000720e089dbe0ae731a1f2dba139d1290146f31 Mon Sep 17 00:00:00 2001
From: Michael Stapelberg 
Date: Thu, 1 Jun 2017 21:28:15 +0200
Subject: [PATCH] add debiman-piuparts-distill

see https://bugs.debian.org/863089
---
 Makefile   |   2 +
 debian/control |   5 +-
 debian/copyright   |  20 
 debian/piuparts-master.install |   1 +
 debiman-piuparts-distill/LICENSE   | 201 +
 debiman-piuparts-distill/atomically.go |  75 
 debiman-piuparts-distill/piuparts.go   | 145 
 7 files changed, 447 insertions(+), 2 deletions(-)
 create mode 100644 debiman-piuparts-distill/LICENSE
 create mode 100644 debiman-piuparts-distill/atomically.go
 create mode 100644 debiman-piuparts-distill/piuparts.go

diff --git a/Makefile b/Makefile
index 6bf7f49c..15ded366 100644
--- a/Makefile
+++ b/Makefile
@@ -51,6 +51,7 @@ build: build-stamp
 build-stamp: $(SCRIPTS_GENERATED) $(DOCS_GENERATED) Makefile
 	$(MAKE) -C instances
 	$(MAKE) python-syntax-check
+	(cd debiman-piuparts-distill && go build)
 	touch $@
 
 build-doc: $(DOCS_GENERATED)
@@ -110,6 +111,7 @@ install-conf-4-running-from-git: build-stamp
 install: build-stamp
 	install -d $(DESTDIR)$(sbindir)
 	install -m 0755 piuparts $(DESTDIR)$(sbindir)/
+	install -m 0755 debiman-piuparts-distill/debiman-piuparts-distill $(DESTDIR)$(sharedir)/piuparts/master/
 
 	install -d $(DESTDIR)$(sharedir)/piuparts
 	install -m 0755 piuparts-slave piuparts-master piuparts-master-backend piuparts-report piuparts-analyze $(DESTDIR)$(sharedir)/piuparts/
diff --git a/debian/control b/debian/control
index f65fd2f4..d58a8226 100644
--- a/debian/control
+++ b/debian/control
@@ -16,7 +16,8 @@ Build-Depends:
  python-debianbts,
  python-yaml,
  python-mox3,
- python-lzma
+ python-lzma,
+ golang-any
 Build-Depends-Indep:
  asciidoc,
  git,
@@ -55,7 +56,7 @@ Description: .deb package installation, upgrading, and removal testing tool
  packages to test them before they upload them to the Debian package archive.
 
 Package: piuparts-master
-Architecture: all
+Architecture: any
 Depends:
  piuparts-common (= ${binary:Version}),
  adduser,
diff --git a/debian/copyright b/debian/copyright
index e430c2db..6639617e 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -49,3 +49,23 @@ License: FreeUse
  from W3C. When the logo usage policy requires that you obtain express written
  permission, please send your request to the W3C Communications Team
  (w3t...@w3.org)
+
+Files: debiman-piuparts-distill
+Copyright: 2017 Michael Stapelberg
+License: Apache-2.0
+
+License: Apache-2.0
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ .
+ On Debian systems, the complete text of the Apache version 2.0 license
+ can be found in "/usr/share/common-licenses/Apache-2.0".
\ No newline at end of file
diff --git a/debian/piuparts-master.install b/debian/piuparts-master.install
index 5e41fa93..c5cb0091 100644
--- a/debian/piuparts-master.install
+++ b/debian/piuparts-master.install
@@ -7,3 +7,4 @@ usr/share/piuparts/piuparts-analyze
 usr/share/piuparts/piuparts-master
 usr/share/piuparts/piuparts-master-backend
 usr/share/piuparts/piuparts-report

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Andreas Beckmann]

On 2017-08-03 19:08, Holger Levsen wrote:
>> And while we are at it, I'd like debsums (>= 2.2.2~) and
>> python-setproctitle as well.
> 
> DSA will very soon update pejacevic to stretch, so that should be fine.

debsums is needed on the slaves, python-setproctitle on the master (but
maybe on the slave as well at some point)


Andreas

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Thu, Aug 03, 2017 at 03:43:26PM +0200, Andreas Beckmann wrote:
> We need to update the metapackages (DSA repo somewhere) that pull in
> packages on the slaves as well.

no, this has already been done for pejacevic and golang is only needed on the
master node (pejaceviv). The issue here is that the install script tries 
to execute golang on the slaves as well. That needs to be fixed.

> And while we are at it, I'd like debsums (>= 2.2.2~) and
> python-setproctitle as well.

DSA will very soon update pejacevic to stretch, so that should be fine.

> Maybe we should add a binary package piuparts-from-git that pulls in th
> needed dependencies for running it from git, should make it more obvious
> what needs to be done in the DSA metapackages as well.

I think it's less work to just provide patches for debian-admin.git.


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Thu, Aug 03, 2017 at 03:35:48PM +0200, Andreas Beckmann wrote:
> > Hmm, so the issue is that /srv/piuparts.debian.org/sbin is not in the PATH,
> > hence it doesn’t find the binary. Looking at /etc/piuparts/piuparts.conf, I
> > can’t find a config variable which points to sbin. Do we need to introduce
> > one, or should we use some other approach entirely?
> 
> Just install the binary into the master binary directory
> $(DESTDIR)$(sharedir)/piuparts/master/
> That is in the path for master's cronjob.

yup, indeed. Michael, please provice a patch which does that.

> BTW, why sbin?

good catch! should be bin indeed.

> >> Finally, please do provide a debian/changelog entry for your work, you
> >> can much better describe it than I and I'm lazy ;-)
> Copyright+License statements are missing as well in the new files.

indeed. Michael, please! :)

Thanks!


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Andreas Beckmann]

On 2017-08-03 09:37, Michael Stapelberg wrote:
> But I already added golang-any to Build-Depends in my patch?
> 
> Also, I thought the update-piuparts-slave-setup script deploys directly
> from git? `go version` on pejacevic seems to work fine, so I’m not sure I
> follow what the issue is.

We need to update the metapackages (DSA repo somewhere) that pull in
packages on the slaves as well.
And while we are at it, I'd like debsums (>= 2.2.2~) and
python-setproctitle as well.

Maybe we should add a binary package piuparts-from-git that pulls in th
needed dependencies for running it from git, should make it more obvious
what needs to be done in the DSA metapackages as well.


Andreas

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Andreas Beckmann]

On 2017-08-03 09:35, Michael Stapelberg wrote:
>> piupartsm@pejacevic:~$ distill_alternatives_log
>> /home/piupartsm/bin/distill_alternatives_log: Zeile 10:
>> debiman-piuparts-distill: Kommando nicht gefunden.
>> piupartsm@pejacevic:~$
>>
>> I guess you'll need to fix this up…
>>
> 
> Hmm, so the issue is that /srv/piuparts.debian.org/sbin is not in the PATH,
> hence it doesn’t find the binary. Looking at /etc/piuparts/piuparts.conf, I
> can’t find a config variable which points to sbin. Do we need to introduce
> one, or should we use some other approach entirely?

Just install the binary into the master binary directory
$(DESTDIR)$(sharedir)/piuparts/master/
That is in the path for master's cronjob.
BTW, why sbin?

>> Finally, please do provide a debian/changelog entry for your work, you
>> can much better describe it than I and I'm lazy ;-)

Copyright+License statements are missing as well in the new files.


Andreas

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

But I already added golang-any to Build-Depends in my patch?

Also, I thought the update-piuparts-slave-setup script deploys directly
from git? `go version` on pejacevic seems to work fine, so I’m not sure I
follow what the issue is.

On Thu, Aug 3, 2017 at 6:09 AM, Holger Levsen  wrote:

> Hey,
>
> adding go-lang any to the build-depends also
> https://jenkins.debian.net/job/piuparts_testsuite_jessie/266/
> not yet sure how to fix this, probably maybe by making it depend on
> golang-any >= $bpo_version, so that it's obvious what needs to be
> done - both on jenkins and when backporting piuparts.
>
> (just a bit too tired to decide this now…)
>
>
> --
> cheers,
> Holger
>



-- 
Best regards,
Michael

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Thu, Aug 3, 2017 at 5:42 AM, Holger Levsen  wrote:

> On Tue, Aug 01, 2017 at 11:30:11AM +0200, Michael Stapelberg wrote:
> > Sure: https://github.com/stapelberg/piuparts/commits/distill
>
> cool, thanks, merged and deployed on pejacevic!
>

Thanks!


>
> > As previously, this is untested, so just let me know if any changes are
> > required, or — probably faster — just perform the changes yourself if you
> > want :).
>
> piupartsm@pejacevic:~$ distill_alternatives_log
> /home/piupartsm/bin/distill_alternatives_log: Zeile 10:
> debiman-piuparts-distill: Kommando nicht gefunden.
> piupartsm@pejacevic:~$
>
> I guess you'll need to fix this up…
>

Hmm, so the issue is that /srv/piuparts.debian.org/sbin is not in the PATH,
hence it doesn’t find the binary. Looking at /etc/piuparts/piuparts.conf, I
can’t find a config variable which points to sbin. Do we need to introduce
one, or should we use some other approach entirely?


>
> I've deployed this using
>
> sudo -u piupartsm -i /srv/piuparts.debian.org/src/
> piuparts/update-piuparts-master-setup develop origin
>
>
> Finally, please do provide a debian/changelog entry for your work, you
> can much better describe it than I and I'm lazy ;-)
>
>
> --
> cheers,
> Holger
>



-- 
Best regards,
Michael

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

Hey,

adding go-lang any to the build-depends also 
https://jenkins.debian.net/job/piuparts_testsuite_jessie/266/
not yet sure how to fix this, probably maybe by making it depend on
golang-any >= $bpo_version, so that it's obvious what needs to be
done - both on jenkins and when backporting piuparts.

(just a bit too tired to decide this now…)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

Hi again,

your change broke deployment on the slaves, please provide a fix:


holger@piu-slave-ubc-01:~$ sudo -u piupartss -i 
/srv/piuparts.debian.org/src/piuparts/update-piuparts-slave-setup develop origin
[...]
make python-syntax-check
make[1]: Entering directory '/srv/piuparts.debian.org/src/piuparts'
+ python -m py_compile piuparts.py
+ python -m py_compile piuparts-analyze.py
+ python -m py_compile piuparts-master-backend.py
+ python -m py_compile piuparts-slave.py
+ python -m py_compile piuparts-report.py
+ python -m py_compile piupartslib/dependencyparser.py
+ python -m py_compile piupartslib/__init__.py
+ python -m py_compile piupartslib/packagesdb.py
+ python -m py_compile piupartslib/conf.py
+ python -m py_compile piupartslib/pkgsummary.py
+ python -m py_compile piupartslib/dwke.py
+ python -m py_compile master-bin/detect_well_known_errors.py
rm -f piuparts.pyc piuparts-analyze.pyc piuparts-master-backend.pyc 
piuparts-slave.pyc piuparts-report.pyc piupartslib/dependencyparser.pyc 
piupartslib/__init__.pyc piupartslib/packagesdb.pyc piupartslib/conf.pyc 
piupartslib/pkgsummary.pyc piupartslib/dwke.pyc 
master-bin/detect_well_known_errors.pyc
make[1]: Leaving directory '/srv/piuparts.debian.org/src/piuparts'
(cd debiman-piuparts-distill && go build)
/bin/sh: 1: go: not found
Makefile:52: recipe for target 'build-stamp' failed
make: *** [build-stamp] Error 127


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Tue, Aug 01, 2017 at 11:30:11AM +0200, Michael Stapelberg wrote:
> Sure: https://github.com/stapelberg/piuparts/commits/distill

cool, thanks, merged and deployed on pejacevic!
 
> As previously, this is untested, so just let me know if any changes are
> required, or — probably faster — just perform the changes yourself if you
> want :).

piupartsm@pejacevic:~$ distill_alternatives_log 
/home/piupartsm/bin/distill_alternatives_log: Zeile 10: 
debiman-piuparts-distill: Kommando nicht gefunden.
piupartsm@pejacevic:~$ 

I guess you'll need to fix this up…

I've deployed this using 

sudo -u piupartsm -i 
/srv/piuparts.debian.org/src/piuparts/update-piuparts-master-setup develop 
origin


Finally, please do provide a debian/changelog entry for your work, you 
can much better describe it than I and I'm lazy ;-)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

Sure: https://github.com/stapelberg/piuparts/commits/distill

As previously, this is untested, so just let me know if any changes are
required, or — probably faster — just perform the changes yourself if you
want :).

Thanks!

On Thu, Jul 20, 2017 at 11:25 PM, Holger Levsen 
wrote:

> Hi Michael,
>
> sorry for the delay here…
>
> On Thu, Jun 01, 2017 at 09:30:50PM +0200, Michael Stapelberg wrote:
> > Alright, then. Find attached a patch against the piuparts git to add
> > debiman-piuparts-distill. You can build it by running “go build” in the
> > debiman-piuparts-distill subdirectory.
>
> cool!
>
> can you please also a.) patch the Makefile, update-piuparts-master-setup
> and
> debian/piuparts-master.install in a meaningful way and b.) push this to
> your
> piuparts.git repo?
>
> Thanks already!
>
>
> --
> cheers,
> Holger
>



-- 
Best regards,
Michael

Bug#863089: [Piuparts-devel] Bug#863089: Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

Hi Michael, 

sorry for the delay here…

On Thu, Jun 01, 2017 at 09:30:50PM +0200, Michael Stapelberg wrote:
> Alright, then. Find attached a patch against the piuparts git to add
> debiman-piuparts-distill. You can build it by running “go build” in the
> debiman-piuparts-distill subdirectory.

cool!

can you please also a.) patch the Makefile, update-piuparts-master-setup and
debian/piuparts-master.install in a meaningful way and b.) push this to your
piuparts.git repo?

Thanks already!


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

Alright, then. Find attached a patch against the piuparts git to add
debiman-piuparts-distill. You can build it by running “go build” in the
debiman-piuparts-distill subdirectory.

On Wed, May 31, 2017 at 11:47 AM, Holger Levsen 
wrote:

> On Wed, May 31, 2017 at 11:32:46AM +0200, Michael Stapelberg wrote:
> > The fact that these manpages are included at all is what the
> > slave-alternative handling got us :)
>
> ah!
>
> > > ok, I'll request this once we got closer… (see below)
> > Thanks :).
>
> done (as you know, but for the record of this bug…)
>
> > The code is authenticated via HTTPS. If you don’t trust GitHub and/or the
> > TLS certificate infrastructure, then we should find an alternative.
>
> I dont trust GitHub, or rather, it's bad enough trusting alioth, I dont
> want
> to also trust Github… (and TLS, no…)
>
> (we should probably rather trust signed tags, but anyway, as we dont have
> that I would prefer to only trust one repo…)
>
> > > Also, I don't expect debiman-piuparts-distill to change very often, do
> you?
> > Agreed, most likely it will not change very often.
>
> ok, cool.
>
> > > does the code need to live in github.com/Debian/debiman at all? One
> copy
> > > in the piuparts repo should be enough, or?
> > That’s correct; I was referring to code that is shared between different
> > debiman components. Specifically,
> > https://github.com/Debian/debiman/blob/54dd6050a6ce8a454c14e172a8687d
> 93d0fd241b/internal/write/atomically.go
>
> ic
>
> > But, if the conclusion is that debiman-piuparts-distill should live in
> the
> > piuparts repo, we can duplicate that code. I don’t expect it to change
> > either.
>
> I think I would really prefer the code to live in the piuparts repo.
>
> Or alternativly, life in your repo and then we deploy it on demand, but not
> everyday. Having it in the piuparts repo would be easier for deployments,
> but
> I can see how its a bit more hassle for you to maintain the code in another
> repo. But then, I know how to be very liberal to accept your code changes
> to
> your code in our piuparts repo :)
>
> Patches welcome! :-D
>
>
> --
> cheers,
> Holger
>



-- 
Best regards,
Michael
From 7001c015ef65f548a0baf5f5488a05d91f2a3a64 Mon Sep 17 00:00:00 2001
From: Michael Stapelberg 
Date: Thu, 1 Jun 2017 21:28:15 +0200
Subject: [PATCH] add debiman-piuparts-distill

see https://bugs.debian.org/863089
---
 debiman-piuparts-distill/atomically.go |  75 +
 debiman-piuparts-distill/piuparts.go   | 145 +
 2 files changed, 220 insertions(+)
 create mode 100644 debiman-piuparts-distill/atomically.go
 create mode 100644 debiman-piuparts-distill/piuparts.go

diff --git a/debiman-piuparts-distill/atomically.go b/debiman-piuparts-distill/atomically.go
new file mode 100644
index ..c1b2d2db
--- /dev/null
+++ b/debiman-piuparts-distill/atomically.go
@@ -0,0 +1,75 @@
+package main
+
+import (
+	"bufio"
+	"compress/gzip"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+)
+
+func tempDir(dest string) string {
+	tempdir := os.Getenv("TMPDIR")
+	if tempdir == "" {
+		// Convenient for development: decreases the chance that we
+		// cannot move files due to /tmp being on a different file
+		// system.
+		tempdir = filepath.Dir(dest)
+	}
+	return tempdir
+}
+
+func writeAtomically(dest string, compress bool, write func(w io.Writer) error) (err error) {
+	f, err := ioutil.TempFile(tempDir(dest), "debiman-")
+	if err != nil {
+		return err
+	}
+	defer func() {
+		// Remove the tempfile if an error occurred
+		if err != nil {
+			os.Remove(f.Name())
+		}
+	}()
+	defer f.Close()
+
+	bufw := bufio.NewWriter(f)
+
+	w := io.Writer(bufw)
+	var gzipw *gzip.Writer
+	if compress {
+		// NOTE(stapelberg): gzip’s decompression phase takes the same
+		// time, regardless of compression level. Hence, we invest the
+		// maximum CPU time once to achieve the best compression.
+		gzipw, err = gzip.NewWriterLevel(bufw, gzip.BestCompression)
+		if err != nil {
+			return err
+		}
+		defer gzipw.Close()
+		w = gzipw
+	}
+
+	if err := write(w); err != nil {
+		return err
+	}
+
+	if compress {
+		if err := gzipw.Close(); err != nil {
+			return err
+		}
+	}
+
+	if err := bufw.Flush(); err != nil {
+		return err
+	}
+
+	if err := f.Chmod(0644); err != nil {
+		return err
+	}
+
+	if err := f.Close(); err != nil {
+		return err
+	}
+
+	return os.Rename(f.Name(), dest)
+}
diff --git a/debiman-piuparts-distill/piuparts.go b/debiman-piuparts-distill/piuparts.go
new file mode 100644
index ..73f773dc
--- /dev/null
+++ b/debiman-piuparts-distill/piuparts.go
@@ -0,0 +1,145 @@
+// debiman-piuparts-distill extracts slave alternative links from
+// LOG-ALTERNATIVES lines found in piuparts logs.
+//
+// See https://github.com/Debian/debiman/issues/12 for more details.
+package main
+
+import (
+	"bufio"
+	"encoding/json"
+	"flag"
+	"io"
+	"log"
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"sync"
+)

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Wed, May 31, 2017 at 11:32:46AM +0200, Michael Stapelberg wrote:
> The fact that these manpages are included at all is what the
> slave-alternative handling got us :)
 
ah!

> > ok, I'll request this once we got closer… (see below)
> Thanks :).
 
done (as you know, but for the record of this bug…)

> The code is authenticated via HTTPS. If you don’t trust GitHub and/or the
> TLS certificate infrastructure, then we should find an alternative.
 
I dont trust GitHub, or rather, it's bad enough trusting alioth, I dont want
to also trust Github… (and TLS, no…) 

(we should probably rather trust signed tags, but anyway, as we dont have
that I would prefer to only trust one repo…)

> > Also, I don't expect debiman-piuparts-distill to change very often, do you?
> Agreed, most likely it will not change very often.
 
ok, cool.

> > does the code need to live in github.com/Debian/debiman at all? One copy
> > in the piuparts repo should be enough, or?
> That’s correct; I was referring to code that is shared between different
> debiman components. Specifically,
> https://github.com/Debian/debiman/blob/54dd6050a6ce8a454c14e172a8687d93d0fd241b/internal/write/atomically.go

ic

> But, if the conclusion is that debiman-piuparts-distill should live in the
> piuparts repo, we can duplicate that code. I don’t expect it to change
> either.

I think I would really prefer the code to live in the piuparts repo.

Or alternativly, life in your repo and then we deploy it on demand, but not
everyday. Having it in the piuparts repo would be easier for deployments, but
I can see how its a bit more hassle for you to maintain the code in another
repo. But then, I know how to be very liberal to accept your code changes to
your code in our piuparts repo :)

Patches welcome! :-D


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Tue, May 30, 2017 at 4:43 PM, Holger Levsen 
wrote:

> On Tue, May 23, 2017 at 09:55:11AM +0200, Michael Stapelberg wrote:
> > I’m not subscribed, so please keep me cc'ed :).
>
> ok!
>
> > > ok, so only parsing 4 suites… how long did these commands run?
> > 294,21s user 212,64s system 30% cpu 27:36,09 total
>
> ok, so minutes rather than hours…
>
> > > hehe, nice! got an example?
> > Sure:
> > https://manpages.debian.org/unstable/postgresql-client-9.
> 6/pg_dump.1.en.html
> > https://manpages.debian.org/unstable/vis/vi.1.en.html
>
> those are the links to other version, languages and conflicting commands?
>

The fact that these manpages are included at all is what the
slave-alternative handling got us :)


>
> > > sounds good, how often would you like to run it?
> > Once a day would be sufficient, I think.
>
> cool.
>
> > Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> > from jessie-backports, please :)
>
> ok, I'll request this once we got closer… (see below)
>

Thanks :).


>
> > Great. Find a patch attached. Note that it’s untested, as golang-go is
> not
> > yet installed and I lack permission to run as piupartsm. I’m not sure
> what
> > your development process is for changes like this, but feel free to fix
> up
> > the patch as necessary.
>
> process wise it's fine to just install it and then fix it up…
>
> but…
>
> +export GOPATH=$(mktemp -d)
> +trap 'rm -rf "${GOPATH}"' TERM INT EXIT QUIT
> +go get -u github.com/Debian/debiman/cmd/debiman-piuparts-distill
> +export PATH=${GOPATH}/bin:$PATH
>
> from distill_alternatives_log from your patch makes me uneasy… because,
> for a
> start, how is the code authenticated?
>

The code is authenticated via HTTPS. If you don’t trust GitHub and/or the
TLS certificate infrastructure, then we should find an alternative.


>
> Also, I don't expect debiman-piuparts-distill to change very often, do you?
>

Agreed, most likely it will not change very often.


>
> > Sure. I’ll add some commentary, and I’m also happy to explain any
> details —
> > just ask.
>
> thanks, the code is much nicer to read now! :)
>
> > It could continue living in github.com/Debian/debiman, or we could move
> it
> > to piuparts. The advantage of keeping it in github.com/Debian/debiman is
> > that we don’t need to duplicate code between different git repositories.
>
> does the code need to live in github.com/Debian/debiman at all? One copy
> in the piuparts repo should be enough, or?


That’s correct; I was referring to code that is shared between different
debiman components. Specifically,
https://github.com/Debian/debiman/blob/54dd6050a6ce8a454c14e172a8687d93d0fd241b/internal/write/atomically.go
.

But, if the conclusion is that debiman-piuparts-distill should live in the
piuparts repo, we can duplicate that code. I don’t expect it to change
either.

-- 
Best regards,
Michael

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

On Tue, May 23, 2017 at 09:55:11AM +0200, Michael Stapelberg wrote:
> I’m not subscribed, so please keep me cc'ed :).
 
ok!

> > ok, so only parsing 4 suites… how long did these commands run?
> 294,21s user 212,64s system 30% cpu 27:36,09 total
 
ok, so minutes rather than hours… 

> > hehe, nice! got an example?
> Sure:
> https://manpages.debian.org/unstable/postgresql-client-9.6/pg_dump.1.en.html
> https://manpages.debian.org/unstable/vis/vi.1.en.html
 
those are the links to other version, languages and conflicting commands?

> > sounds good, how often would you like to run it?
> Once a day would be sufficient, I think.
 
cool.

> Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> from jessie-backports, please :)
 
ok, I'll request this once we got closer… (see below)

> Great. Find a patch attached. Note that it’s untested, as golang-go is not
> yet installed and I lack permission to run as piupartsm. I’m not sure what
> your development process is for changes like this, but feel free to fix up
> the patch as necessary.
 
process wise it's fine to just install it and then fix it up…

but… 

+export GOPATH=$(mktemp -d)
+trap 'rm -rf "${GOPATH}"' TERM INT EXIT QUIT
+go get -u github.com/Debian/debiman/cmd/debiman-piuparts-distill
+export PATH=${GOPATH}/bin:$PATH

from distill_alternatives_log from your patch makes me uneasy… because, for a
start, how is the code authenticated?

Also, I don't expect debiman-piuparts-distill to change very often, do you?

> Sure. I’ll add some commentary, and I’m also happy to explain any details —
> just ask.
 
thanks, the code is much nicer to read now! :)

> It could continue living in github.com/Debian/debiman, or we could move it
> to piuparts. The advantage of keeping it in github.com/Debian/debiman is
> that we don’t need to duplicate code between different git repositories.

does the code need to live in github.com/Debian/debiman at all? One copy
in the piuparts repo should be enough, or?


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Tue, May 23, 2017 at 9:55 AM, Michael Stapelberg 
wrote:

>
>
> On Tue, May 23, 2017 at 3:34 AM, Holger Levsen 
> wrote:
>
>> Hi Michael,
>>
>> are you subscribed to this bug or do you need cc:s?
>>
>
> I’m not subscribed, so please keep me cc'ed :).
>
>
>>
>> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
>> > This is in relation to https://bugs.debian.org/850917, where we added
>> > code to log update-alternatives calls.
>> >
>> > Now that the calls are logged, we need to somehow make Debiman aware
>> > of the calls. To that end, I wrote a program which will read all
>> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
>> > file with the links created by the various binary packages.
>>
>> thanks for the nice summary and proposal!
>>
>> > Find the source of that program at
>> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effc
>> d85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go
>>
>> looks simple and clean enough!
>>
>> > I then ran the program like so on piuparts.debian.org:
>>
>> /me really likes that pejacevic is developer accessable! ;)
>>
>> > % debiman-piuparts-distill \
>> > -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
>> > -output=debiman/testing.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
>> >   -output=debiman/unstable.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
>> >   -output=debiman/experimental.json.gz
>> >
>> > % debiman-piuparts-distill \
>> >   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
>> >   -output=debiman/stable.json.gz
>>
>> ok, so only parsing 4 suites… how long did these commands run?
>>
>
> 294,21s user 212,64s system 30% cpu 27:36,09 total
>
>
>>
>> > I then copied debiman/*.json.gz over to manziarly.debian.org,
>> > re-generated the entire site, and slave alternative manpages now work
>> > as expected :).
>>
>> hehe, nice! got an example?
>>
>
> Sure:
> https://manpages.debian.org/unstable/postgresql-client-9.
> 6/pg_dump.1.en.html
> https://manpages.debian.org/unstable/vis/vi.1.en.html
>
>
>>
>> > Obviously, the above was a one-off manual effort, so now we should
>> > discuss how we’d like to set things up for a more permanent export.
>>
>> sure!
>>
>> > Here’s a strawman proposal. We could:
>> >
>> > 1. Add a crontab entry, which triggers a shell script, which will
>> >update debiman-piuparts-distill, then run it on the logs
>> >directories.
>>
>> sounds good, how often would you like to run it?
>>
>
> Once a day would be sufficient, I think.
>
>
>>
>> > 2. Store the resulting *.json.gz files in a directory called
>> >/srv/piuparts.debian.org/htdocs/for-manpages.d.o
>>
>> I don't really like the "for-manpages.d.o" directory name, but cannot
>> think
>> of a better one either, so…
>>
>> > 3. Download the files via HTTPS on manziarly.d.o.
>>
>> great, so only step 1+2 are on the piuparts side :)
>>
>> > Step ① requires installing the Go compiler
>> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
>> > could avoid this if we’re all okay with just copying the
>> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to
>> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
>> > the piuparts git repository feels somewhat unclean.
>>
>> nah, installing golang-go from jessie or jessie-backports is easy, it
>> just needs a patch for git.debian.org/git/mirror/debian.org.git for the
>> files
>> debian/control and debian/changelog and a mail to admin@rt.d.o with the
>> subject "Debian RT: .*" to generate a ticket… which golang-go version do
>> you want?
>>
>
> Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
> from jessie-backports, please :)
>
>
>>
>> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
>> > a patch.
>>
>> Yup, seems very fine to me! Please do!
>>
>
> Great. Find a patch attached. Note that it’s untested, as golang-go is not
> yet installed and I lack permission to run as piupartsm. I’m not sure what
> your development process is for changes like this, but feel free to fix up
> the patch as necessary.
>
>
>>
>> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in
>> main()
>> and explaining the purpose of the various functions ;-) I was just giving
>> it
>> a 2nd look after wanting to ask you to explain it/go-basics to me the next
>> time we met but then thought I should first try to understand it myself
>> ;-)
>> And then I noticed some more comments explaining/introducing the purpose
>> of the following code lines would be nice…
>>
>
> Sure. I’ll add some commentary, and I’m also happy to explain any details
> — just ask.
>

Commentary added:
https://github.com/Debian/debiman/commit/8306cf79924d18db55a1bc0123d63ac98ad2f2c4

Let me know if anything else deserves an explanation.


>
>
>>
>> 

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

On Tue, May 23, 2017 at 3:34 AM, Holger Levsen 
wrote:

> Hi Michael,
>
> are you subscribed to this bug or do you need cc:s?
>

I’m not subscribed, so please keep me cc'ed :).


>
> On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
> > This is in relation to https://bugs.debian.org/850917, where we added
> > code to log update-alternatives calls.
> >
> > Now that the calls are logged, we need to somehow make Debiman aware
> > of the calls. To that end, I wrote a program which will read all
> > available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
> > file with the links created by the various binary packages.
>
> thanks for the nice summary and proposal!
>
> > Find the source of that program at
> > https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541
> ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go
>
> looks simple and clean enough!
>
> > I then ran the program like so on piuparts.debian.org:
>
> /me really likes that pejacevic is developer accessable! ;)
>
> > % debiman-piuparts-distill \
> > -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
> > -output=debiman/testing.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
> >   -output=debiman/unstable.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
> >   -output=debiman/experimental.json.gz
> >
> > % debiman-piuparts-distill \
> >   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
> >   -output=debiman/stable.json.gz
>
> ok, so only parsing 4 suites… how long did these commands run?
>

294,21s user 212,64s system 30% cpu 27:36,09 total


>
> > I then copied debiman/*.json.gz over to manziarly.debian.org,
> > re-generated the entire site, and slave alternative manpages now work
> > as expected :).
>
> hehe, nice! got an example?
>

Sure:
https://manpages.debian.org/unstable/postgresql-client-9.6/pg_dump.1.en.html
https://manpages.debian.org/unstable/vis/vi.1.en.html


>
> > Obviously, the above was a one-off manual effort, so now we should
> > discuss how we’d like to set things up for a more permanent export.
>
> sure!
>
> > Here’s a strawman proposal. We could:
> >
> > 1. Add a crontab entry, which triggers a shell script, which will
> >update debiman-piuparts-distill, then run it on the logs
> >directories.
>
> sounds good, how often would you like to run it?
>

Once a day would be sufficient, I think.


>
> > 2. Store the resulting *.json.gz files in a directory called
> >/srv/piuparts.debian.org/htdocs/for-manpages.d.o
>
> I don't really like the "for-manpages.d.o" directory name, but cannot think
> of a better one either, so…
>
> > 3. Download the files via HTTPS on manziarly.d.o.
>
> great, so only step 1+2 are on the piuparts side :)
>
> > Step ① requires installing the Go compiler
> > (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
> > could avoid this if we’re all okay with just copying the
> > debiman-piuparts-distill binary onto piuparts.d.o, but referring to
> > e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
> > the piuparts git repository feels somewhat unclean.
>
> nah, installing golang-go from jessie or jessie-backports is easy, it
> just needs a patch for git.debian.org/git/mirror/debian.org.git for the
> files
> debian/control and debian/changelog and a mail to admin@rt.d.o with the
> subject "Debian RT: .*" to generate a ticket… which golang-go version do
> you want?
>

Since piuparts.d.o is running on jessie, we’ll need to use golang-go 1.7
from jessie-backports, please :)


>
> > Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
> > a patch.
>
> Yup, seems very fine to me! Please do!
>

Great. Find a patch attached. Note that it’s untested, as golang-go is not
yet installed and I lack permission to run as piupartsm. I’m not sure what
your development process is for changes like this, but feel free to fix up
the patch as necessary.


>
> cmd/debiman-piuparts-distill/piuparts.go also could use a comments in
> main()
> and explaining the purpose of the various functions ;-) I was just giving
> it
> a 2nd look after wanting to ask you to explain it/go-basics to me the next
> time we met but then thought I should first try to understand it myself ;-)
> And then I noticed some more comments explaining/introducing the purpose
> of the following code lines would be nice…
>

Sure. I’ll add some commentary, and I’m also happy to explain any details —
just ask.


>
> I suppose cmd/debiman-piuparts-distill/piuparts.go should live in
> master-bin
> in piuparts.git or what would you propose? I don't think we wanna compile
> it
> when building the piuparts-master package (for the Debian archive) but
> rather
> only build it when installing on pejacevic…
>

It could continue living in github.com/Debian/debiman, or we could move it
to piuparts. The advantage of keeping it in 

Bug#863089: [Piuparts-devel] Bug#863089: Please provide post-processed logs output for manpages.d.o

[Holger Levsen]

Hi Michael,

are you subscribed to this bug or do you need cc:s?

On Sun, May 21, 2017 at 06:43:54PM +0200, Michael Stapelberg wrote:
> This is in relation to https://bugs.debian.org/850917, where we added
> code to log update-alternatives calls.
> 
> Now that the calls are logged, we need to somehow make Debiman aware
> of the calls. To that end, I wrote a program which will read all
> available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
> file with the links created by the various binary packages.
 
thanks for the nice summary and proposal!

> Find the source of that program at
> https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go

looks simple and clean enough!
 
> I then ran the program like so on piuparts.debian.org:

/me really likes that pejacevic is developer accessable! ;)

> % debiman-piuparts-distill \
> -logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
> -output=debiman/testing.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
>   -output=debiman/unstable.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
>   -output=debiman/experimental.json.gz
> 
> % debiman-piuparts-distill \
>   -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
>   -output=debiman/stable.json.gz

ok, so only parsing 4 suites… how long did these commands run?

> I then copied debiman/*.json.gz over to manziarly.debian.org,
> re-generated the entire site, and slave alternative manpages now work
> as expected :).

hehe, nice! got an example?

> Obviously, the above was a one-off manual effort, so now we should
> discuss how we’d like to set things up for a more permanent export.

sure!

> Here’s a strawman proposal. We could:
> 
> 1. Add a crontab entry, which triggers a shell script, which will
>update debiman-piuparts-distill, then run it on the logs
>directories.

sounds good, how often would you like to run it?

> 2. Store the resulting *.json.gz files in a directory called
>/srv/piuparts.debian.org/htdocs/for-manpages.d.o

I don't really like the "for-manpages.d.o" directory name, but cannot think
of a better one either, so…

> 3. Download the files via HTTPS on manziarly.d.o.

great, so only step 1+2 are on the piuparts side :)

> Step ① requires installing the Go compiler
> (https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
> could avoid this if we’re all okay with just copying the
> debiman-piuparts-distill binary onto piuparts.d.o, but referring to
> e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
> the piuparts git repository feels somewhat unclean.

nah, installing golang-go from jessie or jessie-backports is easy, it
just needs a patch for git.debian.org/git/mirror/debian.org.git for the files
debian/control and debian/changelog and a mail to admin@rt.d.o with the 
subject "Debian RT: .*" to generate a ticket… which golang-go version do
you want?

> Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
> a patch.

Yup, seems very fine to me! Please do!

cmd/debiman-piuparts-distill/piuparts.go also could use a comments in main()
and explaining the purpose of the various functions ;-) I was just giving it 
a 2nd look after wanting to ask you to explain it/go-basics to me the next
time we met but then thought I should first try to understand it myself ;-) 
And then I noticed some more comments explaining/introducing the purpose
of the following code lines would be nice…

I suppose cmd/debiman-piuparts-distill/piuparts.go should live in master-bin
in piuparts.git or what would you propose? I don't think we wanna compile it
when building the piuparts-master package (for the Debian archive) but rather
only build it when installing on pejacevic…


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#863089: Please provide post-processed logs output for manpages.d.o

[Michael Stapelberg]

Package: piuparts
Severity: wishlist

This is in relation to https://bugs.debian.org/850917, where we added
code to log update-alternatives calls.

Now that the calls are logged, we need to somehow make Debiman aware
of the calls. To that end, I wrote a program which will read all
available logs, parse all LOG-ALTERNATIVES lines, and generate a JSON
file with the links created by the various binary packages.

Find the source of that program at
https://github.com/Debian/debiman/blob/b4d79f84b34abfc59effcd85c78541ce8f572b4b/cmd/debiman-piuparts-distill/piuparts.go

I then ran the program like so on piuparts.debian.org:

% debiman-piuparts-distill \
-logs_dir=/srv/piuparts.debian.org/htdocs/stretch \
-output=debiman/testing.json.gz

% debiman-piuparts-distill \
  -logs_dir=/srv/piuparts.debian.org/htdocs/sid \
  -output=debiman/unstable.json.gz

% debiman-piuparts-distill \
  -logs_dir=/srv/piuparts.debian.org/htdocs/experimental \
  -output=debiman/experimental.json.gz

% debiman-piuparts-distill \
  -logs_dir=/srv/piuparts.debian.org/htdocs/jessie \
  -output=debiman/stable.json.gz

I then copied debiman/*.json.gz over to manziarly.debian.org,
re-generated the entire site, and slave alternative manpages now work
as expected :).

Obviously, the above was a one-off manual effort, so now we should
discuss how we’d like to set things up for a more permanent export.

Here’s a strawman proposal. We could:

1. Add a crontab entry, which triggers a shell script, which will
   update debiman-piuparts-distill, then run it on the logs
   directories.

2. Store the resulting *.json.gz files in a directory called
   /srv/piuparts.debian.org/htdocs/for-manpages.d.o

3. Download the files via HTTPS on manziarly.d.o.

Step ① requires installing the Go compiler
(https://packages.debian.org/stretch/golang-go) on piuparts.d.o. We
could avoid this if we’re all okay with just copying the
debiman-piuparts-distill binary onto piuparts.d.o, but referring to
e.g. /home/stapelberg/debiman-piuparts-distill from a cronjob within
the piuparts git repository feels somewhat unclean.

Any thoughts? Would you be okay with my strawman? If so, I’ll prepare
a patch.