Bug#746621: DDPO: Integrate Piuparts (w/ patches)

[Dave Steele]

On Thu, May 1, 2014 at 10:04 PM, David Steele  wrote:
> Package: qa.debian.org
...
> Piuparts has a new summary file format, visible at
> https://piuparts.debian.org/summary.json.
>
> The attached patches use that data to integrate piuparts into the DDPO
> report, to the right of the lintian results...

This is a reminder that patches are available to integrate
piuparts.d.o results into DDPO.

Note that pejacevic currently tests 17 distinct distribution
combinations, and that maintainers must currently inspect each of
these individually to find all published installation errors for their
packages. The patches submitted with this wishlist item integrate
maintainer results across all of the tests into the DDPO output.

Again, comments are welcome.

-- 
"Le mieux est l'ennemi du bien" - Voltaire


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Bug#746621: DDPO: Integrate Piuparts (w/ patches)

[David Steele]

On 05/02/2014 06:25 AM, Holger Levsen wrote:
> Hi,
> 
> On Freitag, 2. Mai 2014, David Steele wrote:
>> Piuparts has a new summary file format, visible at
>> https://piuparts.debian.org/summary.json.
> 
> please note that I've just changed the piuparts URLs within the .json file 
> from http to https - I don't think it should change things, but I thought I'd 
> let you know anyway.
> 

I've verified this doesn't affect the DDPO patches, or overall
functionality.


-- 
"Le mieux est l'ennemi du bien" - Voltaire



signature.asc
Description: OpenPGP digital signature

Bug#746621: DDPO: Integrate Piuparts (w/ patches)

[Holger Levsen]

Hi,

On Freitag, 2. Mai 2014, David Steele wrote:
> Piuparts has a new summary file format, visible at
> https://piuparts.debian.org/summary.json.

please note that I've just changed the piuparts URLs within the .json file 
from http to https - I don't think it should change things, but I thought I'd 
let you know anyway.

(As a side comment: I hardcoded this (choice of protocal) as a.) I'm lazy b.) 
everybody should use https c.) it was a one byte change as opposed to 200 or 
so for doing it properly. Patches to fix this properly are much welcome as I 
now feel bad for pushing such a hack. I'll reply to this to the piuparts devel 
list only...)

> The attached patches use that data to integrate piuparts into the DDPO
> report, to the right of the lintian results.

I think that's an excellent idea, but cannot comment on the DDPO code :)


cheers,
Holger


signature.asc
Description: This is a digitally signed message part.

Bug#746621: DDPO: Integrate Piuparts (w/ patches)

[David Steele]

Package: qa.debian.org
Severity: wishlist
Tags: patch
X-Debbugs-CC: piuparts-de...@lists.alioth.debian.org
thanks

Piuparts has a new summary file format, visible at
https://piuparts.debian.org/summary.json.

The attached patches use that data to integrate piuparts into the DDPO
report, to the right of the lintian results. The contents of the column
indicate the summary test result for the package; either '-' for pass or
unknown, 'F' for fail, 'X' for blocked, or 'W' for waiting. The tooltip
describes the test result, lists the applicable distribution(s), and the
number of blocked packages resulting from a failure, if any. All entries
hyperlink to appropriate pages with further details about the displayed
test result.

If this sounds like a good idea, I'd appreciate a review of the code,
and sponsorship for integration.

Thanks in advance.

0001-piuparts-update-Initial-checkin-of-piuparts-cron.patch
0002-extract-piuparts.py-1st-commit-create-piuparts-db.patch
0003-piuparts-update-Add-db-creation-to-piuparts-cron.patch
0004-developer.wml-Add-skeleton-for-piuparts-column.patch
0005-developer.wml-Populate-piuparts-cell-contents.patch

-- 
"Le mieux est l'ennemi du bien" - Voltaire



From bc2c2a4b0b13f8bb09042e43cb5a78a61770a3b9 Mon Sep 17 00:00:00 2001
From: David Steele 
Date: Fri, 21 Feb 2014 20:15:25 -0500
Subject: [PATCH 1/5] piuparts-update: Initial checkin of piuparts cron

Baby steps. Do no more than download the piuparts summary json into
the data directory. Do it two hours after the two daily piuparts
run. The runs currently take about 80 minutes.
---
 data/cronjobs/piuparts-update | 7 +++
 1 file changed, 7 insertions(+)
 create mode 100755 data/cronjobs/piuparts-update

diff --git a/data/cronjobs/piuparts-update b/data/cronjobs/piuparts-update
new file mode 100755
index 000..91d561c
--- /dev/null
+++ b/data/cronjobs/piuparts-update
@@ -0,0 +1,7 @@
+#!/bin/sh -e
+# CRON=33 2,14 * * *
+
+umask 002
+wget -q https://piuparts.debian.org/summary.json \
+ -O /srv/qa.debian.org/data/piuparts-summary.json
+
-- 
2.0.0.rc0

From 3eca446f2f87ee872e18261a5b37f047b9fe9bb8 Mon Sep 17 00:00:00 2001
From: David Steele 
Date: Fri, 21 Feb 2014 23:56:55 -0500
Subject: [PATCH 2/5] extract-piuparts.py: 1st commit-create piuparts db

Parse a JSON piuparts results summary file, and store in a dbm db
file. Results, stored by package name, consist of a single-character
result flag, a package result inspection url, and tooltip text,
separated by spaces.
---
 data/piuparts/extract_piuparts.py | 142 ++
 1 file changed, 142 insertions(+)
 create mode 100755 data/piuparts/extract_piuparts.py

diff --git a/data/piuparts/extract_piuparts.py b/data/piuparts/extract_piuparts.py
new file mode 100755
index 000..4987e6f
--- /dev/null
+++ b/data/piuparts/extract_piuparts.py
@@ -0,0 +1,142 @@
+#!/usr/bin/python
+#
+# Copyright (C) 2014 David Steele (dste...@gmail.com)
+#
+# This file is part of the Debian Developer's Package Overview (DDPO)
+#
+# This is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# This is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, see .
+
+# This extracts the information from piuparts for each package
+
+import textwrap
+import argparse
+import os
+import sys
+import json
+import dbm
+from collections import defaultdict
+
+
+DBFILE = 'piuparts-new'
+
+SUMMID = "Piuparts Package Test Results Summary"
+SUMMVER = "1."
+DEFSEC = 'overall'
+
+class SummaryException(Exception):
+pass
+
+
+dist_sort_order = defaultdict(lambda: 100, [
+   ('oldstable', 10),
+   ('stable', 20),
+   ('testing', 30),
+   ('unstable', 40),
+   ('experimental', 50),
+   ])
+
+
+flag_description = defaultdict(lambda: 'Unknown', [
+   ('P', 'Passed'),
+   ('X', 'Blocked'),
+   ('W', 'Waiting'),
+   ('F', 'Failed'),
+   ('-', 'Unknown'),
+   ])
+
+def tooltip(summary, pkg):
+"""Returns e.g. "Failed in stable and testing, blocking 5 packages"."""
+
+tip = ''
+pkgdict = summary['packages']
+
+if pkg in pkgd