Bug#892712: [Qa-jenkins-dev] Bug#892712: jenkins.debian.org: reproducible Debian: Please generate and save JSON output from diffoscope

[Chris Lamb]

Hi Holger,

> thanks for the bug report with patch. do you have any estimate how much
> diskspace will be needed to enable this? 4 archs, 4 suites, 2500
> unreproducible packages per (rough) average.

I can't work this out from first principles, but I have no reason to
think it won't be in the same order of magnitude as the size of the
existing .txt outputs.

Perhaps something like:

  $ find /path/to/data/dir -type f -name '*.txt' -print0 | du --files0-from=- 
-hc | tail -n1

… will help.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#892712: [Qa-jenkins-dev] Bug#892712: Bug#892712: jenkins.debian.org: reproducible Debian: Please generate and save JSON output from diffoscope

[Mattia Rizzolo]

On Mon, Mar 12, 2018 at 12:11:48PM +, Holger Levsen wrote:
> Hi Chris,
> 
> On Mon, Mar 12, 2018 at 06:00:34AM +, Chris Lamb wrote:
> >   reproducible Debian: Also generate and save JSON output
> >   from diffoscope to make automatic categorisation easier.
> 
> thanks for the bug report with patch. do you have any estimate how much
> diskspace will be needed to enable this? 4 archs, 4 suites, 2500
> unreproducible packages per (rough) average.
> 
> /dev/mapper/reproducible-userContent  221G  143G   78G  65% 
> /var/lib/jenkins/userContent/reproducible
> seems we are not as low on diskspace as I feared.

Nonetheless, that might be too low anyway.
The textual diffoscope report (gzipped) take 82 GB currently.  I must
assume the json output would take something similar.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#892712: [Qa-jenkins-dev] Bug#892712: jenkins.debian.org: reproducible Debian: Please generate and save JSON output from diffoscope

[Holger Levsen]

Hi Chris,

On Mon, Mar 12, 2018 at 06:00:34AM +, Chris Lamb wrote:
>   reproducible Debian: Also generate and save JSON output
>   from diffoscope to make automatic categorisation easier.

thanks for the bug report with patch. do you have any estimate how much
diskspace will be needed to enable this? 4 archs, 4 suites, 2500
unreproducible packages per (rough) average.

/dev/mapper/reproducible-userContent  221G  143G   78G  65% 
/var/lib/jenkins/userContent/reproducible
seems we are not as low on diskspace as I feared.


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Bug#892712: jenkins.debian.org: reproducible Debian: Please generate and save JSON output from diffoscope

[Chris Lamb]

Package: jenkins.debian.org
Severity: wishlist
Tags: patch

Hi!

Attached is the following:

  commit 4de26dd2adcabd2c59da4b9ab51784cc6d9cc99e
  Author: Chris Lamb 
  Date:   Sun Mar 11 22:55:44 2018 -0700
  
  reproducible Debian: Also generate and save JSON output
  from diffoscope to make automatic categorisation easier.
  
   bin/reproducible_build.sh | 10 +-
   1 file changed, 9 insertions(+), 1 deletion(-)


You can also pull from the "generate-diffoscope-json" branch via 
https://github.com/lamby/jenkins.debian.net:

  https://github.com/lamby/jenkins.debian.net/commits/generate-diffoscope-json


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-
>From 4de26dd2adcabd2c59da4b9ab51784cc6d9cc99e Mon Sep 17 00:00:00 2001
From: Chris Lamb 
Date: Sun, 11 Mar 2018 22:55:44 -0700
Subject: [PATCH] reproducible Debian: Also generate and save JSON output from
 diffoscope to make automatic categorisation easier.

---
 bin/reproducible_build.sh | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/bin/reproducible_build.sh b/bin/reproducible_build.sh
index 96acd93d..7b89bc7f 100755
--- a/bin/reproducible_build.sh
+++ b/bin/reproducible_build.sh
@@ -50,6 +50,7 @@ exit_early_if_debian_is_broken() {
 create_results_dirs() {
 	mkdir -vp $DEBIAN_BASE/dbd/${SUITE}/${ARCH}
 	mkdir -vp $DEBIAN_BASE/dbdtxt/${SUITE}/${ARCH}
+	mkdir -vp $DEBIAN_BASE/dbdjson/${SUITE}/${ARCH}
 	mkdir -vp $DEBIAN_BASE/logs/${SUITE}/${ARCH}
 	mkdir -vp $DEBIAN_BASE/logdiffs/${SUITE}/${ARCH}
 	mkdir -vp $DEBIAN_BASE/rbuild/${SUITE}/${ARCH}
@@ -300,6 +301,10 @@ handle_ftbr() {
 		mv ./$DBDTXT $DEBIAN_BASE/dbdtxt/$SUITE/$ARCH/
 		gzip -9n $DEBIAN_BASE/dbdtxt/$SUITE/$ARCH/$DBDTXT
 	fi
+	if [ -f ./$DBDJSON ] ; then
+		mv ./$DBDJSON $DEBIAN_BASE/dbdjson/$SUITE/$ARCH/
+		gzip -9n $DEBIAN_BASE/dbdjson/$SUITE/$ARCH/$DBDJSON
+	fi
 	calculate_build_duration
 	update_db_and_html "unreproducible"
 }
@@ -396,6 +401,7 @@ call_diffoscope_on_changes_files() {
 		-- sh -c "export TMPDIR=$TEMP ; diffoscope \
 			--html $TMPDIR/${DBDREPORT} \
 			--text $TMPDIR/$DBDTXT \
+			--json $TMPDIR/$DBDJSON \
 			--profile=- \
 			$TMPDIR/b1/${CHANGES} \
 			$TMPDIR/b2/${CHANGES}" \
@@ -403,7 +409,7 @@ call_diffoscope_on_changes_files() {
 	RESULT=$?
 	LOG_RESULT=$(grep '^E: 15binfmt: update-binfmts: unable to open' $TMPLOG || true)
 	if [ ! -z "$LOG_RESULT" ] ; then
-		rm -f $TMPLOG $TMPDIR/${DBDREPORT} $TMPDIR/$DBDTXT
+		rm -f $TMPLOG $TMPDIR/${DBDREPORT} $TMPDIR/$DBDTXT $TMPDIR/$DBDJSON
 		echo "$(date -u) - schroot jenkins-reproducible-${DBDSUITE}-diffoscope not available, will sleep 2min and retry."
 		sleep 2m
 		# remember to also modify the retry diffoscope call 15 lines above
@@ -413,6 +419,7 @@ call_diffoscope_on_changes_files() {
 			-- sh -c "export TMPDIR=$TEMP ; diffoscope \
 --html $TMPDIR/${DBDREPORT} \
 --text $TMPDIR/$DBDTXT \
+--json $TMPDIR/$DBDJSON \
 --profile=- \
 $TMPDIR/b1/${CHANGES} \
 $TMPDIR/b2/${CHANGES}" \
@@ -562,6 +569,7 @@ get_source_package() {
 	EVERSION="$(echo $VERSION | cut -d ':' -f2)"  # EPOCH_FREE_VERSION is too long
 	DBDREPORT="${SRCPACKAGE}_${EVERSION}.diffoscope.html"
 	DBDTXT="${SRCPACKAGE}_${EVERSION}.diffoscope.txt"
+	DBDJSON="${SRCPACKAGE}_${EVERSION}.diffoscope.json"
 	BUILDINFO="${SRCPACKAGE}_${EVERSION}_${ARCH}.buildinfo"
 	BUILDINFO_SIGNED="${BUILDINFO}.asc"
 
-- 
2.16.2