Re: [Reproducible-builds] simple next step for getting .buildinfo files into Debian

2015-12-14 Thread Niels Thykier
Niels Thykier:
> Holger Levsen:
>> Hi ftp folks,
>>
> 
> Hi,
> 
>> while we still appreciate your comments on this proposal as last week 
>> described in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=763822 I'd 
>> like 
>> to make a intermediate very simple proposal, so that reproducible builds in 
>> Debian get one step forward:
>>
> 
> I have started a git branch, build-info-support, available from:
> 
> * ssh://release.debian.org/~nthykier/dak
> 
> [...]
> 
> Thanks,
> ~Niels
> 
> 

Hi,

Thanks to h01ger and jwilk, who spotted a ".changes" that should have
been a ".buildinfo" in regexes.py.  I have rebased the patches (and
resigned them) on to master.  The changed patch is also attached.

Thanks,
~Niels


From ce901421acfc505d9d5787940dc964dceed1b32c Mon Sep 17 00:00:00 2001
From: Niels Thykier 
Date: Mon, 14 Dec 2015 20:46:28 +
Subject: [PATCH 1/2] daklib/upload.py: Silently accept and discard .buildinfo
 files

Besides some very trivial validation of the filename, accept
.buildinfo files without really looking at them.

Signed-off-by: Niels Thykier 
---
 daklib/regexes.py | 4 
 daklib/upload.py  | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/daklib/regexes.py b/daklib/regexes.py
index ae8fd91..16dd7ca 100644
--- a/daklib/regexes.py
+++ b/daklib/regexes.py
@@ -141,6 +141,10 @@ re_file_source = re.compile(_re_file_prefix + r'(?:(?:\.orig(?:-[a-zA-Z0-9-]+)?|
 # Groups: package, version
 re_file_orig = re.compile(_re_file_prefix + r'\.orig(?:-[a-zA-Z0-9-]+)?(?:\.tar\.(?:bz2|gz|xz)|\.asc)')
 
+# Match buildinfo file
+# Groups: package, version, suffix
+re_file_buildinfo = re.compile(_re_file_prefix + r'_(?P[a-zA-Z0-9+]+)\.buildinfo')
+
 ##
 # Patterns matching fields   #
 ##
diff --git a/daklib/upload.py b/daklib/upload.py
index b78d100..4c10f45 100644
--- a/daklib/upload.py
+++ b/daklib/upload.py
@@ -374,6 +374,8 @@ class Changes(object):
 for f in self.files.itervalues():
 if re_file_dsc.match(f.filename) or re_file_source.match(f.filename) or re_file_binary.match(f.filename):
 continue
+if re_file_buildinfo.match(f.filename):
+continue
 if f.section != 'byhand' and f.section[:4] != 'raw-':
 raise InvalidChangesException("{0}: {1} looks like a byhand package, but is in section {2}".format(self.filename, f.filename, f.section))
 byhand.append(f)
-- 
2.6.2



signature.asc
Description: OpenPGP digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] simple next step for getting .buildinfo files into Debian

2015-12-14 Thread Niels Thykier
Holger Levsen:
> Hi ftp folks,
> 

Hi,

> while we still appreciate your comments on this proposal as last week 
> described in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=763822 I'd 
> like 
> to make a intermediate very simple proposal, so that reproducible builds in 
> Debian get one step forward:
> 

I have started a git branch, build-info-support, available from:

* ssh://release.debian.org/~nthykier/dak

I realise that not every one have access to that machine, so the patches
are also attached (@FTP: The branch have signed commits, so you may
prefer merging form that).

> - modify dak, so that it will not rejects uploads with a .buildinfo file 
> included. 

I got patches to have dak accept these and do some trivial validation
(but not every validation proposed).  I will extend my branch as time
permits with additional checks.

> - still, for now, throw the .buildinfo file immediately away.

I have assumed this happens if you do no nothing explicitly with the
file after it being accepted.

 * @FTP: If not, please let me know how I can have dak discard the file.

> - only do this for experimental at the beginning. (maybe this restriction is 
> not even needed/useful.)
> 

 * Given the file is discarded, I have not added any such restrictions
   in my patch series.

> That's it.
> 
> This would allow the dpkg maintainers to enable .buildinfo file creation, at 
> least for builds for experimental.
> 
> What do you think?
> 

FWIW, I agree. :)

> As I see it, this should be a rather trivial code change for dak and yet 
> bring 
> us forward quite enourmously. Also it should be rather uncontroversial as we 
> all agreed in Heidelberg at DebConf15 that we want .buildinfo files in 
> Debian… 
> 
> 
> cheers,
>   Holger
> 


Thanks,
~Niels


From 10bc29c42eb36916aba1290c155126755837d903 Mon Sep 17 00:00:00 2001
From: Niels Thykier 
Date: Mon, 14 Dec 2015 20:46:28 +
Subject: [PATCH 1/2] daklib/upload.py: Silently accept and discard .buildinfo
 files

Besides some very trivial validation of the filename, accept
.buildinfo files without really looking at them.

Signed-off-by: Niels Thykier 
---
 daklib/regexes.py | 4 
 daklib/upload.py  | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/daklib/regexes.py b/daklib/regexes.py
index ae8fd91..63d2ee6 100644
--- a/daklib/regexes.py
+++ b/daklib/regexes.py
@@ -141,6 +141,10 @@ re_file_source = re.compile(_re_file_prefix + r'(?:(?:\.orig(?:-[a-zA-Z0-9-]+)?|
 # Groups: package, version
 re_file_orig = re.compile(_re_file_prefix + r'\.orig(?:-[a-zA-Z0-9-]+)?(?:\.tar\.(?:bz2|gz|xz)|\.asc)')
 
+# Match buildinfo file
+# Groups: package, version, suffix
+re_file_buildinfo = re.compile(_re_file_prefix + r'_(?P[a-zA-Z0-9+]+)\.changes$')
+
 ##
 # Patterns matching fields   #
 ##
diff --git a/daklib/upload.py b/daklib/upload.py
index b78d100..4c10f45 100644
--- a/daklib/upload.py
+++ b/daklib/upload.py
@@ -374,6 +374,8 @@ class Changes(object):
 for f in self.files.itervalues():
 if re_file_dsc.match(f.filename) or re_file_source.match(f.filename) or re_file_binary.match(f.filename):
 continue
+if re_file_buildinfo.match(f.filename):
+continue
 if f.section != 'byhand' and f.section[:4] != 'raw-':
 raise InvalidChangesException("{0}: {1} looks like a byhand package, but is in section {2}".format(self.filename, f.filename, f.section))
 byhand.append(f)
-- 
2.6.2

From 603ab424528a6f698332636ef3c80031a4e8c39c Mon Sep 17 00:00:00 2001
From: Niels Thykier 
Date: Mon, 14 Dec 2015 21:06:55 +
Subject: [PATCH 2/2] Do very basic validation of .buildinfo files

Validate that .buildinfo:

 * Have a valid signature (if signed)
 * Are Deb822 control files (UTF-8 encoded)
 * Have a Source and Version field equal to that of the .changes
   file.
---
 daklib/archive.py |  1 +
 daklib/checks.py  | 25 +
 daklib/upload.py  | 49 +
 3 files changed, 75 insertions(+)

diff --git a/daklib/archive.py b/daklib/archive.py
index 4226ce9..6a044db 100644
--- a/daklib/archive.py
+++ b/daklib/archive.py
@@ -951,6 +951,7 @@ class ArchiveUpload(object):
 checks.BinaryCheck,
 checks.BinaryTimestampCheck,
 checks.SingleDistributionCheck,
+checks.BuildinfoCheck,
 ):
 chk().check(self)
 
diff --git a/daklib/checks.py b/daklib/checks.py
index b187f03..07c43f9 100644
--- a/daklib/checks.py
+++ b/daklib/checks.py
@@ -500,6 +500,31 @@ class SourceCheck(Check):
 
 return True
 
+
+class BuildinfoCheck(Check):
+"""Check buildinfo file for syntax errors."""
+
+def check(self, upload):
+buildinfo_files = upload.c