Re: packages which have not been rebuild since December 2016

[gregor herrmann]

On Wed, 27 Jun 2018 10:09:24 +0200, Andreas Tille wrote:

> I wonder whether this query might be helpful.  It sorts the Debian
> packages according to date in upload_history - you can draw your cutting
> line wherever you want.

Which reminded my of ltnu (long time no upload, in devscripts) which
also queries the udd mirror.

`ltnu %@%'

does something similar (with less columns, and it needs an "email
address" for the sql where clause, hence the '%@%').


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Beatles: While My Guitar Gently Weeps


signature.asc
Description: Digital Signature

Re: packages which have not been rebuild since December 2016

[Chris Lamb]

Hi Andreas,

> > We have the "upload_history" relation but that will only give us an
> > upper limit (roughly 50% of the archive).
> 
> Why do you think upload_history would be incomplete.

(Curiously, I can't recall now.Perhaps something about packages built
on builds..?)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: packages which have not been rebuild since December 2016

[Andreas Tille]

Hi,

sorry, I've spotted this very late - may be it is helpful anyway.

On Wed, May 30, 2018 at 10:29:24PM +0100, Chris Lamb wrote:
> Hi Holger,
> 
> > One issue we forgot to mention there is that all binary packages built
> > with dpkg < 1.18.17 need to be rebuild. Is that something easy to find
> > out with UDD? 
> 
> Unless I'm missing something, I don't think that UDD has this. Whilst
> we have a `packages` relation that can "detect" binNMUs (ie. SELECT *
> FROM packages WHERE version != source_version) there is no timestamp
> on this.
> 
> We have the "upload_history" relation but that will only give us an
> upper limit (roughly 50% of the archive).

Why do you think upload_history would be incomplete.

I wonder whether this query might be helpful.  It sorts the Debian
packages according to date in upload_history - you can draw your cutting
line wherever you want.


export PGPASSWORD="public-udd-mirror"
SERVICE="--host=public-udd-mirror.xvm.mit.edu --username=public-udd-mirror udd"

psql $SERVICE >packages_sort_by_age.txt <= (SELECT sort FROM releases WHERE role = 'stable') OR 
r.sort = 0) -- 0 for experimental
   GROUP BY source, r.release, r.sort
 ) s
 JOIN upload_history u ON s.source = u.source AND s.version = u.version
 WHERE row_number = 1
 ORDER BY date, source
  ;
EOT


May be you want to drop all non-sid releases since they might be
irrelevant. 

Hope this helps

Andreas.

-- 
http://fam-tille.de

Re: packages which have not been rebuild since December 2016

[Chris Lamb]

Chris Lamb wrote:

> Given that we forgot to mention this issue at least once, I would like
> to create a bug in the BTS for it.
> 
> This would have the advantages of being a good place to store the
> latest status as well as being a convenient way to link others to it. 

I went ahead and filed this against release.debian.org as #900837. As
mentioned there, please feel to reassign..


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: packages which have not been rebuild since December 2016

[Chris Lamb]

Hi Holger,

> at the MiniDebConf 2018 in Hamburg we listed a few issues in Debian with
> regards to making Debian Buster reproducible in practice. (*)
> 
> One issue we forgot to mention there is that

Given that we forgot to mention this issue at least once, I would like
to create a bug in the BTS for it.

This would have the advantages of being a good place to store the
latest status as well as being a convenient way to link others to it. 

I'd be happy to summarise the replies to the thread thus far, please
just let me know which package I should assign it to (general?
buildd.debian.org? I'm easy...)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: packages which have not been rebuild since December 2016

[Holger Levsen]

On Sat, Jun 02, 2018 at 10:36:58AM +0200, Emilio Pozuelo Monfort wrote:
> On 01-06-18 16:32, Chris Lamb wrote:
> > … wouldn't we just binNMU these?
> There are many packages in your list that are arch:all only, and those can't 
> be
> binNMU'ed. Still I'm not sure we can do some several thousand binNMUs. But 
> that
> number could get reduced due to maintainer uploads and binNMUs due to 
> unrelated
> transitions.

also, binNMUs break multi-arch, see #894441, so I'm not a fan (of the
current implementation of) binNMUs.


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: packages which have not been rebuild since December 2016

[Emilio Pozuelo Monfort]

On 01-06-18 16:32, Chris Lamb wrote:
> … wouldn't we just binNMU these?

There are many packages in your list that are arch:all only, and those can't be
binNMU'ed. Still I'm not sure we can do some several thousand binNMUs. But that
number could get reduced due to maintainer uploads and binNMUs due to unrelated
transitions.

Cheers,
Emilio

Re: packages which have not been rebuild since December 2016

[Paul Gevers]

Hi Holger,

On 01-06-18 20:51, Holger Levsen wrote:
> Do you by chance still have the old changes file and can check whether
> it included the .buildinfo file?

Attached. It does include the .buildinfo file, which was uploaded as the
log file tells me.

Paul
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Format: 1.8
Date: Sun, 04 Feb 2018 13:49:35 +0100
Source: siridb-connector
Binary: python3-siridb-connector
Architecture: source all
Version: 2.0.5-1
Distribution: unstable
Urgency: medium
Maintainer: SiriDB Maintainers 

Changed-By: Paul Gevers 
Description:
 python3-siridb-connector - Python3 interface for the SiriDB time series 
database server
Closes: 889548
Changes:
 siridb-connector (2.0.5-1) unstable; urgency=medium
 .
   * Initial release. (Closes: #889548)
Checksums-Sha1:
 e7ea31a8a8d0b9793f3c7cc65e59550f7ccb5d5f 1768 siridb-connector_2.0.5-1.dsc
 571f553ad2bc9c1c7d68bbb879ef2f42afef68e5 46113 
siridb-connector_2.0.5.orig.tar.gz
 3603e4fc1664d9ceb9cbf60933d22975ea0f5814 1908 
siridb-connector_2.0.5-1.debian.tar.xz
 314880264c990362f711c45dbac123beca9b05f9 10884 
python3-siridb-connector_2.0.5-1_all.deb
 3423e160b43b4d7ab7f808a5663413b65ec6fb27 5629 
siridb-connector_2.0.5-1_amd64.buildinfo
Checksums-Sha256:
 df3efbf7bca2326cc82d22c074db312501f4b6c46bf95dfc04bb9efec1742ed7 1768 
siridb-connector_2.0.5-1.dsc
 730d96b6ec4cd985382893bf4d1d7cd8ffc5743300a36053b53202ce1f82d101 46113 
siridb-connector_2.0.5.orig.tar.gz
 a4f4e7b509c23933dfe58b3f8a46521284f68313e722e8db9a2c385199d399c2 1908 
siridb-connector_2.0.5-1.debian.tar.xz
 4666f9c093645f740d87990ac2288dcd6d01c66b5729cb578227fef072ba51d7 10884 
python3-siridb-connector_2.0.5-1_all.deb
 9b95c3edb24037427e093cf34d522026c93731f074f170c9530a5ce7d617d44c 5629 
siridb-connector_2.0.5-1_amd64.buildinfo
Files:
 9ee40d041c0309f9baa1e3f623592203 1768 python optional 
siridb-connector_2.0.5-1.dsc
 2410fbeb853ebb6b0e470df0b6daa79a 46113 python optional 
siridb-connector_2.0.5.orig.tar.gz
 9f02623f4de5b587e1bcc24ca47dacc2 1908 python optional 
siridb-connector_2.0.5-1.debian.tar.xz
 7cb24b262aaee9ad96ed4f4b9d0fd002 10884 python optional 
python3-siridb-connector_2.0.5-1_all.deb
 277012676e52417200f83965f8a1d94c 5629 python optional 
siridb-connector_2.0.5-1_amd64.buildinfo

-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEWLZtSHNr6TsFLeZynFyZ6wW9dQoFAlp4uEoACgkQnFyZ6wW9
dQpMsQf/Yz8b78MMUeZ7V5TZRViHfXZt1G99gf1VRz4rHMFrtq80QzCKr8iz3nIg
2H4jrumVnkIhcREVT2UprwzcVnh0ZwGArca67a3Xj0Xg2YEABQbe2lYeDW4u2jwo
R98V8DXiO5jmhltgo9Nj9TXwAaRTM3l4XT+7gCiB0tdRFP5vB89yfT843af+AWZL
ZpE4Yk7lfM74ql9nCT17RQ7crVfCsMy142HgALSKB0zmCFEhDUGdj4izpaQdh9Lj
4kRGigY71KIadA7X5uN3Pq9rfWm/0KtvMTkRuwY8+nTXPKWF7Sq2IADxuqUOvAhy
06QZ/jiQ963/gDKMl8tHMCZyBxFvyg==
=OytF
-END PGP SIGNATURE-
2018-02-05 21:02:45,983 - dput[7283]: uploader.invoke_dput - Uploading 
siridb-connector using ftp to ftp-eu (host: ftp.eu.upload.debian.org; 
directory: /pub/UploadQueue/)
2018-02-05 21:02:45,986 - dput[7283]: hook.run_hook - running 
allowed-distribution: check whether a local profile permits uploads to the 
target distribution
2018-02-05 21:02:46,058 - dput[7283]: hook.run_hook - running 
protected-distribution: warn before uploading to distributions where a special 
policy applies
2018-02-05 21:02:46,090 - dput[7283]: hook.run_hook - running checksum: verify 
checksums before uploading
2018-02-05 21:02:46,101 - dput[7283]: hook.run_hook - running suite-mismatch: 
check the target distribution for common errors
2018-02-05 21:02:46,108 - dput[7283]: hook.run_hook - running check-debs: makes 
sure the upload contains a binary package
2018-02-05 21:02:46,113 - dput[7283]: deb.check_debs_in_upload - Garbage value 
for check-debs/enforce - is source, debs, valid values are `debs` and `source`. 
Skipping checks.
2018-02-05 21:02:46,115 - dput[7283]: hook.run_hook - running gpg: check GnuPG 
signatures before the upload
2018-02-05 21:02:47,387 - dput[7283]: uploader.invoke_dput - Uploading 
siridb-connector_2.0.5-1.dsc
2018-02-05 21:02:47,572 - dput[7283]: uploader.invoke_dput - Uploading 
siridb-connector_2.0.5.orig.tar.gz
2018-02-05 21:02:48,049 - dput[7283]: uploader.invoke_dput - Uploading 
siridb-connector_2.0.5-1.debian.tar.xz
2018-02-05 21:02:48,233 - dput[7283]: uploader.invoke_dput - Uploading 
python3-siridb-connector_2.0.5-1_all.deb
2018-02-05 21:02:48,465 - dput[7283]: uploader.invoke_dput - Uploading 
siridb-connector_2.0.5-1_amd64.buildinfo
2018-02-05 21:02:48,665 - dput[7283]: uploader.invoke_dput - Uploading 
siridb-connector_2.0.5-1_amd64.changes


signature.asc
Description: OpenPGP digital signature

Re: packages which have not been rebuild since December 2016

[Holger Levsen]

Hi Paul,

On Fri, Jun 01, 2018 at 08:17:48PM +0200, Paul Gevers wrote:
> It helps to spot things. I notice:
>siridb-connector (U)
> which wasn't even in Debian until February this year. Your original list

February 2018 even, ouch

> says:
> siridb-connector needs rebuild (no .buildinfo)
> 
> So apparently the (binary) build that I uploaded (this is still the only
> upload) didn't contain the .buildinfo. That makes at least your subject
> (in this case) ambiguous. True, the package hasn't been rebuild ever,
> but maybe some have but are lacking the .buildinfo due to tooling issues
> we had/have? Maybe we need to fix those first?

yes, probably. As it was 2018 (and not 2017) I think it's highly
unlikely that your build didnt produce a .buildinfo file, so it means it
must have gone lost somewhere on the way.

Do you by chance still have the old changes file and can check whether
it included the .buildinfo file?

> Another thing, you probably thought about it, but if we care for the
> buster release, why not use testing instead of sid? Are you afraid stuff
> starts moving to testing suddenly?

We just always tend to look at sid, because that's how buster will be in
the future. From our POV there is not really a difference as (usually)
builds happen in sid only anyway, so the only relevant diff between
buster and sid (for us here) is that buster has a bit less packages, so
the stats look better.

(sid has 28043 packages in main, while buster has 26621.)


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: packages which have not been rebuild since December 2016

[Holger Levsen]

Hi Chris,

On Fri, Jun 01, 2018 at 10:40:18AM +0100, Chris Lamb wrote:
> > I think this should be relatively easy to compute:
> Indeed — 9182/33705 packages need a rebuild in sid.
> (Full output of following script attached.)

hehe, very nice. 

Just the numbers strike me as odd: we currently have 28043 source
packages in sid/main (according to our stats on tests.r-b.o), so…

sid  debian-work:/var/lib/apt/lists$ grep -c ^Package: 
deb.debian.org_debian_dists_sid_main_source_Sources
33168
sid  debian-work:/var/lib/apt/lists$ grep ^Package: 
deb.debian.org_debian_dists_sid_main_source_Sources|sort -u|wc -l
  28049  

ah, ok, that explains, you counted some packages twice.

an then I was curious how big contrib and non-free were:

sid  debian-work:/var/lib/apt/lists$ grep ^Package: 
deb.debian.org_debian_dists_sid_contrib_source_Sources | sort -u |wc -l
156
sid  debian-work:/var/lib/apt/lists$ grep ^Package: 
deb.debian.org_debian_dists_sid_non-free_source_Sources | sort -u |wc -l
325

So in other words a bit less than 33% of the archive still needs a rebuild.


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: packages which have not been rebuild since December 2016

[Paul Gevers]

Hi Chris,

On 01-06-18 16:32, Chris Lamb wrote:
> Hi Paul,
> 
>>> (Full output of following script attached.)
>>
>> Would it be easy to run dd-list on this and share that as well?
> 
> Attached, but…

Thanks.

>> Just in case people care enough to help with rebuilding their
>> own packages (like I would)
> 
> … wouldn't we just binNMU these? Or possibly; would knowing a package
> tha you maintain was on this list cause you to perform an regular
> upload you would not normally have done...?

It helps to spot things. I notice:
   siridb-connector (U)
which wasn't even in Debian until February this year. Your original list
says:
siridb-connector needs rebuild (no .buildinfo)

So apparently the (binary) build that I uploaded (this is still the only
upload) didn't contain the .buildinfo. That makes at least your subject
(in this case) ambiguous. True, the package hasn't been rebuild ever,
but maybe some have but are lacking the .buildinfo due to tooling issues
we had/have? Maybe we need to fix those first?

Another thing, you probably thought about it, but if we care for the
buster release, why not use testing instead of sid? Are you afraid stuff
starts moving to testing suddenly?

Paul



signature.asc
Description: OpenPGP digital signature

Re: packages which have not been rebuild since December 2016

[Paul Gevers]

Hi Chris,

On 01-06-18 11:40, Chris Lamb wrote:
> (Full output of following script attached.)

Would it be easy to run dd-list on this and share that as well? Just in
case people care enough to help with rebuilding their own packages (like
I would).

Paul



signature.asc
Description: OpenPGP digital signature

Re: packages which have not been rebuild since December 2016

[Chris Lamb]

Hi Guillem,

> I think this should be relatively easy to compute:

Indeed — 9182/33705 packages need a rebuild in sid.

(Full output of following script attached.)

#!/usr/bin/env python3

import os
import re
import glob
import bisect
import psycopg2
import collections

from debian import deb822
from debian.debian_support import version_compare

re_installed_build_depends = re.compile(
r'^(?P[^ ]+) \(= (?P.+)\)'
)


def get_buildinfos():
path = '/srv/ftp-master.debian.org/buildinfo/*/*/*/*.buildinfo'
result = collections.defaultdict(list)

for x in glob.glob(path):
src = os.path.basename(x).split('_')[0]
bisect.insort(result[src], x)

return result


def gen_sources():
connection = psycopg2.connect(
  user='udd-mirror',
  dbname='udd',
  password='udd-mirror',
  host='udd-mirror.debian.net',
)

c = connection.cursor()
c.execute("""
SELECT
source
FROM
all_sources
WHERE
distribution = 'debian'
AND
release = 'sid'
ORDER BY
source ASC
""")

for x in c:
yield x[0]


def main():
buildinfos = get_buildinfos()

for src in gen_sources():
try:
with open(buildinfos[src][-1]) as f:
data = deb822.Deb822(f)
except IndexError:
print("{} needs rebuild (no .buildinfo)".format(src))
continue

built_using = {}
for x in data['Installed-Build-Depends'].strip().splitlines():
m = re_installed_build_depends.match(x.strip())
built_using[m.group('package')] = m.group('version')

dpkg_version = built_using.get('dpkg-dev', '0~missing')

if version_compare(dpkg_version, '1.18.17') == -1:
print("{} needs rebuild (built using dpkg-dev {})".format(
  src, dpkg_version))


if __name__ == '__main__':
main()



Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-


output.txt.gz
Description: application/gzip

Re: packages which have not been rebuild since December 2016

[Holger Levsen]

Hi,

On Thu, May 31, 2018 at 05:33:10PM +0200, Guillem Jover wrote:
> > We have the "upload_history" relation but that will only give us an
> > upper limit (roughly 50% of the archive).
> 
> I think this should be relatively easy to compute:
 
yes, that's the easy part, once you have the data :)

>   if ! -e pkg.buildinfo
> → rebuild
>   elif dpkg-dev not in pkg.buildinfo
> → rebuild
>   elfi dpkg-dev < 1.18.17 in pkg.buildinfo
> → rebuild

yup. (and/or s#rebuild#display 'this package needs rebuid'#)

> And, hey, just today I stumbled upon this
>  :), which might perhaps
> be a nice starting point.

oh, indeed, seems David is already colling .buildinfo's somewhere.
David, is that on a public site somewhere?


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: packages which have not been rebuild since December 2016

[Guillem Jover]

Hi!

On Wed, 2018-05-30 at 22:29:24 +0100, Chris Lamb wrote:
> > One issue we forgot to mention there is that all binary packages built
> > with dpkg < 1.18.17 need to be rebuild. Is that something easy to find
> > out with UDD? 
> 
> Unless I'm missing something, I don't think that UDD has this. Whilst
> we have a `packages` relation that can "detect" binNMUs (ie. SELECT *
> FROM packages WHERE version != source_version) there is no timestamp
> on this.
> 
> We have the "upload_history" relation but that will only give us an
> upper limit (roughly 50% of the archive).

I think this should be relatively easy to compute:

  if ! -e pkg.buildinfo
→ rebuild
  elif dpkg-dev not in pkg.buildinfo
→ rebuild
  elfi dpkg-dev < 1.18.17 in pkg.buildinfo
→ rebuild

?

And, hey, just today I stumbled upon this
 :), which might perhaps
be a nice starting point.

Thanks,
Guillem

Re: packages which have not been rebuild since December 2016

[Chris Lamb]

Hi Holger,

> One issue we forgot to mention there is that all binary packages built
> with dpkg < 1.18.17 need to be rebuild. Is that something easy to find
> out with UDD? 

Unless I'm missing something, I don't think that UDD has this. Whilst
we have a `packages` relation that can "detect" binNMUs (ie. SELECT *
FROM packages WHERE version != source_version) there is no timestamp
on this.

We have the "upload_history" relation but that will only give us an
upper limit (roughly 50% of the archive).


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

packages which have not been rebuild since December 2016

[Holger Levsen]

hi,

at the MiniDebConf 2018 in Hamburg we listed a few issues in Debian with
regards to making Debian Buster reproducible in practice. (*)

One issue we forgot to mention there is that all binary packages built
with dpkg < 1.18.17 need to be rebuild. Is that something easy to find
out with UDD? (That's roughly all packages which haven't seen a rebuild
since January 1st 2017.)

https://www.archlinux.org/todo/buildinfo-rebuild/ has such a list since
today, which reminded me of this topic in Debian :)



(*) "93% is a lie. We need infrastructure, processes and policies. (And
testing. Currently we only have testing and a vague goal.)" - explained
in more detail in
https://meetings-archive.debian.net/pub/debian-meetings/2018/miniconf-hamburg/2018-05-20/reproducible_buster.webm
starting at 07:45min.


-- 
cheers,
Holger


signature.asc
Description: PGP signature