Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
I unfortunately don’t have time to test this package before uploading. If anyone has the time to either contribute a working and comprehensive test suite for the package and/or take over maintenance, that’d be much appreciated. I’m merely uploading these patches as-is to avoid blocking progress on the package. On Tue, Apr 19, 2016 at 11:10 PM, Dmitry Smirnov wrote: > On Tuesday, 19 April 2016 10:32:58 PM AEST Michael Stapelberg wrote: > > Dmitry, please let us know if the patch works for you and I’ll be happy > to > > upload it. > > Hi Michael, > > Unfortunately I won't be able to spare time for testing at least till > weekend. > Please upload if patch looks good for you -- I doubt it could make > situation > worse than it already is. > > I should trust you to remember to test this important package before every > upload. ;) > > Uploaded or not I'll have a look at first opportunity. Thanks. > > -- > Cheers, > Dmitry Smirnov. > > --- > > Truth — Something somehow discreditable to someone. > -- H. L. Mencken, 1949 > -- Best regards, Michael
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
New patch. Builds everything on Dmitry's list without any stderr from dh_golang and the built-using headers produced look reasonable. Cheers, mwh On 20 April 2016 at 11:13, Michael Hudson-Doyle wrote: > On 20 April 2016 at 09:05, Dmitry Smirnov wrote: >> On Tuesday, 19 April 2016 12:02:10 PM AEST Michael Hudson-Doyle wrote: >>> Are there any other packages you think would be particularly good to >>> try to build? >> >> You can check the following packages, starting from top: > > Thanks for this list. > >> golang-github-aws-aws-sdk-go > > This manages to trigger an "Argument list too long" in my patch. New > one coming in a little while. > > Cheers, > mwh > >> docker-swarm >> influxdb >> golang-github-unknwon-com >> cadvisor >> golang-github-revel-revel >> runc >> >> Thanks. >> >> -- >> Cheers, >> Dmitry Smirnov. From 10cd725d7996c71b1ff4784e7d0894c21d3f9d6d Mon Sep 17 00:00:00 2001 From: Michael Hudson-Doyle Date: Tue, 19 Apr 2016 11:59:51 +1200 Subject: [PATCH] More dh_golang fixes --- debian/changelog | 15 +++ script/dh_golang | 58 2 files changed, 48 insertions(+), 25 deletions(-) diff --git a/debian/changelog b/debian/changelog index a4b3689..451dd19 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,18 @@ +dh-golang (1.16) UNRELEASED; urgency=medium + + * Make dh_golang more robust: +- Initialize the buildsystem more correctly, so --builddirectory=_build + works (Closes: 821350) +- Exit with an error if any of the 'go list' or 'dpkg-search' commands + fail. +- Quote the current working directory in the regexp used to filter out + files from the build directory. +- Store package / directory lists in files and use xargs to avoid + constructing over-long command lines. + * Also trim dh_golang's use statements. + + -- Michael Hudson-Doyle Tue, 19 Apr 2016 11:42:26 +1200 + dh-golang (1.15) unstable; urgency=medium [ Michael Hudson-Doyle ] diff --git a/script/dh_golang b/script/dh_golang index 5e1e71d..d39523b 100755 --- a/script/dh_golang +++ b/script/dh_golang @@ -9,14 +9,8 @@ dh_golang - Generates Built-Using substvar use strict; use Cwd qw(realpath); use Debian::Debhelper::Dh_Lib; # not in core -use Dpkg; # not in core -use Dpkg::Control; # not in core -use Dpkg::Control::Info; # not in core -use Dpkg::Deps; # not in core -use Dpkg::Gettext; # not in core -use Scalar::Util qw(blessed); # in core since v5.7.3 -use List::Util qw(first); # in core since v5.7.3 -use Debian::Debhelper::Buildsystem::golang; +use Debian::Debhelper::Dh_Buildsystems; # not in core +use File::Temp qw(tempdir); =head1 SYNOPSIS @@ -35,34 +29,48 @@ The best way to invoke B is by using B. =cut -init(); - # Generate misc:Built-Using substvar. -my $bs = Debian::Debhelper::Buildsystem::golang->new(); +buildsystems_init(); +my $bs = load_buildsystem("golang"); -$bs->_set_dh_gopkg(); $bs->_set_gopath(); my @targets = $bs->get_targets(); -my $godeps = `go list -f '{{ range .Deps }}{{.}} {{ end }}' @targets`; -$godeps =~ s/\n/ /g; -my @godirs = split /\n/, `go list -f '{{ .Dir }}' $godeps`; -my $realgodirs; -my $cwd = $bs->{cwd}; -for my $godir (@godirs) { -my $realpath = realpath($godir); -# @godirs will include the directories of the package being built, so exclude them. -if ($realpath !~ /^$bs->{cwd}/) { -$realgodirs .= realpath($godir) . " "; +my $tmpl = '{{ range .Deps }}{{.}} +{{ end }}'; + +my $tmpdir = tempdir("dh_golang_XXX", TMPDIR => 1, CLEANUP => 1); + +system("go list -f \"$tmpl\" @targets > $tmpdir/godeps") == 0 +or die "go list of targets failed with code $?, $!"; + +system("sort -u $tmpdir/godeps | xargs go list -f '{{ .Dir }}' > $tmpdir/godirs") == 0 +or die "go list of dependencies failed with code $?, $!"; + +open(my $inp, "<", "$tmpdir/godirs"); +open(my $outp, ">", "$tmpdir/realgodirs"); +while (<$inp>) { +chomp; +my $realpath = realpath($_); +# godirs will include the directories of the package being built, so exclude them. +if ($realpath !~ /^\Q$bs->{cwd}\E/) { +printf $outp "%s\n", $realpath; } } -my $pkgs = `dpkg-query --search $realgodirs | cut -d: -f1`; -$pkgs =~ s/\n/ /g; -my $built_using = `dpkg-query -f='\${source:Package} (= \${source:Version}), ' -W $pkgs`; +close($inp); +close($outp); + +system("cat $tmpdir/realgodirs | xargs dpkg-query --search > $tmpdir/pkgs") == 0 +or die "dpkg-query --search failed with code $?, $!"; + +my $built_using = `cut -d: -f1 $tmpdir/pkgs | sort -u | xargs dpkg-query -f='\${source:Package} (= \${source:Version}), ' -W`; +if ($? != 0) { +die "dpkg-query -W failed with code $?, $!"; +} # If there is an easier way to have a universal misc:Bui
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
On 20 April 2016 at 09:05, Dmitry Smirnov wrote: > On Tuesday, 19 April 2016 12:02:10 PM AEST Michael Hudson-Doyle wrote: >> Are there any other packages you think would be particularly good to >> try to build? > > You can check the following packages, starting from top: Thanks for this list. > golang-github-aws-aws-sdk-go This manages to trigger an "Argument list too long" in my patch. New one coming in a little while. Cheers, mwh > docker-swarm > influxdb > golang-github-unknwon-com > cadvisor > golang-github-revel-revel > runc > > Thanks. > > -- > Cheers, > Dmitry Smirnov.
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
On Tuesday, 19 April 2016 10:32:58 PM AEST Michael Stapelberg wrote: > Dmitry, please let us know if the patch works for you and I’ll be happy to > upload it. Hi Michael, Unfortunately I won't be able to spare time for testing at least till weekend. Please upload if patch looks good for you -- I doubt it could make situation worse than it already is. I should trust you to remember to test this important package before every upload. ;) Uploaded or not I'll have a look at first opportunity. Thanks. -- Cheers, Dmitry Smirnov. --- Truth — Something somehow discreditable to someone. -- H. L. Mencken, 1949 signature.asc Description: This is a digitally signed message part.
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
On Tuesday, 19 April 2016 12:02:10 PM AEST Michael Hudson-Doyle wrote: > Are there any other packages you think would be particularly good to > try to build? You can check the following packages, starting from top: golang-github-aws-aws-sdk-go docker-swarm influxdb golang-github-unknwon-com cadvisor golang-github-revel-revel runc Thanks. -- Cheers, Dmitry Smirnov. signature.asc Description: This is a digitally signed message part.
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
Dmitry, please let us know if the patch works for you and I’ll be happy to upload it. On Tue, Apr 19, 2016 at 2:02 AM, Michael Hudson-Doyle < michael.hud...@canonical.com> wrote: > The attached should fix this. I get this for rkt's Built-Using now: > > Built-Using: docker-registry (= 2.3.1~ds1-1), golang (= 2:1.6.1-2), > golang-context (= 0.0~git20140604.1.14f550f-1), golang-github-appc-cni > (= 0.2.0~rc0+dfsg-1), golang-github-appc-docker2aci (= 0.9.3+dfsg-1), > golang-github-appc-spec (= 0.7.4+dfsg-1), > golang-github-coreos-go-systemd (= 5-1), > golang-github-coreos-ioprogress (= 0.0~git20151023.0.4637e49-1), > golang-github-cznic-b (= 0.0~git20151027.0.01b13d7-1), > golang-github-cznic-bufs (= 0.0~git20140818.0.3dcccbd-1), > golang-github-cznic-fileutil (= 0.0~git20150708.0.1c9c88f-1), > golang-github-cznic-mathutil (= 0.0~git20150605.0.a804f0f-1), > golang-github-cznic-ql (= 1.0.2-1), golang-github-cznic-sortutil (= > 0.0~git20150617.0.4c73428-1), golang-github-cznic-strutil (= > 0.0~git20150430.0.1eb03e3-1), golang-github-cznic-zappy (= > 0.0~git20160305.0.4f5e6ef-1), golang-github-dustin-go-humanize (= > 0.0~git20151125.0.8929fe9-1), golang-github-google-btree (= > 0.0~git20150413.0.cc6329d-1), golang-github-gorilla-mux (= > 0.0~git20150814.0.f7b6aaa-1), golang-github-hashicorp-errwrap (= > 0.0~git20141028.0.7554cd9-1), golang-github-pborman-uuid (= > 0.0+git20150824.0.cccd189-1), golang-github-peterbourgon-diskv (= > 2.0.0-1), golang-github-spf13-cobra (= 0.0~git20160117.0.8e91712-1), > golang-github-spf13-pflag (= 0.0~git20151218.0.7f60f83-2), > golang-go-semver (= 0.0~git20150304-1), golang-go.crypto (= > 1:0.0~git20151201.0.7b85b09-2), golang-gocapability-dev (= > 0.0~git20150506.1.66ef2aa-1), golang-golang-x-net-dev (= > 1:0.0+git20160110.4fd4a9f-1), golang-google-grpc (= > 0.0~git20151002.0.3e7b7e5-1), golang-goprotobuf (= 0.0~git20160330-1), > golang-speter-go-exp-math-dec-inf (= 0.0~git20140417.0.42ca6cd-2) > > which looks at least plausible. > > On 19 April 2016 at 10:28, Dmitry Smirnov wrote: > > On Tuesday, 19 April 2016 10:19:56 AM AEST Michael Hudson-Doyle wrote: > >> Wow, I'm not sure that package gets much from using dh-golang at all? > >> But I think the problem is the " --builddirectory=_build" in the > >> default target, somehow that needs to get funnelled into the right > >> place. Will have a look. > > > > Thanks. We actually have many Golang packages with > "--builddirectory=_build" > > so fixing that is very important. > > Are there any other packages you think would be particularly good to > try to build? > > Cheers, > mwh > -- Best regards, Michael
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
The attached should fix this. I get this for rkt's Built-Using now: Built-Using: docker-registry (= 2.3.1~ds1-1), golang (= 2:1.6.1-2), golang-context (= 0.0~git20140604.1.14f550f-1), golang-github-appc-cni (= 0.2.0~rc0+dfsg-1), golang-github-appc-docker2aci (= 0.9.3+dfsg-1), golang-github-appc-spec (= 0.7.4+dfsg-1), golang-github-coreos-go-systemd (= 5-1), golang-github-coreos-ioprogress (= 0.0~git20151023.0.4637e49-1), golang-github-cznic-b (= 0.0~git20151027.0.01b13d7-1), golang-github-cznic-bufs (= 0.0~git20140818.0.3dcccbd-1), golang-github-cznic-fileutil (= 0.0~git20150708.0.1c9c88f-1), golang-github-cznic-mathutil (= 0.0~git20150605.0.a804f0f-1), golang-github-cznic-ql (= 1.0.2-1), golang-github-cznic-sortutil (= 0.0~git20150617.0.4c73428-1), golang-github-cznic-strutil (= 0.0~git20150430.0.1eb03e3-1), golang-github-cznic-zappy (= 0.0~git20160305.0.4f5e6ef-1), golang-github-dustin-go-humanize (= 0.0~git20151125.0.8929fe9-1), golang-github-google-btree (= 0.0~git20150413.0.cc6329d-1), golang-github-gorilla-mux (= 0.0~git20150814.0.f7b6aaa-1), golang-github-hashicorp-errwrap (= 0.0~git20141028.0.7554cd9-1), golang-github-pborman-uuid (= 0.0+git20150824.0.cccd189-1), golang-github-peterbourgon-diskv (= 2.0.0-1), golang-github-spf13-cobra (= 0.0~git20160117.0.8e91712-1), golang-github-spf13-pflag (= 0.0~git20151218.0.7f60f83-2), golang-go-semver (= 0.0~git20150304-1), golang-go.crypto (= 1:0.0~git20151201.0.7b85b09-2), golang-gocapability-dev (= 0.0~git20150506.1.66ef2aa-1), golang-golang-x-net-dev (= 1:0.0+git20160110.4fd4a9f-1), golang-google-grpc (= 0.0~git20151002.0.3e7b7e5-1), golang-goprotobuf (= 0.0~git20160330-1), golang-speter-go-exp-math-dec-inf (= 0.0~git20140417.0.42ca6cd-2) which looks at least plausible. On 19 April 2016 at 10:28, Dmitry Smirnov wrote: > On Tuesday, 19 April 2016 10:19:56 AM AEST Michael Hudson-Doyle wrote: >> Wow, I'm not sure that package gets much from using dh-golang at all? >> But I think the problem is the " --builddirectory=_build" in the >> default target, somehow that needs to get funnelled into the right >> place. Will have a look. > > Thanks. We actually have many Golang packages with "--builddirectory=_build" > so fixing that is very important. Are there any other packages you think would be particularly good to try to build? Cheers, mwh From 5877dffe5bcb36a4afd724c31530d4f747f887d5 Mon Sep 17 00:00:00 2001 From: Michael Hudson-Doyle Date: Tue, 19 Apr 2016 11:59:51 +1200 Subject: [PATCH] More dh_golang fixes --- debian/changelog | 13 + script/dh_golang | 30 -- 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/debian/changelog b/debian/changelog index a4b3689..0dcca6c 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,16 @@ +dh-golang (1.16) UNRELEASED; urgency=medium + + * Make dh_golang more robust: +- Initialize the buildsystem more correctly, so --builddirectory=_build + works (Closes: 821350) +- Exit with an error if any of the 'go list' or 'dpkg-search' commands + fail. +- Quote the current working directory in the regexp used to filter out + files from the build directory. + * Also trim dh_golang's use statements. + + -- Michael Hudson-Doyle Tue, 19 Apr 2016 11:42:26 +1200 + dh-golang (1.15) unstable; urgency=medium [ Michael Hudson-Doyle ] diff --git a/script/dh_golang b/script/dh_golang index 5e1e71d..b2c6d99 100755 --- a/script/dh_golang +++ b/script/dh_golang @@ -9,14 +9,7 @@ dh_golang - Generates Built-Using substvar use strict; use Cwd qw(realpath); use Debian::Debhelper::Dh_Lib; # not in core -use Dpkg; # not in core -use Dpkg::Control; # not in core -use Dpkg::Control::Info; # not in core -use Dpkg::Deps; # not in core -use Dpkg::Gettext; # not in core -use Scalar::Util qw(blessed); # in core since v5.7.3 -use List::Util qw(first); # in core since v5.7.3 -use Debian::Debhelper::Buildsystem::golang; +use Debian::Debhelper::Dh_Buildsystems; # not in core =head1 SYNOPSIS @@ -35,34 +28,43 @@ The best way to invoke B is by using B. =cut -init(); - # Generate misc:Built-Using substvar. -my $bs = Debian::Debhelper::Buildsystem::golang->new(); +buildsystems_init(); +my $bs = load_buildsystem("golang"); -$bs->_set_dh_gopkg(); $bs->_set_gopath(); my @targets = $bs->get_targets(); my $godeps = `go list -f '{{ range .Deps }}{{.}} {{ end }}' @targets`; +if ($? != 0) { +die "go list of targets failed"; +} $godeps =~ s/\n/ /g; my @godirs = split /\n/, `go list -f '{{ .Dir }}' $godeps`; +if ($? != 0) { +die "go list of dependencies failed"; +} my $realgodirs; -my $cwd = $bs->{cwd}; for my $godir (@godirs) { my $realpath = realpath($godir); # @godirs will include the directories of the package being built, so exclude them. -if ($realpath !~ /^$b
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
On Tuesday, 19 April 2016 10:19:56 AM AEST Michael Hudson-Doyle wrote: > Wow, I'm not sure that package gets much from using dh-golang at all? > But I think the problem is the " --builddirectory=_build" in the > default target, somehow that needs to get funnelled into the right > place. Will have a look. Thanks. We actually have many Golang packages with "--builddirectory=_build" so fixing that is very important. -- All the best, Dmitry Smirnov. --- It has become almost a cliché to remark that nobody boasts of ignorance of literature, but it is socially acceptable to boast ignorance of science. -- Richard Dawkins signature.asc Description: This is a digitally signed message part.
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
Wow, I'm not sure that package gets much from using dh-golang at all? But I think the problem is the " --builddirectory=_build" in the default target, somehow that needs to get funnelled into the right place. Will have a look. On 19 April 2016 at 09:05, Michael Stapelberg wrote: > Michael, can you take a look at this issue please? > > On Mon, Apr 18, 2016 at 12:45 AM, Dmitry Smirnov wrote: >> >> Package: dh-golang >> Version: 1.15 >> Severity: serious >> >> Recent upload of "rkt" was rejected >> >> rkt_1.4.0+dfsg-1_amd64.deb: Built-Using refers to non-existing source >> package apt (= 1.0.9.10) >> >> due to rubbish in Built-Using field: >> >> apt (= 1.0.9.10), apt (= 1.0.10.2), apt (= 1.2.10) >> >> Also invocation of dh-golang logged the following: >> >> >>dh_golang -O--buildsystem=golang -O--builddirectory=_build >> can't load package: package github.com/coreos/rkt/api/v1alpha: cannot find >> package "github.com/coreos/rkt/api/v1alpha" in any of: >> /usr/lib/go/src/github.com/coreos/rkt/api/v1alpha (from $GOROOT) >> >> /build/rkt-1.4.0+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/rkt/api/v1alpha >> (from $GOPATH) >> can't load package: package github.com/coreos/rkt/rkt: cannot find package >> "github.com/coreos/rkt/rkt" in any of: >> /usr/lib/go/src/github.com/coreos/rkt/rkt (from $GOROOT) >> >> /build/rkt-1.4.0+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/rkt/rkt >> (from $GOPATH) >> can't load package: package .: no buildable Go source files in >> /build/rkt-1.4.0+dfsg >> can't load package: package .: no buildable Go source files in >> /build/rkt-1.4.0+dfsg >> dpkg-query: error: --search needs at least one file name pattern argument >> >> Use --help for help about querying packages. >> >> >> This is regression from "dh-golang" v1.12. >> >> Please investigate. >> >> -- >> All the best, >> Dmitry Smirnov >> GPG key : 4096R/53968D1B >> >> --- >> >> Truth never damages a cause that is just. >> -- Mahatma Gandhi > > > > > -- > Best regards, > Michael
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
Michael, can you take a look at this issue please? On Mon, Apr 18, 2016 at 12:45 AM, Dmitry Smirnov wrote: > Package: dh-golang > Version: 1.15 > Severity: serious > > Recent upload of "rkt" was rejected > > rkt_1.4.0+dfsg-1_amd64.deb: Built-Using refers to non-existing source > package apt (= 1.0.9.10) > > due to rubbish in Built-Using field: > > apt (= 1.0.9.10), apt (= 1.0.10.2), apt (= 1.2.10) > > Also invocation of dh-golang logged the following: > > >dh_golang -O--buildsystem=golang -O--builddirectory=_build > can't load package: package github.com/coreos/rkt/api/v1alpha: cannot > find package "github.com/coreos/rkt/api/v1alpha" in any of: > /usr/lib/go/src/github.com/coreos/rkt/api/v1alpha (from $GOROOT) > /build/rkt-1.4.0+dfsg/obj-x86_64-linux-gnu/src/ > github.com/coreos/rkt/api/v1alpha (from $GOPATH) > can't load package: package github.com/coreos/rkt/rkt: cannot find > package "github.com/coreos/rkt/rkt" in any of: > /usr/lib/go/src/github.com/coreos/rkt/rkt (from $GOROOT) > /build/rkt-1.4.0+dfsg/obj-x86_64-linux-gnu/src/ > github.com/coreos/rkt/rkt (from $GOPATH) > can't load package: package .: no buildable Go source files in > /build/rkt-1.4.0+dfsg > can't load package: package .: no buildable Go source files in > /build/rkt-1.4.0+dfsg > dpkg-query: error: --search needs at least one file name pattern argument > > Use --help for help about querying packages. > > > This is regression from "dh-golang" v1.12. > > Please investigate. > > -- > All the best, > Dmitry Smirnov > GPG key : 4096R/53968D1B > > --- > > Truth never damages a cause that is just. > -- Mahatma Gandhi > -- Best regards, Michael
Bug#821350: dh-golang: generates rubbish in Built-Using; errors on invocation
Package: dh-golang Version: 1.15 Severity: serious Recent upload of "rkt" was rejected rkt_1.4.0+dfsg-1_amd64.deb: Built-Using refers to non-existing source package apt (= 1.0.9.10) due to rubbish in Built-Using field: apt (= 1.0.9.10), apt (= 1.0.10.2), apt (= 1.2.10) Also invocation of dh-golang logged the following: dh_golang -O--buildsystem=golang -O--builddirectory=_build can't load package: package github.com/coreos/rkt/api/v1alpha: cannot find package "github.com/coreos/rkt/api/v1alpha" in any of: /usr/lib/go/src/github.com/coreos/rkt/api/v1alpha (from $GOROOT) /build/rkt-1.4.0+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/rkt/api/v1alpha (from $GOPATH) can't load package: package github.com/coreos/rkt/rkt: cannot find package "github.com/coreos/rkt/rkt" in any of: /usr/lib/go/src/github.com/coreos/rkt/rkt (from $GOROOT) /build/rkt-1.4.0+dfsg/obj-x86_64-linux-gnu/src/github.com/coreos/rkt/rkt (from $GOPATH) can't load package: package .: no buildable Go source files in /build/rkt-1.4.0+dfsg can't load package: package .: no buildable Go source files in /build/rkt-1.4.0+dfsg dpkg-query: error: --search needs at least one file name pattern argument Use --help for help about querying packages. This is regression from "dh-golang" v1.12. Please investigate. -- All the best, Dmitry Smirnov GPG key : 4096R/53968D1B --- Truth never damages a cause that is just. -- Mahatma Gandhi signature.asc Description: This is a digitally signed message part.