Re: [pkg-go] Minutes for the DebConf17 BoF

2018-01-25 Thread Martín Ferrari
On 24/01/18 21:45, Michael Stapelberg wrote:
> I filed a wishlist bug for the remote configuration:
> https://bugs.debian.org/888313

Thanks!

> To import it, AIUI, I should tag the latest upstream branch commit
> (7cafcd837844e784b526369c9bce262804aebc60) with tag
> upstream/0.0_git20160503.7cafcd8, then merge the upstream branch into
> the master branch with the merge strategy “theirs”.

What I do, is to first merge upstream at the point of the current
version (with --allow-unrelated-histories), which "should" be identical
to the current upstream, and after that merges are automatic.

> However, I can’t find a git-buildpackage command to take care of this.
> How do you do it? Manually?

Yup :)

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2018-01-25 Thread Martín Ferrari
On 23/01/18 21:48, Michael Stapelberg wrote:
> Status update: I now have tooling to perform the “delete existing

Yay!

> Also, I’m wondering how we can accomplish automated git remote
> configuration, i.e. how do people obtain an upstream branch which tracks
> the correct upstream remote branch? AFAICT, git-buildpackage has no
> solution for this (yet?). Does anyone know by chance?

For us, 99% of the time, the homepage field in d/control is what we
need. But it would be nicer to have something proper.

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2018-01-24 Thread Michael Stapelberg
I filed a wishlist bug for the remote configuration:
https://bugs.debian.org/888313

My tooling is now able to work with almost all repositories — there are
about 20 which still require some fixes, I’ll send out emails where I can’t
do the fix myself.

I have a beginner question regarding the git upstream workflow we’re
targeting:
How exactly would I import a new upstream version?

I looked at the golang-text repository (gbp clone vcsgit:golang-text),
removed the existing upstream branch, configured the correct remote and
tracked its master branch:
% git remote add upstream https://github.com/kr/text
% git fetch upstream
% git branch -D upstream
% git branch --track upstream upstream/master

The upstream branch indeed contains one commit which is not yet reflected
in the debian branch (“master”).

To import it, AIUI, I should tag the latest upstream branch commit
(7cafcd837844e784b526369c9bce262804aebc60) with tag
upstream/0.0_git20160503.7cafcd8, then merge the upstream branch into the
master branch with the merge strategy “theirs”.

However, I can’t find a git-buildpackage command to take care of this. How
do you do it? Manually?

On Tue, Jan 23, 2018 at 10:48 PM, Michael Stapelberg 
wrote:

> Status update: I now have tooling to perform the “delete existing upstream
> branch, create a tracking upstream branch in correct version from Go import
> path” step. The tooling still fails on about 60 of 901 repositories. I’ll
> take a look at the failures and give another status update.
>
> The VCS composition is:
> 889 git
> 3 hg
> 2 bzr
>
> I’ll need to look into how to use git-remote-{hg,bzr}.
>
> Also, I’m wondering how we can accomplish automated git remote
> configuration, i.e. how do people obtain an upstream branch which tracks
> the correct upstream remote branch? AFAICT, git-buildpackage has no
> solution for this (yet?). Does anyone know by chance?
>
> On Thu, Nov 9, 2017 at 10:06 PM, Michael Stapelberg  > wrote:
>
>> Status update: I discovered (and documented) gbp’s postclone=origtargz
>> option. Also, I prepared (and documented/referenced) two git hooks.
>>
>> I’m still not entirely certain about how to approach the “upstream
>> branch contains upstream git history” change in the best way.
>> Suggestions/tips welcome.
>>
>> On Thu, Nov 9, 2017 at 3:57 PM, Michael Stapelberg
>>  wrote:
>> > On Thu, Nov 9, 2017 at 6:44 AM, Martín Ferrari 
>> wrote:
>> >> On 09/11/17 04:24, Michael Stapelberg wrote:
>> >>
>> >>> At least for our transition period, we’ll have to use origtargz.
>> >>
>> >> Yes, but only for -2 and up releases. For -1, origtargz would do the
>> >> same as gbp (AFAIK).
>> >
>> > Yes, for -1 releases, users supply the orig tarball :)
>> >
>> >>
>> >>> I’m happy to pro-actively add compression algorithm/level options and
>> >>> evaluate at a later time whether that worked. I think just testing
>> >>> across different machines is a good start, but we should also pass the
>> >>> test of time — perhaps we can easily simulate that by testing on
>> >>> stable/oldstable.
>> >> Not sure if we need to devote a lot of time to this, after all, the
>> >> worst that can happen is a rejected upload because of checksum
>> mismatches.
>> >
>> > Personally, I find rejected uploads super frustrating, especially
>> > because of the large delay in between the action and the result (up to
>> > half an hour).
>> >
>> > I’d really like to define a workflow which makes them a thing of the
>> > past. For the time being, this is using origtargz.
>> >
>> > --
>> > Best regards,
>> > Michael
>>
>>
>>
>> --
>> Best regards,
>> Michael
>>
>
>
>
> --
> Best regards,
> Michael
>



-- 
Best regards,
Michael
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2018-01-23 Thread Michael Stapelberg
Status update: I now have tooling to perform the “delete existing upstream
branch, create a tracking upstream branch in correct version from Go import
path” step. The tooling still fails on about 60 of 901 repositories. I’ll
take a look at the failures and give another status update.

The VCS composition is:
889 git
3 hg
2 bzr

I’ll need to look into how to use git-remote-{hg,bzr}.

Also, I’m wondering how we can accomplish automated git remote
configuration, i.e. how do people obtain an upstream branch which tracks
the correct upstream remote branch? AFAICT, git-buildpackage has no
solution for this (yet?). Does anyone know by chance?

On Thu, Nov 9, 2017 at 10:06 PM, Michael Stapelberg 
wrote:

> Status update: I discovered (and documented) gbp’s postclone=origtargz
> option. Also, I prepared (and documented/referenced) two git hooks.
>
> I’m still not entirely certain about how to approach the “upstream
> branch contains upstream git history” change in the best way.
> Suggestions/tips welcome.
>
> On Thu, Nov 9, 2017 at 3:57 PM, Michael Stapelberg
>  wrote:
> > On Thu, Nov 9, 2017 at 6:44 AM, Martín Ferrari 
> wrote:
> >> On 09/11/17 04:24, Michael Stapelberg wrote:
> >>
> >>> At least for our transition period, we’ll have to use origtargz.
> >>
> >> Yes, but only for -2 and up releases. For -1, origtargz would do the
> >> same as gbp (AFAIK).
> >
> > Yes, for -1 releases, users supply the orig tarball :)
> >
> >>
> >>> I’m happy to pro-actively add compression algorithm/level options and
> >>> evaluate at a later time whether that worked. I think just testing
> >>> across different machines is a good start, but we should also pass the
> >>> test of time — perhaps we can easily simulate that by testing on
> >>> stable/oldstable.
> >> Not sure if we need to devote a lot of time to this, after all, the
> >> worst that can happen is a rejected upload because of checksum
> mismatches.
> >
> > Personally, I find rejected uploads super frustrating, especially
> > because of the large delay in between the action and the result (up to
> > half an hour).
> >
> > I’d really like to define a workflow which makes them a thing of the
> > past. For the time being, this is using origtargz.
> >
> > --
> > Best regards,
> > Michael
>
>
>
> --
> Best regards,
> Michael
>



-- 
Best regards,
Michael
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-09 Thread Michael Stapelberg
Status update: I discovered (and documented) gbp’s postclone=origtargz
option. Also, I prepared (and documented/referenced) two git hooks.

I’m still not entirely certain about how to approach the “upstream
branch contains upstream git history” change in the best way.
Suggestions/tips welcome.

On Thu, Nov 9, 2017 at 3:57 PM, Michael Stapelberg
 wrote:
> On Thu, Nov 9, 2017 at 6:44 AM, Martín Ferrari  wrote:
>> On 09/11/17 04:24, Michael Stapelberg wrote:
>>
>>> At least for our transition period, we’ll have to use origtargz.
>>
>> Yes, but only for -2 and up releases. For -1, origtargz would do the
>> same as gbp (AFAIK).
>
> Yes, for -1 releases, users supply the orig tarball :)
>
>>
>>> I’m happy to pro-actively add compression algorithm/level options and
>>> evaluate at a later time whether that worked. I think just testing
>>> across different machines is a good start, but we should also pass the
>>> test of time — perhaps we can easily simulate that by testing on
>>> stable/oldstable.
>> Not sure if we need to devote a lot of time to this, after all, the
>> worst that can happen is a rejected upload because of checksum mismatches.
>
> Personally, I find rejected uploads super frustrating, especially
> because of the large delay in between the action and the result (up to
> half an hour).
>
> I’d really like to define a workflow which makes them a thing of the
> past. For the time being, this is using origtargz.
>
> --
> Best regards,
> Michael



-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-09 Thread Martín Ferrari
On 09/11/17 04:24, Michael Stapelberg wrote:

> At least for our transition period, we’ll have to use origtargz.

Yes, but only for -2 and up releases. For -1, origtargz would do the
same as gbp (AFAIK).

> I’m happy to pro-actively add compression algorithm/level options and
> evaluate at a later time whether that worked. I think just testing
> across different machines is a good start, but we should also pass the
> test of time — perhaps we can easily simulate that by testing on
> stable/oldstable.
Not sure if we need to devote a lot of time to this, after all, the
worst that can happen is a rejected upload because of checksum mismatches.

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-08 Thread Michael Stapelberg
Thanks for looking into this.

I think mandating compression algorithms and levels is necessary for
this approach indeed, but I’m not sure if it’s sufficient.

At least for our transition period, we’ll have to use origtargz.

I’m happy to pro-actively add compression algorithm/level options and
evaluate at a later time whether that worked. I think just testing
across different machines is a good start, but we should also pass the
test of time — perhaps we can easily simulate that by testing on
stable/oldstable.

On Thu, Nov 9, 2017 at 1:27 AM, Martín Ferrari  wrote:
> On 08/11/17 21:01, Martín Ferrari wrote:
>> The best test would be to use gbp to create the tarballs under different
>> conditions (machine, user name, path, manually touch()ing files locally)
>> and see if they are really reproducible.
>
> For one data point, I just tried this on two different machines (same
> arch, though), on different paths, one a fresh clone, other my usual
> work dir, and after some random touch() of files, I get always the same tar.
>
> $ gbp buildpackage --git-force-create --git-no-pristine-tar
> --git-compression=gzip --git-compression-level=9
>
> $ sha256sum ../build-area/prometheus_1.8.1+ds.orig.tar.gz
> 726f7c392f99b48b63a85bc8f873fbdecbf6fabbb167a2dd7be312bdcf56d60c
> ../build-area/prometheus_1.8.1+ds.orig.tar.gz
>
>
> Which, notably, does not match what's on the archive. It seems I had
> different default values for the compression level on different
> machines, so I had to pass the parameters explicitly.
>
> If I use compression level 6, I get that exact SHA:
>
> $ sha256sum ../build-area/prometheus_1.8.1+ds.orig.tar.gz
> 726f7c392f99b48b63a85bc8f873fbdecbf6fabbb167a2dd7be312bdcf56d60c
> ../build-area/prometheus_1.8.1+ds.orig.tar.gz
>
> I think if we mandate some fixed parameters (by policy or inclusion in
> debian/gbp.conf), this approach would be feasible.
>
> --
> Martín Ferrari (Tincho)



-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-08 Thread Martín Ferrari
On 08/11/17 21:01, Martín Ferrari wrote:
> The best test would be to use gbp to create the tarballs under different
> conditions (machine, user name, path, manually touch()ing files locally)
> and see if they are really reproducible.

For one data point, I just tried this on two different machines (same
arch, though), on different paths, one a fresh clone, other my usual
work dir, and after some random touch() of files, I get always the same tar.

$ gbp buildpackage --git-force-create --git-no-pristine-tar
--git-compression=gzip --git-compression-level=9

$ sha256sum ../build-area/prometheus_1.8.1+ds.orig.tar.gz
726f7c392f99b48b63a85bc8f873fbdecbf6fabbb167a2dd7be312bdcf56d60c
../build-area/prometheus_1.8.1+ds.orig.tar.gz


Which, notably, does not match what's on the archive. It seems I had
different default values for the compression level on different
machines, so I had to pass the parameters explicitly.

If I use compression level 6, I get that exact SHA:

$ sha256sum ../build-area/prometheus_1.8.1+ds.orig.tar.gz
726f7c392f99b48b63a85bc8f873fbdecbf6fabbb167a2dd7be312bdcf56d60c
../build-area/prometheus_1.8.1+ds.orig.tar.gz

I think if we mandate some fixed parameters (by policy or inclusion in
debian/gbp.conf), this approach would be feasible.

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-08 Thread Martín Ferrari
On 08/11/17 17:55, Michael Stapelberg wrote:

> Quote from the commit message
> (https://anonscm.debian.org/git/pkg-go/website.git/commit/?id=866810cfeea8086dbdfc0176b148f7a063e3ac0b):
One comment on this:

+NOTE: Using `--git-upstream-tree=TAG` (the default) is not sufficient
to obtain
+a byte-for-byte equal orig .tar.gz file. In stapelberg’s tests, only 5% of
+pkg-go’s git repositories would match the orig .tar.gz in the archive.

I think this test is not useful, because probably that 95% of packages
matches those whose tars were not created by gbp, but by upstream or by
pristine-tar.

If you are using gbp, the tars are meant to be reproducible, but not
necesarily equal to pristine-tar. I think it uses last commit time to
set the mtime of each file, and that could surely be different in a
tarball created by upstream.

The best test would be to use gbp to create the tarballs under different
conditions (machine, user name, path, manually touch()ing files locally)
and see if they are really reproducible.

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-08 Thread Martín Ferrari
On 08/11/17 17:55, Michael Stapelberg wrote:
> I started a document describing the changes we agreed on, the
> rationale behind them, the old/new workflows and the migration
> strategy.

Thanks!! I am very glad we are moving forward with this!

> I hope I can make some progress on this in the next few days. Any help
> is welcome.

I will try to squeeze some contributions.

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-08 Thread Michael Stapelberg
I started a document describing the changes we agreed on, the
rationale behind them, the old/new workflows and the migration
strategy.

Quote from the commit message
(https://anonscm.debian.org/git/pkg-go/website.git/commit/?id=866810cfeea8086dbdfc0176b148f7a063e3ac0b):

> Quite a number of points haven’t been entirely researched yet.
>
> Any help is welcome.
>
> We can do mass-migrations of packages once the document is in a good shape and
> the steps have been tested on a few packages.

Find the HTML version at https://pkg-go.alioth.debian.org/workflow-changes.html

I hope I can make some progress on this in the next few days. Any help
is welcome.

On Mon, Nov 6, 2017 at 7:15 PM, Martín Ferrari  wrote:
> On 06/11/17 15:04, Martín Ferrari wrote:
>
>> I have added something that I think it is widely accepted, but was not
>> explicitly discussed, and that I would like to be set on policy: should
>> we mandate merging of upstream code in debian branch?
>
> Just to clarify: I don't think this new point should delay the rest of
> the decisions. I think the other points had enough time by now to be
> considered settled.
>
>
> --
> Martín Ferrari (Tincho)



-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-06 Thread Martín Ferrari
On 06/11/17 15:04, Martín Ferrari wrote:

> I have added something that I think it is widely accepted, but was not
> explicitly discussed, and that I would like to be set on policy: should
> we mandate merging of upstream code in debian branch?

Just to clarify: I don't think this new point should delay the rest of
the decisions. I think the other points had enough time by now to be
considered settled.


-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-06 Thread Martín Ferrari
On 06/11/17 04:04, Michael Stapelberg wrote:
> Reminder: The deadline is approaching in a few hours. Please have a final 
> look.

Thanks!

I have added something that I think it is widely accepted, but was not
explicitly discussed, and that I would like to be set on policy: should
we mandate merging of upstream code in debian branch?

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-11-05 Thread Michael Stapelberg
Reminder: The deadline is approaching in a few hours. Please have a final look.

You’ll hear from me this evening,

On Wed, Oct 25, 2017 at 10:03 AM, Michael Stapelberg
 wrote:
> Everyone, thanks for your updates, that helped clarify things!
>
> I have updated the document to reflect the new positions. It seems
> like we’re agreeing on having the upstream git history in the upstream
> branch, and that also means importing new upstream versions no longer
> needs special tooling.
>
> Remaining undecided items are how we repackage non-DFSG free upstream,
> how we manage patches, and whether/in which way we should support
> dgit.
>
> Please continue to add your comments/positions in the pad.
>
> On Tue, Oct 24, 2017 at 10:05 PM, Martín Ferrari  wrote:
>> On 21/10/17 17:10, Michael Stapelberg wrote:
>>
>>> Please give it a look at
>>> https://oasis.sandstorm.io/shared/l4YsLYdwUgnhzzwmKS_TGwWyVQ9Ol8EKiMR8qhpHaS2
>>> and add any remaining discussion points you would like to bring up
>>> within the next 14 days, i.e. no later than 2017-11-04.
>>
>> I added my comments, and updated my preferences (mainly, that since
>> albertito pointed me to gbp's ability to create reproducible tarballs, I
>> have stopped using pristine-tar).
>>
>> What I feel is not too clear there is that many discussion points depend
>> on others; the most important one being to follow upstream git history
>> or not.
>>
>> In any case, it is important that other people check it out and comment.
>> So far, I only see edits my Michael and me. Having a clear policy is
>> very important to make the team more productive and easy to work with!
>>
>> --
>> Martín Ferrari (Tincho)
>>
>> ___
>> Pkg-go-maintainers mailing list
>> Pkg-go-maintainers@lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
>
>
>
> --
> Best regards,
> Michael



-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-10-25 Thread Michael Stapelberg
Everyone, thanks for your updates, that helped clarify things!

I have updated the document to reflect the new positions. It seems
like we’re agreeing on having the upstream git history in the upstream
branch, and that also means importing new upstream versions no longer
needs special tooling.

Remaining undecided items are how we repackage non-DFSG free upstream,
how we manage patches, and whether/in which way we should support
dgit.

Please continue to add your comments/positions in the pad.

On Tue, Oct 24, 2017 at 10:05 PM, Martín Ferrari  wrote:
> On 21/10/17 17:10, Michael Stapelberg wrote:
>
>> Please give it a look at
>> https://oasis.sandstorm.io/shared/l4YsLYdwUgnhzzwmKS_TGwWyVQ9Ol8EKiMR8qhpHaS2
>> and add any remaining discussion points you would like to bring up
>> within the next 14 days, i.e. no later than 2017-11-04.
>
> I added my comments, and updated my preferences (mainly, that since
> albertito pointed me to gbp's ability to create reproducible tarballs, I
> have stopped using pristine-tar).
>
> What I feel is not too clear there is that many discussion points depend
> on others; the most important one being to follow upstream git history
> or not.
>
> In any case, it is important that other people check it out and comment.
> So far, I only see edits my Michael and me. Having a clear policy is
> very important to make the team more productive and easy to work with!
>
> --
> Martín Ferrari (Tincho)
>
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers



-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-10-24 Thread Martín Ferrari
On 21/10/17 17:10, Michael Stapelberg wrote:

> Please give it a look at
> https://oasis.sandstorm.io/shared/l4YsLYdwUgnhzzwmKS_TGwWyVQ9Ol8EKiMR8qhpHaS2
> and add any remaining discussion points you would like to bring up
> within the next 14 days, i.e. no later than 2017-11-04.

I added my comments, and updated my preferences (mainly, that since
albertito pointed me to gbp's ability to create reproducible tarballs, I
have stopped using pristine-tar).

What I feel is not too clear there is that many discussion points depend
on others; the most important one being to follow upstream git history
or not.

In any case, it is important that other people check it out and comment.
So far, I only see edits my Michael and me. Having a clear policy is
very important to make the team more productive and easy to work with!

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-10-21 Thread Michael Stapelberg
Sorry for taking so long to reply with this, but I’ve had little time recently.

I think that, at this point, it makes sense to structure the
discussion a little bit more. To that end, I would like to step up as
moderator, and have also created an etherpad, which should allow for a
more pleasant form of collaborative communication around documented
goals than a seemingly endless email thread (I can’t keep it in my
head).

Please give it a look at
https://oasis.sandstorm.io/shared/l4YsLYdwUgnhzzwmKS_TGwWyVQ9Ol8EKiMR8qhpHaS2
and add any remaining discussion points you would like to bring up
within the next 14 days, i.e. no later than 2017-11-04.

We can continue the discussion longer than that, but we need to cap
the volume of what we talk about at some point.

In case we can’t decide on all pending topics within November, I
suggest we do a vote in early December, then find volunteers to help
with implementing the decisions we make — New Year’s resolutions? ;-)

I’ll periodically remind people of outstanding work items because the
etherpad has no email notifications.

On Sat, Oct 7, 2017 at 12:08 PM, Alberto Bertogli
 wrote:
> On Tue, Aug 15, 2017 at 11:02:39PM +0200, Michael Stapelberg wrote:
>> 6. I have come to appreciate pristine-tar, as it is the only reliable way
>> (that I know of) to prepare uploads for packages which already have their
>> orig tarball in Debian. Whenever I come across a package which isn’t using
>> pristine-tar, the generated orig tarball will have a different checksum,
>> and my upload will be rejected. I’d suggest we codify that pristine-tar is
>> a requirement.
>
> As someone very new to packaging things for Debian, here are my 2 cents.
>
> For projects with an upstream git repository, I find packaging using two
> branches to be much easier and practical:
>
>  - One branch "upstream" which follows uptream directly.
>  - One branch "debian" which is based on "upstream" + patches to the
>debian/ directory.
>To update the package, I just do "git merge upstream" and then make
>the required changes to the debian/ directory.
>
> I find this result in a repo with clean and useful history, as I can
> clearly see in the "debian" log the points where the package was brought
> up to date, and what changes it required. It also means there's no easy
> way to accidentally import a mutated upstream (which can happen if one
> just takes snapshots) so it's harder to make mistakes.
>
>
> No pristine tar branches, those I find terribly confusing and
> unnecessary: if everything is in the git repository already, why take
> it, turn it into a tar, and shove it back into git again? This requires
> additional tooling and maintenance for, what I can see, no real gain.
>
> I noticed that "gbp buildpackage" works just fine without it, and can
> build the tar based on the git repo just fine, and in a reproducible
> way. The generated tree is 100% reproducible, the points of
> non-reproducibility are tar and compression so they're should be rare.
>
> Asking around, it seems that needing the tars is not that common; and
> when needed, they can be downloaded from the archives anyway if tar/gz
> happened to have changed.
>
>
>> > dh-make-golang
>> > --
>> >
>> > A few times people expressed the desire for dh-make-golang to grow an
>> > `--update` option, as most packages are trivial to update, but tedious
>> > to do so.
>
> This * 1000 :)
>
> I imagine it would require some assumptions about the way the packages
> are handled (branch naming, how upstream is tracked, etc.), which makes
> having some degree of consensus about it even more important.
>
>
>> > API changes in upstream
>> > ---
>> >
>> > We ranted at length about upstreams, and noted that we need a system
>> > that provides early warning of API breakages. We discussed using ratt
>> > and autopkgtest for that purpose.
>> >
>> > Aviau pointed out that he usually requests upstreams to make releases
>> > and that he is usually successful. Tincho pointed out the problems with
>> > meaningless releases and with upstreams releasing once and then
>> > forgetting to do it when needed.
>> >
>> > We discussed the possibility of changing "soname"s by renaming packages
>> > when we detect API incompatibilities, but concluded that in general it
>> > is too much work and that it makes more sense to try and fix
>> > reverse-dependencies by bugging upstream or patching them ourselves.
>
> For C, there were at some point public API checkers that take your code
> and will send you a warning if you make breaking changes.
>
> I think such a tool for Go would go a long way with this. You can give
> it a "go get"able package, and it would track it and report when the
> public API changes, and warn of incompatible changes.
>
>
>> >   * aviau volunteers to document dh_golang options.
>
> As a newbie, I'd really appreciate this!
>
> I often just rely on asking on IRC or to Tincho directly, because the
> 

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-10-07 Thread Alberto Bertogli
On Tue, Aug 15, 2017 at 11:02:39PM +0200, Michael Stapelberg wrote:
> 6. I have come to appreciate pristine-tar, as it is the only reliable way
> (that I know of) to prepare uploads for packages which already have their
> orig tarball in Debian. Whenever I come across a package which isn’t using
> pristine-tar, the generated orig tarball will have a different checksum,
> and my upload will be rejected. I’d suggest we codify that pristine-tar is
> a requirement.

As someone very new to packaging things for Debian, here are my 2 cents.

For projects with an upstream git repository, I find packaging using two
branches to be much easier and practical:

 - One branch "upstream" which follows uptream directly.
 - One branch "debian" which is based on "upstream" + patches to the
   debian/ directory.
   To update the package, I just do "git merge upstream" and then make
   the required changes to the debian/ directory.

I find this result in a repo with clean and useful history, as I can
clearly see in the "debian" log the points where the package was brought
up to date, and what changes it required. It also means there's no easy
way to accidentally import a mutated upstream (which can happen if one
just takes snapshots) so it's harder to make mistakes.


No pristine tar branches, those I find terribly confusing and
unnecessary: if everything is in the git repository already, why take
it, turn it into a tar, and shove it back into git again? This requires
additional tooling and maintenance for, what I can see, no real gain.

I noticed that "gbp buildpackage" works just fine without it, and can
build the tar based on the git repo just fine, and in a reproducible
way. The generated tree is 100% reproducible, the points of
non-reproducibility are tar and compression so they're should be rare.

Asking around, it seems that needing the tars is not that common; and
when needed, they can be downloaded from the archives anyway if tar/gz
happened to have changed.


> > dh-make-golang
> > --
> >
> > A few times people expressed the desire for dh-make-golang to grow an
> > `--update` option, as most packages are trivial to update, but tedious
> > to do so.

This * 1000 :)

I imagine it would require some assumptions about the way the packages
are handled (branch naming, how upstream is tracked, etc.), which makes
having some degree of consensus about it even more important.


> > API changes in upstream
> > ---
> >
> > We ranted at length about upstreams, and noted that we need a system
> > that provides early warning of API breakages. We discussed using ratt
> > and autopkgtest for that purpose.
> >
> > Aviau pointed out that he usually requests upstreams to make releases
> > and that he is usually successful. Tincho pointed out the problems with
> > meaningless releases and with upstreams releasing once and then
> > forgetting to do it when needed.
> >
> > We discussed the possibility of changing "soname"s by renaming packages
> > when we detect API incompatibilities, but concluded that in general it
> > is too much work and that it makes more sense to try and fix
> > reverse-dependencies by bugging upstream or patching them ourselves.

For C, there were at some point public API checkers that take your code
and will send you a warning if you make breaking changes.

I think such a tool for Go would go a long way with this. You can give
it a "go get"able package, and it would track it and report when the
public API changes, and warn of incompatible changes.


> >   * aviau volunteers to document dh_golang options.

As a newbie, I'd really appreciate this!

I often just rely on asking on IRC or to Tincho directly, because the
way it works seems quite opaque to me.  Luckily most of the things I've
touched so far have been simple enough not to need this.


Thanks!
Alberto


___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
On 20/08/17 19:21, Michael Stapelberg wrote:

>> Gccgo has many quirks. One is that it does not use the vendor directory
>> (I need to check if this is true with the latest version), so you might
>> need to copy vendor into the builddirectory.
> 
> …hopefully only temporarily, though, right? Ideally, we wouldn’t have
> any vendored source in our packages.

Gah! Another thing I forgot to talk about! go packaging is hard.. :)

Yes, most of the time I kill all vendoring, but there have been some
exceptions: 1) small, useless libraries that I see no point in packaging
separately; and 2) self-contained parts of a library, that avoid
dragging 100s of dependencies.

The latter happened to me recently with prometheus: I had removed the
vendoring of the consul API, but when I tried to backport that, I
realised I'd need to backport consul, docker, and way too many dependencies.

Sadly, even if the client API is in a different package, the source
packages have long dependency chains. I am starting to think that for
some of these packages, having a separate source package with client
APIs would make sense.


>> BUILDFLAGS := -ldflags \
>>   " -X $(METAPKG)/version.Version=$(VERSION)\
>> -X $(METAPKG)/version.Revision=$(REV)\
>> -X $(METAPKG)/version.Branch=$(BRANCH)\
>> -X $(METAPKG)/version.BuildUser=$(USER)\
>> -X $(METAPKG)/version.BuildDate=$(BUILD_DATE)\
>> -X $(METAPKG)/version.GoVersion=$(GO_VERSION)"
> 
> What does METAPKG resolve to? We should consider centralizing these
> definitions somewhere.

Ah, this is prometheus-specific. All the meta info is stored in the
prometheus/common namespace, so I defined that earlier in the rules file.


-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Michael Stapelberg
On Sun, Aug 20, 2017 at 7:18 PM, Martín Ferrari  wrote:
> On 20/08/17 17:35, Martín Ferrari wrote:
>> So, my turn to describe workflows.
>
> Some things I forgot in my previous email:
>
> * I am not married to the idea of dch + debcommit, specially when I have
> merge conflicts. I understand the merits of git commit + gbp dch.
> * The export=WC option in gbp.conf is to make sure my uncommitted
> changes are taken into account when building, which allows me to
> experiment and test before committing.
>
> Some notes not about workflow, but about packaging strategies:
>
> 1. debian/control
>
> I wrap and sort all multi-entry fields, with a trailing comma at every
> line (to minimise diffs). I always did this by hand (or with my
> pkg-go-common-fixes script), but a friend just recommended the
> wrap-and-sort script from devscripts, and I think we should all use that
> with the same options.
>
> Even if this does not match what I am currently doing (indent after
> colon), I think it makes more sense:
>
> $ wrap-and-sort -st
>
> Add myself as Uploader for any package where I do some non-trivial
> amount of work. Although I think this usage of Uploaders is being
> challenged project-wise.

Strong +1. Let’s auto-format all things we can reasonably auto-format.
debian/control files are a good candidate for that :).

>
> Add Testsuite: autopkgtest-pkg-go to every package.
>
> 2. debian/rules
>
> Keep it as minimal as possible.
>
> To add extra needed files (for tests and the such) use
> DH_GOLANG_INSTALL_EXTRA, no manual copying and no DH_GOLANG_INSTALL_ALL.
>
> To avoid compilation and testing of some packages use DH_GOLANG_EXCLUDES.
>
> If I need to do something in the build directory, pass --builddirectory
> to dh, so it is a known location (as opposed to getting the weird path
> from debhelper)
>
> Gccgo has many quirks. One is that it does not use the vendor directory
> (I need to check if this is true with the latest version), so you might
> need to copy vendor into the builddirectory.

…hopefully only temporarily, though, right? Ideally, we wouldn’t have
any vendored source in our packages.

>
> For detecting gccgo, and doing special things:
>
> GCCGO  := $(strip $(shell go version | grep gccgo))
> ifneq ($(GCCGO),)
> ..
> endif
>
> Many programs include versions, build info, date, etc, through linker
> variables. To make it consistent and reproducible I use this:
>
> DEBVERS?= $(shell dpkg-parsechangelog -SVersion)
> VERSION?= $(shell echo '$(DEBVERS)' | sed 's/^[[:digit:]]*://;
> s/[-].*//')
> DEBDATE?= $(shell dpkg-parsechangelog -SDate)
> REV:= $(DEBVERS)
> BRANCH := debian/sid
> USER   := pkg-go-maintainers@lists.alioth.debian.org
> HOSTNAME   := debian
> BUILD_DATE := $(shell date --utc --date='$(DEBDATE)' +%Y%m%d-%H:%M:%S)
> GO_VERSION := $(shell go version | sed 's/go version \(\S*\).*/\1/')
> BUILDFLAGS := -ldflags \
>   " -X $(METAPKG)/version.Version=$(VERSION)\
> -X $(METAPKG)/version.Revision=$(REV)\
> -X $(METAPKG)/version.Branch=$(BRANCH)\
> -X $(METAPKG)/version.BuildUser=$(USER)\
> -X $(METAPKG)/version.BuildDate=$(BUILD_DATE)\
> -X $(METAPKG)/version.GoVersion=$(GO_VERSION)"

What does METAPKG resolve to? We should consider centralizing these
definitions somewhere.


-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Michael Stapelberg
On Sun, Aug 20, 2017 at 7:05 PM, Martín Ferrari  wrote:
> On 20/08/17 18:46, Michael Stapelberg wrote:
>
>> Side note, not meant to persuade anyone one way or the other: I just
>> realized why I never saw any appeal in that argument: I find git
>> packaging (or git in general?) too brittle and confusing to keep what
>> I consider are multiple projects in the same repository.
>
> Uhm.. I don't really have that feeling. Could you elaborate more?

I don’t want to derail this thread too much, so I’ll keep it brief:

Regarding repos being brittle: in the past, I’ve often found that when
messing up (e.g. when incorrectly importing a new upstream version, or
doing an incorrect merge of some sort), the easiest way to undo is to
delete the repository and start from scratch.

Regarding multiple projects being confusing: I find the history of git
repositories often not that easy to understand, even though I work
with git daily since over 9 years. When multiple branches are
involved, this becomes even harder. Before running any command in an
upstream-tracking git packaging repo, I need to stop and think about
how this affects the repo as a whole.

Maybe this gets easier over time, but I do like the mental simplicity
of having everything upstream-related nicely contained in a tarball.

>
>> When I need to find out something about upstream repositories, I
>> usually use the GitHub web interface, or my local gopath. I never use
>> the git packaging repos, regardless of whether they have history or
>> not.
>
> Heh, I hate the github web interface, can't compare to gitk, git log, etc :)
>
> Also, I don't even have a go path. To this day I get confused every time
> I try to build things by hand!
>
>
>> git config --add remote.origin.push "+refs/heads/*:refs/heads/*"
>> git config --add remote.origin.push "+refs/tags/*:refs/tags/*"
>
> The problem with this is that you push all tags and branches, even if
> they are coming from upstream (I know, not relevant for you). I try to
> keep the alioth repo free from that.
>
>> But note that gbp recently gained “gbp push”:
>> https://git.sigxcpu.org/cgit/git-buildpackage//commit/?id=cbacdfb40ca35633da06e9e05497ac0fb56cc4f9
>> It’s included in 0.9.0~exp2, but I haven’t tried it out yet.
>> Hopefully, it makes both our extra setup steps unnecessary :).
>
> Oh, cool, I should try it!
>
>> Given that you _also_ maintain history in git, using gbp dch seems
>> like significantly cutting down the number of commands. Is there any
>> rationale behind your decision to not use gbp dch, or are you just
>> used to this way? :)
>
> Mostly historical reasons and muscle memory :)
>
>
> --
> Martín Ferrari (Tincho)
>
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers



-- 
Best regards,
Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
On 20/08/17 17:35, Martín Ferrari wrote:
> So, my turn to describe workflows.

Some things I forgot in my previous email:

* I am not married to the idea of dch + debcommit, specially when I have
merge conflicts. I understand the merits of git commit + gbp dch.
* The export=WC option in gbp.conf is to make sure my uncommitted
changes are taken into account when building, which allows me to
experiment and test before committing.

Some notes not about workflow, but about packaging strategies:

1. debian/control

I wrap and sort all multi-entry fields, with a trailing comma at every
line (to minimise diffs). I always did this by hand (or with my
pkg-go-common-fixes script), but a friend just recommended the
wrap-and-sort script from devscripts, and I think we should all use that
with the same options.

Even if this does not match what I am currently doing (indent after
colon), I think it makes more sense:

$ wrap-and-sort -st

Add myself as Uploader for any package where I do some non-trivial
amount of work. Although I think this usage of Uploaders is being
challenged project-wise.

Add Testsuite: autopkgtest-pkg-go to every package.

2. debian/rules

Keep it as minimal as possible.

To add extra needed files (for tests and the such) use
DH_GOLANG_INSTALL_EXTRA, no manual copying and no DH_GOLANG_INSTALL_ALL.

To avoid compilation and testing of some packages use DH_GOLANG_EXCLUDES.

If I need to do something in the build directory, pass --builddirectory
to dh, so it is a known location (as opposed to getting the weird path
from debhelper)

Gccgo has many quirks. One is that it does not use the vendor directory
(I need to check if this is true with the latest version), so you might
need to copy vendor into the builddirectory.

For detecting gccgo, and doing special things:

GCCGO  := $(strip $(shell go version | grep gccgo))
ifneq ($(GCCGO),)
..
endif

Many programs include versions, build info, date, etc, through linker
variables. To make it consistent and reproducible I use this:

DEBVERS?= $(shell dpkg-parsechangelog -SVersion)
VERSION?= $(shell echo '$(DEBVERS)' | sed 's/^[[:digit:]]*://;
s/[-].*//')
DEBDATE?= $(shell dpkg-parsechangelog -SDate)
REV:= $(DEBVERS)
BRANCH := debian/sid
USER   := pkg-go-maintainers@lists.alioth.debian.org
HOSTNAME   := debian
BUILD_DATE := $(shell date --utc --date='$(DEBDATE)' +%Y%m%d-%H:%M:%S)
GO_VERSION := $(shell go version | sed 's/go version \(\S*\).*/\1/')
BUILDFLAGS := -ldflags \
  " -X $(METAPKG)/version.Version=$(VERSION)\
-X $(METAPKG)/version.Revision=$(REV)\
-X $(METAPKG)/version.Branch=$(BRANCH)\
-X $(METAPKG)/version.BuildUser=$(USER)\
-X $(METAPKG)/version.BuildDate=$(BUILD_DATE)\
-X $(METAPKG)/version.GoVersion=$(GO_VERSION)"


-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
On 20/08/17 18:46, Michael Stapelberg wrote:

> Side note, not meant to persuade anyone one way or the other: I just
> realized why I never saw any appeal in that argument: I find git
> packaging (or git in general?) too brittle and confusing to keep what
> I consider are multiple projects in the same repository.

Uhm.. I don't really have that feeling. Could you elaborate more?

> When I need to find out something about upstream repositories, I
> usually use the GitHub web interface, or my local gopath. I never use
> the git packaging repos, regardless of whether they have history or
> not.

Heh, I hate the github web interface, can't compare to gitk, git log, etc :)

Also, I don't even have a go path. To this day I get confused every time
I try to build things by hand!


> git config --add remote.origin.push "+refs/heads/*:refs/heads/*"
> git config --add remote.origin.push "+refs/tags/*:refs/tags/*"

The problem with this is that you push all tags and branches, even if
they are coming from upstream (I know, not relevant for you). I try to
keep the alioth repo free from that.

> But note that gbp recently gained “gbp push”:
> https://git.sigxcpu.org/cgit/git-buildpackage//commit/?id=cbacdfb40ca35633da06e9e05497ac0fb56cc4f9
> It’s included in 0.9.0~exp2, but I haven’t tried it out yet.
> Hopefully, it makes both our extra setup steps unnecessary :).

Oh, cool, I should try it!

> Given that you _also_ maintain history in git, using gbp dch seems
> like significantly cutting down the number of commands. Is there any
> rationale behind your decision to not use gbp dch, or are you just
> used to this way? :)

Mostly historical reasons and muscle memory :)


-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Michael Stapelberg
While we’re on the topic:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859867 discusses
making sbuild easier to install/use :)

On Sun, Aug 20, 2017 at 6:56 PM, Martín Ferrari  wrote:
> On 20/08/17 18:36, Michael Stapelberg wrote:
>> I use gbp with sbuild, and I do see different behavior with/without
>> exporting. Take for example the freeradius repository, where the build
>> fails without git-export-dir: https://paste.debian.net/982241/
>
> I guess the difference is with not having an export option at all, which
> I have never tried..
>
 2. My ~/.gbp.conf reads https://paste.debian.net/hidden/a48afca2/
 

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
On 20/08/17 18:36, Michael Stapelberg wrote:
> I use gbp with sbuild, and I do see different behavior with/without
> exporting. Take for example the freeradius repository, where the build
> fails without git-export-dir: https://paste.debian.net/982241/

I guess the difference is with not having an export option at all, which
I have never tried..

>>> 2. My ~/.gbp.conf reads https://paste.debian.net/hidden/a48afca2/
>>> 

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Michael Stapelberg
On Sun, Aug 20, 2017 at 2:30 PM, Martín Ferrari  wrote:
> Hi Mickael,
>
> I haven't yet got the time to write down properly my workflow, but I
> will still reply to some points. :)
>
> On 15/08/17 23:02, Michael Stapelberg wrote:
>
>> 1. I store packaging in e.g. ~/d/pkg/gocryptfs and build using `gbp
>> buildpackage --git-export-dir=~/d/out/gocryptfs`. By using
>> --git-export-dir, my working copy always stays clean. By collecting the
>> output files per package, I can easily debdiff between versions. This
>> point is informational and shouldn’t have any bearing on a canonical
>> workflow.
>
> Why do you need this? I use gbp with cowbuilder, and so the working copy
> is never touched. Looking at my gbp.conf, I see my default is to export
> to '../build-area', but probably that does not change much.

I use gbp with sbuild, and I do see different behavior with/without
exporting. Take for example the freeradius repository, where the build
fails without git-export-dir: https://paste.debian.net/982241/

>
>> 2. My ~/.gbp.conf reads https://paste.debian.net/hidden/a48afca2/
>> 

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
So, my turn to describe workflows.

I use gbp, pristine-tar, cowbuilder (but planning to move to sbuild),
dch, debcommit as my main tools. I have not really used dh-make-golang much.

My global gbp configuration is as follows:

[DEFAULT]
pristine-tar = True
sign-tags = True

[buildpackage]
export = WC
postbuild = lintian $GBP_CHANGES_FILE && debsign $GBP_CHANGES_FILE
export-dir = ../build-area/
tarball-dir = ../tarballs/
ignore-new = True
pbuilder = True

[import-orig]
postimport = dch -v%(version)s New upstream release.


In general, I try to keep existing structures in git, but for packages I
prepare, I use DEP14 branches and tags:

 * HEAD is debian/sid, other dists are debian/.
 * upstream/ tag for unpacked upstream source.
 * debian/ tag for uploaded packages.
 * I keep an "upstream" branch for unpackaged upstream sources, which is
tagged and then merged into the main packaging branch.

Depending on upstream having releases or not, I either use gbp
import-orig --uscan, or follow git history with an "upstream" remote. In
both cases, the sources end in a git branch.

For repackaging, I either use "excluded-files" in debian/changelog for
released projects, or create a second upstream branch "repackaged",
"unvendored", or somesuch, where I make changes and merge from
"upstream" after every import.

For the go ecosystem, I am feeling it is much more useful and productive
to keep complete upstream history than using the releases, and for some
packages with releases I started using this. It pays off every time I
need to troubleshoot api changes, backport fixes, and the such..

When I am doing this, I add this gbp.conf snippet in the debian/ directory:

[buildpackage]
dist = DEP14
upstream-tag = upstream/%(version)s
pristine-tar = True
pristine-tar-commit = True

For simplifying pushes, I also add this config:

$ git config --add remote.debian.push 'refs/heads/debian/*'
$ git config --add remote.debian.push 'refs/heads/upstream'
$ git config --add remote.debian.push 'refs/heads/pristine-tar'
$ git config --add remote.debian.push 'refs/tags/debian/*'
$ git config --add remote.debian.push 'refs/tags/upstream/*'
$ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'


My preference for debian/changelog is:
 * distribution=UNRELEASED until the package is ready to upload.
 * I use dch for every change, and debcommit to commit with that same
message.
 * dch -r when package is ready for upload (changes dist).
 * debcommit -r after uploading (creates signed tag).


My workflow with upstream history is as this:

1. Init

(on alioth)

$ /git/pkg-go/setup-repository foo
$ git --git-dir=/git/pkg-go/packages/... symbolic-ref HEAD
refs/heads/debian/sid

(locally)
$ git remote add upstream https://...
$ git remote add debian ssh://git.debian.org/git/pkg-go/packages/...
$ git fetch --all
$ git checkout -b upstream upstream/master # or upstream tag
$ git tag upstream/0.0+gitMMDD.NNN
# optional, if repackaging is needed.
$ git ckeckout -b repackaged
$ git rm -rf vendor/; git commit -m repackage
$ git tag upstream/0.0+gitMMDD.NNN-ds
$ git checkout -b debian/sid
# create debian/* stuff, with dh-make-golang, or (usually) copy from
another package :-). Add gbp.conf snippet, git config, etc

2. Update

$ git fetch --all
$ git checkout debian/sid
$ git tag upstream/0.0+gitMMDD.NNN NNN
$ git merge upstream/0.0+gitMMDD.NNN
$ dch -v 0.0+gitMMDD.NNN-1 'New upstream snapshot.'
$ git add debian/changelog; debcommit
$ gbp buildpackage  # This also creates the pristine tar from the tag.

3. Work on the package

# Fix stuff
$ dch 'debian/foo: fix bar'; debcommit -a

# When finished
$ dch -r
$ git add debian/changelog; debcommit
$ gbp buildpackage

$ dput 
$ debcommit -r
$ git push debian


For backports is all the same, except that instead of using debian/sid,
I fork a debian/stretch-backports and work from there.

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers


Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
Hey,

On 16/08/17 04:54, Michael Hudson-Doyle wrote:

> I think I /slightly/ prefer the upstream branch to be upstream's git
> history not a series of imports of tarballs. But I'm not set on it, and
> gbp doesn't really get on with this approach if you're just packaging a
> random commit rather than a tag AFAICS.

Actually, it works wonderfully: you just need to tag the right commit
with -say- `upstream/0.0+git20170808.deadbee`, and then `dch -v
0.0+git20170808.deadbeef-1`. With the right options, it creates the orig
tarball and commits it to pristine tar during the first build.

> Makes sense. Which gccgo arches are really being used today though? MIPS
> I guess, but that really should get golang-go once we figure out how to
> do that.

Not sure about being used, but we compile for mips* and arm*.

> https://nm.debian.org/process/198 :)

yay!!

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-20 Thread Martín Ferrari
Hi Mickael,

I haven't yet got the time to write down properly my workflow, but I
will still reply to some points. :)

On 15/08/17 23:02, Michael Stapelberg wrote:

> 1. I store packaging in e.g. ~/d/pkg/gocryptfs and build using `gbp
> buildpackage --git-export-dir=~/d/out/gocryptfs`. By using
> --git-export-dir, my working copy always stays clean. By collecting the
> output files per package, I can easily debdiff between versions. This
> point is informational and shouldn’t have any bearing on a canonical
> workflow.

Why do you need this? I use gbp with cowbuilder, and so the working copy
is never touched. Looking at my gbp.conf, I see my default is to export
to '../build-area', but probably that does not change much.

> 2. My ~/.gbp.conf reads https://paste.debian.net/hidden/a48afca2/
> 

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-19 Thread Dmitry Smirnov
On Tuesday, 15 August 2017 11:02:39 PM AEST Michael Stapelberg wrote:
> 6. I have come to appreciate pristine-tar, as it is the only reliable way
> (that I know of) to prepare uploads for packages which already have their
> orig tarball in Debian.

Then perhaps you should learn about `origtargz` utility from devscripts.

Also you can use 

apt(-get) source {packagename}

to fetch uploaded package with tarball.


> I’d suggest we codify that pristine-tar is a requirement.

I found it disturbing and depressing to read proposals to enforce pristine-
tar workflow here and there...

IMHO pristine-tar is counter productive and maintaining it requires extra 
effort. It should not be made mandatory just like GBP should not be required 
to maintain Debian packages.
I'll expand further in separate email when I can...

Regards,
 Dmitry Smirnov.

---

The truth is incontrovertible, malice may attack it, ignorance may deride
it, but in the end; there it is.
-- Winston Churchill


signature.asc
Description: This is a digitally signed message part.
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-18 Thread Sascha Steinbiss
Hi all,

just a quick comment to one of Michael's questions:

> Satta requested an option to disable SSL verification for badly
> configured redirection sites.
> 
> 
> Which sites are affected? Did we try contacting the maintainers and fix
> their config? In the days of LetsEncrypt, it seems easier to just fix
> the site.

True, and of course that would be preferable. I know that I had this
problem multiple times, but the only one I can name at the moment is
golang-github-dhowett-go-plist, located at:

  https://github.com/DHowett/go-plist

which in one of its files contained an (old?) import path of

  howett.net/plist

which wouldn't come through due to SSL issues.

Incidentally this upstream contacted me out of the blue recently and
asked for comments on how he might make things easier for us, so I'll
let them know about this particular problem.
I guess at least this one might work itself out on its own :)

Cheers
Sascha





signature.asc
Description: OpenPGP digital signature
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-15 Thread Shengjing Zhu
On Sat, Aug 12, 2017 at 4:01 AM, Martín Ferrari  wrote:
> Git workflows
> -
>
> It was discussed about the problem of having so many different
> workflows, as it makes it difficult to work on packages prepared by
> other team members.
>
> The agreement was to find one standard and make it part of the team's
> policy and incorporate into dh-make-golang.
>
> To that end, it is requested that everyone sends an email to the mailing
> list describing their preferred workflow, and after a period of
> discussion we agree to a conclusion.
>

I just use gbp in the workflow.
* use `gbp import-orig` to update the upstream/pristine-tar/master
branch when new upstream released.
  The reason to use this, is we often need to exclude some files from
upstream. Thus I think we can't just pull all upstream commits to
upstream branch.
* use `gbp pq` to handle patches.
  It integrates the git workflow well.
* use `gbp buildpackge` to build

Currently I'm just not comfortable when I need to create an orig
tarball if the upstream doesn't have a TAG.
I hope #811565 can be fixed soon to get rid of this pain :)

-- 
Best regards,
Shengjing Zhu

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-15 Thread Michael Hudson-Doyle
On 12 August 2017 at 08:01, Martín Ferrari  wrote:

> Pkg-go BoF meeting minutes
> ==
>
> On Tuesday, we had the first in-person meeting of the team. We met for 2
> hours to discuss our current issues and to plan for the future.
>
> People present
> --
>
> Alexandre Viau (aviau)
> Martín Ferrari (Tincho)
> Paul Tagliamonte (paultag)
> Sascha Steinbiss (satta)
>
> Test files
> --
>
> We discussed the issues raised about shipping test sources and fixtures
> in the -dev packages. It was pointed out that they are not really needed
> for autopkgtest or for reverse-dependencies, but that it will involve a
> lot of work to achieve, so we decided to keep them for now.
>

Makes sense tbh.


> Using -dev packages for development
> ---
>
> Due to the friction that can bring with upstreams, it was agreed to
> continue discouraging to use -dev packages for everyday golang development.
>

+1


> Outdated packages and binNMUs
> -
>
> Paultag proposed automating the detection of packages which have been
> compiled with old versions of libraries. He will implement a first
> version that just sends emails to remind of needed binNMUs, with the
> idea of some day automatically triggering wanna-build.
>
> He also indicated that he wants this automation to detect and warn about
> circular dependencies.
>

Sounds good.


> Git workflows
> -
>
> It was discussed about the problem of having so many different
> workflows, as it makes it difficult to work on packages prepared by
> other team members.
>
> The agreement was to find one standard and make it part of the team's
> policy and incorporate into dh-make-golang.
>
> To that end, it is requested that everyone sends an email to the mailing
> list describing their preferred workflow, and after a period of
> discussion we agree to a conclusion.
>

I think I /slightly/ prefer the upstream branch to be upstream's git
history not a series of imports of tarballs. But I'm not set on it, and gbp
doesn't really get on with this approach if you're just packaging a random
commit rather than a tag AFAICS.


> dh-make-golang
> --
>
> A few times people expressed the desire for dh-make-golang to grow an
> `--update` option, as most packages are trivial to update, but tedious
> to do so.
>

Oh yes please.  Although the ideas that would let gbp import-orig --uscan
work without upstream tarballs would also be fine.


> Satta requested an option to disable SSL verification for badly
> configured redirection sites.
>
> x/tools package
> ---
>
> We discussed the current breakage in x/tools, and agreed that it is a
> core package and that we should make it a shared responsibility to keep
> it in a good shape.
>

It is also a random bag of stuff, it's not really the best bit of factoring
from upstream :)


> gccgo support
> -
>
> We talked about the status of gccgo, paultag explained how mainline
> golang promises the compiler will always be buildable by gccgo, and how
> that makes bootstrapping and cross-building way simpler.
>
> We agreed on working towards making it a first-class citizen in the
> future, using golang-any by default, and only reverting to golang-go
> when needed.
>

Makes sense. Which gccgo arches are really being used today though? MIPS I
guess, but that really should get golang-go once we figure out how to do
that.


> API changes in upstream
> ---
>
> We ranted at length about upstreams, and noted that we need a system
> that provides early warning of API breakages. We discussed using ratt
> and autopkgtest for that purpose.
>

It would be good to have some automation and central resources around this
for sure.


> Aviau pointed out that he usually requests upstreams to make releases
> and that he is usually successful. Tincho pointed out the problems with
> meaningless releases and with upstreams releasing once and then
> forgetting to do it when needed.
>
> We discussed the possibility of changing "soname"s by renaming packages
> when we detect API incompatibilities, but concluded that in general it
> is too much work


Yeah, don't do this :)


> and that it makes more sense to try and fix
> reverse-dependencies by bugging upstream or patching them ourselves.
>




> Team collaboration
> --
>
> On the topic of team collaboration, we agreed to avoid using the DM ACL
> mechanism too much, and instead help active contributors to become DDs.
>

https://nm.debian.org/process/198 :)

(Although the bottleneck is me answering my DAM currently...)


> We also revisited the policy on team uploads, and concluded that we want
> to continue with avoiding hard ownership of packages and that by default
> everything is team-maintained.
>

+100, _especially_ for -dev packages.

Cheers,
mwh

Plan of action
> --
>
> There was some talk about things to do in the immediate future 

Re: [pkg-go] Minutes for the DebConf17 BoF

2017-08-15 Thread Michael Stapelberg
Thanks for sending these out. Comments inline:

On Fri, Aug 11, 2017 at 10:01 PM, Martín Ferrari  wrote:

> Pkg-go BoF meeting minutes
> ==
>
> On Tuesday, we had the first in-person meeting of the team. We met for 2
> hours to discuss our current issues and to plan for the future.
>
> People present
> --
>
> Alexandre Viau (aviau)
> Martín Ferrari (Tincho)
> Paul Tagliamonte (paultag)
> Sascha Steinbiss (satta)
>
> Test files
> --
>
> We discussed the issues raised about shipping test sources and fixtures
> in the -dev packages. It was pointed out that they are not really needed
> for autopkgtest or for reverse-dependencies, but that it will involve a
> lot of work to achieve, so we decided to keep them for now.
>
> Using -dev packages for development
> ---
>
> Due to the friction that can bring with upstreams, it was agreed to
> continue discouraging to use -dev packages for everyday golang development.
>
> Outdated packages and binNMUs
> -
>
> Paultag proposed automating the detection of packages which have been
> compiled with old versions of libraries. He will implement a first
> version that just sends emails to remind of needed binNMUs, with the
> idea of some day automatically triggering wanna-build.
>
> He also indicated that he wants this automation to detect and warn about
> circular dependencies.
>
> Git workflows
> -
>
> It was discussed about the problem of having so many different
> workflows, as it makes it difficult to work on packages prepared by
> other team members.
>
> The agreement was to find one standard and make it part of the team's
> policy and incorporate into dh-make-golang.
>
> To that end, it is requested that everyone sends an email to the mailing
> list describing their preferred workflow, and after a period of
> discussion we agree to a conclusion.
>

This sounds very good.

I have come to appreciate the following properties of working with
git-buildpackage:

1. I store packaging in e.g. ~/d/pkg/gocryptfs and build using `gbp
buildpackage --git-export-dir=~/d/out/gocryptfs`. By using
--git-export-dir, my working copy always stays clean. By collecting the
output files per package, I can easily debdiff between versions. This point
is informational and shouldn’t have any bearing on a canonical workflow.

2. My ~/.gbp.conf reads https://paste.debian.net/hidden/a48afca2/
(informational)

3. To import a new upstream version, I run `gbp import-orig --uscan`. This
of course only works with tagged releases until #811565 is fixed (hopefully
soon).

4. To update debian/changelog, I run `gbp dch -R --commit`. Note that this
goes against our current policy of editing debian/changelog with an
UNRELEASED entry — when using gbp-dch, the changelog is entirely
auto-generated from git (but you do have the option to amend it before
committing). Hence, I’d suggest we update that policy and start using
gbp-dch.

5. I use quilt for managing patches, and am reasonably happy with that,
i.e. I personally don’t see the need for anything more fancy than that,
especially given that we usually only carry very few patches per package.
Some of our packages use branches which only contain debian/, not the
upstream source. This breaks quilt and confuses me. I’d suggest we codify
that the branch must contain the upstream sources plus debian/.

Here are some more thoughts, not directly related to how I maintain my own
packages:

6. I have come to appreciate pristine-tar, as it is the only reliable way
(that I know of) to prepare uploads for packages which already have their
orig tarball in Debian. Whenever I come across a package which isn’t using
pristine-tar, the generated orig tarball will have a different checksum,
and my upload will be rejected. I’d suggest we codify that pristine-tar is
a requirement.

7. We don’t currently have a guideline with regards to branch naming,
especially when maintaining branches for multiple debian versions (e.g.
stretch, buster, stretch-backports, …). I’d suggest we adopt the
debian/ branch naming scheme, e.g. debian/buster is the default
branch, backports can be found in debian/stretch-backports, etc.

8. (Optional/best effort) I recently came to understand that dgit is
thought of as a universal approach for new users/maintainers to easily
contribute to packaging (you get the same style of git checkout of any
package in the archive). We should verify the above constraints still leave
us in a place where dgit works well — it will work for any package, but it
will work better for packages which are `dgit push`ed. I don’t yet know
what the requirements for that are.

Thoughts? :)


>
> dh-make-golang
> --
>
> A few times people expressed the desire for dh-make-golang to grow an
> `--update` option, as most packages are trivial to update, but tedious
> to do so.
>

See point ③ above — I’m optimistic that we can use uscan for that