Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Sun, Jan 14, 2018 at 08:25:05PM +0100, Michael Stapelberg wrote:
> Thanks for your work on this!
> 
> I gave the attached patch a shot, and it seems to work fine for me. I tried it
> on two packages:
> 
> “upspin” (untagged git, non-GitHub host, Files-Excluded in debian/copyright):
> 
> # uscan(1) configuration file.
> version=4
> opts="mode=git,pgpmode=none" \
> https://upspin.googlesource.com/upspin \
> HEAD debian uupdate
> 
> “golang-github-coreos-go-oidc” (typical Go package on GitHub):
> 
> # uscan(1) configuration file.
> version=4
> opts="mode=git,pgpmode=none" \
> https://github.com/coreos/go-oidc.git \
> HEAD debian uupdate
> 
> So, from my end, this looks good. I hope these examples can serve as test
> cases?
> 
> Let me know if you need anything else.

I cleaned-up/rewrote code and made serious debugs ..

But I still see several rough edges.

Also I need to ensure remote branch head tracking and -3-g1f526bd
like autogenerated tag from git describe o work.

Then I can set up clean syntax for git.

Give me more time.

Osamu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Thanks for your work on this!

I gave the attached patch a shot, and it seems to work fine for me. I tried
it on two packages:

“upspin” (untagged git, non-GitHub host, Files-Excluded in
debian/copyright):

# uscan(1) configuration file.
version=4
opts="mode=git,pgpmode=none" \
https://upspin.googlesource.com/upspin \
HEAD debian uupdate

“golang-github-coreos-go-oidc” (typical Go package on GitHub):

# uscan(1) configuration file.
version=4
opts="mode=git,pgpmode=none" \
https://github.com/coreos/go-oidc.git \
HEAD debian uupdate

So, from my end, this looks good. I hope these examples can serve as test
cases?

Let me know if you need anything else.


On Sat, Jan 13, 2018 at 8:32 PM, Osamu Aoki  wrote:

> Hi,
>
> On Thu, Jan 11, 2018 at 09:56:01AM +0100, Michael Stapelberg wrote:
> > Happy new year!
> >
> > Osamu, any update on this? We’re still really interested.
>
> In progress ... partially working
>
> I need to clean up codes ;-)
> I also need couple test cases to test these out.
>
> git served from github, git over WEBDAV, ...
>
> Latest local diff is attached.
>
> Osamu
>



-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Thu, Jan 11, 2018 at 09:56:01AM +0100, Michael Stapelberg wrote:
> Happy new year!
> 
> Osamu, any update on this? We’re still really interested.

In progress ... partially working

I need to clean up codes ;-)
I also need couple test cases to test these out.

git served from github, git over WEBDAV, ...

Latest local diff is attached.

Osamu
>From a635d170bfc8b09218c778ea1f7d3866b0776531 Mon Sep 17 00:00:00 2001
From: Osamu Aoki 
Date: Sat, 13 Jan 2018 21:24:34 +0900
Subject: [PATCH] Improve git (wip)

Now git and git-dumb exists

Need to clean up temporary directory etc.

Signed-off-by: Osamu Aoki 
---
 scripts/uscan.pl | 214 +--
 1 file changed, 160 insertions(+), 54 deletions(-)

diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index 1b47da63..208115cc 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -357,14 +357,42 @@ either B or B by URL.
 =item B
 
 This mode accesses the upstream git archive directly with the B command
-and packs the source tree with the specified tag into
+and packs the source tree with the specified tag via I into
 IB<.tar.xz>.
 
 If the upstream publishes the released tarball via its web interface, please
 use it instead of using this mode.  This mode is the last resort method.
 
+If I is set to B, the pertinent I is
+automatically generated with the date and hush of the B of the git
+repository.
+
+=item B
+
+B.
+
+When the upstream git archive is a dumb HTTP server which doesn't allow shallow
+checkout, this B mode should be used instead.
+
+Other than the fact that this mode makes full clone of the upstream repository
+as needed, this mode behaves exactly the same as the B mode.
+
 =back
 
+=item 

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Happy new year!

Osamu, any update on this? We’re still really interested.

On Fri, Oct 13, 2017 at 7:06 PM, Michael Stapelberg 
wrote:

> Almost a month has passed. What’s the current status? The pkg-go group
> is still eagerly waiting for this. If there is any way we can help,
> please let us know.
>
> On Mon, Sep 18, 2017 at 3:04 AM, Osamu Aoki  wrote:
> > Hi,
> >
> > On Mon, Sep 18, 2017 at 01:20:22PM +0800, Shengjing Zhu wrote:
> >> Hi,
> >>
> >> I want to know if it's still possible to support GitHub's commit via
> >> different way, rather than do a `git clone`.
> >>
> >> I find GitHub has RSS feed for the commit, the url is like
> >> https://github.com/Debian/dh-make-golang/commits/master.atom
> >> So uscan can parse that xml feed, and get the commit data, id, and
> >> finally forms the download link like
> >> https://github.com/Debian/dh-make-golang/archive/
> 71736daa55a06e466cdcc6c0347f5b9489471fe3.tar.gz,
> >> and the version is 0.0~git.
> >> Besides, the RSS feed url is not an API url, and will not have the API
> >> rate problem.
> >
> > Good.
> >
> >> That's simpler than doing `git clone` locally, though the shortage is
> >> site specific.
> >> BTW, gitlab also provides such feed, the url format is like
> >> https://gitlab.com/inkscape/inkscape/commits/master?format=atom
> >
> > I have been busy with other higher priority issues with devscripts
> > lately.
> >
> > There are two types of git archives.  I don't want to make unorganized
> > addition.  Please let me have some time on this.  It is living in my
> > private git now.
> >
> > Osamu
>
>
>
> --
> Best regards,
> Michael
>



-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Almost a month has passed. What’s the current status? The pkg-go group
is still eagerly waiting for this. If there is any way we can help,
please let us know.

On Mon, Sep 18, 2017 at 3:04 AM, Osamu Aoki  wrote:
> Hi,
>
> On Mon, Sep 18, 2017 at 01:20:22PM +0800, Shengjing Zhu wrote:
>> Hi,
>>
>> I want to know if it's still possible to support GitHub's commit via
>> different way, rather than do a `git clone`.
>>
>> I find GitHub has RSS feed for the commit, the url is like
>> https://github.com/Debian/dh-make-golang/commits/master.atom
>> So uscan can parse that xml feed, and get the commit data, id, and
>> finally forms the download link like
>> https://github.com/Debian/dh-make-golang/archive/71736daa55a06e466cdcc6c0347f5b9489471fe3.tar.gz,
>> and the version is 0.0~git.
>> Besides, the RSS feed url is not an API url, and will not have the API
>> rate problem.
>
> Good.
>
>> That's simpler than doing `git clone` locally, though the shortage is
>> site specific.
>> BTW, gitlab also provides such feed, the url format is like
>> https://gitlab.com/inkscape/inkscape/commits/master?format=atom
>
> I have been busy with other higher priority issues with devscripts
> lately.
>
> There are two types of git archives.  I don't want to make unorganized
> addition.  Please let me have some time on this.  It is living in my
> private git now.
>
> Osamu



-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Mon, Sep 18, 2017 at 01:20:22PM +0800, Shengjing Zhu wrote:
> Hi,
> 
> I want to know if it's still possible to support GitHub's commit via
> different way, rather than do a `git clone`.
> 
> I find GitHub has RSS feed for the commit, the url is like
> https://github.com/Debian/dh-make-golang/commits/master.atom
> So uscan can parse that xml feed, and get the commit data, id, and
> finally forms the download link like
> https://github.com/Debian/dh-make-golang/archive/71736daa55a06e466cdcc6c0347f5b9489471fe3.tar.gz,
> and the version is 0.0~git.
> Besides, the RSS feed url is not an API url, and will not have the API
> rate problem.

Good.

> That's simpler than doing `git clone` locally, though the shortage is
> site specific.
> BTW, gitlab also provides such feed, the url format is like
> https://gitlab.com/inkscape/inkscape/commits/master?format=atom

I have been busy with other higher priority issues with devscripts
lately.  

There are two types of git archives.  I don't want to make unorganized
addition.  Please let me have some time on this.  It is living in my
private git now.

Osamu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Shengjing Zhu]

Hi,

I want to know if it's still possible to support GitHub's commit via
different way, rather than do a `git clone`.

I find GitHub has RSS feed for the commit, the url is like
https://github.com/Debian/dh-make-golang/commits/master.atom
So uscan can parse that xml feed, and get the commit data, id, and
finally forms the download link like
https://github.com/Debian/dh-make-golang/archive/71736daa55a06e466cdcc6c0347f5b9489471fe3.tar.gz,
and the version is 0.0~git.
Besides, the RSS feed url is not an API url, and will not have the API
rate problem.

That's simpler than doing `git clone` locally, though the shortage is
site specific.
BTW, gitlab also provides such feed, the url format is like
https://gitlab.com/inkscape/inkscape/commits/master?format=atom

Thanks
Shengjing Zhu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Fri, Aug 11, 2017 at 08:09:39PM +0200, Michael Stapelberg wrote:
> Ah, thanks for the correction. In that case, we should probably try a shallow
> clone first, then fall back to a non-shallow clone :).

That's my plan.  The same thing can be said for existing git archive
command.  So I am refactoring code too.

Osamu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Ah, thanks for the correction. In that case, we should probably try a
shallow clone first, then fall back to a non-shallow clone :).

On Fri, Aug 11, 2017 at 7:41 PM, Shengjing Zhu  wrote:

> On Fri, Aug 11, 2017 at 9:55 PM, Michael Stapelberg
>  wrote:
> > I think the git HTTP transport does never allow shallow clones. You can
> use
> > the git transport protocol (also with GitHub), but note that the git
> > transport protocol is not authenticated (whereas https is, of course).
> >
>
> Only dumb http server doesn't support shallow clone.
> The smart http protocol supports, like `git clone --depth=1
> https://github.com/xxx`
>
> --
> Best regards,
> Shengjing Zhu
>



-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Shengjing Zhu]

On Fri, Aug 11, 2017 at 9:55 PM, Michael Stapelberg
 wrote:
> I think the git HTTP transport does never allow shallow clones. You can use
> the git transport protocol (also with GitHub), but note that the git
> transport protocol is not authenticated (whereas https is, of course).
>

Only dumb http server doesn't support shallow clone.
The smart http protocol supports, like `git clone --depth=1
https://github.com/xxx`

-- 
Best regards,
Shengjing Zhu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

I think the git HTTP transport does never allow shallow clones. You can use
the git transport protocol (also with GitHub), but note that the git
transport protocol is not authenticated (whereas https is, of course).

So, I think the solution is to only use --depth=1 for git:// URLs.

For GitHub, I’d recommend first translating https URLs into git URLs, then
(later) use the GitHub API.

On Fri, Aug 11, 2017 at 6:51 AM, Osamu Aoki  wrote:

> Hi,
>
> On Thu, Aug 10, 2017 at 09:37:33AM +0200, Michael Stapelberg wrote:
> > Here’s an example: https://github.com/Debian/dh-make-golang/. The
> corresponding
>
> Tested ...
>
> $ git clone --depth=1 --bare https://github.com/Debian/dh-make-golang/ boo
> Cloning into bare repository 'boo'...
> remote: Counting objects: 13, done.
> remote: Compressing objects: 100% (12/12), done.
> remote: Total 13 (delta 0), reused 6 (delta 0), pack-reused 0
> Unpacking objects: 100% (13/13), done.
> $ git clone --depth=1 --bare http://git.ao2.it/tweeper.git foo
> Cloning into bare repository 'foo'...
> fatal: dumb http transport does not support shallow capabilities
> $ git clone  --bare http://git.ao2.it/tweeper.git foo
> Cloning into bare repository 'foo'...
> $
>
> Interesting.  So not all site can work with shallow checkout.
>
> This reminds me of problems I had with git-archive, too.
>
> Before coding further, let me check and test all these sites.
>
>
> Osamu
>
>
>


-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Thu, Aug 10, 2017 at 09:37:33AM +0200, Michael Stapelberg wrote:
> Here’s an example: https://github.com/Debian/dh-make-golang/. The 
> corresponding

Tested ...

$ git clone --depth=1 --bare https://github.com/Debian/dh-make-golang/ boo
Cloning into bare repository 'boo'...
remote: Counting objects: 13, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13 (delta 0), reused 6 (delta 0), pack-reused 0
Unpacking objects: 100% (13/13), done.
$ git clone --depth=1 --bare http://git.ao2.it/tweeper.git foo
Cloning into bare repository 'foo'...
fatal: dumb http transport does not support shallow capabilities
$ git clone  --bare http://git.ao2.it/tweeper.git foo
Cloning into bare repository 'foo'...
$ 

Interesting.  So not all site can work with shallow checkout.

This reminds me of problems I had with git-archive, too.

Before coding further, let me check and test all these sites.


Osamu

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

On Thu, Aug 10, 2017 at 9:34 AM, Osamu Aoki  wrote:

> Hi,
>
> On Thu, Aug 10, 2017 at 08:19:20AM +0200, Michael Stapelberg wrote:
> > On Thu, Aug 10, 2017 at 6:50 AM, Osamu Aoki  wrote:
> >
> > The functionality in the tool is exactly what was already communicated
> in the
> > bug :).
> >
> > If you need any help with that bug, just let me know.
>
> #1: exact use case example URL.  (Project which hopefully exists after 4
> years.)
>

Here’s an example: https://github.com/Debian/dh-make-golang/. The
corresponding Debian binary package is
https://packages.debian.org/buster/dh-make-golang; there’s no debian/watch
file yet because of the current lack of support for git :).

Please let me know if you need anything else. Thanks!


>
> #2 Yes, I see ... but backticks  I think I need to do it slightly
> different
>
> backticks("git", "clone", "--quiet", "--bare", "--depth=1", $url,
> $dest);
>
> my $commit_data = backticks("git", "--git-dir=$dest", "log", "-1",
> "--date=format:%Y%m%d", "--format=%h %cd");
>
> chomp($commit_data);
> $commit_data =~ /^([0-9a-z]{7}) ([0-9]{8})$/m
> or die("Invalid git response: $commit_data");
> return ($1, $2);
>
> Thank
>
> Osamu
>
>


-- 
Best regards,
Michael

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Thu, Aug 10, 2017 at 08:19:20AM +0200, Michael Stapelberg wrote:
> On Thu, Aug 10, 2017 at 6:50 AM, Osamu Aoki  wrote:
> 
> The functionality in the tool is exactly what was already communicated in the
> bug :).
> 
> If you need any help with that bug, just let me know.

#1: exact use case example URL.  (Project which hopefully exists after 4
years.)

#2 Yes, I see ... but backticks  I think I need to do it slightly
different

backticks("git", "clone", "--quiet", "--bare", "--depth=1", $url,
$dest);

my $commit_data = backticks("git", "--git-dir=$dest", "log", "-1",
"--date=format:%Y%m%d", "--format=%h %cd");

chomp($commit_data);
$commit_data =~ /^([0-9a-z]{7}) ([0-9]{8})$/m
or die("Invalid git response: $commit_data");
return ($1, $2);

Thank

Osamu

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

On Thu, Aug 10, 2017 at 6:50 AM, Osamu Aoki  wrote:

> Hi,
>
>
> On Wed, Aug 09, 2017 at 07:54:05AM -0700, Michael Stapelberg wrote:
> > Thanks for sharing your tool!
>
> Thank you but I don't use go nor perl regulary :-(  I wish uscan was in
> Python.)
>
> But supporting github for go or other project is high on my list.
>
>
> > Hence, I think extending uscan is a much much more elegant route to
> achieve our
> > goal, and I’d like to ask people to hold off providing/using custom
> services as
> > a stop-gap measure. Thanks!
>
> I am thinking to add features to support use case for github for go as
> much as it can be as generic solution extendible by user.  (normally,
> this is the last thing on my list since uscan is already too
> complicated.  The last thing we need is project/site specific brute
> force work around.)
>
> If you explain in a plain english (or in patch) text what you wish!
> (Please don't send me complicated perl code. I am no perl speaker.  My
> perl skill is woose than my English one.)
>
> I know there is a bug report on uscan support of github which I should
> be working on.  :-)
>

The functionality in the tool is exactly what was already communicated in
the bug :).

If you need any help with that bug, just let me know.

Thanks!

-- 
Best regards,
Michael

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,


On Wed, Aug 09, 2017 at 07:54:05AM -0700, Michael Stapelberg wrote:
> Thanks for sharing your tool!

Thank you but I don't use go nor perl regulary :-(  I wish uscan was in
Python.)

But supporting github for go or other project is high on my list.   
  
 
> Hence, I think extending uscan is a much much more elegant route to achieve 
> our
> goal, and I’d like to ask people to hold off providing/using custom services 
> as
> a stop-gap measure. Thanks!

I am thinking to add features to support use case for github for go as
much as it can be as generic solution extendible by user.  (normally,
this is the last thing on my list since uscan is already too
complicated.  The last thing we need is project/site specific brute
force work around.)

If you explain in a plain english (or in patch) text what you wish!
(Please don't send me complicated perl code. I am no perl speaker.  My
perl skill is woose than my English one.)

I know there is a bug report on uscan support of github which I should
be working on.  :-)

Osamu

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

On Wed, Aug 9, 2017 at 6:54 PM, Shengjing Zhu  wrote:

> Thanks for the comment.
>
> On Wed, Aug 9, 2017 at 10:54 PM, Michael Stapelberg
>  wrote:
> > 1. I think that infrastructure which the pkg-go team critically and very
> > visibly depends on should eventually be hosted by DSA under debian.org.
> I
> > don’t see them hosting this special “workaround” service, when there
> already
> > is infrastructure in place to run uscan.
>
> Well it can be hosted by DSA, or even don't use web service. Maybe
> uscan can just call its cli tool.
>

As much as I’d like to see more Go code within Debian, I think it might be
best to stick with Perl for uscan :).


> I do hope someone can implement it in perl and bring it to uscan. But
> it's hard for me to hack 4k lines perl.
>

I can understand that, and I’m not asking you to work on uscan — Osamu
already seems to be on that.


>
> Anyway, it's an exploration for using API rather than `git clone` locally.
> And I intend to get it to support more Git services, maybe gopkg.in,
> gitlab, etc.
>
> PS, gopkg.in will point to some specific branch, and
> github.com///tags doesn't work well even I append a
> '?after=' suffix.
>
>
> >
> > 2. I have concerns regarding the scalability of such a service if we
> > actually adopted this approach: the GitHub quota permits 5000 requests
> per
> > hour (when authenticated). This sounds like a lot at first glance, but
> > consider that we already have 845 Go packages. Your code does 4 requests
> per
> > repository (IIUC), so already we are fairly close to reaching the limit,
> if
> > we don’t take any precautions.
>
> I haven't considered rate-limit, but do we check so frequently indeed?


I don’t actually know what the rate of uscan checks is behind the Debian
Package Tracker. I can imagine that other places do run uscan, too, though
(think Ubuntu, or other Debian derivatives). In fact, I’m working on a
dashboard myself which does run uscan fairly frequently.

-- 
Best regards,
Michael

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Shengjing Zhu]

Thanks for the comment.

On Wed, Aug 9, 2017 at 10:54 PM, Michael Stapelberg
 wrote:
> 1. I think that infrastructure which the pkg-go team critically and very
> visibly depends on should eventually be hosted by DSA under debian.org. I
> don’t see them hosting this special “workaround” service, when there already
> is infrastructure in place to run uscan.

Well it can be hosted by DSA, or even don't use web service. Maybe
uscan can just call its cli tool.
I do hope someone can implement it in perl and bring it to uscan. But
it's hard for me to hack 4k lines perl.

Anyway, it's an exploration for using API rather than `git clone` locally.
And I intend to get it to support more Git services, maybe gopkg.in,
gitlab, etc.

PS, gopkg.in will point to some specific branch, and
github.com///tags doesn't work well even I append a
'?after=' suffix.


>
> 2. I have concerns regarding the scalability of such a service if we
> actually adopted this approach: the GitHub quota permits 5000 requests per
> hour (when authenticated). This sounds like a lot at first glance, but
> consider that we already have 845 Go packages. Your code does 4 requests per
> repository (IIUC), so already we are fairly close to reaching the limit, if
> we don’t take any precautions.

I haven't considered rate-limit, but do we check so frequently indeed?


-- 
Best regards,
Shengjing Zhu

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Thanks for sharing your tool!

I also considered implementing such a tool, but ultimately decided against
it for a number of reasons:

1. I think that infrastructure which the pkg-go team critically and very
visibly depends on should eventually be hosted by DSA under debian.org. I
don’t see them hosting this special “workaround” service, when there
already is infrastructure in place to run uscan.

2. I have concerns regarding the scalability of such a service if we
actually adopted this approach: the GitHub quota permits 5000 requests per
hour (when authenticated). This sounds like a lot at first glance, but
consider that we already have 845 Go packages. Your code does 4 requests
per repository (IIUC), so already we are fairly close to reaching the
limit, if we don’t take any precautions.

Most likely, point ② could be addressed with some careful limiting on our
end, and changing the processing model from generating a response upon
end-user request to iterating through all Go packages in Debian and
querying GitHub in a rate-limited fashion. This significantly complicates
the program, though, to the point where we duplicate the logic behind the
Debian Package Tracker. Worse, it introduces accidental complexity, not
inherent complexity :).

Hence, I think extending uscan is a much much more elegant route to achieve
our goal, and I’d like to ask people to hold off providing/using custom
services as a stop-gap measure. Thanks!

On Wed, Aug 9, 2017 at 7:38 AM, Shengjing Zhu  wrote:

> Hi all,
>
> I spent some time playing around GitHub api, and results a small tool,
> https://github.com/zhsj/git-watch
>
> I didn't implement it in uscan. But it can work well with uscan.
>
> A demo service is at https://watch.zhsj.me/
>
> Take one of packages I maintained,
> https://tracker.debian.org/golang-github-xiaq-persistent
> https://watch.zhsj.me/github/xiaq/persistent is the watch url.
> And the following d/watch works fine,
>
> version=4
> opts="filenamemangle=s%(?:.*?)?([^/]*)\.tar\.gz%golang-githu
> b-xiaq-persistent-$1.tar.gz%"
> \
>   https://watch.zhsj.me/github/xiaq/persistent \
>   (?:.*?/)?([^/]*)\.tar\.gz
>
> This tool works both as web service and cli.
>
> I hope you find this tool useful.
>
> Yours,
> Shengjing Zhu
>



-- 
Best regards,
Michael

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Shengjing Zhu]

Hi all,

I spent some time playing around GitHub api, and results a small tool,
https://github.com/zhsj/git-watch

I didn't implement it in uscan. But it can work well with uscan.

A demo service is at https://watch.zhsj.me/

Take one of packages I maintained,
https://tracker.debian.org/golang-github-xiaq-persistent
https://watch.zhsj.me/github/xiaq/persistent is the watch url.
And the following d/watch works fine,

version=4
opts="filenamemangle=s%(?:.*?)?([^/]*)\.tar\.gz%golang-github-xiaq-persistent-$1.tar.gz%"
\
  https://watch.zhsj.me/github/xiaq/persistent \
  (?:.*?/)?([^/]*)\.tar\.gz

This tool works both as web service and cli.

I hope you find this tool useful.

Yours,
Shengjing Zhu

Bug#811565: [pkg-go] Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Martín Ferrari]

On 29/07/17 17:44, Michael Stapelberg wrote:


> Given that we are talking about repositories which do not use tags, we
> could specify --depth=1 when cloning to get a shallow clone, i.e. only
> the latest commit. That saves bandwidth and disk space, but has the
> downside that we cannot do any additional validation, i.e. we can’t
> detect if upstream ever starts using tags — unfortunately, that is a
> plausible scenario, so I would suggest doing a full clone.

As a data point, I wrote a script a while ago to do exactly this
locally. I used the shallow clone on a temporary directory:

backticks("git", "clone", "--quiet", "--bare", "--depth=1", $url,
$dest);

my $commit_data = backticks("git", "--git-dir=$dest", "log", "-1",
"--date=format:%Y%m%d", "--format=%h %cd");

chomp($commit_data);
$commit_data =~ /^([0-9a-z]{7}) ([0-9]{8})$/m
or die("Invalid git response: $commit_data");
return ($1, $2);

-- 
Martín Ferrari (Tincho)

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

On Sun, Jul 30, 2017 at 6:10 AM, Osamu Aoki  wrote:

> Hi,
>
> (I switched my ISP.  No more osamua...@e01.itscom.net  Thanks for the
> reminder)
>
> On Sat, Jul 29, 2017 at 06:44:43PM +0200, Michael Stapelberg wrote:
> > Hi Osamu,
> >
> > Sorry for the late reply, and thanks for looking into this! Replies
> > inline:
>
> It's good time to make feature enhancements now.
>
> > Osamu Aoki  writes:
> > > How should we explicitly specify such variables, I guess it should be
> > > through "opts=..." such as:
> > >
> > >  opts="mode=git, pretty=0.0~git%cd.%h, date=%Y%m%d%H%M"
> >
> > Sounds good.
>
> I had to read the whole thread to recall what I was thinking ... OK ;-)
>
> > > But this "git log" needs to have local clone of git repository.
> > >
> > > I wonder if I can do without cloning first.
> >
> > After reading the git protocol and searching on the web for a little
> > bit, my conclusion is that no, you cannot use “git log” without having a
> > clone of the repository.
> >
> > Given that we are talking about repositories which do not use tags, we
> > could specify --depth=1 when cloning to get a shallow clone, i.e. only
> > the latest commit. That saves bandwidth and disk space, but has the
> > downside that we cannot do any additional validation, i.e. we can’t
> > detect if upstream ever starts using tags — unfortunately, that is a
> > plausible scenario, so I would suggest doing a full clone.
>
> OK with FULL clone.  (I need to rethink details though... I totally lost
> my memory on this topic)
>
> The thing to consider is what git local repository looks like and how
> you clone such remote tree. "upstream" branch used by git-buildpackage
> is not really the upstream git repository but its series of commits from
> the released upstream tarballs.  Maybe clone it into "upstream-git"
> branch...
>

Wouldn’t it be cleaner to not modify the local repository at all, i.e.
clone in a separate, temporary directory? Aside from a new orig tarball,
uscan doesn’t leave files behind usually, does it?


>
> > For GitHub, we can apply an optimization: the GitHub HTTP API exposes
> > repository details, such as:
> >
> > 1. The default_branch of the repo, in
> >https://developer.github.com/v3/repos/#get
> >
> > 2. The latest commit of the branch, in
> >https://developer.github.com/v3/repos/branches/#get-branch
> >
> > For interactive use by individual developers, we could send these HTTP
> > requests unauthenticated. For a setup which does many uscan calls, we’d
> > need to create a GitHub account to get the higher rate limit. See
> > https://godoc.org/github.com/google/go-github/github#hdr-Rate_Limiting
> > for details.
>
> (This optimization is a bit more work than I can do immediately.)
>

That’s fair. I’m happy to help with a patch for uscan to apply this
optimization, once the foundation for it is done.


>
> > > Adding support to the number of commits is complicated.  Let's be happy
> > > to use hash to be unique commit.  I do not think we upload more than 2
> > > Debian upstream tarball in a minute.
> >
> > In a day, not in a minute. But regardless, you are probably right. I
> > asked in the pkg-go IRC channel to see whether people are okay with
> > removing that part from the version number, so barring any objections,
> > we can probably get that done within the next few days.
>
> Why in a day?
>
> %cd is committer date and this format respects --date= option.
> --date option I suggested was %Y%m%d%H%M" which specified down to
> minutes;-)
> If you insist, I can add seconds ;-)
>

Ah, now I see where you’re coming from. We’re currently using day
granularity, and don’t want to change that, so we’re restricted to 1 upload
per day :).


>
> > > As for "git describe" like nearest tag feature, it's a interesting
> > > thought but it may make things more complicate.  So unless someone
> > > strongly request with patch, I would like to skip it.
> >
> > Agreed — if we get rid of the number of commits, we shouldn’t need git
> > describe, not even in dh-make-golang.
> >
> > It seems like you have a good handle on implementing this in uscan. Do
> > you need any additional details? Do you prefer an external patch from
> > us over implementing this yourself? I’d be happy to give you feedback on
> > a proposed patch or git commit.
>
> OK.  I guess this will be a nice project during My Debconf17 travel for
> me.


Sounds great! I can’t make it to this DebConf, but I wish you safe travels
and a great conference!

Thanks in advance,

-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

(I switched my ISP.  No more osamua...@e01.itscom.net  Thanks for the
reminder)

On Sat, Jul 29, 2017 at 06:44:43PM +0200, Michael Stapelberg wrote:
> Hi Osamu,
> 
> Sorry for the late reply, and thanks for looking into this! Replies
> inline:

It's good time to make feature enhancements now.
 
> Osamu Aoki  writes:
> > How should we explicitly specify such variables, I guess it should be
> > through "opts=..." such as:
> >
> >  opts="mode=git, pretty=0.0~git%cd.%h, date=%Y%m%d%H%M"
> 
> Sounds good.

I had to read the whole thread to recall what I was thinking ... OK ;-)

> > But this "git log" needs to have local clone of git repository.
> >
> > I wonder if I can do without cloning first.
> 
> After reading the git protocol and searching on the web for a little
> bit, my conclusion is that no, you cannot use “git log” without having a
> clone of the repository.
> 
> Given that we are talking about repositories which do not use tags, we
> could specify --depth=1 when cloning to get a shallow clone, i.e. only
> the latest commit. That saves bandwidth and disk space, but has the
> downside that we cannot do any additional validation, i.e. we can’t
> detect if upstream ever starts using tags — unfortunately, that is a
> plausible scenario, so I would suggest doing a full clone.

OK with FULL clone.  (I need to rethink details though... I totally lost
my memory on this topic)

The thing to consider is what git local repository looks like and how
you clone such remote tree. "upstream" branch used by git-buildpackage
is not really the upstream git repository but its series of commits from
the released upstream tarballs.  Maybe clone it into "upstream-git"
branch...

> For GitHub, we can apply an optimization: the GitHub HTTP API exposes
> repository details, such as:
> 
> 1. The default_branch of the repo, in
>https://developer.github.com/v3/repos/#get
> 
> 2. The latest commit of the branch, in
>https://developer.github.com/v3/repos/branches/#get-branch
> 
> For interactive use by individual developers, we could send these HTTP
> requests unauthenticated. For a setup which does many uscan calls, we’d
> need to create a GitHub account to get the higher rate limit. See
> https://godoc.org/github.com/google/go-github/github#hdr-Rate_Limiting
> for details.

(This optimization is a bit more work than I can do immediately.)

> > Adding support to the number of commits is complicated.  Let's be happy
> > to use hash to be unique commit.  I do not think we upload more than 2
> > Debian upstream tarball in a minute.
> 
> In a day, not in a minute. But regardless, you are probably right. I
> asked in the pkg-go IRC channel to see whether people are okay with
> removing that part from the version number, so barring any objections,
> we can probably get that done within the next few days.

Why in a day?

%cd is committer date and this format respects --date= option.
--date option I suggested was %Y%m%d%H%M" which specified down to minutes;-)
If you insist, I can add seconds ;-)

> > As for "git describe" like nearest tag feature, it's a interesting
> > thought but it may make things more complicate.  So unless someone
> > strongly request with patch, I would like to skip it.
> 
> Agreed — if we get rid of the number of commits, we shouldn’t need git
> describe, not even in dh-make-golang.
> 
> It seems like you have a good handle on implementing this in uscan. Do
> you need any additional details? Do you prefer an external patch from
> us over implementing this yourself? I’d be happy to give you feedback on
> a proposed patch or git commit.

OK.  I guess this will be a nice project during My Debconf17 travel for
me.

Osamu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Hi Osamu,

Sorry for the late reply, and thanks for looking into this! Replies
inline:

Osamu Aoki  writes:
> How should we explicitly specify such variables, I guess it should be
> through "opts=..." such as:
>
>  opts="mode=git, pretty=0.0~git%cd.%h, date=%Y%m%d%H%M"

Sounds good.

>
> But this "git log" needs to have local clone of git repository.
>
> I wonder if I can do without cloning first.

After reading the git protocol and searching on the web for a little
bit, my conclusion is that no, you cannot use “git log” without having a
clone of the repository.

Given that we are talking about repositories which do not use tags, we
could specify --depth=1 when cloning to get a shallow clone, i.e. only
the latest commit. That saves bandwidth and disk space, but has the
downside that we cannot do any additional validation, i.e. we can’t
detect if upstream ever starts using tags — unfortunately, that is a
plausible scenario, so I would suggest doing a full clone.

For GitHub, we can apply an optimization: the GitHub HTTP API exposes
repository details, such as:

1. The default_branch of the repo, in
   https://developer.github.com/v3/repos/#get

2. The latest commit of the branch, in
   https://developer.github.com/v3/repos/branches/#get-branch

For interactive use by individual developers, we could send these HTTP
requests unauthenticated. For a setup which does many uscan calls, we’d
need to create a GitHub account to get the higher rate limit. See
https://godoc.org/github.com/google/go-github/github#hdr-Rate_Limiting
for details.

> Adding support to the number of commits is complicated.  Let's be happy
> to use hash to be unique commit.  I do not think we upload more than 2
> Debian upstream tarball in a minute.

In a day, not in a minute. But regardless, you are probably right. I
asked in the pkg-go IRC channel to see whether people are okay with
removing that part from the version number, so barring any objections,
we can probably get that done within the next few days.

> As for "git describe" like nearest tag feature, it's a interesting
> thought but it may make things more complicate.  So unless someone
> strongly request with patch, I would like to skip it.

Agreed — if we get rid of the number of commits, we shouldn’t need git
describe, not even in dh-make-golang.

It seems like you have a good handle on implementing this in uscan. Do
you need any additional details? Do you prefer an external patch from
us over implementing this yourself? I’d be happy to give you feedback on
a proposed patch or git commit.

Thank you very much!

-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi

On Sat, Jun 11, 2016 at 02:20:09PM -0400, James McCoy wrote:
> $ git log -1 --pretty='format:%cd' --date='format:%Y%m%d'

Nice:  It seems we can add hush etc as we wish:

$ git log -1 --pretty=format:0.0~git%cd.%h --date=format:%Y%m%d%H%M

So let's make these as configurable variables with default being above.

How should we explicitly specify such variables, I guess it should be
through "opts=..." such as:

 opts="mode=git, pretty=0.0~git%cd.%h, date=%Y%m%d%H%M"

But this "git log" needs to have local clone of git repository.

I wonder if I can do without cloning first.

(After all, we eventually need to download contents for the tarball.
So this is not very high priority but I am curious.)

So if the matching pattern starts with ref/tags/, we keep current
pattern.  If it starts with ref/heads, we use this new auto generated
tag.

Adding support to the number of commits is complicated.  Let's be happy
to use hash to be unique commit.  I do not think we upload more than 2
Debian upstream tarball in a minute.  

At least we can reduce cloning cost by only cloning particular branch.

As for "git describe" like nearest tag feature, it's a interesting
thought but it may make things more complicate.  So unless someone
strongly request with patch, I would like to skip it.

(One can always add squashed patch and make a new Debian revision)

Osamu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[James McCoy]

On Sun, Jun 12, 2016 at 02:47:52AM +0900, Osamu Aoki wrote:
> Let me recall your last part of comment again:
> > downloading the last object, and parsing it for the date.
> 
> Can you show me this "parsing it for the date" part with an example
> shell activity?

The below will show the commit date of the last commit.  Adjust
accordingly to get the information for a specific commit.

$ git log -1 --pretty='format:%cd' --date='format:%Y%m%d'

Cheers,
-- 
James
GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7  2D23 DFE6 91AE 331B A3DB

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Osamu Aoki]

Hi,

On Tue, Jan 19, 2016 at 04:30:01PM -0500, Alexandre Viau wrote:
> Package: devscripts
> Version: 2.15.10
> Severity: wishlist
> Usertags: uscan
> 
> In the Go Packaging Team[1], we face the issue that the Golang community
> does not tag releases for library packages. This practice is encouraged
> by the golang tools and by the community. Whether that is a good thing
> or not is the subject of a broader discussion.
> 
> When used with git mode, I would like for uscan to generate a version
> number even though the repository is not tagged.
> 
> One of our tools, dh-make-golang, is able to do this. For example:
> "0.0~git20151028.0.2a60fc2-1":
> 
>  - 0.0 is the latest tag. If there are no tags found, 0.0 is the default.
>  - git20151028 represents the date of the last commit
>  - 0 represents the number of commits that day (counting from 0)
>  - 2a60fc2 is the commit id
> 
> It does not have to be this exact versioning scheme, others can work too.
> 
> I have done a quick research and it looks like this can be done with a
> combination of git ls-remote, downloading the last object, and parsing
> it for the date.

The uscan current uscan does "git ls-remote" and try to use tag as if it
is a version name and locate the latest version using mangling rules.

The current code only use refs/tags/ and designed for it.

> This would help us a great deal to keep up with our packages, and would
> surely be useful for other teams/packages.

Yes.

My current method does not pay attention to branch and simply relies on
the remote tag.  This approach is used to filter out unwanted tags.

After reading Paul's comment, I thought it is a good idea ... but it was
not simple.  Let me describe ...

If you wish to track particular remore branch HEAD, the simplest thing I
can do is to pull that branch and use "git describe" to get the nice tag
if there is a nice previous tag.  (Hush is 97d8771)

$ git describe
v2.16.4-9-g97d8771

Here comes complication when there is no tag.

$ git describe
fatal: No names found, cannot describe anything.
$ git describe --all
heads/master
$ git describe --tags
fatal: No names found, cannot describe anything.
$ git describe --always
c58401b

So I need to do the following to get consistent tag with 0.0~.

$ git describe 2>/dev/null|| ( echo -n "0.0~g" ; git describe --always )
0.0~gc58401b

OK, problem solved ... No, the package version generated in this way
will not be in the sortable series expected by the Debian packaging
system.

We need ito fix this problem using your suggestion to embed date in the
version.

If uscan is in git mode and matching rule is "HEAD" or start with
"ref/heads/", we need to pull the matched object and get the date on it.

Let me recall your last part of comment again:
> downloading the last object, and parsing it for the date.

Can you show me this "parsing it for the date" part with an example
shell activity?

(Yes, I am addicted to gitk.  So I never bothered to use fancy git
copmmands to get date etc.)

Osamu

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Paul Wise]

On Tue, 19 Jan 2016 16:30:01 -0500 Alexandre Viau wrote:

> It does not have to be this exact versioning scheme, others can work too.

I would recommend using the output of `git describe` where there are
tags and there is no tag, act as if there were a 0 tag at the root.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Michael Stapelberg]

Hi Alexandre,

Alexandre Viau  writes:
> This was discussed in dh-make-golang's issue tracker (adding Michael to
> CC as he has expressed a friend of his would like to code this):
>  - https://github.com/Debian/dh-make-golang/issues/8

I’ve checked back with said friend, and his life is too busy to work on
this anytime soon. So, if anyone else wants to step in, that’d be
appreciated :).

-- 
Best regards,
Michael

Bug#811565: [uscan] git mode: allow for scanning repositories without tags

[Alexandre Viau]

Package: devscripts
Version: 2.15.10
Severity: wishlist
Usertags: uscan

In the Go Packaging Team[1], we face the issue that the Golang community
does not tag releases for library packages. This practice is encouraged
by the golang tools and by the community. Whether that is a good thing
or not is the subject of a broader discussion.

When used with git mode, I would like for uscan to generate a version
number even though the repository is not tagged.

One of our tools, dh-make-golang, is able to do this. For example:
"0.0~git20151028.0.2a60fc2-1":

 - 0.0 is the latest tag. If there are no tags found, 0.0 is the default.
 - git20151028 represents the date of the last commit
 - 0 represents the number of commits that day (counting from 0)
 - 2a60fc2 is the commit id

It does not have to be this exact versioning scheme, others can work too.

I have done a quick research and it looks like this can be done with a
combination of git ls-remote, downloading the last object, and parsing
it for the date.

This would help us a great deal to keep up with our packages, and would
surely be useful for other teams/packages.

This was discussed in dh-make-golang's issue tracker (adding Michael to
CC as he has expressed a friend of his would like to code this):
 - https://github.com/Debian/dh-make-golang/issues/8

[1]:
https://qa.debian.org/developer.php?login=pkg-go-maintain...@lists.alioth.debian.org

Cheers,

-- 
Alexandre Viau
av...@debian.org



signature.asc
Description: OpenPGP digital signature