Re: uscan roadmap

[Yadd]

On 05/12/2021 10:26, Timo Röhling wrote:

Hi Yadd,

thank you very much for your work on uscan. That new version 5
format looks really promising.

* Yadd  [2021-12-01 09:11]:

* Version 5:
  * Main (first) paragraph contains "Version: 5" and optional options
    that change default values for source-paragraph
  * URL and regex are separated
  * Some default values change. For example, `dversionmangle` default
    value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

[...]

Of course, comments are welcome!
I have a feature request regarding signature verification. As luck would 
have it, I maintain three packages with upstream

signatures; one of them is me being my own upstream, and the other
two do not use the "standard" approach with one GnuPG signature per
source tarball:

- cmake releases its sources in multiple archive formats and signs
   them indirectly (a text file with SHA256 hashes) [1].
- liblzf uses the BSD signify tool [2] and only GnuPG-signs the
   signify key.

I don't know if any of these schemes are used elsewhere (more likely
for the CMake approach, less likely for liblzf, I'd guess), but it
would be nice if uscan offered some support for this; maybe a hook
to run the signature verification by an external script with
autopkgtest semantics (fail if output occurs on stderr the script
returns with a non-zero exit code).


Hi,

this was ~impossible using previous format. With new format, things like 
pgpmode=previous/next will still be accepted (to avoid regressions) but 
new things will be possible. Example (not yet implemented):


  Version: 5

  Source: https://url...
  Version-Regex: @PACKAGE@@ANY_VERSION@@ARCHIVE_EXT@
  Pgp-Version-Regex:  @PACKAGE@@SAME_VERSION@@ARCHIVE_EXT@.asc

Then it will be possible to implement new signature workflows, maybe 
something like:


  Version: 5

  Source: https://url...
  Version-Regex: @PACKAGE@@ANY_VERSION@@ARCHIVE_EXT@
  Pgp-Mode: sha256-hashes
  Sha256-Version-Regex: .*@SAME_VERSION@.sig

Cheers,
Yadd

Re: uscan roadmap

[Timo Röhling]

Hi Yadd,

thank you very much for your work on uscan. That new version 5
format looks really promising.

* Yadd  [2021-12-01 09:11]:

* Version 5:
  * Main (first) paragraph contains "Version: 5" and optional options
that change default values for source-paragraph
  * URL and regex are separated
  * Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

[...]

Of course, comments are welcome!
I have a feature request regarding signature verification. 
As luck would have it, I maintain three packages with upstream

signatures; one of them is me being my own upstream, and the other
two do not use the "standard" approach with one GnuPG signature per
source tarball:

- cmake releases its sources in multiple archive formats and signs
  them indirectly (a text file with SHA256 hashes) [1].
- liblzf uses the BSD signify tool [2] and only GnuPG-signs the
  signify key.

I don't know if any of these schemes are used elsewhere (more likely
for the CMake approach, less likely for liblzf, I'd guess), but it
would be nice if uscan offered some support for this; maybe a hook
to run the signature verification by an external script with
autopkgtest semantics (fail if output occurs on stderr the script
returns with a non-zero exit code).


Cheers
Timo

[1] https://cmake.org/install/#download-verification
[2] http://dist.schmorp.de/signing-key.txt

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: uscan roadmap

[Yadd]

On 01/12/2021 22:16, Yadd wrote:

On 01/12/2021 21:07, Patrice wrote:


Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)


Yes but without optimization neither scheme (except some few fields). 
Example:


   version=4

   opts=uversionmangle=s/-/~/g,pgpmode=none \
    https://...  .*(\d[\d\.]*)@ARCHIVE_EXT@

will be translated into:

   Version: 5

   Source: https://...
   Regex: .*(\d[\d\.]*)@ARCHIVE_EXT@
   Uversionmangle: s/-/~/g

You'll have to manually modify it into

   Version: 5
   Scheme: stable

   Source: https://...


Done, uscan from 
https://salsa.debian.org/debian/devscripts/-/merge_requests/251 is already:


 * natively working with watchFiles version 5
 * able to read old versions transparently (using a wrapper)
 * able to convert files from version 3/4 to 5 (using the same
   wrapper), and probably versions 1/2 (not fully tested):

 uscan --update-watchfile

   Don't us it for now except for tests, key names will probably change

Looking at test results: no regression, all passed

Re: uscan roadmap

[Paul Wise]

On Thu, 2021-12-02 at 10:16 +0100, Yadd wrote:

> Yes but the redirector often responded with 500 codes

500 codes probably just mean bugs in the redirector, which should be
easy to fix for anyone with access to the redirector source code.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Gard Spreemann]

Jonas Smedegaard  writes:

> Quoting Gard Spreemann (2021-12-02 13:09:17)
>> 
>> Jonas Smedegaard  writes:
>> 
>> > Quoting Gard Spreemann (2021-12-02 12:31:30)
>> >> 
>> >> Paul Wise  writes:
>> >> 
>> >> > I also wonder if it is time to split debian/watch out of Debian 
>> >> > source packages, since upstream download locations generally 
>> >> > change independently of the Debian package and so information 
>> >> > about upstream download locations probably should be maintained 
>> >> > independently.
>> >> 
>> >> I very much agree. I don't understand the logic of tying upstream 
>> >> checking to a particular version of a source package. The fact that 
>> >> some version of a package once upon a time could locate and 
>> >> download new upstream versions using a particular recipe is of no 
>> >> use if said recipe becomes outdated at any later time.
>> >> 
>> >> It makes a lot more sense to provide this service in a way that 
>> >> allows it to be modified/updated/improved/fixed with no regards to 
>> >> the actual packages that may use it. That could be as simple as a 
>> >> uscan service with watch files that can be individually and 
>> >> independently modified.
>> >
>> > I find it helpful for our packages to include information about 
>> > where and how (at the time of its release) that package was 
>> > monitoring for upstream releases.  It helps working decentralized - 
>> > both for preparing packages for Debian and for working on 
>> > Debian-derived packages, both without needing access to somewhere 
>> > central for this "watch" information.
>> 
>> Would it make sense for a package to contain a snapshot of the 
>> relevant metadata in the hypothetical "centralized-and-often-updated 
>> watch service" at the time in enters into the archives?
>
> Not _instead_ of current location: What I find helpful is to have the 
> watch file available with the source package.  I am unaware if there 
> would be some benefit of _additionally_ embedding the watch file in 
> binary packages (if that's what you meant).
>
>
>> > Therefore I like the proposal to have Debian project scanners 
>> > aggressively look for _newest_ watch file for a packaging project, 
>> > including looking up declared Vcs-* hints for not-yet-released work.
>> 
>> Indeed, that sounds like a better idea than what I suggest above!
>
> Not sure if you noticed, but (since I won't steal credit) I basically 
> emphasized Pabs' suggestion in last paragraph of what you previously 
> quoted:
>
> Quoting Paul Wise (2021-12-02 00:47:44)
>> Alternatively, perhaps we could workaround outdated debian/watch files
>> by having vcswatch extract debian/watch files from the repo specified
>> in the Vcs-* URLs.

Apologies; I somehow thought that he meant auto-generating watch files
from *upstream* VCSs. My bad, and thanks for clarifying!

 -- Gard
 


signature.asc
Description: PGP signature

Re: uscan roadmap

[Julien Puydt]

Hi

Le jeu. 2 déc. 2021 à 11:36, Yadd  a écrit :

>
> Another idea to have a compromise:
>   * uscan is released with versioned schemes (GitHub.json, sf.json,...)
>   * when launched, it tries to download new version from a new Debian API
> (static json files)
> * if no response or no new version, uscan uses its own scheme or a
>   previously downloaded update (verifying signature)
> * if a new version is available from new redirector:
>   * it verifies GPG signature of new scheme
> * if not OK, it warns and uses cached scheme
> * if OK, it stores it with signature in ~/.cache/uscan/schemes
>

What I don't like is that it will need time to check new profiles on a
central site, which looks like an invitation for DoS situations.

I propose a variation of this: an explicit
"uscan --update" will update the profiles, and all other calls will use the
known profiles.

Cheers,

J. Puydt

Re: uscan roadmap

[Jonas Smedegaard]

Quoting Gard Spreemann (2021-12-02 13:09:17)
> 
> Jonas Smedegaard  writes:
> 
> > Quoting Gard Spreemann (2021-12-02 12:31:30)
> >> 
> >> Paul Wise  writes:
> >> 
> >> > I also wonder if it is time to split debian/watch out of Debian 
> >> > source packages, since upstream download locations generally 
> >> > change independently of the Debian package and so information 
> >> > about upstream download locations probably should be maintained 
> >> > independently.
> >> 
> >> I very much agree. I don't understand the logic of tying upstream 
> >> checking to a particular version of a source package. The fact that 
> >> some version of a package once upon a time could locate and 
> >> download new upstream versions using a particular recipe is of no 
> >> use if said recipe becomes outdated at any later time.
> >> 
> >> It makes a lot more sense to provide this service in a way that 
> >> allows it to be modified/updated/improved/fixed with no regards to 
> >> the actual packages that may use it. That could be as simple as a 
> >> uscan service with watch files that can be individually and 
> >> independently modified.
> >
> > I find it helpful for our packages to include information about 
> > where and how (at the time of its release) that package was 
> > monitoring for upstream releases.  It helps working decentralized - 
> > both for preparing packages for Debian and for working on 
> > Debian-derived packages, both without needing access to somewhere 
> > central for this "watch" information.
> 
> Would it make sense for a package to contain a snapshot of the 
> relevant metadata in the hypothetical "centralized-and-often-updated 
> watch service" at the time in enters into the archives?

Not _instead_ of current location: What I find helpful is to have the 
watch file available with the source package.  I am unaware if there 
would be some benefit of _additionally_ embedding the watch file in 
binary packages (if that's what you meant).


> > Therefore I like the proposal to have Debian project scanners 
> > aggressively look for _newest_ watch file for a packaging project, 
> > including looking up declared Vcs-* hints for not-yet-released work.
> 
> Indeed, that sounds like a better idea than what I suggest above!

Not sure if you noticed, but (since I won't steal credit) I basically 
emphasized Pabs' suggestion in last paragraph of what you previously 
quoted:

Quoting Paul Wise (2021-12-02 00:47:44)
> Alternatively, perhaps we could workaround outdated debian/watch files
> by having vcswatch extract debian/watch files from the repo specified
> in the Vcs-* URLs.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: uscan roadmap

[Gard Spreemann]

Jonas Smedegaard  writes:

> Quoting Gard Spreemann (2021-12-02 12:31:30)
>> 
>> Paul Wise  writes:
>> 
>> > I also wonder if it is time to split debian/watch out of Debian 
>> > source packages, since upstream download locations generally change 
>> > independently of the Debian package and so information about 
>> > upstream download locations probably should be maintained 
>> > independently.
>> 
>> I very much agree. I don't understand the logic of tying upstream 
>> checking to a particular version of a source package. The fact that 
>> some version of a package once upon a time could locate and download 
>> new upstream versions using a particular recipe is of no use if said 
>> recipe becomes outdated at any later time.
>> 
>> It makes a lot more sense to provide this service in a way that allows 
>> it to be modified/updated/improved/fixed with no regards to the actual 
>> packages that may use it. That could be as simple as a uscan service 
>> with watch files that can be individually and independently modified.
>
> I find it helpful for our packages to include information about where 
> and how (at the time of its release) that package was monitoring for 
> upstream releases.  It helps working decentralized - both for preparing 
> packages for Debian and for working on Debian-derived packages, both 
> without needing access to somewhere central for this "watch" 
> information.

Would it make sense for a package to contain a snapshot of the relevant
metadata in the hypothetical "centralized-and-often-updated watch
service" at the time in enters into the archives?

> Therefore I like the proposal to have Debian project scanners 
> aggressively look for _newest_ watch file for a packaging project, 
> including looking up declared Vcs-* hints for not-yet-released work.

Indeed, that sounds like a better idea than what I suggest above!


 -- Gard
 


signature.asc
Description: PGP signature

Re: uscan roadmap

[Jonas Smedegaard]

Quoting Gard Spreemann (2021-12-02 12:31:30)
> 
> Paul Wise  writes:
> 
> > I also wonder if it is time to split debian/watch out of Debian 
> > source packages, since upstream download locations generally change 
> > independently of the Debian package and so information about 
> > upstream download locations probably should be maintained 
> > independently.
> 
> I very much agree. I don't understand the logic of tying upstream 
> checking to a particular version of a source package. The fact that 
> some version of a package once upon a time could locate and download 
> new upstream versions using a particular recipe is of no use if said 
> recipe becomes outdated at any later time.
> 
> It makes a lot more sense to provide this service in a way that allows 
> it to be modified/updated/improved/fixed with no regards to the actual 
> packages that may use it. That could be as simple as a uscan service 
> with watch files that can be individually and independently modified.

I find it helpful for our packages to include information about where 
and how (at the time of its release) that package was monitoring for 
upstream releases.  It helps working decentralized - both for preparing 
packages for Debian and for working on Debian-derived packages, both 
without needing access to somewhere central for this "watch" 
information.

Therefore I like the proposal to have Debian project scanners 
aggressively look for _newest_ watch file for a packaging project, 
including looking up declared Vcs-* hints for not-yet-released work.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: uscan roadmap

[Gard Spreemann]

Paul Wise  writes:

> I also wonder if it is time to split debian/watch out of Debian source
> packages, since upstream download locations generally change
> independently of the Debian package and so information about upstream
> download locations probably should be maintained independently.

I very much agree. I don't understand the logic of tying upstream
checking to a particular version of a source package. The fact that some
version of a package once upon a time could locate and download new
upstream versions using a particular recipe is of no use if said recipe
becomes outdated at any later time.

It makes a lot more sense to provide this service in a way that allows
it to be modified/updated/improved/fixed with no regards to the actual
packages that may use it. That could be as simple as a uscan service
with watch files that can be individually and independently modified.


 Best,
 Gard
 


signature.asc
Description: PGP signature

Re: uscan roadmap

[Geert Stappers]

On Thu, Dec 02, 2021 at 11:36:08AM +0100, Yadd wrote:
> On 02/12/2021 10:16, Yadd wrote:
> > On 02/12/2021 00:34, Paul Wise  wrote:
> > > On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:
> > > 
> > > > Personally I dislike redirectors.
> > > 
> > > A redirector service is superior to including the redirector code
> > > within uscan itself or within a debian/watch file, since when the
> > > upstream website breaks the existing code, a service can be updated in
> > > one place immediately, while uscan in Debian stable will be broken
> > > until the next point release if it gets fixed at all and one in
> > > debian/watch requires every package using the site to get updated.

So true


> > 
> > Yes but the redirector often responded with 500 codes
> 
> Another idea to have a compromise:
>  * uscan is released with versioned schemes (GitHub.json, sf.json,...)
>  * when launched, it tries to download new version from a new Debian API
>(static json files)
>* if no response or no new version, uscan uses its own scheme or a
>  previously downloaded update (verifying signature)
>* if a new version is available from new redirector:
>  * it verifies GPG signature of new scheme
>* if not OK, it warns and uses cached scheme
>* if OK, it stores it with signature in ~/.cache/uscan/schemes
> 
> Then:
>  * no more redirector with an heavy load, but just some JSON schemes
>statically stored
>  * uscan still works if Debian website doesn't respond
> 
> What do you think about this idea?


Way too optimistic   :-)

The original problem was (and is) dealing with various upstream websites.

Putting a translator, a redirector, between uscan and a single upstream
website solves the problem for that particular website.

IMNSHO is building (hard to upgrade and distribute) "solutions"
for redirectors with 500s or whatever error effort at the wrong place.

Explaining to the user (us, debian maintainers) what is happing is a
better approach.   Especial when the redirector can explain the 500 is
due problems with the actual upstream website.



Groeten
Geert Stappers
-- 
Silence is hard to parse

Re: uscan roadmap

[Yadd]

On 02/12/2021 10:16, Yadd wrote:

On 02/12/2021 00:34, Paul Wise  wrote:

On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:


Personally I dislike redirectors.


A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.



Yes but the redirector often responded with 500 codes


Another idea to have a compromise:
 * uscan is released with versioned schemes (GitHub.json, sf.json,...)
 * when launched, it tries to download new version from a new Debian API
   (static json files)
   * if no response or no new version, uscan uses its own scheme or a
 previously downloaded update (verifying signature)
   * if a new version is available from new redirector:
 * it verifies GPG signature of new scheme
   * if not OK, it warns and uses cached scheme
   * if OK, it stores it with signature in ~/.cache/uscan/schemes

Then:
 * no more redirector with an heavy load, but just some JSON schemes
   statically stored
 * uscan still works if Debian website doesn't respond
 * GPG permits to be sure that scheme isn't corrupted (released files
   are as protected as uscan itself because owned by root)
 * easy update if upstream store changes its behavior: just to update
   one JSON file

What do you think about this idea? Which GPG keys will be accepted?

More than one scheme could be used. Example:

  Schemes: GitHub, stable

  Schemes: GitHub, semver

GitHub scheme:

  {
"Version": "1.0",
"filenamemangle": "auto",
"regex": "archive/.*/v?(@DEFAULT_VERSION_REGEX@)@ARCHIVE_EXT@$",
...
  }

"stable" scheme:

  {
"Version": "1.0",
"Default-Version-Regex": "(?:0|[1-9]\d*)(?:0|[1-9]\d*)*"
  }

Re: uscan roadmap

[Yadd]

Le 2 décembre 2021 00:34:27 GMT+01:00, Paul Wise  a écrit :
>On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:
>
>> Personally I dislike redirectors.
>
>A redirector service is superior to including the redirector code
>within uscan itself or within a debian/watch file, since when the
>upstream website breaks the existing code, a service can be updated in
>one place immediately, while uscan in Debian stable will be broken
>until the next point release if it gets fixed at all and one in
>debian/watch requires every package using the site to get updated.
>

Yes but the redirector often responded with 500 codes
-- 
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma 
brièveté.

Re: uscan roadmap

[Paul Wise]

On Wed, 2021-12-01 at 09:11 +0100, Yadd wrote:

> after few discussions with some devscripts maintainers, we decided to
> build a new "version=5" format for debian/watch.

It might be a idea to look at how other distributions do checking for
new upstream releases and adopt some of their improvements.

I note Fedora uses a service (that isn't Fedora specific) for this:

https://release-monitoring.org
https://docs.fedoraproject.org/en-US/package-maintainers/Upstream_Release_Monitoring/

Another idea would be to use the Repology service to notice when other
distros include a newer version of a package than Debian does. 

https://repology.org/

I also wonder if it is time to split debian/watch out of Debian source
packages, since upstream download locations generally change
independently of the Debian package and so information about upstream
download locations probably should be maintained independently.

Alternatively, perhaps we could workaround outdated debian/watch files
by having vcswatch extract debian/watch files from the repo specified
in the Vcs-* URLs.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Paul Wise]

On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:

> sf.net because it needs JS interpretation

The sf.net redirector uses the RSS feed of the files.
This is documented at the top of the redirector HTML:

$ curl -s https://qa.debian.org/watch/sf.php/NSIS/ | grep -i rss
(https://sourceforge.net/projects/NSIS/rss?limit=1000;>RSS)
pages. File listing converted from the RSS.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Paul Wise]

On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:

> Personally I dislike redirectors.

A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Yadd]

On 01/12/2021 21:07, Patrice wrote:


Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)


Yes but without optimization neither scheme (except some few fields). 
Example:


  version=4

  opts=uversionmangle=s/-/~/g,pgpmode=none \
   https://...  .*(\d[\d\.]*)@ARCHIVE_EXT@

will be translated into:

  Version: 5

  Source: https://...
  Regex: .*(\d[\d\.]*)@ARCHIVE_EXT@
  Uversionmangle: s/-/~/g

You'll have to manually modify it into

  Version: 5
  Scheme: stable

  Source: https://...

Re: uscan roadmap

[Patrice]

Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)

Wishes,
Patrice

Re: uscan roadmap

[Tomas Pospisek]

On 01.12.21 12:50, Mattia Rizzolo wrote:


Likewise, I would love if uscan could just learn how github, gitlab,
launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:

   Source: GitHub
   Source-Options:
namespace: trendmicro
project: tlsh
match-on: tags|releases


Excellent idea: +1

However at the very moment that abstractions get introduced (which is 
+1), please, please, please do keep the poor users in mind when stuff 
*does not* work. I.e. please make uscan trivially debuggable.


Something like:

$ uscan --verbose
... found GitHub source definition:
Source: GitHub
Source-Options:
  namespace: trendmicro
  project: tlsh
  match-on: tags|releases
... using that GitHub source definition to find new release under 
https://github.com/foo/bar/releases/bar-1.2.3.tar (uscan.py:line 3498)


One of the regularly recurring frustrations I am encountering is that 
I'm using some SW that has abstracted something and I know in principle 
how the thing it has abstracted works, but am completely unable to find 
out where the abstraction gets all wired up and falls on its face. (Hi 
k8s!). The famous "computer says no" moment.


OK, this is just me whining and not contributing any code at all, so 
please take it as just that: a wishlist item.

*t

Re: uscan roadmap

[Yadd]

On 01/12/2021 18:39, Thomas Goirand wrote:

Hi Yadd,

Thanks a lot for working on this. What you are proposing (ie: using a
mime thing, which is easy to parse instead of the dirty command-line
oriented thingy of version 3 and 4) feels much nicer than what we
currently have.

On 12/1/21 12:53 PM, Yadd wrote:

Fix: will be

   Version: 5

   Source: https://qa.debian.org/watch/sourceforge/
   Regex: -(.+)\.tar\.gz


That's much nicer than previous proposal!


And I don't think "uupdate" is still useful.


IMO, it is needed. That's what is nice with calling scripts: it can take
care with programming of things you didn't even think of. If you remove
it, the risk is that maintainers will continue to use version 3 or 4,
because they still need an update script.

How about:

Update-Script: uupdate

?


Looks good to me

Re: uscan roadmap

[Thomas Goirand]

Hi Yadd,

Thanks a lot for working on this. What you are proposing (ie: using a
mime thing, which is easy to parse instead of the dirty command-line
oriented thingy of version 3 and 4) feels much nicer than what we
currently have.

On 12/1/21 12:53 PM, Yadd wrote:
> Fix: will be
> 
>   Version: 5
> 
>   Source: https://qa.debian.org/watch/sourceforge/
>   Regex: -(.+)\.tar\.gz

That's much nicer than previous proposal!

> And I don't think "uupdate" is still useful.

IMO, it is needed. That's what is nice with calling scripts: it can take
care with programming of things you didn't even think of. If you remove
it, the risk is that maintainers will continue to use version 3 or 4,
because they still need an update script.

How about:

Update-Script: uupdate

?

Cheers,

Thomas Goirand (zigo)

Re: uscan roadmap

[Russ Allbery]

Yadd  writes:

> after few discussions with some devscripts maintainers, we decided to
> build a new "version=5" format for debian/watch.

> Principles:
>  * keep compatibility with versions 3 and 4, no need to change all
>debian/watch files
>  * new version 5 format using the same syntax than other debian/* files
>(rfc822 + "# comments")

I have no detailed feedback on your proposal, but just wanted to say thank
you so much for doing this.  Being able to write a uscan file using a
clear key/value syntax will make me very happy.

-- 
Russ Allbery (r...@debian.org)  

Re: uscan roadmap

[Yadd]

On 01/12/2021 13:14, Jonas Smedegaard wrote:

Quoting Yadd (2021-12-01 13:04:09)

On 01/12/2021 12:50, Mattia Rizzolo wrote:

Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json.  Though I accepted it
because I saw some value, I'm totally in awe of universes where that
is actually needed..

This is very useful for package with components. This is the only way
to be able to `uscan --download-current-version`.


Speakin of components, it would be quite helpful if possible to handle
versions of components - e.g. upgrade all components except
SomeComponent like this:

   `uscan --download-current-version-SomeComponent`


Yes, maybe with a --download-compat-version-SomeComponent which will 
accept upgrades only if it is semver-compatible (no major updates)


Note that this will be in a separated MR (not related to version change)

Re: uscan roadmap

[Jonas Smedegaard]

Quoting Yadd (2021-12-01 13:04:09)
> On 01/12/2021 12:50, Mattia Rizzolo wrote:
> > Possibly, I'm indeed kind of unimpressed that we grew a parse for 
> > nodejs' package.json and perl's META.json.  Though I accepted it 
> > because I saw some value, I'm totally in awe of universes where that 
> > is actually needed..
> This is very useful for package with components. This is the only way 
> to be able to `uscan --download-current-version`.

Speakin of components, it would be quite helpful if possible to handle 
versions of components - e.g. upgrade all components except 
SomeComponent like this:

  `uscan --download-current-version-SomeComponent`


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: uscan roadmap

[Yadd]

On 01/12/2021 12:50, Mattia Rizzolo wrote:

On Wed, Dec 01, 2021 at 12:39:41PM +0100, Geert Stappers wrote:

Summary: unhide redirectors


And not only.


On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:

after few discussions with some devscripts maintainers, we decided to build
a new "version=5" format for debian/watch.


To be clear, this is a *very* non-ready proposal that we are getting
through the wider community.  Nothing of this is implemented anywhere.


* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...


I honestly would like to add website-aware functionalities to uscan,
such as exactly this.


I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.

[..]

  Version: 5
  Source: https://qa.debian.org/watch/sourceforge/ 
-(.+)\.tar\.gz debian uupdate


I would like something like:
   Source: qa-redirector
   Source-Options:
name: sourceforge
project: 


Likewise, I would love if uscan could just learn how github, gitlab,
launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:

   Source: GitHub
   Source-Options:
namespace: trendmicro
project: tlsh
match-on: tags|releases


OK, "Source" will be:
 * a keyword known by uscan DB
 * a URL for other websites

And what about default version value, "stable" (only digits), "semver" 
or "any" (== @ANY_VERSION@)



To go either matching on https://github.com/trendmicro/tlsh/tags or
https://github.com/trendmicro/tlsh/releases. using then Scheme (a name
that, tbh, I don't particularly like right now) for the tags or releases
regex.


And I think such change will allow removal of

bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.

from the uscan code and uscan manual page  (they are in /usr/bin/uscan )


The goal is to have documented that there are extra components being used.
Avoiding nasty surprises.


this feels like the opposite direction I'm proposing above :D


Seconded


Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter.


Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json.  Though I accepted it because
I saw some value, I'm totally in awe of universes where that is actually
needed..
This is very useful for package with components. This is the only way to 
be able to `uscan --download-current-version`.

Re: uscan roadmap

[Yadd]

On 01/12/2021 12:39, Geert Stappers wrote:

Summary: unhide redirectors

On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:

Hi,

after few discussions with some devscripts maintainers, we decided to build
a new "version=5" format for debian/watch.

Principles:
  * keep compatibility with versions 3 and 4, no need to change all
debian/watch files
  * new version 5 format using the same syntax than other debian/* files
(rfc822 + "# comments")
  * no option renaming (becomes case-insensitive to be compliant with
all formats)
  * Version 5:
* Main (first) paragraph contains "Version: 5" and optional options
  that change default values for source-paragraph
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

Example:

   Version: 5


   


Of course, comments are welcome!



I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.


Example

  version=4
  https://sf.net// -(.+)\.tar\.gz debian uupdate


becomes something like

  Version: 5
  Source: https://qa.debian.org/watch/sourceforge/ 
-(.+)\.tar\.gz debian uupdate


Fix: will be

  Version: 5

  Source: https://qa.debian.org/watch/sourceforge/
  Regex: -(.+)\.tar\.gz

And I don't think "uupdate" is still useful.

Personally I dislike redirectors. For npmjs, I wrote a 
"serachMode=plain" to be able to parse npm JSON result, also because 
there are a lot of npmjs packages and redirector was often unable to 
respond. This is different of course for sf.net because it needs JS 
interpretation



And I think such change will allow removal of

bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.

from the uscan code and uscan manual page  (they are in /usr/bin/uscan )


Yes, doc will be updated


The goal is to have documented that there are extra components being used.
Avoiding nasty surprises.

Groeten
Geert Stappers


Thanks!


P.S.
Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter

Re: uscan roadmap

[Mattia Rizzolo]

On Wed, Dec 01, 2021 at 12:39:41PM +0100, Geert Stappers wrote:
> Summary: unhide redirectors

And not only.

> On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
> > after few discussions with some devscripts maintainers, we decided to build
> > a new "version=5" format for debian/watch.

To be clear, this is a *very* non-ready proposal that we are getting
through the wider community.  Nothing of this is implemented anywhere.

> >* URL and regex are separated
> >* Some default values change. For example, `dversionmangle` default
> >  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
> >  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

I honestly would like to add website-aware functionalities to uscan,
such as exactly this.

> I think the move from v4 to v5 is an excellent opportunity
> to express in the watch file that there is a dependency on a redirector.
[..]
>  Version: 5
>  Source: https://qa.debian.org/watch/sourceforge/ 
> -(.+)\.tar\.gz debian uupdate

I would like something like:
  Source: qa-redirector
  Source-Options:
   name: sourceforge
   project: 


Likewise, I would love if uscan could just learn how github, gitlab,
launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:

  Source: GitHub
  Source-Options:
   namespace: trendmicro
   project: tlsh
   match-on: tags|releases

To go either matching on https://github.com/trendmicro/tlsh/tags or
https://github.com/trendmicro/tlsh/releases. using then Scheme (a name
that, tbh, I don't particularly like right now) for the tags or releases
regex.

> And I think such change will allow removal of
> 
>bare
>Disable all site specific special case code such as URL
>redirector uses and page content alterations.
> 
> from the uscan code and uscan manual page  (they are in /usr/bin/uscan )
> 
> 
> The goal is to have documented that there are extra components being used.
> Avoiding nasty surprises.

this feels like the opposite direction I'm proposing above :D

> Awareness of redirectors will get us more redirectors.
> Those redirectors will help us to prevent that `uscan`
> must get a javascript interpreter.

Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json.  Though I accepted it because
I saw some value, I'm totally in awe of universes where that is actually
needed..

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: uscan roadmap

[Geert Stappers]

Summary: unhide redirectors

On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
> Hi,
> 
> after few discussions with some devscripts maintainers, we decided to build
> a new "version=5" format for debian/watch.
> 
> Principles:
>  * keep compatibility with versions 3 and 4, no need to change all
>debian/watch files
>  * new version 5 format using the same syntax than other debian/* files
>(rfc822 + "# comments")
>  * no option renaming (becomes case-insensitive to be compliant with
>all formats)
>  * Version 5:
>* Main (first) paragraph contains "Version: 5" and optional options
>  that change default values for source-paragraph
>* URL and regex are separated
>* Some default values change. For example, `dversionmangle` default
>  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
>  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
> 
> Example:
> 
>   Version: 5
> 
  
> 
> Of course, comments are welcome!


I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.


Example

 version=4
 https://sf.net// -(.+)\.tar\.gz debian uupdate


becomes something like

 Version: 5
 Source: https://qa.debian.org/watch/sourceforge/ 
-(.+)\.tar\.gz debian uupdate



And I think such change will allow removal of

   bare
   Disable all site specific special case code such as URL
   redirector uses and page content alterations.

from the uscan code and uscan manual page  (they are in /usr/bin/uscan )


The goal is to have documented that there are extra components being used.
Avoiding nasty surprises.




Groeten
Geert Stappers


P.S.
Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter.


-- 
Silence is hard to parse