Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[Osamu Aoki]

Hi,

Thanks for checking.
On Sun, Jan 31, 2016 at 09:14:57PM +0100, Uwe Kleine-König wrote:
...
> I tested your change, and it works fine here now. \o/
> 
> > Script started on Sun 31 Jan 2016 05:23:24 PM JST
> > [...]
> > uscan info: Matching pattern:
> >
> > (?:(?:http://www.kernel.org)?\/pub\/linux\/utils\/rt\-tests\/)?rt-tests-(.*)\.tar\.xz
> >  
> > (?:(?:https://www.kernel.org)?\/pub\/linux\/utils\/rt\-tests\/)?rt-tests-(.*)\.tar\.xz
> 
> Using (?:.)? at the start of a regexp doesn't give any advantage,

Why.  href in a web page may be written without "http://www.kernel.org;
We certainly want to match such href string.

> and probably can be dropped without any loss. If you want to keep it:
> Why is the - quoted? You might want to make "http:" optional, because
> otherwise
> 
>   //www.kernel.org/pub/linux/utils/rt-tests/...
> 
> isn't matched.

Do you want to match such href?  I do not understand your point here.

(Quite frankly this part of logic is inherited.  Unless someone explain
me and take responsibility of such changes, I am reluctant to change
logic around this.  base page magic etc. is a bit too complicated for
me.)

Osamu

Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[Uwe Kleine-König]

Hello Osamu,

On 01/31/2016 09:41 AM, Osamu Aoki wrote:
> I think I have fix for this bug report.

Thanks for you quick reaction.

> On Sun, Jan 31, 2016 at 08:03:22AM +0900, Osamu Aoki wrote:
>> On Wed, Jan 27, 2016 at 10:26:49PM -0500, James McCoy wrote:
> 
> Your comment on --force-download is correct.
> 
>>> Thanks for the report.  There are a few things going on here.
>>>
>>> On Wed, Jan 27, 2016 at 11:36:52AM +0100, Uwe Kleine-König wrote:
 now running
> 
> [snip]
>  
uscan: Successfully downloaded package rt-tests-0.96.tar.xz
Could not read ../rt-tests-0.96.tar.xz: No such file or directory at 
 /usr/bin/mk-origtargz line 361.
uscan: error: mk-origtargz --package rt-tests --version 0.96 
 --compression gzip --directory .. --copyright-file debian/copyright 
 ../rt-tests-0.96.tar.xz gave error exit status 2

 where the problem seems to be that uscan decompresses the archive but in
 the same go removes the tar.xz for mk-origtargz.
>>>
>>> Actually, it keeps the tar.xz when it should be passing the filename as
>>> rt-tests-0.96.tar, if the current verification behavior isn't changed.
> 
> uscan keeps filename for tar.xz in its internal variable but
> gunzip/unxz/bunzip2 were invoked without --keep in uscan

I tested your change, and it works fine here now. \o/

> Script started on Sun 31 Jan 2016 05:23:24 PM JST
> [...]
> uscan info: Matching pattern:
>
> (?:(?:http://www.kernel.org)?\/pub\/linux\/utils\/rt\-tests\/)?rt-tests-(.*)\.tar\.xz
>  
> (?:(?:https://www.kernel.org)?\/pub\/linux\/utils\/rt\-tests\/)?rt-tests-(.*)\.tar\.xz

Using (?:.)? at the start of a regexp doesn't give any advantage,
and probably can be dropped without any loss. If you want to keep it:
Why is the - quoted? You might want to make "http:" optional, because
otherwise

//www.kernel.org/pub/linux/utils/rt-tests/...

isn't matched.

Best regards
Uwe



signature.asc
Description: OpenPGP digital signature

Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[Osamu Aoki]

Hi,

Excuseme I was a bit confused with #812417.  This is #812860.

I think I have fix for this bug report.

On Sun, Jan 31, 2016 at 08:03:22AM +0900, Osamu Aoki wrote:
> On Wed, Jan 27, 2016 at 10:26:49PM -0500, James McCoy wrote:

Your comment on --force-download is correct.

> > Thanks for the report.  There are a few things going on here.
> > 
> > On Wed, Jan 27, 2016 at 11:36:52AM +0100, Uwe Kleine-König wrote:
> > > now running

[snip]
 
> > >   uscan: Successfully downloaded package rt-tests-0.96.tar.xz
> > >   Could not read ../rt-tests-0.96.tar.xz: No such file or directory at 
> > > /usr/bin/mk-origtargz line 361.
> > >   uscan: error: mk-origtargz --package rt-tests --version 0.96 
> > > --compression gzip --directory .. --copyright-file debian/copyright 
> > > ../rt-tests-0.96.tar.xz gave error exit status 2
> > > 
> > > where the problem seems to be that uscan decompresses the archive but in
> > > the same go removes the tar.xz for mk-origtargz.
> > 
> > Actually, it keeps the tar.xz when it should be passing the filename as
> > rt-tests-0.96.tar, if the current verification behavior isn't changed.

uscan keeps filename for tar.xz in its internal variable but
gunzip/unxz/bunzip2 were invoked without --keep in uscan

> > > Is this just me using uscan in a wrong way, or is there something fishy
> > > with uscan? In the first case an example would be great.
> > 
> > There are some issues to work out from the major rework of uscan, but
> > hopefully some of the above helps.

I added --keep and now works fine.

(Found another bug as "$options{'pgpsigurlmangle'}=ARRAY(0x2ddfb88)")

Here is my log:

Script started on Sun 31 Jan 2016 05:23:24 PM JST
osamu@goofy: ~/ssd/rt-tests$ debcheckout rt-tests
declared git repository at git://git.pengutronix.de/git/ukl/rt-tests.git
git clone git://git.pengutronix.de/git/ukl/rt-tests.git rt-tests ...
Cloning into 'rt-tests'...
Checking connectivity... done.
/home/osamu/ssd/rt-tests/rt-tests
/home/osamu/ssd/rt-tests/rt-tests
osamu@goofy: ~/ssd/rt-tests$ cd rt-tests
/home/osamu/ssd/rt-tests/rt-tests
osamu@goofy: ~/ssd/rt-tests/rt-tests$ echo  > debian/watch 'version=4'
osamu@goofy: ~/ssd/rt-tests/rt-tests$ echo >> debian/watch
osamu@goofy: ~/ssd/rt-tests/rt-tests$ echo >> debian/watch 
'opts="pgpsigurlmangle=s%.xz$%.sign%, decompress" \'
osamu@goofy: ~/ssd/rt-tests/rt-tests$
osamu@goofy: ~/ssd/rt-tests/rt-tests$ echo >> debian/watch 
'http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-(.*)\.tar\.xz'
osamu@goofy: ~/ssd/rt-tests/rt-tests$ uscan --debug --force-download
uscan info: uscan (version 2.16.1) See uscan(1) for help
uscan info: Scan watch files in .
uscan debug: Found ./debian
uscan info: Check debian/watch and debian/changelog in .
uscan info: package="rt-tests" version="0.96-1" (as seen in debian/changelog)
uscan info: package="rt-tests" version="0.96" (no epoch/revision)
uscan info: ./debian/changelog sets package="rt-tests" version="0.96"
uscan info: Process ./debian/watch (package=rt-tests version=0.96)
uscan info: Found upstream signing keyring: debian/upstream/signing-key.asc
uscan info: opts: pgpsigurlmangle=s%.xz$%.sign%, decompress
uscan info: line: 
http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-(.*)\.tar\.xz
uscan info: Parsing pgpsigurlmangle=s%.xz$%.sign%
uscan info: Parsing  decompress
uscan info: line: 
http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-(.*)\.tar\.xz
uscan debug: $options{'pgpmode'}=mangle, 
$options{'pgpsigurlmangle'}=ARRAY(0x2ddfb88)
uscan info: Last orig.tar.* tarball version (from debian/changelog): 0.96
uscan info: Last orig.tar.* tarball version (dversionmangled): 0.96
uscan info: Requesting URL:
   http://www.kernel.org/pub/linux/utils/rt-tests/
uscan info: redirections: https://www.kernel.org/pub/linux/utils/rt-tests/
uscan debug: received content:


 
  Index of /pub/linux/utils/rt-tests
 
 
Index of /pub/linux/utils/rt-tests
NameLast 
modified  Size  Parent Directory -
older/  22-Oct-2015 10:44-
rt-tests-0.96.tar.gz22-Oct-2015 10:43  
112K
rt-tests-0.96.tar.sign  22-Oct-2015 10:43  
819
rt-tests-0.96.tar.xz22-Oct-2015 10:43   
83K
sha256sums.asc  22-Oct-2015 11:01  1.0K



[End of received content] by HTTP
uscan debug: processed content:


 
  Index of /pub/linux/utils/rt-tests
 
 
Index of /pub/linux/utils/rt-tests
NameLast 
modified  Size  Parent Directory -
older/  22-Oct-2015 10:44-
rt-tests-0.96.tar.gz22-Oct-2015 10:43  
112K
rt-tests-0.96.tar.sign  22-Oct-2015 10:43  
819
rt-tests-0.96.tar.xz22-Oct-2015 10:43   
83K
sha256sums.asc  22-Oct-2015 11:01  1.0K



[End of processed content] by fix bad HTML code
uscan info: Matching pattern:
   
(?:(?:http://www.kernel.org)?\/pub\/linux\/utils\/rt\-tests\/)?rt-tests-(.*)\.tar\.xz
 
(?:(?:https://www.kernel.org)?\/pub\/linux\/utils\/rt\-tests\/)?rt-tests-(.*)\.tar\.xz
uscan 

Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[Osamu Aoki]

Hi,

On Wed, Jan 27, 2016 at 10:26:49PM -0500, James McCoy wrote:
> Thanks for the report.  There are a few things going on here.
> 
> On Wed, Jan 27, 2016 at 11:36:52AM +0100, Uwe Kleine-König wrote:
> > now running

[snip]

> > uscan: Successfully downloaded package rt-tests-0.96.tar.xz
> > Could not read ../rt-tests-0.96.tar.xz: No such file or directory at 
> > /usr/bin/mk-origtargz line 361.
> > uscan: error: mk-origtargz --package rt-tests --version 0.96 
> > --compression gzip --directory .. --copyright-file debian/copyright 
> > ../rt-tests-0.96.tar.xz gave error exit status 2
> > 
> > where the problem seems to be that uscan decompresses the archive but in
> > the same go removes the tar.xz for mk-origtargz.
> 
> Actually, it keeps the tar.xz when it should be passing the filename as
> rt-tests-0.96.tar, if the current verification behavior isn't changed.
> 
> > Is this just me using uscan in a wrong way, or is there something fishy
> > with uscan? In the first case an example would be great.
> 
> There are some issues to work out from the major rework of uscan, but
> hopefully some of the above helps.

I see.  So the crash reported is fixed in previous commit by Antonio and
version ordering problem seen on the log is fixed but I still need to
work on passing the proper file name mk-origtargz.

Together with verbosety issues reported, these needs immediate
attention.

Osamu 

Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[Osamu Aoki]

Hi,

On Wed, Jan 27, 2016 at 11:36:52AM +0100, Uwe Kleine-König wrote:
> Package: devscripts
> Version: 2.15.10
> Severity: normal
> File: /usr/bin/uscan
> Control: user adn+...@diwi.org
> Control: usertag -1 + uscan
> 
> Hello,
> 
> I started experimenting with uscan's pgp mechanism to verfiy the
> signature of rt-tests. You can reproduce my tests using:
> 
>   debcheckout rt-tests
>   cd rt-tests
>   echo  > debian/watch 'version=4'
>   echo >> debian/watch
>   echo >> debian/watch 'opts="pgpsigurlmangle=s%.xz$%.sign%, decompress" 
> \'
>   echo >> debian/watch 
> 'http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-(.*)\.tar\.xz'
... 
> where the problem seems to be that uscan decompresses the archive but in
> the same go removes the tar.xz for mk-origtargz.
> 
> Without decompress in the options the signature verification obviously
> fails.

You are right.  uscan should keep the compressed file when decompressing
it for the signature verification.

> Is this just me using uscan in a wrong way, or is there something fishy
> with uscan? In the first case an example would be great.

No it is uscan problem I created.

Osamu

Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[James McCoy]

Thanks for the report.  There are a few things going on here.

On Wed, Jan 27, 2016 at 11:36:52AM +0100, Uwe Kleine-König wrote:
> now running
> 
>   uscan --debug
> 
> ends in

You omitted these important lines:

uscan: Newest version on remote site is 0.96, local version is 0.96
uscan:=> Package is up to date
uscan: Don't downloading upstream package: rt-tests-0.96.tar.xz

By default, uscan only downloads the upstream archive if it is *newer*
than your source package.  You need to use --force-download to download
even when the newest remote version matches the current version.

>   uscan: Downloading OpenPGP signature from
>  
> http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.sign 
> (pgpsigurlmangled)
>  as rt-tests-0.96.tar.xz.pgp
>   uscan info: Requesting URL:
>  http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.sign
>   uscan warn: FAIL Checking OpenPGP signature (no upstream tarball 
> downloaded).
>   uscan info: Scan finished
> 
> (Here I would have expected a more verbose output to explain the FAIL.)

However, we still downloaded the signature ... I think this might be
related to the request to be able to re-verify an existing archive,
which is the behavior you end up using later on.

> My expectations is that uscan downloads rt-tests-0.96.tar.xz and
> rt-tests-0.96.tar.sign, does something like:
> 
>   zcat rt-tests-0.96.tar.xz | gpg --verify rt-tests-0.96.tar.sign - 
> 
> with the right keyring added to the mix and then links it to
> rt-tests_0.96.orig.tar.xz.

That's the behavior I'd expect as well.  The current behavior
decompresses the archive on disk and then passes that to gpg.

> When doing:
> 
>   cd ..
>   wget http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.xz
>   cd rt-tests
> 
> and starting uscan again I get:
> 
> [snip]
>   uscan: Successfully downloaded package rt-tests-0.96.tar.xz
>   Could not read ../rt-tests-0.96.tar.xz: No such file or directory at 
> /usr/bin/mk-origtargz line 361.
>   uscan: error: mk-origtargz --package rt-tests --version 0.96 
> --compression gzip --directory .. --copyright-file debian/copyright 
> ../rt-tests-0.96.tar.xz gave error exit status 2
> 
> where the problem seems to be that uscan decompresses the archive but in
> the same go removes the tar.xz for mk-origtargz.

Actually, it keeps the tar.xz when it should be passing the filename as
rt-tests-0.96.tar, if the current verification behavior isn't changed.

> Is this just me using uscan in a wrong way, or is there something fishy
> with uscan? In the first case an example would be great.

There are some issues to work out from the major rework of uscan, but
hopefully some of the above helps.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy 

Bug#812860: /usr/bin/uscan: [uscan] failure to download and verify package.tar.xz with package.sign

[Uwe Kleine-König]

Package: devscripts
Version: 2.15.10
Severity: normal
File: /usr/bin/uscan
Control: user adn+...@diwi.org
Control: usertag -1 + uscan

Hello,

I started experimenting with uscan's pgp mechanism to verfiy the
signature of rt-tests. You can reproduce my tests using:

debcheckout rt-tests
cd rt-tests
echo  > debian/watch 'version=4'
echo >> debian/watch
echo >> debian/watch 'opts="pgpsigurlmangle=s%.xz$%.sign%, decompress" 
\'
echo >> debian/watch 
'http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-(.*)\.tar\.xz'

now running

uscan --debug

ends in

uscan: Downloading OpenPGP signature from
   
http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.sign 
(pgpsigurlmangled)
   as rt-tests-0.96.tar.xz.pgp
uscan info: Requesting URL:
   http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.sign
uscan warn: FAIL Checking OpenPGP signature (no upstream tarball 
downloaded).
uscan info: Scan finished

(Here I would have expected a more verbose output to explain the FAIL.)

My expectations is that uscan downloads rt-tests-0.96.tar.xz and
rt-tests-0.96.tar.sign, does something like:

zcat rt-tests-0.96.tar.xz | gpg --verify rt-tests-0.96.tar.sign - 

with the right keyring added to the mix and then links it to
rt-tests_0.96.orig.tar.xz.

When doing:

cd ..
wget http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.xz
cd rt-tests

and starting uscan again I get:

uscan: uscan (version 2.15.10) See uscan(1) for help
uscan: Scan watch files in .
uscan: ./debian/changelog sets package="rt-tests" version="0.96"
uscan: Newest version on remote site is 0.96, local version is 0.96
uscan:=> Package is up to date
uscan: Don't download and use the existing file: rt-tests-0.96.tar.xz
uscan: Downloading OpenPGP signature from
   
http://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-0.96.tar.sign 
(pgpsigurlmangled)
   as rt-tests-0.96.tar.pgp
gpgv: Signature made Thu 22 Oct 2015 12:41:14 PM CEST using RSA key ID 
639D2D16
gpgv: Good signature from "John Kacur "
gpgv: aka "John Kacur "
uscan: Successfully downloaded package rt-tests-0.96.tar.xz
Could not read ../rt-tests-0.96.tar.xz: No such file or directory at 
/usr/bin/mk-origtargz line 361.
uscan: error: mk-origtargz --package rt-tests --version 0.96 
--compression gzip --directory .. --copyright-file debian/copyright 
../rt-tests-0.96.tar.xz gave error exit status 2

where the problem seems to be that uscan decompresses the archive but in
the same go removes the tar.xz for mk-origtargz.

Without decompress in the options the signature verification obviously
fails.

Is this just me using uscan in a wrong way, or is there something fishy
with uscan? In the first case an example would be great.

Best regards
Uwe

-- Package-specific info:

--- /etc/devscripts.conf ---

--- ~/.devscripts ---
BTS_CACHE=no
DEBCHANGE_RELEASE_HEURISTIC=changelog
DEBSIGN_KEYID=32669bd6

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (800, 'testing'), (600, 'unstable'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.3.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages devscripts depends on:
ii  dpkg-dev 1.18.4
ii  libc62.21-6
ii  perl 5.22.1-4
pn  python3:any  

Versions of packages devscripts recommends:
ii  apt 1.2
ii  at  3.1.18-2
ii  curl7.46.0-1
ii  dctrl-tools 2.24-1
ii  debian-keyring  2016.01.20
ii  dput-ng [dput]  1.10
ii  equivs  2.0.9+nmu1
ii  fakeroot1.20.2-1
ii  file1:5.25-2
ii  gnupg   1.4.20-1
ii  gnupg2  2.0.28-3
ii  libdistro-info-perl 0.14
ii  libencode-locale-perl   1.05-1
ii  libjson-perl2.90-1
ii  liblwp-protocol-https-perl  6.06-2
ii  libsoap-lite-perl   1.19-1
ii  liburi-perl 1.71-1
ii  libwww-perl 6.15-1
ii  lintian 2.5.39.1
ii  man-db  2.7.5-1
ii  patch   2.7.5-1
ii  patchutils  0.3.4-1
ii  python3-debian  0.1.27
ii  python3-magic   1:5.25-2
ii  sensible-utils  0.0.9
ii  strace  4.10-3
ii  unzip   6.0-20
ii  wdiff   1.2.2-1+b1
ii  wget1.17.1-1
ii  xz-utils5.1.1alpha+20120614-2.1

Versions of packages devscripts suggests: