Re: [Bug-wget] [PATCH 01/25] Add two Metalink/XML tests
On Tue, 13 Sep 2016 10:10:49 +0200 Giuseppe Scrivano wrote: > Hi Matthew, > > Matthew White writes: > > > Next patches add more conditions to testenv/Test-metalink-xml-relpath.py. > > > > Also variants are introduced, like: > > * testenv/Test-metalink-xml-trust.py > > * testenv/Test-metalink-xml-homepath.py > > * testenv/Test-metalink-xml-homepath-trust.py > > * testenv/Test-metalink-xml-abspath-trust.py > > * testenv/Test-metalink-xml-relpath-trust.py > > > > And there will be other tests with other meanings too. > > > > So, I don't know if it's a good idea to mix specific tests together. > > > > This topic also applies to Patch 05/25, Patch 10/25, and Patch 17/25: > > http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00046.html > > http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00050.html > > http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00057.html > > > > WDYT? > > Could we have a base file, that is shared by these files? I agree it is > good to have a separate file for each different test, but if the > difference is only for one line, I think we can refactor them a bit. It > will make maintainance easier later. I'll work on it. > > Regards, > Giuseppe Regards, Matthew -- Matthew White pgpHumFkSxT3S.pgp Description: PGP signature
Re: [Bug-wget] [PATCH 01/25] Add two Metalink/XML tests
Hi Matthew, Matthew White writes: > Next patches add more conditions to testenv/Test-metalink-xml-relpath.py. > > Also variants are introduced, like: > * testenv/Test-metalink-xml-trust.py > * testenv/Test-metalink-xml-homepath.py > * testenv/Test-metalink-xml-homepath-trust.py > * testenv/Test-metalink-xml-abspath-trust.py > * testenv/Test-metalink-xml-relpath-trust.py > > And there will be other tests with other meanings too. > > So, I don't know if it's a good idea to mix specific tests together. > > This topic also applies to Patch 05/25, Patch 10/25, and Patch 17/25: > http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00046.html > http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00050.html > http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00057.html > > WDYT? Could we have a base file, that is shared by these files? I agree it is good to have a separate file for each different test, but if the difference is only for one line, I think we can refactor them a bit. It will make maintainance easier later. Regards, Giuseppe
Re: [Bug-wget] [PATCH 01/25] Add two Metalink/XML tests
On Sun, 11 Sep 2016 21:52:16 +0200
Giuseppe Scrivano wrote:
> Hi Matthew,
>
> thanks for all your work, I am going to comment on each patch separately:
>
> Matthew White writes:
>
>
> > index 000..041d772
> > --- /dev/null
> > +++ b/testenv/Test-metalink-xml-relpath.py
> > @@ -0,0 +1,87 @@
> > +#!/usr/bin/env python3
> > +from sys import exit
> > +from test.http_test import HTTPTest
> > +from misc.wget_file import WgetFile
> > +import re
> > +import hashlib
> > +
> > +"""
> > +This is to test if Metalink XML file escapes current directory.
> > +"""
> > +# File Definitions
> > ###
> > +File1 = "Would you like some Tea?"
> > +File1_lowPref = "Do not take this"
> > +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest ()
> > +MetaXml = \
> > +"""
> > +http://www.metalinker.org/";>
> > +
> > +GNU Wget
> > +
> > +
> > +GNU GPL
> > +http://www.gnu.org/licenses/gpl.html
> > +
> > + Wget Test File 1
> > + 1.2.3
> > + Wget Test File 1 description
> > +
> > +
>
> If the difference between the two files is only here, could we have only
> one test file and another template substition for file name? Somethging
> like ?
Next patches add more conditions to testenv/Test-metalink-xml-relpath.py.
Also variants are introduced, like:
* testenv/Test-metalink-xml-trust.py
* testenv/Test-metalink-xml-homepath.py
* testenv/Test-metalink-xml-homepath-trust.py
* testenv/Test-metalink-xml-abspath-trust.py
* testenv/Test-metalink-xml-relpath-trust.py
And there will be other tests with other meanings too.
So, I don't know if it's a good idea to mix specific tests together.
This topic also applies to Patch 05/25, Patch 10/25, and Patch 17/25:
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00046.html
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00050.html
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00057.html
WDYT?
>
>
> > +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml)
> > +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml)
> > +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml)
> > +MetaFile.content = MetaXml
>
> should be enough to use replace instead of the re module?
>
> MetaXml = MetaXml.replace('{{FILE1_HASH}}', File1_sha256)
Fixed. Posting after final decisions are taken about open topics in this series
of patches.
>
> Thanks,
> Giuseppe
Regards,
Matthew
--
Matthew White
pgpWPg7fpxWDk.pgp
Description: PGP signature
Re: [Bug-wget] [PATCH 01/25] Add two Metalink/XML tests
Hi Matthew,
thanks for all your work, I am going to comment on each patch separately:
Matthew White writes:
> index 000..041d772
> --- /dev/null
> +++ b/testenv/Test-metalink-xml-relpath.py
> @@ -0,0 +1,87 @@
> +#!/usr/bin/env python3
> +from sys import exit
> +from test.http_test import HTTPTest
> +from misc.wget_file import WgetFile
> +import re
> +import hashlib
> +
> +"""
> +This is to test if Metalink XML file escapes current directory.
> +"""
> +# File Definitions
> ###
> +File1 = "Would you like some Tea?"
> +File1_lowPref = "Do not take this"
> +File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest ()
> +MetaXml = \
> +"""
> +http://www.metalinker.org/";>
> +
> +GNU Wget
> +
> +
> +GNU GPL
> +http://www.gnu.org/licenses/gpl.html
> +
> + Wget Test File 1
> + 1.2.3
> + Wget Test File 1 description
> +
> +
If the difference between the two files is only here, could we have only
one test file and another template substition for file name? Somethging
like ?
> +MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml)
> +MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml)
> +MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml)
> +MetaFile.content = MetaXml
should be enough to use replace instead of the re module?
MetaXml = MetaXml.replace('{{FILE1_HASH}}', File1_sha256)
Thanks,
Giuseppe
[Bug-wget] [PATCH 01/25] Add two Metalink/XML tests
[Coverity Scan is ok, make syntax-check is ok, make check-valgrind is ok,
contrib/check-hard is ok]
Patch provided by Tim, slightly amended to make the tests to return a proper
result that conforms to the RFC5854 standard [1].
References:
[1] https://tools.ietf.org/html/rfc5854#section-4.1.2.1
This adds two new Metalink tests to testenv:
* testenv/Test-metalink-xml-relpath.py: Test if Metalink/XML forbids relative
paths
* testenv/Test-metalink-xml-abspath.py: Test if Metalink/XML forbids absolute
paths
Regards,
Matthew
--
Matthew White
>From 6dda91458cf0e8978be8de029b4b8a84bab449ce Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Tim=20R=C3=BChsen?=
Date: Tue, 9 Aug 2016 21:07:19 +0200
Subject: [PATCH 01/25] Add two Metalink/XML tests
* testenv/Test-metalink-xml-abspath.py: Reject absolute paths
* testenv/Test-metalink-xml-relpath.py: Reject relative paths
* testenv/Makefile.am: Add both new files to metalink tests
---
testenv/Makefile.am | 6 ++-
testenv/Test-metalink-xml-abspath.py | 87
testenv/Test-metalink-xml-relpath.py | 87
3 files changed, 178 insertions(+), 2 deletions(-)
create mode 100755 testenv/Test-metalink-xml-abspath.py
create mode 100755 testenv/Test-metalink-xml-relpath.py
diff --git a/testenv/Makefile.am b/testenv/Makefile.am
index deef18e..41bf902 100644
--- a/testenv/Makefile.am
+++ b/testenv/Makefile.am
@@ -27,8 +27,10 @@
if METALINK_IS_ENABLED
- METALINK_TESTS = Test-metalink-xml.py \
-Test-metalink-http.py
+ METALINK_TESTS = Test-metalink-xml.py \
+Test-metalink-http.py \
+Test-metalink-xml-relpath.py\
+Test-metalink-xml-abspath.py
else
METALINK_TESTS =
endif
diff --git a/testenv/Test-metalink-xml-abspath.py b/testenv/Test-metalink-xml-abspath.py
new file mode 100755
index 000..62aabb9
--- /dev/null
+++ b/testenv/Test-metalink-xml-abspath.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from misc.wget_file import WgetFile
+import re
+import hashlib
+
+"""
+This is to test if Metalink XML file escapes current directory.
+"""
+# File Definitions ###
+File1 = "Would you like some Tea?"
+File1_lowPref = "Do not take this"
+File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest ()
+MetaXml = \
+"""
+http://www.metalinker.org/";>
+
+GNU Wget
+
+
+GNU GPL
+http://www.gnu.org/licenses/gpl.html
+
+ Wget Test File 1
+ 1.2.3
+ Wget Test File 1 description
+
+
+
+{{FILE1_HASH}}
+
+
+http://broken.example/File1
+http://{{SRV_HOST}}:{{SRV_PORT}}/File1_lowPref
+http://{{SRV_HOST}}:{{SRV_PORT}}/File1
+
+
+
+
+"""
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File1_lowPref", File1_lowPref)
+MetaFile = WgetFile ("test.meta4", MetaXml)
+
+WGET_OPTIONS = "--input-metalink test.meta4"
+WGET_URLS = [[]]
+
+Files = [[A_File, B_File]]
+Existing_Files = [MetaFile]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [MetaFile]
+
+ Pre and Post Test Hooks #
+pre_test = {
+"ServerFiles" : Files,
+"LocalFiles": Existing_Files
+}
+test_options = {
+"WgetCommands" : WGET_OPTIONS,
+"Urls" : WGET_URLS
+}
+post_test = {
+"ExpectedFiles" : ExpectedDownloadedFiles,
+"ExpectedRetcode" : ExpectedReturnCode
+}
+
+http_test = HTTPTest (
+pre_hook=pre_test,
+test_params=test_options,
+post_hook=post_test,
+)
+
+http_test.server_setup()
+### Get and use dynamic server sockname
+srv_host, srv_port = http_test.servers[0].server_inst.socket.getsockname ()
+
+MetaXml = re.sub (r'{{FILE1_HASH}}', File1_sha256, MetaXml)
+MetaXml = re.sub (r'{{SRV_HOST}}', srv_host, MetaXml)
+MetaXml = re.sub (r'{{SRV_PORT}}', str (srv_port), MetaXml)
+MetaFile.content = MetaXml
+
+err = http_test.begin ()
+
+exit (err)
diff --git a/testenv/Test-metalink-xml-relpath.py b/testenv/Test-metalink-xml-relpath.py
new file mode 100755
index 000..041d772
--- /dev/null
+++ b/testenv/Test-metalink-xml-relpath.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from misc.wget_file import WgetFile
+import re
+import hashlib
+
+"""
+This is to test if Metalink XML file escapes current directory.
+"""
+# File Definitions ###
+File1 = "Would you like some Tea?"
+File1_lowPref = "Do not take this"
+File1_sha256 = hashlib.sha256 (File1.encode ('UTF-8')).hexdigest ()
+MetaXml = \
+"""
+http://www.metalinker.org/";>
+
+GNU Wget
+
+
+GNU GPL
+http://www.gnu.org/licenses/gpl.html
+
+ Wget Test File 1
+ 1.2.3
+ Wget Test File 1 description
+
+
+
+{{FILE1_HASH}}
+
