Re: Bug#853189: tracker.debian.org: Ecnoding issue / Code injection through Maintainer field (and probably others)

2017-02-01 Thread Niels Thykier 
Christophe Siraut:
> Niels Thykier wrote:
>>  * tracker.d.o does *not* import excuses.yaml but update_excuses.html
>>(as far as I am informed at least)
> 
> True.
> 
> Here is a patch for tracker to parse YAML instead of HTML.
> 
> Cheers,
> Christophe
> 

Hi Christophe,

Thanks for looking into this issue. :)

As the maintainer of Britney, I am a bit concerned that this patch
appears to be relying on the "excuses"-field inside.  That is a
"non-machine"-parsable format (basically all raw HTML notes) that I
would like to eventually phase out of the excuses.yaml.

If there is data in that field that tracker needs, then it should
preferably be extracted to another field.  (FTR, the format is still a
bit WIP)

Thanks,
~Niels

Re: Bug#853189: tracker.debian.org: Ecnoding issue / Code injection through Maintainer field (and probably others)

2017-02-01 Thread Christophe Siraut 
Niels Thykier wrote:
>  * tracker.d.o does *not* import excuses.yaml but update_excuses.html
>(as far as I am informed at least)

True.

Here is a patch for tracker to parse YAML instead of HTML.

Cheers,
Christophe
>From 04692b5c65124b930a94f668cd2b409269d186c5 Mon Sep 17 00:00:00 2001
From: Christophe Siraut 
Date: Wed, 1 Feb 2017 17:05:05 +0100
Subject: [PATCH] Use excuses.yaml instead of parsing HTML. Closes: #853189

---
 .../vendor/debian/tests-data/update_excuses-1.html |  11 ---
 .../vendor/debian/tests-data/update_excuses-1.yaml |  11 +++
 .../vendor/debian/tests-data/update_excuses-2.html |  11 ---
 .../vendor/debian/tests-data/update_excuses-2.yaml |  12 +++
 distro_tracker/vendor/debian/tests.py  |  11 +--
 distro_tracker/vendor/debian/tracker_tasks.py  | 106 ++---
 6 files changed, 56 insertions(+), 106 deletions(-)
 delete mode 100644 distro_tracker/vendor/debian/tests-data/update_excuses-1.html
 create mode 100644 distro_tracker/vendor/debian/tests-data/update_excuses-1.yaml
 delete mode 100644 distro_tracker/vendor/debian/tests-data/update_excuses-2.html
 create mode 100644 distro_tracker/vendor/debian/tests-data/update_excuses-2.yaml

diff --git a/distro_tracker/vendor/debian/tests-data/update_excuses-1.html b/distro_tracker/vendor/debian/tests-data/update_excuses-1.html
deleted file mode 100644
index c23541e..000
--- a/distro_tracker/vendor/debian/tests-data/update_excuses-1.html
+++ /dev/null
@@ -1,11 +0,0 @@
-http://www.w3.org/TR/REC-html40/strict.dtd;>
-excuses...
-Generated: 2013.08.12 10:03:22 +
-
-dummy-package (1.0.0 to 2.0.0)
-
-Maintainer: Some Maintainer
-20 days old (needed 10 days)
-Not considered
-
-
diff --git a/distro_tracker/vendor/debian/tests-data/update_excuses-1.yaml b/distro_tracker/vendor/debian/tests-data/update_excuses-1.yaml
new file mode 100644
index 000..bb0d86e
--- /dev/null
+++ b/distro_tracker/vendor/debian/tests-data/update_excuses-1.yaml
@@ -0,0 +1,11 @@
+generated-date: 2017-02-01 06:47:18.195464
+sources:
+- excuses:
+  - 20 days old (needed 10 days)
+  hints:
+  is-candidate:
+  item-name: dummy-package
+  new-version: 2.0.0
+  old-version: 1.0.0
+  reason: []
+  source: dummy-package
diff --git a/distro_tracker/vendor/debian/tests-data/update_excuses-2.html b/distro_tracker/vendor/debian/tests-data/update_excuses-2.html
deleted file mode 100644
index 4666c7b..000
--- a/distro_tracker/vendor/debian/tests-data/update_excuses-2.html
+++ /dev/null
@@ -1,11 +0,0 @@
-http://www.w3.org/TR/REC-html40/strict.dtd;>
-excuses...
-Generated: 2013.08.12 10:03:22 +
-
-dummy-package (1.0.0 to 2.0.0)
-
-Maintainer: Some Maintainer
-10 days old (needed 10 days)
-Not considered
-
-
diff --git a/distro_tracker/vendor/debian/tests-data/update_excuses-2.yaml b/distro_tracker/vendor/debian/tests-data/update_excuses-2.yaml
new file mode 100644
index 000..f3e74be
--- /dev/null
+++ b/distro_tracker/vendor/debian/tests-data/update_excuses-2.yaml
@@ -0,0 +1,12 @@
+generated-date: 2017-02-01 06:47:18.195464
+sources:
+- excuses:
+  - 10 days old (needed 10 days)
+  hints:
+  is-candidate:
+  item-name: dummy-package
+  new-version: 2.0.0
+  old-version: 1.0.0
+  reason: []
+  source: dummy-package
+
diff --git a/distro_tracker/vendor/debian/tests.py b/distro_tracker/vendor/debian/tests.py
index b67271e..5a56566 100644
--- a/distro_tracker/vendor/debian/tests.py
+++ b/distro_tracker/vendor/debian/tests.py
@@ -1760,15 +1760,14 @@ class UpdateExcusesTaskActionItemTest(TestCase):
 
 def set_update_excuses_content(self, content):
 """
-Sets the stub content of the update_excuses.html that the task will
+Sets the stub content of the update_excuses.yaml that the task will
 have access to.
 """
-self.task._get_update_excuses_content.return_value = iter(
-content.splitlines())
+self.task._get_update_excuses_content.return_value = content
 
 def set_update_excuses_content_from_file(self, file_name):
 """
-Sets the stub content of the update_excuses.html that the task will
+Sets the stub content of the update_excuses.yaml that the task will
 have access to based on the content of the test file with the given
 name.
 """
@@ -1786,7 +1785,7 @@ class UpdateExcusesTaskActionItemTest(TestCase):
 Tests that an action item is created when a package has not moved to
 testing after the allocated period.
 """
-self.set_update_excuses_content_from_file('update_excuses-1.html')
+self.set_update_excuses_content_from_file('update_excuses-1.yaml')
 # Sanity check: no action items currently
 self.assertEqual(0, ActionItem.objects.count())
 expected_data = {
@@ -1834,7 +1833,7 @@ class UpdateExcusesTaskActionItemTest(TestCase):
 package=self.package_name,
 item_type=self.get_action_item_type(),

Re: Bug#853189: tracker.debian.org: Ecnoding issue / Code injection through Maintainer field (and probably others)

2017-01-30 Thread Adrian Bunk 
On Mon, Jan 30, 2017 at 04:48:55PM +0100, Mattia Rizzolo wrote:
> On Mon, Jan 30, 2017 at 03:43:44PM +0100, Dominik George wrote:
> > tracker.debian.org apparently has encoding issues, not of the “schei�
> > encoding” kind, but it even seems to break the HTML completely and even
> > introduces new elements into the DOM in some way…
> > 
> > أحمد المحمودي (Ahmed El-Mahmoudy), e.g., in the Maintainer field of
> > python-whoosh [1] triggers the issue in the “testing migrations” pane
> > (but not in the Maintainer field itself…).
> 
> That's coming from the excuses.yaml coming from
> https://release.debian.org/britney/excuses.yaml (debian-released CCed):
>...

Niels correctly stated in IRC that the tracker is actually using
update_excuses.html

My guess regarding the cause would be that the tracker fails to properly
parse bi-directional text in update_excuses (Arabic is right-to-left).

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Bug#853189: tracker.debian.org: Ecnoding issue / Code injection through Maintainer field (and probably others)

2017-01-30 Thread Niels Thykier 
Mattia Rizzolo:
> On Mon, Jan 30, 2017 at 03:43:44PM +0100, Dominik George wrote:
>> tracker.debian.org apparently has encoding issues, not of the “schei�
>> encoding” kind, but it even seems to break the HTML completely and even
>> introduces new elements into the DOM in some way…
>>
>> أحمد المحمودي (Ahmed El-Mahmoudy), e.g., in the Maintainer field of
>> python-whoosh [1] triggers the issue in the “testing migrations” pane
>> (but not in the Maintainer field itself…).
> 
> That's coming from the excuses.yaml coming from
> https://release.debian.org/britney/excuses.yaml (debian-released CCed):
> 
> [...]

Sorry, but I am afraid that is incorrect.

 * excuses.yaml is valid UTF-8 AFAICT
 * tracker.d.o does *not* import excuses.yaml but update_excuses.html
   (as far as I am informed at least)
 * Even update_excuses.html us valid UTF-8 (but it uses "meta
   http-equiv" tag to declare that rather than a HTTP header).

So I am not (yet?) convinced that the problem is on the d-release side.

Thanks,
~Niels

Re: Bug#853189: tracker.debian.org: Ecnoding issue / Code injection through Maintainer field (and probably others)

2017-01-30 Thread Mattia Rizzolo 
On Mon, Jan 30, 2017 at 03:43:44PM +0100, Dominik George wrote:
> tracker.debian.org apparently has encoding issues, not of the “schei�
> encoding” kind, but it even seems to break the HTML completely and even
> introduces new elements into the DOM in some way…
> 
> أحمد المحمودي (Ahmed El-Mahmoudy), e.g., in the Maintainer field of
> python-whoosh [1] triggers the issue in the “testing migrations” pane
> (but not in the Maintainer field itself…).

That's coming from the excuses.yaml coming from
https://release.debian.org/britney/excuses.yaml (debian-released CCed):

- excuses:
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=amd64=python-whoosh=2.7.0-1.1;
target="_blank">amd64: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=amd64=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=i386=python-whoosh=2.7.0-1.1;
target="_blank">i386: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=i386=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=arm64=python-whoosh=2.7.0-1.1;
target="_blank">arm64: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=arm64=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=armel=python-whoosh=2.7.0-1.1;
target="_blank">armel: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=armel=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=armhf=python-whoosh=2.7.0-1.1;
target="_blank">armhf: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=armhf=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=mips=python-whoosh=2.7.0-1.1;
target="_blank">mips: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=mips=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=mips64el=python-whoosh=2.7.0-1.1;
target="_blank">mips64el: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=mips64el=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=mipsel=python-whoosh=2.7.0-1.1;
target="_blank">mipsel: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=mipsel=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=ppc64el=python-whoosh=2.7.0-1.1;
target="_blank">ppc64el: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=ppc64el=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - 'missing build on https://buildd.debian.org/status/logs.php?arch=s390x=python-whoosh=2.7.0-1.1;
target="_blank">s390x: python-whoosh, python3-whoosh (from https://buildd.debian.org/status/logs.php?arch=s390x=python-whoosh=2.7.0-1;
target="_blank">2.7.0-1)'
  - Piuparts tested OK - https://piuparts.debian.org/sid/source/p/python-whoosh.html;>https://piuparts.debian.org/sid/source/p/python-whoosh.html
  is-candidate: false
  item-name: python-whoosh
  maintainer: أحمد المحمودي (Ahmed El-Mahmoudy)
  missing-builds:
on-architectures:
- amd64
- arm64
- armel
- armhf
- i386
- mips
- mips64el
- mipsel
- ppc64el
- s390x
on-unimportant-architectures: []
  new-version: 2.7.0-1.1
  old-binaries:
2.7.0-1:
- python-whoosh
- python3-whoosh
  old-version: 2.7.0-1
  policy_info:
age:
  age-requirement: 10
  current-age: 0
piuparts:
  piuparts-test-url: 
https://piuparts.debian.org/sid/source/p/python-whoosh.html
  test-results: pass
rc-bugs:
  shared-bugs:
  - '812768'
  unique-source-bugs: []
  unique-target-bugs: []
  reason: []
  source: python-whoosh

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature