Bug#959231: Bug#929228: Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Francesco Poli]

On Fri, 1 May 2020 16:46:21 +0200 Salvatore Bonaccorso wrote:

[...]
> Thanks, applied and deployed.

Wow, this looks fixed: thanks to you all, for your prompt
reaction!   :-)

Bye.

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpKcBrzjUy1p.pgp
Description: PGP signature

Bug#929228: Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Salvatore Bonaccorso]

Hi Florian,

On Fri, May 01, 2020 at 04:01:39PM +0200, Florian Weimer wrote:
> * Salvatore Bonaccorso:
> 
> > Hi Florian,
> >
> > On Fri, May 01, 2020 at 02:33:21PM +0200, Florian Weimer wrote:
> >> * Salvatore Bonaccorso:
> >> 
> >> > Hi Florian,
> >> >
> >> > On Fri, May 01, 2020 at 02:11:50PM +0200, Florian Weimer wrote:
> >> >> * Florian Weimer:
> >> >> 
> >> >> > * Francesco Poli:
> >> >> >
> >> >> >> Please note that the CVE is mentioned in [DSA-4667-1].
> >> >> >>
> >> >> >> [DSA-4667-1]: 
> >> >> >> 
> >> >> >>
> >> >> >> What's wrong with that tracker page?
> >> >> >
> >> >> > It's something in the NVD data that breaks the HTML escaping.
> >> >> 
> >> >> This patch adds basic Unicode support to the web framework.  I'm not
> >> >> sure if it is the right direction to move in, but it fixes the issue.
> >> >> 
> >> >> An alternative fix would be to change the NVD importer not to put
> >> >> Unicode strings into the database, by encoding them as byte strings
> >> >> first.
> >> >
> >> > Do you want to deploy that or rather investigate an alternative?
> >> 
> >> I'd appreciate if you could spot-check the changes (e.g., do we still
> >> do HTML escaping properly?) and deploy it.  It looks like I have
> >> forgotten how to do it.
> >
> > Looks good to me, and yes can deploy it if you want me to. Please have
> > a look at at attache git format-patch'ed version if you agree with the
> > slight rewrite, since I do not want to commit something in your name
> > you would not agree with).
> 
> Still looks fine.
> 
> Signed-off-by: Florian Weimer 

Thanks, applied and deployed.

Regards,
Salvatore

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Florian Weimer]

* Salvatore Bonaccorso:

> Hi Florian,
>
> On Fri, May 01, 2020 at 02:33:21PM +0200, Florian Weimer wrote:
>> * Salvatore Bonaccorso:
>> 
>> > Hi Florian,
>> >
>> > On Fri, May 01, 2020 at 02:11:50PM +0200, Florian Weimer wrote:
>> >> * Florian Weimer:
>> >> 
>> >> > * Francesco Poli:
>> >> >
>> >> >> Please note that the CVE is mentioned in [DSA-4667-1].
>> >> >>
>> >> >> [DSA-4667-1]: 
>> >> >> 
>> >> >>
>> >> >> What's wrong with that tracker page?
>> >> >
>> >> > It's something in the NVD data that breaks the HTML escaping.
>> >> 
>> >> This patch adds basic Unicode support to the web framework.  I'm not
>> >> sure if it is the right direction to move in, but it fixes the issue.
>> >> 
>> >> An alternative fix would be to change the NVD importer not to put
>> >> Unicode strings into the database, by encoding them as byte strings
>> >> first.
>> >
>> > Do you want to deploy that or rather investigate an alternative?
>> 
>> I'd appreciate if you could spot-check the changes (e.g., do we still
>> do HTML escaping properly?) and deploy it.  It looks like I have
>> forgotten how to do it.
>
> Looks good to me, and yes can deploy it if you want me to. Please have
> a look at at attache git format-patch'ed version if you agree with the
> slight rewrite, since I do not want to commit something in your name
> you would not agree with).

Still looks fine.

Signed-off-by: Florian Weimer 

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Salvatore Bonaccorso]

Hi Florian,

On Fri, May 01, 2020 at 02:33:21PM +0200, Florian Weimer wrote:
> * Salvatore Bonaccorso:
> 
> > Hi Florian,
> >
> > On Fri, May 01, 2020 at 02:11:50PM +0200, Florian Weimer wrote:
> >> * Florian Weimer:
> >> 
> >> > * Francesco Poli:
> >> >
> >> >> Please note that the CVE is mentioned in [DSA-4667-1].
> >> >>
> >> >> [DSA-4667-1]: 
> >> >> <https://lists.debian.org/debian-security-announce/2020/msg00071.html>
> >> >>
> >> >> What's wrong with that tracker page?
> >> >
> >> > It's something in the NVD data that breaks the HTML escaping.
> >> 
> >> This patch adds basic Unicode support to the web framework.  I'm not
> >> sure if it is the right direction to move in, but it fixes the issue.
> >> 
> >> An alternative fix would be to change the NVD importer not to put
> >> Unicode strings into the database, by encoding them as byte strings
> >> first.
> >
> > Do you want to deploy that or rather investigate an alternative?
> 
> I'd appreciate if you could spot-check the changes (e.g., do we still
> do HTML escaping properly?) and deploy it.  It looks like I have
> forgotten how to do it.

Looks good to me, and yes can deploy it if you want me to. Please have
a look at at attache git format-patch'ed version if you agree with the
slight rewrite, since I do not want to commit something in your name
you would not agree with).

Salvatore
>From 88268c60d683edcda8aa2ae9b339bd46c59e58ec Mon Sep 17 00:00:00 2001
From: Florian Weimer 
Date: Fri, 1 May 2020 14:11:50 +0200
Subject: [PATCH] Add basic Unicode support to the web framework

As mentioned in Debian bug #959231 ("security-tracker: Proxy Error on
CVE-2020-11565 tracker page"):

* Florian Weimer:

> * Francesco Poli:
>
>> Please note that the CVE is mentioned in [DSA-4667-1].
>>
>> [DSA-4667-1]: <https://lists.debian.org/debian-security-announce/2020/msg00071.html>
>>
>> What's wrong with that tracker page?
>
> It's something in the NVD data that breaks the HTML escaping.

This patch adds basic Unicode support to the web framework.  I'm not
sure if it is the right direction to move in, but it fixes the issue.

An alternative fix would be to change the NVD importer not to put
Unicode strings into the database, by encoding them as byte strings
first.

[carnil: Slightly rewrite the commit message]
BugLink: https://bugs.debian.org/929228
BugLink: https://bugs.debian.org/959231
Signed-off-by: Salvatore Bonaccorso 
---
 lib/python/web_support.py | 42 ---
 1 file changed, 22 insertions(+), 20 deletions(-)

diff --git a/lib/python/web_support.py b/lib/python/web_support.py
index 5752f34b5f77..116cbec2bea3 100644
--- a/lib/python/web_support.py
+++ b/lib/python/web_support.py
@@ -220,27 +220,25 @@ class URLFactory:
 def updateParams(self, **args):
 self.updateParamsDict(args)
 
-charToHTML = map(chr, range(256))
-charToHTMLattr = map(chr, range(256))
-def _initStringToHTML(s):
-for (ch, repl) in (('<', ''),
-   ('>', ''),
-   ('&', '')):
-s[ord(ch)] = repl
-_initStringToHTML(charToHTML)
-_initStringToHTML(charToHTMLattr)
-charToHTMLattr[ord('"')] = '&34;'
-del _initStringToHTML
+charToHTML = {
+'<' : '',
+'>' : '',
+'&' : '',
+}
+charToHTMLattr = {
+'&' : '',
+'"' : '&34;',
+}
 
 def escapeHTML(str):
-'''Replaces the characters <>&" in the passed strings with their
+'''Replaces the characters <>& in the passed strings with their
 HTML entities.'''
+return ''.join([charToHTML.get(ch, ch) for ch in str])
 
-result = []
-append = result.append
-for ch in str:
-append(charToHTML[ord(ch)])
-return ''.join(result)
+def escapeHTMLattr(str):
+'''Replaces the characters &" in the passed strings with their
+HTML entities.'''
+return ''.join([charToHTMLattr.get(ch, ch) for ch in str])
 
 class HTMLBase:
 def flatten(self, write):
@@ -310,8 +308,7 @@ class Tag(HTMLBase):
 else:
 append(key)
 append('="')
-for ch in str(value):
-append(charToHTMLattr[ord(ch)])
+append(escapeHTMLattr(str(value)))
 append('"')
 self.__attribs = ''.join(attrs)
 self.contents = contents
@@ -659,7 +656,12 @@ class HTMLResult(Result):
 buf = cStringIO.StringIO()
 buf.write(self.doctype)
 buf.write('\n')
-self.contents.flatten(buf.write)
+def write_both(s):
+if type(s) == types.UnicodeType:
+buf.write(s.encode('UTF-8'))
+else:
+buf.write(s)
+self.contents.flatten(write_both)
 buf = buf.getvalue()
 self.headers['Content-Length'] = str(len(buf))
 def later(req):
-- 
2.26.2

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Florian Weimer]

* Salvatore Bonaccorso:

> Hi Florian,
>
> On Fri, May 01, 2020 at 02:11:50PM +0200, Florian Weimer wrote:
>> * Florian Weimer:
>> 
>> > * Francesco Poli:
>> >
>> >> Please note that the CVE is mentioned in [DSA-4667-1].
>> >>
>> >> [DSA-4667-1]: 
>> >> 
>> >>
>> >> What's wrong with that tracker page?
>> >
>> > It's something in the NVD data that breaks the HTML escaping.
>> 
>> This patch adds basic Unicode support to the web framework.  I'm not
>> sure if it is the right direction to move in, but it fixes the issue.
>> 
>> An alternative fix would be to change the NVD importer not to put
>> Unicode strings into the database, by encoding them as byte strings
>> first.
>
> Do you want to deploy that or rather investigate an alternative?

I'd appreciate if you could spot-check the changes (e.g., do we still
do HTML escaping properly?) and deploy it.  It looks like I have
forgotten how to do it.

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Salvatore Bonaccorso]

Hi Florian,

On Fri, May 01, 2020 at 02:11:50PM +0200, Florian Weimer wrote:
> * Florian Weimer:
> 
> > * Francesco Poli:
> >
> >> Please note that the CVE is mentioned in [DSA-4667-1].
> >>
> >> [DSA-4667-1]: 
> >> 
> >>
> >> What's wrong with that tracker page?
> >
> > It's something in the NVD data that breaks the HTML escaping.
> 
> This patch adds basic Unicode support to the web framework.  I'm not
> sure if it is the right direction to move in, but it fixes the issue.
> 
> An alternative fix would be to change the NVD importer not to put
> Unicode strings into the database, by encoding them as byte strings
> first.

Do you want to deploy that or rather investigate an alternative?

Salvatore

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Salvatore Bonaccorso]

Control: forcemerge 929228 959231

Hi,

On Fri, May 01, 2020 at 01:36:40PM +0200, Florian Weimer wrote:
> * Francesco Poli:
> 
> > Please note that the CVE is mentioned in [DSA-4667-1].
> >
> > [DSA-4667-1]: 
> > 
> >
> > What's wrong with that tracker page?
> 
> It's something in the NVD data that breaks the HTML escaping.

Yes it's likely due to the used "" and "" in the
description, whe have seen that on other occurences, never came to it
to fix the bug.

This is same/similar to #929228.

Regards,
Salvatore

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Florian Weimer]

* Florian Weimer:

> * Francesco Poli:
>
>> Please note that the CVE is mentioned in [DSA-4667-1].
>>
>> [DSA-4667-1]: 
>> 
>>
>> What's wrong with that tracker page?
>
> It's something in the NVD data that breaks the HTML escaping.

This patch adds basic Unicode support to the web framework.  I'm not
sure if it is the right direction to move in, but it fixes the issue.

An alternative fix would be to change the NVD importer not to put
Unicode strings into the database, by encoding them as byte strings
first.

diff --git a/lib/python/web_support.py b/lib/python/web_support.py
index 5752f34b5f..116cbec2be 100644
--- a/lib/python/web_support.py
+++ b/lib/python/web_support.py
@@ -220,27 +220,25 @@ class URLFactory:
 def updateParams(self, **args):
 self.updateParamsDict(args)
 
-charToHTML = map(chr, range(256))
-charToHTMLattr = map(chr, range(256))
-def _initStringToHTML(s):
-for (ch, repl) in (('<', ''),
-   ('>', ''),
-   ('&', '')):
-s[ord(ch)] = repl
-_initStringToHTML(charToHTML)
-_initStringToHTML(charToHTMLattr)
-charToHTMLattr[ord('"')] = '&34;'
-del _initStringToHTML
+charToHTML = {
+'<' : '',
+'>' : '',
+'&' : '',
+}
+charToHTMLattr = {
+'&' : '',
+'"' : '&34;',
+}
 
 def escapeHTML(str):
-'''Replaces the characters <>&" in the passed strings with their
+'''Replaces the characters <>& in the passed strings with their
 HTML entities.'''
+return ''.join([charToHTML.get(ch, ch) for ch in str])
 
-result = []
-append = result.append
-for ch in str:
-append(charToHTML[ord(ch)])
-return ''.join(result)
+def escapeHTMLattr(str):
+'''Replaces the characters &" in the passed strings with their
+HTML entities.'''
+return ''.join([charToHTMLattr.get(ch, ch) for ch in str])
 
 class HTMLBase:
 def flatten(self, write):
@@ -310,8 +308,7 @@ class Tag(HTMLBase):
 else:
 append(key)
 append('="')
-for ch in str(value):
-append(charToHTMLattr[ord(ch)])
+append(escapeHTMLattr(str(value)))
 append('"')
 self.__attribs = ''.join(attrs)
 self.contents = contents
@@ -659,7 +656,12 @@ class HTMLResult(Result):
 buf = cStringIO.StringIO()
 buf.write(self.doctype)
 buf.write('\n')
-self.contents.flatten(buf.write)
+def write_both(s):
+if type(s) == types.UnicodeType:
+buf.write(s.encode('UTF-8'))
+else:
+buf.write(s)
+self.contents.flatten(write_both)
 buf = buf.getvalue()
 self.headers['Content-Length'] = str(len(buf))
 def later(req):

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Florian Weimer]

* Francesco Poli:

> Please note that the CVE is mentioned in [DSA-4667-1].
>
> [DSA-4667-1]: 
> 
>
> What's wrong with that tracker page?

It's something in the NVD data that breaks the HTML escaping.

Bug#959231: security-tracker: Proxy Error on CVE-2020-11565 tracker page

[Francesco Poli (wintermute)]

Package: security-tracker
Severity: normal

Hi all!

I noticed that the tracker page for [CVE-2020-11565] fails to display
and returns the following error:

| Proxy Error
| 
| The proxy server received an invalid response from an upstream server.
| The proxy server could not handle the request
| 
| Reason: Error reading from remote server
| 
| Apache Server at security-tracker.debian.org Port 443

[CVE-2020-11565]: 

Please note that the CVE is mentioned in [DSA-4667-1].

[DSA-4667-1]: 


What's wrong with that tracker page?
Please fix anything that's missing.

Thanks for your time and dedication!