Dear all,
i have committed a change to achieve a more consistent and compliant
behavior.
Since all form and query processing of NaviServer happens via API
(ns_urldecode, ns_getform), the current architecture does not allow
direct error messages. The NaviServer philosophy is that the (Tcl)
developer should have the option to handle such cases application
specific.
We had recently changes to address this (mostly driven by vulnerability
scanners) by letting e.g. "ns_urldecode" to raise an exception when
this happens. This change completes this by also raising an exception
for "ns_getform" in such conditions. Note that raising an exception
might be a potential incompatibility for invalid data (which was
"swallowed" before).
The regression test was extended to handle such cases.
There is one more thing (in ns_connchan, so far, not able
to reproduce) that i would like to have a look on
before making the next release candidate available.
all the best
-gn
On 02.05.22 14:29, Gustaf Neumann wrote:
Dear David and all,
I looked into this issue, and I do not like the current situation either.
In the current snapshot, a GET request with invalid coded
query variables is rejected, while the POST request leads just
to the warning, and the invalid entry is omitted.
W3C [1] says in the reference for Multilingual form encoding:
> If non-UTF-8 data is received, an error message should be sent back.
This means, that the only defensible logic is to reject in both cases
the request as invalid. One can certainly send single-byte funny character
data in URLs, which is invalid UTF8 (e.g. "%9C" or "%E6" etc.),
but for these requests, the charset has to be specified, either
via content type, or via the default URL encoding in the NaviServer
configuration... see example (2) below.
As mentioned earlier, there are increasingly many attacks with invalid
UTF-8 data (also by vulnerability scanners), so we to be strict here.
I will try to address the outstanding issues ASAP and provide then
another RC.
All the best
-gn
[1] https://www.w3.org/International/questions/qa-forms-utf-8
# POST request with already encoded form data (x-www-form-urlencoded)
$ curl -X POST -d "p1=a%C5%93Cb&p2=a%E6b" localhost:8100/upload.tcl
# POST request with already encoded form data, but proper encoding
$ curl -X POST -H "Content-Type: application/x-www-form-urlencoded; charset=iso-8859-1"
-d "p2=a%E6b" localhost:8100/upload.tcl
# POST + x-www-form-urlencoded, but let curl do the encoding
$ curl -X POST -d "p1=aœb" -d $(echo -e 'p2=a\xE6b') localhost:8100/upload.tcl
# POST + multipart/form-data, let curl do the encoding
$ curl -X POST -F "p1=aœb" -F $(echo -e 'p2=a\xE6b') localhost:8100/upload.tcl
POST request with already encoded form data (x-www-form-urlencoded)
$ curl -X GET "localhost:8100/upload.tcl?p1=a%C5%93Cb&p2=a%E6b"
On 28.04.22 17:45, David Osborne wrote:
Hi Gustaf,
We've been testing *4.99.24 rc1* and it seems pretty solid so far.
Thanks for all the work that went into it.
One change of behaviour that is causing us issues is the handling of
invalid UTF8 characters.
We have a system which regularly POSTs data to NaviServer - sometimes
(for reasons we're looking into) the POST'ed data received by
NaviServer can contain urlencoded characters which don't exist in
UTF8 ( for example *%9C* instead of *%C5%93*).
In previous versions of NaviServer, this causes an invalid character
to be embedded in the data when we save it.
Now, in version 4.99.24 we, rightly, get the warning "*Warning:
decoded string is invalid UTF-8:*".
But the additional behaviour is that the entire form variable seems
to be dropped.
I just wanted to query if that is the intended behaviour?
I've seen some servers convert such invalid characters to *\ufffd*
(\ufffd being "replacement character" - "used to replace an incoming
character whose value is unknown or unrepresentable in Unicode") -
but not sure which is the correct behaviour.
Regards,
Dave
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
