Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
Shanti Suresh wrote: Greetings, On Wed, Jun 26, 2013 at 4:08 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, But, even when sending UTF-8 encoded data according to this principle, they are *not* indicating that it is UTF-8 data, which is basically wrong, because the standard HTTP/HTML character set is iso-8859-1, and they *should* indicate it when that is not what they are sending. But that is the reality. No, as much as it pains me to do so, I agree with with Mozilla folks on this one: adding a charset attribute to an application/x-form-urlencoded Content-Type violates the spec. There is no good solution. ... We really need an RFC for HTTP 2.0, with UTF-8 as the default charset/encoding. +1 Maybe they can clear-up Tomcat logging configuration while they are at it :) Thank you! This discussion was quite informational. You are welcome. Further as relatively [OT], in some other - non-Tomcat, non-Java - applications, we solve the general issue as follows (taking into account the deficiencies of the RFCs, the servers, the browsers, and the users) : - when programmers create the html documents containing the forms, they must make sure that they use a tool which really saves the html document in the charset/encoding that corresponds to their wishes - the html page should contain a declaration like : (where is the correct charset/encoding, like "UTF-8") - each form that is sent to the browser is sent by the server with an explicit HTTP header : Content-type: text/html; charset= (that normally happens automatically, but you should nevertheless check that it matches) - the tag of the form should contain the "accept-charset" attribute with the expected character set as value, like - the form itself contains a hidden parameter like : (where y is a character sequence which is so that, seen as bytes, its length would be different depending on the real character set used. E.g., for Europe, "ÖöÜüÄä") - the application which receives the form submit data, must first check if the string received for the "charset-test" parameter matches its expectations. In other words, if the application expects UTF-8, then it should check that the received string has a byte length of 12 and a character length of 6, and matches a Unicode string "ÖöÜüÄä") And if it doesn't, then it should take appropriate action (abort the action, or try another charset) (if the form sent by the server contains additional data coming from a back-end database system, then one should also check that the charset of that data matches the one of the form of course). This may look a bit like overkill, but it is the result of long and painful real-world experience with multi-lingual applications used with multiple browsers and multiple types of users in multiple countries doing cut-and-paste of all kinds of stuff into forms. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
Greetings, On Wed, Jun 26, 2013 at 4:08 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > André, > > > > > But, even when sending UTF-8 encoded data according to this > > principle, they are *not* indicating that it is UTF-8 data, which > > is basically wrong, because the standard HTTP/HTML character set is > > iso-8859-1, and they *should* indicate it when that is not what > > they are sending. But that is the reality. > > No, as much as it pains me to do so, I agree with with Mozilla folks > on this one: adding a charset attribute to an > application/x-form-urlencoded Content-Type violates the spec. There is > no good solution. > ... > > > We really need an RFC for HTTP 2.0, with UTF-8 as the default > > charset/encoding. > > +1 > > Maybe they can clear-up Tomcat logging configuration while they are at > it :) > > Thank you! This discussion was quite informational. -Shanti
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 6/26/13 11:40 AM, André Warnier wrote: > Shanti Suresh wrote: >> Hi Chris, >> >> This is such an interesting discussion. I am not sure what to >> make of this person's comment: >> >> --- TAXI 2012-10-09 09:03:59 PDT >> >> Wow, no fix since 8 years... >> >> And this is a real bug: If the HTTP header says the file is >> encoded in ISO-8859-1 the common way to override this with HTML >> is: >> >> >> >> Firefox reads the body in UTF-8 then, which is fine, but the >> charset used in forms is still ISO-8859-1, so you have to add >> accept-charset="utf-8" to the form just for firefox (other >> browser automatically use UTF-8 or send the charset with the >> content-type). >> >> So: Why the hell is nobody fixing this bug? --- >> >> >> So the questions I have are: (1) Firefox is not properly sending >> UTF-8 in the POST request even if it reads the HTML page in >> UTF-8? And other browsers are now sending "charset=utf-8" based >> on the the HTML META tag? (2) Firefox has started respecting the >> accept-charset="utf-8" attribute in forms now such that it adds >> charset to the Content-Type header of the POST request? I'm >> confused. I thought Mozilla was not going to fix this issue. >> >> Thanks for any clarifications. >> > > I think that you are still confused.. :-) (As are, in part, some of > the people who posted on that Mozilla bug). > > (1) browsers, in general, are *not* sending a "charset" attribute > in their POST submissions (whether form-url-encoded or multipart). > This is a real pity, because it is the source of much confusion, > and the real reason why servers have to go through loops to figure > out (or force) the character set/encoding of the data that they are > getting from browser POSTs. It is a shame, but you're running into a hole in the spec: only text/* should have a charset, and we're not talking about text/* unfortunately. So, the spec technically forbids fixing the problem. > And the Mozilla people seem to say that it is that way, because > when they tried to add this "charset" attribute, it broke a number > of server applications at the time (8 years ago), and they see no > reason to think that it would not still be the same today, so they > arer not trying it again. It's more like 15 years ago at this point. But, how many times have we said "Tomcat is spec-compliant and your client is not, so you can go #@$*(& yourself"? The same is true here for the browser: they are being spec-compliant, even if it is rather stupid. It would be nice to introduce a new HTTP header like "Form-Content-Type" or whatever and it's always valid (because old, stupid software will ignore it and new software will respect it). > (1a) what browsers *will* do, in general, is to send POST data in > the same character set/encoding as the one of the HTML *page* which > contains the form being posted. It's worth pointing out that this is merely convention, and only because it obviously makes sense to operate this way. > But, even when sending UTF-8 encoded data according to this > principle, they are *not* indicating that it is UTF-8 data, which > is basically wrong, because the standard HTTP/HTML character set is > iso-8859-1, and they *should* indicate it when that is not what > they are sending. But that is the reality. No, as much as it pains me to do so, I agree with with Mozilla folks on this one: adding a charset attribute to an application/x-form-urlencoded Content-Type violates the spec. There is no good solution. > (2) the "accept-charset" attribute of a does not mean that > this will *send* data according to that charset/encoding. It /should/. Why do you think it wouldn't? > It indicates that any data that is entered in the form's input > boxes will be interpreted as being in that charset. I think that's roughly the same thing. The browser would be stupid not to send the data using that character encoding. > So the fact of adding an "accept-charset" attribute to your > tags does not make it so that the browser will magically change its > behaviour when POSTing data. Whether it's magic or not, accept-charset should cause the form to be sent in one of the charsets listed. If you only list one, it should be sent using that character encoding or the browser should give the user an error saying it doesn't know how to encode that way. All browsers know how to do UTF-8. > In other words, it's a mess, and the mess is mainly due to some > lack of precision in the original RFC's, but it is being > perpetuated now by the fear of browser developers of breaking > server applications by doing things right. Which is rather funny in > a way, considering all the things that browser developers do all > the time anyway which do break existing applications. > > We really need an RFC for HTTP 2.0, with UTF-8 as the default > charset/encoding. +1 Maybe they can clear-up Tomcat logging configuration
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 6/26/13 11:00 AM, Shanti Suresh wrote: > Hi Chris, > > This is such an interesting discussion. I am not sure what to make > of this person's comment: > > --- TAXI 2012-10-09 09:03:59 PDT > > Wow, no fix since 8 years... > > And this is a real bug: If the HTTP header says the file is encoded > in ISO-8859-1 the common way to override this with HTML is: > > > > Firefox reads the body in UTF-8 then, which is fine, but the > charset used in forms is still ISO-8859-1, so you have to add > accept-charset="utf-8" to the form just for firefox (other browser > automatically use UTF-8 or send the charset with the > content-type). > > So: Why the hell is nobody fixing this bug? --- > > > So the questions I have are: (1) Firefox is not properly sending > UTF-8 in the POST request even if it reads the HTML page in UTF-8? > And other browsers are now sending "charset=utf-8" based on the the > HTML META tag? This is just a convention: it's not defined by the spec. One could argue that META should override HEADER, but the spec says the opposite, actually. > (2) Firefox has started respecting the accept-charset="utf-8" > attribute in forms now such that it adds charset to the > Content-Type header of the POST request? I'm confused. I thought > Mozilla was not going to fix this issue. No, browsers will use accept-charset as the charset they use to encode the POST request. The will still not likely add the charset attribute to the Content-Type header, because the spec says not to do so. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRy0i2AAoJEBzwKT+lPKRYUfEQAKpBWbhF6sDKY1uA6v7HgqQ3 eRbANUtnknEPiOscJqjMcW8ZWdKJl1mucvKigPNXwJMYpTe3Q/c3r8RjVLV9TqtC jgtWwxlS/GF3W8gO9BuLKva59dTbQtE+xyG12nNCJyxMHPow7WJWUIo4KVB4t0IB vCTXwQT65DxMXZGwRMoc6eFI9idu5M0t1Luj+wnf5PxqV338WoZDx4N7QrSKS3Cj z8S1OeA6uEqPHjSotl4Y7jgRBSSutP4AOwdnFjhBU81JWIZOl5+oaIemFcZgTcwm 0ZlcKThYud+BcH10mPv3nCieUSj1LHo6deEcCYMoNi3sEfSKlfvCW3XsQJI2T7xk 3jj+OplKqhYd8oMhg3sy7M2D25IenG0LJxnCADZutVhQ7U+qO3xKMeovymffNeXn rLOLlxliUiZ0EhPpQSqbtloAOvO/klyECdZlHlILSkFFtMmiQf6yDYtXQLAQfgqK o4/OAD9M7IqMLITmF4sUdt1fXfBjU7g/02tqmJWOP479XWbvqYHESG81XD6dR+PR mFQu+gpIMAZJpIGcVWwMKvG7hMKP9jeNRBp+w5yOBj2BPg9s8Z6vBaonXVMAFN5c VPzows5r1tx6YqKJkfEVpbswzRU4rx8HJHZUpYYBDpUhRztCQhcsFqKRj0d754cV l9hb19ydVgu1Iz0ztsFZ =MUYp -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
2013/6/26 Shanti Suresh : > Hi Chris, > > This is such an interesting discussion. I am not sure what to make of this > person's comment: > > --- > TAXI 2012-10-09 09:03:59 PDT > > Wow, no fix since 8 years... > > And this is a real bug: If the HTTP header says the file is encoded in > ISO-8859-1 the common way to override this with HTML is: > > > > (...) > --- That person is plainly wrong. You must not use different values in "Content-Type" HTTP header and in "Content-Type" META tag. 1. The HTML spec explicitly says that the HTTP header takes priority. http://www.w3.org/TR/html401/charset.html#h-5.2.2 2. Actually different browsers behave differently. Some may fall back to autodetection. (You may guess what browsers do not follow the specification). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
Shanti Suresh wrote: Hi Chris, This is such an interesting discussion. I am not sure what to make of this person's comment: --- TAXI 2012-10-09 09:03:59 PDT Wow, no fix since 8 years... And this is a real bug: If the HTTP header says the file is encoded in ISO-8859-1 the common way to override this with HTML is: Firefox reads the body in UTF-8 then, which is fine, but the charset used in forms is still ISO-8859-1, so you have to add accept-charset="utf-8" to the form just for firefox (other browser automatically use UTF-8 or send the charset with the content-type). So: Why the hell is nobody fixing this bug? --- So the questions I have are: (1) Firefox is not properly sending UTF-8 in the POST request even if it reads the HTML page in UTF-8? And other browsers are now sending "charset=utf-8" based on the the HTML META tag? (2) Firefox has started respecting the accept-charset="utf-8" attribute in forms now such that it adds charset to the Content-Type header of the POST request? I'm confused. I thought Mozilla was not going to fix this issue. Thanks for any clarifications. I think that you are still confused.. :-) (As are, in part, some of the people who posted on that Mozilla bug). (1) browsers, in general, are *not* sending a "charset" attribute in their POST submissions (whether form-url-encoded or multipart). This is a real pity, because it is the source of much confusion, and the real reason why servers have to go through loops to figure out (or force) the character set/encoding of the data that they are getting from browser POSTs. And the Mozilla people seem to say that it is that way, because when they tried to add this "charset" attribute, it broke a number of server applications at the time (8 years ago), and they see no reason to think that it would not still be the same today, so they arer not trying it again. (1a) what browsers *will* do, in general, is to send POST data in the same character set/encoding as the one of the HTML *page* which contains the form being posted. But, even when sending UTF-8 encoded data according to this principle, they are *not* indicating that it is UTF-8 data, which is basically wrong, because the standard HTTP/HTML character set is iso-8859-1, and they *should* indicate it when that is not what they are sending. But that is the reality. (2) the "accept-charset" attribute of a does not mean that this will *send* data according to that charset/encoding. It indicates that any data that is entered in the form's input boxes will be interpreted as being in that charset. So the fact of adding an "accept-charset" attribute to your tags does not make it so that the browser will magically change its behaviour when POSTing data. In other words, it's a mess, and the mess is mainly due to some lack of precision in the original RFC's, but it is being perpetuated now by the fear of browser developers of breaking server applications by doing things right. Which is rather funny in a way, considering all the things that browser developers do all the time anyway which do break existing applications. We really need an RFC for HTTP 2.0, with UTF-8 as the default charset/encoding. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
Hi Chris, This is such an interesting discussion. I am not sure what to make of this person's comment: --- TAXI 2012-10-09 09:03:59 PDT Wow, no fix since 8 years... And this is a real bug: If the HTTP header says the file is encoded in ISO-8859-1 the common way to override this with HTML is: Firefox reads the body in UTF-8 then, which is fine, but the charset used in forms is still ISO-8859-1, so you have to add accept-charset="utf-8" to the form just for firefox (other browser automatically use UTF-8 or send the charset with the content-type). So: Why the hell is nobody fixing this bug? --- So the questions I have are: (1) Firefox is not properly sending UTF-8 in the POST request even if it reads the HTML page in UTF-8? And other browsers are now sending "charset=utf-8" based on the the HTML META tag? (2) Firefox has started respecting the accept-charset="utf-8" attribute in forms now such that it adds charset to the Content-Type header of the POST request? I'm confused. I thought Mozilla was not going to fix this issue. Thanks for any clarifications. -Shanti On Wed, Jun 26, 2013 at 9:58 AM, Christopher Schultz > Here's the long-standing bug in Mozilla: > https://bugzilla.mozilla.org/show_bug.cgi?id=241540 > > ...and one referenced from it with better spec-research: > https://bugzilla.mozilla.org/show_bug.cgi?id=289060#c8 > > The bottom line is two specific items: > 1. Content-Type should only have a "charset" attribute for "text/" types > 2. Adding "charset" to Content-Type for application/x-form-urlencoded > has broken some stuff in the past, so Mozilla has chosen not to > re-enable it > > - -chris >
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 6/26/13 8:01 AM, André Warnier wrote: > Jan Vávra wrote: >> Hello, > When I create user with password with czech String > "ŽežUlička.1" the browser sends correctly this string as: > > POST http://localhost:70/myapp/j_security_check HTTP/1.1 > Content-Type: application/x-www-form-urlencoded > > j_username=p&j_password=%C5%BDe%C5%BEUli%C4%8Dka.1 >>> The browser is not sending that correctly. The password is >>> UTF-8 encoded but the Content-Type fails to specify the >>> character set used. It it did, Tomcat would treat the password >>> as UTF-8. >>> >>> This is a common failing of browsers and is covered in the FAQ. >>> [1] >> Well I have tried IE, Firefox, Chrome. None of them is appending >> charset in Content-Type. I have manually modified the request >> header to: Content-Type: application/x-www-form-urlencoded; >> charset=utf-8 and Tomcat gives me the letters in the correct >> form. Ok, good to know. >> >>> > Any idea how to tell tomcat to use utf-8 in form based > authentication? It's tomcat 7.0.34 on Czech Windows 7 32 > bit with default ansi code page set as Windows-1250. >>> Authentication is tricky because the processing happens before >>> any user code runs. The best / only option is to set the >>> characterEncoding attribute for the Authenticator [2] to UTF-8 >>> and hope that the browsers are consistent in their failing to >>> follow the specification and use whatever encoding the page is >>> encoded with. >>> >>> HTH, >>> >>> Mark >>> >>> >>> [1] http://wiki.apache.org/tomcat/FAQ/CharacterEncoding [2] >>> http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Form_Authenticator_Valve/Attributes >>> >>> >> >>> As you have referred in [2] I have added to my app's context xml >> > className="org.apache.catalina.authenticator.FormAuthenticator" >> characterEncoding="utf-8"/> and Czech letters are in the correct >> form. This is a solution. >> >> Thanks for an advice. > > By the way, referring to this basic failing of browsers : this is > something that is clearly contrary to the specs, yet since years > all major browsers have consistently ignored this issue. This > failure of adding the character set/encoding to HTTP POST's is > causing problems in multi-lingual web applications, and by itself > forcing multiple workarounds which themselves are per force > inconsistent. Does anyone have an idea why browsers keep on > ignoring this issue, version after version ? > > (I would imagine that Apache httpd and Tomcat devs must have > regular contacts with whomever develop browsers, so did anyone ever > ask ?) Here's the long-standing bug in Mozilla: https://bugzilla.mozilla.org/show_bug.cgi?id=241540 ...and one referenced from it with better spec-research: https://bugzilla.mozilla.org/show_bug.cgi?id=289060#c8 The bottom line is two specific items: 1. Content-Type should only have a "charset" attribute for "text/" types 2. Adding "charset" to Content-Type for application/x-form-urlencoded has broken some stuff in the past, so Mozilla has chosen not to re-enable it - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRyvNoAAoJEBzwKT+lPKRYlucQAJVVLDiXE03vtrmTp6hjpgGQ G9PBDZiPapLjRRqtfzfZ1Wz6koOB5kfZJngopt2xP+Ha9H7rGHCfcCYW3tIk5WNi 9c+6G/9Mzx54NCpIYLklx4AZZH8FG/egMmgTsdnAQ9P+Z9DBzPX9luD0DTaXf9dE aDOPG0aSNMPl9Q4VXhFLV22PoEAdDVGDPnglOLRBoLgGzmVYVPNBqcDu8zfDLZKf ZHBBIhcOUGf5YdJlwhqOSTwqo6Uez0BfkZT126FrYh3y4MNMXVTO9p2cpOHNfGZn f7Ld1G0V3CB5bcsXIPNRQ53dugt/vOyN9diUaZ1WZwjqUUKjQoNw18qdzsolXpmn gWsigotQE0Cko7UQzTKsGJRNMPMGlPmKNG3QKG62LNfYQc3ugkeeNJIEDFKcIC5c 1KUaGFdim2/13XXLonbrxF8Fx0k1VTynhqZHmDWsgQbibx/6gPs6HTthLYO/bowW x26oHAC9o2BQLeJvJediaGwylHfogqOBbjFoHvIgyGYkwNj1Tav9ompHkh3bsHin OkT3KOuTx36oiZRF5DPhCuRviB8UgbQCvWubkGfIzSnj6QI+vdQbHLMqbL1Y5D7w TyJIzNZX4alGvfJDNk4Zh58h/MlUuRTkzGtRmNtOjHW4GRiILxuYS/cKsxPG/zvR euKbE0Lk4BcrOOSu+9h6 =UKAo -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [slightly OT] FORM based authentication and utf-8 encoding of credentials
Jan Vávra wrote: Hello, When I create user with password with czech String "ŽežUlička.1" the browser sends correctly this string as: POST http://localhost:70/myapp/j_security_check HTTP/1.1 Content-Type: application/x-www-form-urlencoded j_username=p&j_password=%C5%BDe%C5%BEUli%C4%8Dka.1 The browser is not sending that correctly. The password is UTF-8 encoded but the Content-Type fails to specify the character set used. It it did, Tomcat would treat the password as UTF-8. This is a common failing of browsers and is covered in the FAQ. [1] Well I have tried IE, Firefox, Chrome. None of them is appending charset in Content-Type. I have manually modified the request header to: Content-Type: application/x-www-form-urlencoded; charset=utf-8 and Tomcat gives me the letters in the correct form. Ok, good to know. Any idea how to tell tomcat to use utf-8 in form based authentication? It's tomcat 7.0.34 on Czech Windows 7 32 bit with default ansi code page set as Windows-1250. Authentication is tricky because the processing happens before any user code runs. The best / only option is to set the characterEncoding attribute for the Authenticator [2] to UTF-8 and hope that the browsers are consistent in their failing to follow the specification and use whatever encoding the page is encoded with. HTH, Mark [1] http://wiki.apache.org/tomcat/FAQ/CharacterEncoding [2] http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Form_Authenticator_Valve/Attributes As you have referred in [2] I have added to my app's context xml characterEncoding="utf-8"/> and Czech letters are in the correct form. This is a solution. Thanks for an advice. By the way, referring to this basic failing of browsers : this is something that is clearly contrary to the specs, yet since years all major browsers have consistently ignored this issue. This failure of adding the character set/encoding to HTTP POST's is causing problems in multi-lingual web applications, and by itself forcing multiple workarounds which themselves are per force inconsistent. Does anyone have an idea why browsers keep on ignoring this issue, version after version ? (I would imagine that Apache httpd and Tomcat devs must have regular contacts with whomever develop browsers, so did anyone ever ask ?) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
