Re: [Laszlo-dev] [Updated] For Review: Change 20090608-raju-Y Summary: Fix for CSS parser uses incorrect file encoding

Tue, 09 Jun 2009 15:58:51 -0700 

Sorry, Amy, just saw that mail now. I'll do that tomorrow.

Best,
Raju

On Jun 9, 2009, at 4:41 PM, Amy Muntz wrote:


Hi Raju,
Please check in the trunk fix for 4.4 right away. It may take an additional fix for 4.0.x, but that's just fine. 
We can re-open the bug for 4.0.x and it can be fixed at a later time.
Thanks,
Amy


On Mon, Jun 8, 2009 at 7:08 PM , Raju Bitter wrote:


Tucker, Wanli,
I've done some testing with OL 4.0.15 and Webtop running 1.5.2 running on OL 4.0.15. While the fix works for OL trunk, it doesn't work for 4.0.15. Here's the debug output for test file "test/css/ encoding/utf8_with_BOM_no_charset_attr.lzx". The UTF-8 encoded data is read correctly. 

http://localhost:8080/4.0.15/test/css/encoding/utf8_with_BOM_no_charset_attr.lzx
09 Jun 2009 01:03:06 (127.0.0.1 15) INFO compiler.StyleSheetCompiler - StyleSheetCompiler.compile called! 09 Jun 2009 01:03:06 (127.0.0.1 15) INFO compiler.StyleSheetCompiler - @charset=utf-8 found on stylesheet tag 09 Jun 2009 01:03:06 (127.0.0.1 15) INFO compiler.StyleSheetCompiler - reading in stylesheet from src="utf8_with_BOM_no_charset_attr.css" 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.FileResolver - Resolving pathname: utf8_with_BOM_no_charset_attr.css and base: / Users/rajubitter/src/svn/openlaszlo/4.0.15/test/css/encoding 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.FileResolver - Resolved utf8_with_BOM_no_charset_attr.css to /Users/rajubitter/ src/ svn/openlaszlo/4.0.15/test/css/encoding/ utf8_with_BOM_no_charset_attr.css 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG cm.DependencyTracker - addFile Path is /Users/rajubitter/src/svn/openlaszlo/4.0.15/test/ css/ encoding/utf8_with_BOM_no_charset_attr.css 09 Jun 2009 01:03:06 (127.0.0.1 15) INFO css.CSSHandler - creating CSSHandler 09 Jun 2009 01:03:06 (127.0.0.1 15) INFO css.CSSHandler - Trying to parse CSS with charset setting of utf-8 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG utils.FileUtils - Read the following bytes: text[name='german'] { 
   backgroundColor: #ffcc00;
   buttonText: "öäüÖÄܧ";
}

text[nam
09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG utils.FileUtils - Testing for UTF-8 BOM! 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG utils.FileUtils - Found BOM on file, encoding is UTF-8 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG css.CSSHandler - Opening CSS file /Users/rajubitter/src/svn/openlaszlo/4.0.15/test/ css/encoding/utf8_with_BOM_no_charset_attr.css using encoding UTF-8 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG css.CSSHandler - Skip first 3 bytes containing UTF-8 BOM 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - compiling CSSHandler using new unique names 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Conditional selector: [name="german"] 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Attribute condition 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - simple selector:text 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Cond string: { attrname: "name", attrvalue: "german", simpleselector: "text"} 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Conditional selector: [name="korean"] 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Attribute condition 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - simple selector:text 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Cond string: { attrname: "name", attrvalue: "korean", simpleselector: "text"} 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Conditional selector: [name="chinese"] 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Attribute condition 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - simple selector:text 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - Cond string: { attrname: "name", attrvalue: "chinese", simpleselector: "text"} 09 Jun 2009 01:03:06 (127.0.0.1 15) DEBUG compiler.StyleSheetCompiler - whole stylesheet as css $lzc $style._addRule(new $lzc$rule({ attrname: "name", attrvalue: "german", simpleselector: "text"}, {backgroundColor: 0xFFCC00, buttonText: "???????"})); $lzc$style._addRule(new $lzc$rule({ attrname: "name", attrvalue: "korean", simpleselector: "text"}, {backgroundColor: 0xFF00CC, buttonText: "? ?? ??? ?? ?? ?? ?? ??"})); $lzc$style._addRule(new $lzc$rule({ attrname: "name", attrvalue: "chinese", simpleselector: "text"}, {backgroundColor: 0x00FFCC, buttonText: "???????????????"}))
But the debug console in the browser shows the following error message: 
ERROR: Unknown cohort for rule: #0 * (undefined)
ERROR: Unknown cohort for rule: #1 * (undefined)
ERROR: Unknown cohort for rule: #2 * (undefined)
WARNING @utf8_with_BOM_no_charset_attr.lzx#7: No CSS value found for node «lz.text#3| #german» for property name buttonText WARNING @utf8_with_BOM_no_charset_attr.lzx#7: No CSS value found for node «lz.text#3| #german» for property name backgroundColor WARNING @utf8_with_BOM_no_charset_attr.lzx#8: No CSS value found for node «lz.text#4| #korean» for property name buttonText WARNING @utf8_with_BOM_no_charset_attr.lzx#8: No CSS value found for node «lz.text#4| #korean» for property name backgroundColor WARNING @utf8_with_BOM_no_charset_attr.lzx#9: No CSS value found for node «lz.text#5| #chinese» for property name buttonText WARNING @utf8_with_BOM_no_charset_attr.lzx#9: No CSS value found for node «lz.text#5| #chinese» for property name backgroundColor 

Any idea what this might be?

- Raju


On Jun 8, 2009, at 9:42 PM, P T Withington wrote:


[cc-ing Wanli]

Wanli,
Maybe you also want to review (or have someone on your team review) as this issue affects you? 

---

Raju,

This looks good.
I think you might have trouble trying checking in your utf-16 examples to svn. I'm not sure svn will do the right thing. It looks like it is treating them as binary files. Maybe that will work. I don't know.
It looks to me like you may also have some tab characters in your sources. You need to expand tabs to check in, otherwise the pre- commit filter will reject your checkin.
I approve this change. You might want to check in your fix separately from your tests, just in case the UTF-16 files cause svn issues. 

On 2009-06-08, at 14:44EDT, Raju Bitter wrote:


Added a "+" in debug output string concatenation.
Change 20090608-raju-Y by [email protected] on 2009-06-08 20:09:02 CEST 
 in /Users/rajubitter/src/svn/openlaszlo/trunk-cssunicode
 for http://[email protected]/openlaszlo/trunk

Summary: Fix for CSS parser uses incorrect file encoding
New Features: Adds an optional @charset to the stylesheet tag, in case the user wants to use a CSS file in a different encoding then utf-8 

Bugs Fixed: LPP-8045

Technical Reviewer: ptw
QA Reviewer: (pending)
Doc Reviewer: (pending)

Documentation:

Release Notes:

Details:
+ StyleSheetCompiler.java: Handling for @charset on stylesheet tag added. Added a 2nd parameter with the encoding value to the CSSHandler.parse() call. 

+ CSSHandler.java:
parse() method takes the encoding from the LZX stylesheet tag as a 2nd parameter. 
getInputSource() method does a few more things now:
1) checks for a possible BOM on the CSS file
2) checks if a possible BOM conflicts with the value of the stylesheet tag's @charset 3) if there's a BOM, the BOM bytes are removed from the input stream 

+ FileUtils.java:
Added method public static String detectBOMEncoding(BufferedInputStream in) The method returns the BOM marker interpreted as one of the following strings: 
UTF-8
UTF-16LE
UTF-16BE

Tests:
+ test files in folder test/css/encoding
The following test exist:
1) iso8859-1_with_charset_attr.lzx
Reading an iso-8859-2 encoded CSS file with some German special chars 
2) utf16BE_with_BOM.lzx
  Reading an utf-16 BE CSS file with BOM marker
3) utf16LE_with_BOM.lzx
  Reading an utf-16 LE CSS file with BOM marker
4) utf8_with_BOM_no_charset_attr.lzx
Reading an utf-8 CSS file with BOM and no charset attribute on the stylesheet tag 
5) utf8_with_BOM_conflicting_charset_attr.lzx
Reading a CSS with @charset value of utf-16, but CSS having a UTF-8 BOM marker, will 
  throw a compile error

Files:
A       test/css/encoding
A       test/css/encoding/iso8859-1_with_charset_attr.lzx
A       test/css/encoding/utf16BE_with_BOM.lzx
A       test/css/encoding/utf8_with_BOM_no_charset_attr.css
A       test/css/encoding/utf16LE_with_BOM.css
A       test/css/encoding/utf8_no_BOM_no_charset_attr.css
A test/css/encoding/ utf8_with_BOM_conflicting_charset_attr.css 
A       test/css/encoding/utf8_with_BOM_no_charset_attr.lzx
A       test/css/encoding/utf16LE_with_BOM.lzx
A       test/css/encoding/utf8_no_BOM_no_charset_attr.lzx
A       test/css/encoding/iso8859-1_with_charset_attr.css
A       test/css/encoding/utf16BE_with_BOM.css
A test/css/encoding/ utf8_with_BOM_conflicting_charset_attr.lzx 
M       WEB-INF/lps/server/src/org/openlaszlo/css/CSSHandler.java
M       WEB-INF/lps/server/src/org/openlaszlo/utils/FileUtils.java
M WEB-INF/lps/server/src/org/openlaszlo/compiler/ StyleSheetCompiler.java 

Changeset: http://svn.openlaszlo.org/openlaszlo/patches/20090608-raju-Y.tar

Reply via email to