[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Timothy Perrett
I agree - it does seem like we should really be doing this by default. Even chinese, hebrew and double byte languages will be good using UTF-8 right? Is there a reason someone might want to set it to another encoding / collation other than UTF-8? I cant think of one right now... @chas - from

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Charles F. Munat
Just Jetty on the server. Maven/Jetty while developing. (I'm not that dumb.) :-) Chas. Timothy Perrett wrote: I agree - it does seem like we should really be doing this by default. Even chinese, hebrew and double byte languages will be good using UTF-8 right? Is there a reason someone

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Timothy Perrett
Phew :) Out of interest, why do you want to use glashfish rather than jetty? Tim On 16/03/2009 10:08, Charles F. Munat c...@munat.com wrote: Just Jetty on the server. Maven/Jetty while developing. (I'm not that dumb.) :-) Chas. --~--~-~--~~~---~--~~

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Timothy Perrett
Im hosting several sites on a single jetty install - its working perfectly right now. Are you not familiar with the virtual hosting options in jetty? Its pretty well documented on their wiki and will let you host from the root context. Someone can correct me if im wrong, but until servlet 3.0

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread David Pollak
On Mon, Mar 16, 2009 at 6:39 AM, Timothy Perrett timo...@getintheloop.euwrote: Im hosting several sites on a single jetty install - its working perfectly right now. Are you not familiar with the virtual hosting options in jetty? Its pretty well documented on their wiki and will let you host

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Timothy Perrett
Sorry! My bad ­ we¹ve had so many convo¹s about this and I had become muddled :-) I was talking about continuations as you say, not the the comet support! Sorry again! Doh! Cheers, Tim On 16/03/2009 19:15, David Pollak feeder.of.the.be...@gmail.com wrote: On Mon, Mar 16, 2009 at 6:39

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Charles F. Munat
That's good to know. But now that Tim has made me aware of the possibilities of Jetty, I might be persuaded to stick with it. Need to figure out how to host multiple sites in one instance, and discover where this context deployer is hidden. If I can get that running, I write a brief tutorial

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Charles F. Munat
Where's Lassie when you need her? Timothy Perrett wrote: Sorry! My bad – we’ve had so many convo’s about this and I had become muddled :-) I was talking about continuations as you say, not the the comet support! Sorry again! Doh! Cheers, Tim On 16/03/2009 19:15, David Pollak

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Timothy Perrett
Lol! Lassie?! What?! Haha. Check out this in my jetty.xml: New class=org.mortbay.jetty.webapp.WebAppContext ArgRef id=Contexts//Arg ArgSystemProperty name=jetty.home//webapps/myapplication.war/Arg Arg//Arg Set name=defaultsDescriptorSystemProperty name=jetty.home

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Charles F. Munat
So will this do the virtual hosting? (At first glance, I'm not seeing how.) Right now I use Apache to forward to the port and I run each Jetty on a different port. And if I need to reboot one application, do I have to reboot them all? (You do remember Lassie, right? Always getting Timmy out

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-16 Thread Timothy Perrett
So will this do the virtual hosting? (At first glance, I'm not seeing how.) Correct :-) Its all on the jetty wiki - http://docs.codehaus.org/display/JETTY/Virtual+hosts And if I need to reboot one application, do I have to reboot them all? Hmm good question - right now im not 100% sure

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
I'm not. Lift is. I'm just saving text into a database, and then pulling it back out again and displaying it. I'm not doing anything special. Everything, as far as I can tell is UTF-8, but still I get gibberish back out. Marc Boschma wrote: Which parser are you using? Quick tests with

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
OK, I can replicate this in our PocketChange app (also going against a PostgreSQL DB). Let me dig a bit. Derek On Sun, Mar 15, 2009 at 3:58 AM, Charles F. Munat c...@munat.com wrote: This might help, but I don't think I was clear. I have an online form. My clients enter text into it. Their

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
This is really interesting. I've narrowed it down to something on form submission. The database shows gibberish, too, and if I manually enter the correct value in the DB it works fine on display. If I print the UTF-8 byte values of the string I get from the browser for my description when I submit

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
I'm not going to have much time to work on this today, so I opened a ticket: http://liftweb.lighthouseapp.com/projects/26102/tickets/20-utf-8-form-submission-broken in case anyone else wants to look at it. Derek On Sun, Mar 15, 2009 at 8:08 AM, Derek Chen-Becker dchenbec...@gmail.comwrote:

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Marc Boschma
Just looking at http://jeppesn.dk/utf-8.html , I found the following lines: Character Latin1 Unicode UTF-8 Latin1 codeinterpr. ç E7 00 E7 C3 A7 ç à is C38C, § is

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Marc Boschma
excuse the typo: On 16/03/2009, at 6:23 AM, Marc Boschma wrote: Just looking at http://jeppesn.dk/utf-8.html , I found the following lines: Character Latin1 Unicode UTF-8 Latin1 codeinterpr. ç

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Marc Boschma
Now I have some breakfast in me, to be clear it appears that UTF-8 byte stream is being interpreted as Latin1 and then converted to unicode... Marc On 16/03/2009, at 6:25 AM, Marc Boschma wrote: excuse the typo: On 16/03/2009, at 6:23 AM, Marc Boschma wrote: Just looking at

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
That was my thinking. It doesn't explain why ccedil; in gets changed to amp;ccedil;, but it explains why ç in becomes ç out. So I think there are two separate issues here. The ç can be created in two different ways in UTF-8. One is the single c with a cedilla character. The second is a c

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
The scala XML syntax automatically converts any in embedded strings to amp;. You have to put the string inside a scala.xml.Unparsed node to prevent that from happening. Derek On Sun, Mar 15, 2009 at 1:59 PM, Charles F. Munat c...@munat.com wrote: That was my thinking. It doesn't explain why

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
Sorry, I'm not suggesting that this is the appropriate method for users; they should just be able to type. I was just trying to explain why the is getting expanded. I think that the current behavior is not really what anyone wants, and hopefully we can fix it in a transparent manner. Derek On

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Marc Boschma
On 16/03/2009, at 6:59 AM, Charles F. Munat wrote: That was my thinking. It doesn't explain why ccedil; in gets changed to amp;ccedil;, but it explains why ç in becomes ç out. So I think there are two separate issues here. I tend to agree. The ç can be created in two different

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
My understanding was that *all* ampersands get converted in raw embedded strings: scala val m = span{ amp; }/span m: scala.xml.Elem = spanamp;amp;/span scala val m = span{ }/span m: scala.xml.Elem = spanamp;/span If you don't embed the string it doesn't do validation on the entities: scala

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
Oh, sorry, Derek. My bad. I didn't mean to imply that you were saying that the situation was optimal. I understood where you were coming from. Actually, I wasn't really addressing your comment after my first sentence. I should have made that clear. Haven't had my coffee yet... This is kind

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
I just went back and tried changing it in the database itself, and that worked fine. So now I have a workaround, but it's one that creates a huge amount of work for me... :-( Chas. Charles F. Munat wrote: Oh, sorry, Derek. My bad. I didn't mean to imply that you were saying that the

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
On Sun, Mar 15, 2009 at 2:57 PM, Charles F. Munat c...@munat.com wrote: Oh, sorry, Derek. My bad. I didn't mean to imply that you were saying that the situation was optimal. I understood where you were coming from. Actually, I wasn't really addressing your comment after my first sentence. I

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
Marc Boschma wrote: When I use ccedil; instead, the problem is that it is *not* converted to ç as it goes into the database, and then on the way out the XML interpreter does not recognize it as a character entity reference and so converts the to amp;. I think this is due to using the

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
Crapola: http://jira.codehaus.org/browse/JETTY-958 I think I've confirmed that this is not lift. I added a non-lift input text element to an existing lift form: input name=testthis type=text / Then I use the following code, which I believe should be getting direct access to Jetty's

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread marius d.
just out of curiosity are you setting manually in the HTTP header Content-Type: text/html; charset=UTF-8 and it's still broken? P.S. Sometimes HTTP equiv from HTML header just doesn't do the trick. Br's, Marius On Mar 15, 11:20 pm, Derek Chen-Becker dchenbec...@gmail.com wrote: Crapola:

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
Yes, both with Maven for development and on the server (though I'd really love to switch to Glassfish if I can just figure out the context-in-the-url issue). Derek Chen-Becker wrote: Crapola: http://jira.codehaus.org/browse/JETTY-958 I think I've confirmed that this is not lift. I added

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
Everything is being served UTF-8... that means the header is UTF-8, the xml process directive is UTF-8, and there's a content type meta tag setting it to UTF-8, and I can confirm in Firefox via both the View Character Encoding menu an Firebug that the page is being served as UTF-8, so even

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread David Pollak
Folks, Please make sure you've got this method in your Boot.scala class: /** * Force the request to be UTF-8 */ private def makeUtf8(req: HttpServletRequest) { req.setCharacterEncoding(UTF-8) } And also in the boot method, put: LiftRules.early.append(makeUtf8) By default,

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread David Pollak
On Mon, Mar 16, 2009 at 12:30 AM, Charles F. Munat c...@munat.com wrote: Everything is being served UTF-8... that means the header is UTF-8, the xml process directive is UTF-8, and there's a content type meta tag setting it to UTF-8, and I can confirm in Firefox via both the View Character

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Charles F. Munat
That's got it. I added it to the FAQ on the wiki. Thanks, David! Wish I'd been smart enough to ask this a week ago! Chas. David Pollak wrote: Folks, Please make sure you've got this method in your Boot.scala class: /** * Force the request to be UTF-8 */ private def

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread David Pollak
On Mon, Mar 16, 2009 at 2:07 AM, Charles F. Munat c...@munat.com wrote: That's got it. I added it to the FAQ on the wiki. Thanks, David! Wish I'd been smart enough to ask this a week ago! I bloodies my head with that one for a good couple of weeks. Glad it's working. Chas. David

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-15 Thread Derek Chen-Becker
Argh. I even thought of that but setting it *after* the request had been accessed (by Lift internals) appears to have no effect. I suppose there's some caching going on there. Any possibility we could add a control to LiftRules? Something like: var totallyBrokenDefaultPostCharsetHandling = false

[Lift] Re: xml parser, utf-8, special characters... kill me now

2009-03-14 Thread Marc Boschma
Which parser are you using? Quick tests with PCDataXmlParser seems to indicate all ccedil; get mapped to the unicode character. In fact it does that for all entities in object HtmlEntities (line 26 onwards in lift-util/src/main/scala/ net/liftweb/util/PCDataMarkupParser.scala) If I enter