Re: DIH and UTF-8

2010-12-29 Thread Mark
Sure thing. In my database.yml I was missing the encoding: utf8 option. If one were to add unicode characters within rails (console, web form, etc) the characters would appear to be saved correctly... ie when trying to retrieve them back, everything looked perfect. The characters also

Re: DIH and UTF-8

2010-12-29 Thread Jonathan Rochkind
I haven't tried it yet, but I _think_ in Rails if you are using the 'mysql2' adapter (now standard with Rails3) instead of 'mysql', it might handle utf-8 better with less areas for gotchas. I think if the underlying mysql database is set to use utf-8, then, at least with mysql2 adapter, you

Re: DIH and UTF-8

2010-12-28 Thread Darx Oman
if you are using tomcat modify server.xml Connector port=*8080* protocol=*HTTP/1.1* connectionTimeout=*2*redirectPort =*8443* URIEncoding=*UTF-8* / check the URIEncoding=*UTF-8* is set

Re: DIH and UTF-8

2010-12-28 Thread Mark
It was due to the way I was writing to the DB using our rails application. Everythin looked correct but when retrieving it using the JDBC driver it was all managled. On 12/27/10 4:38 PM, Glen Newton wrote: Is it possible your browser is not set up to properly display the chinese characters?

Re: DIH and UTF-8

2010-12-28 Thread Glen Newton
Hi Mark, Could you offer a more technical explanation of the Rails problem, so that if others encounter a similar problem your efforts in finding the issue will be available to them? :-) Thanks, Glen PS. This has wandered somewhat off-topic to this list: apologies thanks for the patience of

Re: DIH and UTF-8

2010-12-27 Thread Erick Erickson
More data please. Which jdbc driver? Have you tried just printing out the results of using that driver in a simple Java program? Solr should handle UTF-8 just fine, but the servlet container may have to have some settings tweaked, which one of those are you using? What version of Solr? Best

Re: DIH and UTF-8

2010-12-27 Thread Mark
Solr: 1.4.1 JDBC driver: Connector/J 5.1.14 Looks like its the JDBC driver because It doesn't even work with a simple java program. I know this is a little off subject now, but do you have any clues? Thanks again On 12/27/10 1:58 PM, Erick Erickson wrote: More data please. Which jdbc

Re: DIH and UTF-8

2010-12-27 Thread Glen Newton
1 - Verify your mysql is set up using UTF-8 2 - Does your JDBC connect string contain: useUnicode=truecharacterEncoding=UTF-8 See: http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-charsets.html Glen http://zzzoot.blogspot.com/ On Mon, Dec 27, 2010 at 5:15 PM, Mark

Re: DIH and UTF-8

2010-12-27 Thread Mark
I tried both of those with no such luck. On 12/27/10 2:49 PM, Glen Newton wrote: 1 - Verify your mysql is set up using UTF-8 2 - Does your JDBC connect string contain: useUnicode=truecharacterEncoding=UTF-8 See: http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-charsets.html Glen

Re: DIH and UTF-8

2010-12-27 Thread Glen Newton
Try this in mysql: mysql SHOW VARIABLES LIKE 'character_set%'; From: http://forums.mysql.com/read.php?45,362257,362257 From the same page: do you have utf8 set in your 'my.cnf' (shouldn't effect JDBC but will effect the server)? [client] default-character-set = utf8 [mysql]

Re: DIH and UTF-8

2010-12-27 Thread Mark
Just like the user of that thread... i have my database, table, columns and system variables all set but it still doesnt work as expected. Server version: 5.0.67 Source distribution Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql SHOW VARIABLES LIKE 'collation%';

Re: DIH and UTF-8

2010-12-27 Thread Glen Newton
Is it possible your browser is not set up to properly display the chinese characters? (I am assuming you are looking at things through your browser) Do you have any problems viewing other chinese documents properly in your browser? Using mysql, can you see these characters properly? What happens

Re: DIH and UTF-8

2010-12-27 Thread Lance Norskog
Also if you can show your DIH script. Does the Chinese text come out ok when you do a database query? http://wiki.apache.org/solr/DataImportHandler#Troubleshooting On Mon, Dec 27, 2010 at 4:38 PM, Glen Newton glen.new...@gmail.com wrote: Is it possible your browser is not set up to properly