Re: [AOLSERVER] chinese characters and oracle driver
Also, try first solving the problem of reading from the database. So if you have some data in a table that you know is correctly stored, try a sql query from AOLserver, and see if you can pinpoint where it gets converted. Once you have that sorted, writing to the database should be solvable. Brian - Reply message - From: Janine Ohmer jan...@furfly.net To: AOLSERVER@LISTSERV.AOL.COM AOLSERVER@LISTSERV.AOL.COM Subject: [AOLSERVER] chinese characters and oracle driver Date: Mon, Jul 18, 2011 8:27 pm Make sure that any vars set in your shell environment that relate to this are also set in your nsd wrapper script. I wish I knew for sure if that is enough for them to be effective, but I don't. I vaguely recall that the C API the driver uses is called Pro-C on the Oracle side - you might want to see if you can find some docs on that for your Oracle version and see if they can shed any light on this. Maybe you have to do something in the driver to make sure your language settings are acted upon. janine On Jul 18, 2011, at 12:03 PM, Brad Chick wrote: Well, I reached an impasse. I grabbed the latest source from CVS, added support for the 2 datatypes (NCHAR, NVARCHAR2), and recompiled. Then, I turned on debugging for the driver. The logs look great. The driver constructs the queries appropriately: e.g. [18/Jul/2011:14:17:30][10744.1099659584][-default:1-] Notice: nsoracle.c:3169:Ns_OracleExec: entry (dbh 0xd8392b0, sql insert into test_zhs (foo, bar) values (N'男孩儿 儿男孩 Yo', N'男孩儿 儿男孩 Yo')) But the data somehow gets mangled in oracle. And that same insert statement works perfectly well in sqlplus: insert into test_zhs (foo, bar) values (N'男孩儿 儿男孩 Yo', N'男孩儿 儿男孩 Yo'); (and i've tried it a million ways (e.g. without the leading 'N'')). When viewed from sqplus, the sqplus added data looks fine, while the stuff that comes in through the driver is mangled. Dunno. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
Also take a look at the CharExpansion parameter. I had some issues with that a few years ago (and the documentation seems to be back to front just to make it even more confusing!). :-) http://www.mail-archive.com/aolserver@listserv.aol.com/msg09664.html From: Fenton, Brian Sent: 19 July 2011 09:12 To: AOLserver Discussion Subject: Re: [AOLSERVER] chinese characters and oracle driver Also, try first solving the problem of reading from the database. So if you have some data in a table that you know is correctly stored, try a sql query from AOLserver, and see if you can pinpoint where it gets converted. Once you have that sorted, writing to the database should be solvable. Brian - Reply message - From: Janine Ohmer jan...@furfly.net To: AOLSERVER@LISTSERV.AOL.COM AOLSERVER@LISTSERV.AOL.COM Subject: [AOLSERVER] chinese characters and oracle driver Date: Mon, Jul 18, 2011 8:27 pm Make sure that any vars set in your shell environment that relate to this are also set in your nsd wrapper script. I wish I knew for sure if that is enough for them to be effective, but I don't. I vaguely recall that the C API the driver uses is called Pro-C on the Oracle side - you might want to see if you can find some docs on that for your Oracle version and see if they can shed any light on this. Maybe you have to do something in the driver to make sure your language settings are acted upon. janine On Jul 18, 2011, at 12:03 PM, Brad Chick wrote: Well, I reached an impasse. I grabbed the latest source from CVS, added support for the 2 datatypes (NCHAR, NVARCHAR2), and recompiled. Then, I turned on debugging for the driver. The logs look great. The driver constructs the queries appropriately: e.g. [18/Jul/2011:14:17:30][10744.1099659584][-default:1-] Notice: nsoracle.c:3169:Ns_OracleExec: entry (dbh 0xd8392b0, sql insert into test_zhs (foo, bar) values (N'男孩儿 儿男孩 Yo', N'男孩儿 儿男孩 Yo')) But the data somehow gets mangled in oracle. And that same insert statement works perfectly well in sqlplus: insert into test_zhs (foo, bar) values (N'男孩儿 儿男孩 Yo', N'男孩儿 儿男孩 Yo'); (and i've tried it a million ways (e.g. without the leading 'N'')). When viewed from sqplus, the sqplus added data looks fine, while the stuff that comes in through the driver is mangled. Dunno. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
I don't think it's tcl at this point. I can see that aolserver/tcl can bring in the unicode characters and return them to the client 'as is'. And you can see what encoding system tcl is using via this command as well: [encoding system] which shows that tcl is using UTF-8, which is what is desired. As well, I have another stack that has the same architecture, with the exception of the db/driver: aolserver 4.51/tcl 8.5/mysql/nsmsql and that works right out of the box. I can pass in any unicode characters (e.g. Braille, Farsi, Gaelic, etc.) and everything works. The strings are put in mysql and pulled out perfectly. In the oracle case, it's simple to see that it's not the db: with sqlplus, I can insert/select into/from oracle and everything looks fine. So, it seems like all that is left is the oracle driver. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
It might be that the Oracle driver doesn't handle those column types. Is there anything in the AOLserver log? Maybe turn on debug to get more info. Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Brad Chick [b...@chickcentral.com] Sent: 18 July 2011 16:19 To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] chinese characters and oracle driver I don't think it's tcl at this point. I can see that aolserver/tcl can bring in the unicode characters and return them to the client 'as is'. And you can see what encoding system tcl is using via this command as well: [encoding system] which shows that tcl is using UTF-8, which is what is desired. As well, I have another stack that has the same architecture, with the exception of the db/driver: aolserver 4.51/tcl 8.5/mysql/nsmsql and that works right out of the box. I can pass in any unicode characters (e.g. Braille, Farsi, Gaelic, etc.) and everything works. The strings are put in mysql and pulled out perfectly. In the oracle case, it's simple to see that it's not the db: with sqlplus, I can insert/select into/from oracle and everything looks fine. So, it seems like all that is left is the oracle driver. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
I just grabbed the latest oracle driver from cvs and you are right: there is no explicit support for either NCHAR or NVARCHAR2 - which oracle requires to store unicode characters. So, I will try to update the driver and report back. Thanks -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
Well, I reached an impasse. I grabbed the latest source from CVS, added support for the 2 datatypes (NCHAR, NVARCHAR2), and recompiled. Then, I turned on debugging for the driver. The logs look great. The driver constructs the queries appropriately: e.g. [18/Jul/2011:14:17:30][10744.1099659584][-default:1-] Notice: nsoracle.c:3169:Ns_OracleExec: entry (dbh 0xd8392b0, sql insert into test_zhs (foo, bar) values (N'ç·å©å¿ å¿ç·å© Yo', N'ç·å©å¿ å¿ç·å© Yo')) But the data somehow gets mangled in oracle. And that same insert statement works perfectly well in sqlplus: insert into test_zhs (foo, bar) values (N'ç·å©å¿ å¿ç·å© Yo', N'ç·å©å¿ å¿ç·å© Yo'); (and i've tried it a million ways (e.g. without the leading 'N'')). When viewed from sqplus, the sqplus added data looks fine, while the stuff that comes in through the driver is mangled. Dunno. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
Make sure that any vars set in your shell environment that relate to this are also set in your nsd wrapper script. I wish I knew for sure if that is enough for them to be effective, but I don't. I vaguely recall that the C API the driver uses is called Pro-C on the Oracle side - you might want to see if you can find some docs on that for your Oracle version and see if they can shed any light on this. Maybe you have to do something in the driver to make sure your language settings are acted upon. janine On Jul 18, 2011, at 12:03 PM, Brad Chick wrote: Well, I reached an impasse. I grabbed the latest source from CVS, added support for the 2 datatypes (NCHAR, NVARCHAR2), and recompiled. Then, I turned on debugging for the driver. The logs look great. The driver constructs the queries appropriately: e.g. [18/Jul/2011:14:17:30][10744.1099659584][-default:1-] Notice: nsoracle.c:3169:Ns_OracleExec: entry (dbh 0xd8392b0, sql insert into test_zhs (foo, bar) values (N'男孩儿 儿男孩 Yo', N'男孩儿 儿男孩 Yo')) But the data somehow gets mangled in oracle. And that same insert statement works perfectly well in sqlplus: insert into test_zhs (foo, bar) values (N'男孩儿 儿男孩 Yo', N'男孩儿 儿男孩 Yo'); (and i've tried it a million ways (e.g. without the leading 'N'')). When viewed from sqplus, the sqplus added data looks fine, while the stuff that comes in through the driver is mangled. Dunno. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
Just a quick shot in the dark here. I have ran into encoding issues in the past because by default TCL will assume everything is Latin-1 and it is not always straight forward when a conflict will happen. You can set a variable like $first_names to be a UTF-8 string and it can be written back out as a UTF-8 string until you do something to to. For example, do you have a wrapper that will clean a database value to prevent sqlinjection like ns_dbquotevalue? If you do then that string's data gets converted to Latin 1. Try something like this: set first_names [encoding convertto utf-8 $first_names] Insert it into your database, something like: insert into test_zhs (foo) values ([ns_dbquotevalue $first_name]); Check from the command line if the data is in there correctly. If it is you will probably have some issues getting it out for the same reason. Look at the TCL's encoding: http://tcl.activestate.com/man/tcl8.5/TclCmd/encoding.htm Like I said, just a shot in the dark at what the issue may be. _Peter Date: Fri, 15 Jul 2011 16:01:19 -0400 From: b...@chickcentral.com Subject: [AOLSERVER] chinese characters and oracle driver To: AOLSERVER@LISTSERV.AOL.COM I am having trouble getting Chinese characters in/out of oracle with AOLserver. Here is my stack: Oracle 11g TCL 8.5 AOLserver 4.51 Oracle Driver version 2.7 The existing database has a database character set of WE8ISO8859P1. But we are using NCHAR and NVARCHAR2 datatypes to store Unicode characters. The NLS_NCHAR_CHARACTERSET is properly set to AL16UTF16. I am setting the following in the environment as an nsd wrapper: export NLS_LANG=_.UTF8 I can login to sqlplus and insert and select chinese characters: insert into test_zhs (foo) values (N'男孩儿男孩儿'); SQL select * from test_zhs; FOO 男孩儿男孩儿 I also am sure that aolserver/tcl are treating the characters appropriately. For example, this form takes whatever characters are inputed into the form, tries to insert them, and spits them back out. When chinese characters are inputed, that is how the server returns them. http://jp.xacte.com:8181/test/db/myform.tcl On the other hand, no matter what I try, I can't get aolserver to get them into oracle properly. I suspect it's the oracle driver, but people have suggested that it is possible to put unicode characters into oracle using that driver. Any help would be way helpful. Thanks -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] chinese characters and oracle driver
I've had to deal with Chinese Characters and Postgres. I don't recall all the details anymore but I do recall what Peter is saying, that Tcl was a culprit more than the database. I would use sqlplus to check what myform.tcl is inserting into the database. That will at least tell you whether Tcl is munging it on the way in or the way out (could be both) and give you a place to start tweaking. janine On Jul 15, 2011, at 1:33 PM, Peter Sadlon wrote: Just a quick shot in the dark here. I have ran into encoding issues in the past because by default TCL will assume everything is Latin-1 and it is not always straight forward when a conflict will happen. You can set a variable like $first_names to be a UTF-8 string and it can be written back out as a UTF-8 string until you do something to to. For example, do you have a wrapper that will clean a database value to prevent sqlinjection like ns_dbquotevalue? If you do then that string's data gets converted to Latin 1. Try something like this: set first_names [encoding convertto utf-8 $first_names] Insert it into your database, something like: insert into test_zhs (foo) values ([ns_dbquotevalue $first_name]); Check from the command line if the data is in there correctly. If it is you will probably have some issues getting it out for the same reason. Look at the TCL's encoding: http://tcl.activestate.com/! man/tcl8.5/TclCmd/encoding.htm Like I said, just a shot in the dark at what the issue may be. _Peter Date: Fri, 15 Jul 2011 16:01:19 -0400 From: b...@chickcentral.com Subject: [AOLSERVER] chinese characters and oracle driver To: AOLSERVER@LISTSERV.AOL.COM I am having trouble getting Chinese characters in/out of oracle with AOLserver. Here is my stack: Oracle 11g TCL 8.5 AOLserver 4.51 Oracle Driver version 2.7 The existing database has a database character set of WE8ISO8859P1. But we are using NCHAR and NVARCHAR2 datatypes to store Unicode characters. The NLS_NCHAR_CHARACTERSET is properly set to AL16UTF16. I am setting the following in the environment as an nsd wrapper: export NLS_LANG=_.UTF8 I can login to sqlplus and insert and select chinese characters: insert into test_zhs (foo) values (N'男孩儿男孩儿� (B'); SQL select * from test_zhs; FOO 男孩儿男孩儿 I also am sure that aolserver/tcl are treating the characters appropriately. For example, this form takes whatever characters are inputed into the form, tries to insert them, and spits them back out. When chinese characters are inputed, that is how the server returns them. http://jp.xacte.com:8181/test/db/myform.tcl On the other hand, no matter what I try, I can't get aolserver to get them into oracle properly. I suspect it's the oracle driver, but people have suggested that it is possible to put unicode characters into oracle using that driver. Any help would be way helpful. Thanks -- AOLserver - http://www.aolserver.! com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
