Re: [AOLSERVER] AOLSERVER Digest - 22 Sep 2011 to 24 Sep 2011 (#2011-57)
Hi Guan, What is the value of [encoding system] on a regular page or in the context where you call ns_returnfile? "[encoding system]" returns utf-8 just before running "ns_returnfile 200 $type $file" If I set "encoding system iso8859-1" before "ns_returnfile 200 $type $file" rp_serve_concrete_file also fails. ./k On 25/09/2011 6:20, Guan Yang wrote: What is the value of [encoding system] on a regular page or in the context where you call ns_returnfile? On Sep 25, 2011, at 0:00, AOLSERVER automatic digest system lists...@listserv.aol.com wrote: tx everybody for the very useful input! Using ns_returnfp is (for several reasons) not an option for us. Recompiling tcl using --with-encoding utf-8 did not resolve the problem. This time I used AOLserver 4.5.1 / tcl8.5.10. I gave it a try since some of our users do upload files using tools such as WINSCP. Since these files would also need to be accessible through the ]project-open[ file manager, we need to come up with a slightly more complex solution to detect files with special chars. ./k On 23/09/2011 1:32, Jeff Rogers wrote: Another thing you could do is to set tcl's default encoding to utf-8, so that the filenames passed to Ns_ConnReturnFile are the same encoding as what the core tcl commands do. Set the default encoding with encoding system utf-8 in some tcl file. It's possible this could have some side effects. I'm trying to figure out what needs to be patched in the server for this to do the right thing, but encoding is a messy business. -J Jim Davidson wrote: Howdy, Looking at the code, ns_returnfile passes the filename through to the core Ns_ConnReturnFile without any of the care that core Tcl does handling filenames. You may be able to replace ns_returnfile with ns_returnfp, passing a file handle returned from Tcl's open command which should be smarter with filesystem encoding. The downside is that would bypass some of the smarts within the underlying Ns_ConnReturnFile used by ns_returnfile: It appears to set Last-Modified headers, optionally cache small files, mmap large files, etc. All this may not matter if you're doing one-shot type work, e.g., returning a custom crafted file for just one user, just once. -Jim On Sep 22, 2011, at 3:27 PM, Peter Sadlon wrote: Just a guess here, but by default, TCL is compiled with Latin-1 encoding. This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset. You could attempt to recompile TCL with --with-encoding utf-8 But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases. Date: Thu, 22 Sep 2011 22:59:02 +0200 From: klaus.hofed...@project-open.com mailto:klaus.hofed...@project-open.com Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars To: AOLSERVER@LISTSERV.AOL.COM mailto:AOLSERVER@LISTSERV.AOL.COM Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final)
[AOLSERVER] AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
Hi Guan, How confident are you that the underlying file name is UTF-8? [file readable $file] returns '1' - this might be evidence enough for a UTF-8 file name ns_returnfile 200 $type [encoding convertto utf-8 $file] negative :( tx for looking into this ! ./k On 25/09/2011 16:57, Guan Yang wrote: On Sunday, September 25, 2011 at 06:42 , Klaus Hofeditz ]project-open[ wrote: Hi Guan, - Show quoted message - "[encoding system]" returns utf-8 just before running "ns_returnfile 200 $type $file" If I set "encoding system iso8859-1" before "ns_returnfile 200 $type $file" rp_serve_concrete_file also fails. ./k I have had some experiences like this in the past but they were always fine when encoding system returned utf-8 in the relevant context. Pardon me if you have been through this, but a couple of diagnostic questions: 1) How confident are you that the underlying file name is UTF-8? I know you used convmv, but have you tried reading the file name directly with a C program that uses system calls? 2) Have you tried converting the file name from Tcl's internal 16-bit encoding to utf-8 first? Like this: ns_returnfile 200 $type [encoding convertto utf-8 $file] /Guan -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- Klaus Hofeditz mailto:klaus.hofed...@project-open.com http://www.project-open.com -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
tx everybody for the very useful input! Using ns_returnfp is (for several reasons) not an option for us. Recompiling tcl using --with-encoding utf-8 did not resolve the problem. This time I used AOLserver 4.5.1 / tcl8.5.10. I gave it a try since some of our users do upload files using tools such as WINSCP. Since these files would also need to be accessible through the ]project-open[ file manager, we need to come up with a slightly more complex solution to detect files with special chars. ./k On 23/09/2011 1:32, Jeff Rogers wrote: Another thing you could do is to set tcl's default encoding to utf-8, so that the filenames passed to Ns_ConnReturnFile are the same encoding as what the core tcl commands do. Set the default encoding with encoding system utf-8 in some tcl file. It's possible this could have some side effects. I'm trying to figure out what needs to be patched in the server for this to do the right thing, but encoding is a messy business. -J Jim Davidson wrote: Howdy, Looking at the code, ns_returnfile passes the filename through to the core Ns_ConnReturnFile without any of the care that core Tcl does handling filenames. You may be able to replace ns_returnfile with ns_returnfp, passing a file handle returned from Tcl's open command which should be smarter with filesystem encoding. The downside is that would bypass some of the smarts within the underlying Ns_ConnReturnFile used by ns_returnfile: It appears to set Last-Modified headers, optionally cache small files, mmap large files, etc. All this may not matter if you're doing one-shot type work, e.g., returning a custom crafted file for just one user, just once. -Jim On Sep 22, 2011, at 3:27 PM, Peter Sadlon wrote: Just a guess here, but by default, TCL is compiled with Latin-1 encoding. This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset. You could attempt to recompile TCL with --with-encoding utf-8 But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases. Date: Thu, 22 Sep 2011 22:59:02 +0200 From: klaus.hofed...@project-open.com mailto:klaus.hofed...@project-open.com Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars To: AOLSERVER@LISTSERV.AOL.COM mailto:AOLSERVER@LISTSERV.AOL.COM Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final) We suddenly encountered the problem that ns_returnfile can't find a file which filname contains special chars such as 'umlaute' (, , etc.) --- convmv tells me that the file name is utf-8 encoded. --- Config.tcl has the standard settings: ns_param HackContentType 1 ns_param DefaultCharset utf-8 ns_param HttpOpenCharset utf-8 ns_param OutputCharset utf-8 ns_param URLCharset utf-8 --- Following CentOS locale: LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8"
[AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final) We suddenly encountered the problem that ns_returnfile can't find a file which filname contains special chars such as 'umlaute' (, , etc.) --- convmv tells me that the file name is utf-8 encoded. --- Config.tcl has the standard settings: ns_param HackContentType1 ns_param DefaultCharset utf-8 ns_param HttpOpenCharsetutf-8 ns_param OutputCharset utf-8 ns_param URLCharset utf-8 --- Following CentOS locale: LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= --- File System is 'ext3' --- Files have been created with " exec /bin/cp $filename $dest_path" out of a tcl file or using 'touch/emacs' on OS level. Same results. --- We did have the problems using AOLserver 3.3 Anybody an idea what the cause might be? Probably its better to always encode file names of uploaded filesto avoid trouble when creating multi-plattform applications. Thanks for your support! Klaus -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
