Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
Klaus Hofeditz ]project-open[ wrote: I gave it a try since some of our users do upload files using tools such as WINSCP. Since these files would also need to be accessible through the ]project-open[ file manager, we need to come up with a slightly more complex solution to detect files with special chars. I have a patch (not checked in yet) that will let ns_returnfile use the same files as native tcl commands (by doing the same utf-8 to external decoding that tcl does internally). It would make sense to apply this to all file-handling commands. But it would make more sense to replace these where possible (and simple) with tcl versions that map to the builtin tcl commands?. I think the docs even say that many of these date back to the time when tcl did not include these functions. Would anyone have any objection to replacing these c-coded commands with tcl ones? The commands that could be easily replaced are ns_chmod, ns_cp, ns_rename, ns_link, ns_mkdir, ns_rename, ns_rmdir, ns_symlink, ns_unlink, and ns_ftruncate. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
On Sunday, September 25, 2011 at 06:42 , Klaus Hofeditz ]project-open[ wrote: Hi Guan, - Show quoted message - [encoding system] returns utf-8 just before running ns_returnfile 200 $type $file If I set encoding system iso8859-1 before ns_returnfile 200 $type $file rp_serve_concrete_file also fails. ./k I have had some experiences like this in the past but they were always fine when encoding system returned utf-8 in the relevant context. Pardon me if you have been through this, but a couple of diagnostic questions: 1) How confident are you that the underlying file name is UTF-8? I know you used convmv, but have you tried reading the file name directly with a C program that uses system calls? 2) Have you tried converting the file name from Tcl's internal 16-bit encoding to utf-8 first? Like this: ns_returnfile 200 $type [encoding convertto utf-8 $file] /Guan -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
tx everybody for the very useful input! Using ns_returnfp is (for several reasons) not an option for us. Recompiling tcl using --with-encoding utf-8 did not resolve the problem. This time I used AOLserver 4.5.1 / tcl8.5.10. I gave it a try since some of our users do upload files using tools such as WINSCP. Since these files would also need to be accessible through the ]project-open[ file manager, we need to come up with a slightly more complex solution to detect files with special chars. ./k On 23/09/2011 1:32, Jeff Rogers wrote: Another thing you could do is to set tcl's default encoding to utf-8, so that the filenames passed to Ns_ConnReturnFile are the same encoding as what the core tcl commands do. Set the default encoding with encoding system utf-8 in some tcl file. It's possible this could have some side effects. I'm trying to figure out what needs to be patched in the server for this to do the right thing, but encoding is a messy business. -J Jim Davidson wrote: Howdy, Looking at the code, ns_returnfile passes the filename through to the core Ns_ConnReturnFile without any of the care that core Tcl does handling filenames. You may be able to replace ns_returnfile with ns_returnfp, passing a file handle returned from Tcl's open command which should be smarter with filesystem encoding. The downside is that would bypass some of the smarts within the underlying Ns_ConnReturnFile used by ns_returnfile: It appears to set Last-Modified headers, optionally cache small files, mmap large files, etc. All this may not matter if you're doing one-shot type work, e.g., returning a custom crafted file for just one user, just once. -Jim On Sep 22, 2011, at 3:27 PM, Peter Sadlon wrote: Just a guess here, but by default, TCL is compiled with Latin-1 encoding. This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset. You could attempt to recompile TCL with --with-encoding utf-8 But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases. Date: Thu, 22 Sep 2011 22:59:02 +0200 From: klaus.hofed...@project-open.com mailto:klaus.hofed...@project-open.com Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars To: AOLSERVER@LISTSERV.AOL.COM mailto:AOLSERVER@LISTSERV.AOL.COM Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final) We suddenly encountered the problem that ns_returnfile can't find a file which filname contains special chars such as 'umlaute' (, , etc.) --- convmv tells me that the file name is utf-8 encoded. --- Config.tcl has the standard settings: ns_param HackContentType 1 ns_param DefaultCharset utf-8 ns_param HttpOpenCharset utf-8 ns_param OutputCharset utf-8 ns_param URLCharset utf-8 --- Following CentOS locale: LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8"
Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
Just a guess here, but by default, TCL is compiled with Latin-1 encoding. This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset. You could attempt to recompile TCL with --with-encoding utf-8 But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases. Date: Thu, 22 Sep 2011 22:59:02 +0200 From: klaus.hofed...@project-open.com Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars To: AOLSERVER@LISTSERV.AOL.COM Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final) We suddenly encountered the problem that ns_returnfile can't find a file which filname contains special chars such as 'umlaute' (ä, ö, etc.) --- convmv tells me that the file name is utf-8 encoded. --- Config.tcl has the standard settings: ns_param HackContentType1 ns_param DefaultCharset utf-8 ns_param HttpOpenCharsetutf-8 ns_param OutputCharset utf-8 ns_param URLCharset utf-8 --- Following CentOS locale: LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_ADDRESS=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_ALL= --- File System is 'ext3' --- Files have been created with exec /bin/cp $filename $dest_path out of a tcl file or using 'touch/emacs' on OS level. Same results. --- We did have the problems using AOLserver 3.3 Anybody an idea what the cause might be? Probably its better to always encode file names of uploaded filesto avoid trouble when creating multi-plattform applications. Thanks for your support! Klaus -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
Howdy, Looking at the code, ns_returnfile passes the filename through to the core Ns_ConnReturnFile without any of the care that core Tcl does handling filenames. You may be able to replace ns_returnfile with ns_returnfp, passing a file handle returned from Tcl's open command which should be smarter with filesystem encoding. The downside is that would bypass some of the smarts within the underlying Ns_ConnReturnFile used by ns_returnfile: It appears to set Last-Modified headers, optionally cache small files, mmap large files, etc. All this may not matter if you're doing one-shot type work, e.g., returning a custom crafted file for just one user, just once. -Jim On Sep 22, 2011, at 3:27 PM, Peter Sadlon wrote: Just a guess here, but by default, TCL is compiled with Latin-1 encoding. This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset. You could attempt to recompile TCL with --with-encoding utf-8 But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases. Date: Thu, 22 Sep 2011 22:59:02 +0200 From: klaus.hofed...@project-open.com Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars To: AOLSERVER@LISTSERV.AOL.COM Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final) We suddenly encountered the problem that ns_returnfile can't find a file which filname contains special chars such as 'umlaute' (ä, ö, etc.) --- convmv tells me that the file name is utf-8 encoded. --- Config.tcl has the standard settings: ns_param HackContentType1 ns_param DefaultCharset utf-8 ns_param HttpOpenCharsetutf-8 ns_param OutputCharset utf-8 ns_param URLCharset utf-8 --- Following CentOS locale: LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_ADDRESS=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_ALL= --- File System is 'ext3' --- Files have been created with exec /bin/cp $filename $dest_path out of a tcl file or using 'touch/emacs' on OS level. Same results. --- We did have the problems using AOLserver 3.3 Anybody an idea what the cause might be? Probably its better to always encode file names of uploaded filesto avoid trouble when creating multi-plattform applications. Thanks for your support! Klaus -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
Another thing you could do is to set tcl's default encoding to utf-8, so that the filenames passed to Ns_ConnReturnFile are the same encoding as what the core tcl commands do. Set the default encoding with encoding system utf-8 in some tcl file. It's possible this could have some side effects. I'm trying to figure out what needs to be patched in the server for this to do the right thing, but encoding is a messy business. -J Jim Davidson wrote: Howdy, Looking at the code, ns_returnfile passes the filename through to the core Ns_ConnReturnFile without any of the care that core Tcl does handling filenames. You may be able to replace ns_returnfile with ns_returnfp, passing a file handle returned from Tcl's open command which should be smarter with filesystem encoding. The downside is that would bypass some of the smarts within the underlying Ns_ConnReturnFile used by ns_returnfile: It appears to set Last-Modified headers, optionally cache small files, mmap large files, etc. All this may not matter if you're doing one-shot type work, e.g., returning a custom crafted file for just one user, just once. -Jim On Sep 22, 2011, at 3:27 PM, Peter Sadlon wrote: Just a guess here, but by default, TCL is compiled with Latin-1 encoding. This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset. You could attempt to recompile TCL with --with-encoding utf-8 But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases. Date: Thu, 22 Sep 2011 22:59:02 +0200 From: klaus.hofed...@project-open.com mailto:klaus.hofed...@project-open.com Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars To: AOLSERVER@LISTSERV.AOL.COM mailto:AOLSERVER@LISTSERV.AOL.COM Hi all at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 5.6.0 on CentOS release 5.3 (Final) We suddenly encountered the problem that ns_returnfile can't find a file which filname contains special chars such as 'umlaute' (ä, ö, etc.) --- convmv tells me that the file name is utf-8 encoded. --- Config.tcl has the standard settings: ns_param HackContentType1 ns_param DefaultCharset utf-8 ns_param HttpOpenCharsetutf-8 ns_param OutputCharset utf-8 ns_param URLCharset utf-8 --- Following CentOS locale: LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_ADDRESS=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_ALL= --- File System is 'ext3' --- Files have been created with exec /bin/cp $filename $dest_path out of a tcl file or using 'touch/emacs' on OS level. Same results. --- We did have the problems using AOLserver 3.3 Anybody an idea what the cause might be? Probably its better to always encode file names of uploaded filesto avoid trouble when creating multi-plattform applications. Thanks for your support! Klaus -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com mailto:lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com mailto:lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
