Re: incantation for uploading shift-jis files
At 9:10 am +0900 26/12/05, Joel Rees wrote: I note your are running both the script and the HTML in Unicode UTF-8. There is wisdom in that, of course, and I may rethink my choice of running these scripts in shift-JIS. (I like to avoid conversions that require tables and context decisions as much as possible.) I simply hate legacy encodings. Having worked with Chinese on the Mac since system 6, I would have switched to Windows NT if Apple had not finally implemented Unicode in Mac OS 10. Having waited so long for Unicode, I use nothing else now. I also notice you are saving the file back to disk so you can re-open it as shift-JIS. I want to avoid that, since perl is already saving it once to a temporary directory anyway... Yes, my script was simply a very badly implemented proof of concept. Having looked at the question more deeply, I'm now half way through doing a proper job making proper use of CGI.pm. (Muttering to self -- can perl open strings as streams like Java?) Yes. I have now had a better look at CGI.pm and come up with a solution that works, I think, as you want it. The curious thing is that some browsers had no difficulty even before I used Encode-from_to. To see the script go to http://bd8.com/temp/ and view uploadj.pl.txt To try it out go to: http://ccgi.bd8.com/cgi-bin/uploadj.pl At 2:27 am + 26/12/05, John Delacour wrote: Do you know of a way to tell perl, or, rather, the CGI module to open the file handle as shift-JIS? open F, :encoding(shift_jis), $f Clearly the Christmas goose had affected my reason! As you were about to tell me, CGI.pm has already opened the file handle; so if you want to avoid writing the contents to another file and then opening that as above, it is necessary to use Encode as shown http://bd8.com/temp/uploadj.pl.txt JD
Re: incantation for uploading shift-jis files
Do you know of a way to tell perl, or, rather, the CGI module to open
the file handle as shift-JIS?
open F, :encoding(shift_jis), $f
Okay, okay, I see I have shown my laziness.
However, I have just dug into chapter 8 of the Cookbook (Second
edition) and found a couple of relevant bits, and
use open IO = :raw :encoding(Shift_JIS);
at the top of the script, before the use encoding( 'Shift_JIS' ); and
binmode( $fh, :raw :encoding(Shift_JIS) );
immediately after grabbing the file handle with $fh = $query-upload(
'file-to-send' ) do not help. They do seem to alter the error messages
in apache's error_log.
So, here's the error messages, and the output where I should be getting
空欄を埋めたり、完全な文書で質問に答えたり、一番適切に思う解答を〇で記したりする。
Error messages with neither of the above attempts to push the io layers
around:
---errors--
Wide character in print at (eval 10) line 85.
\x{fffd} does not map to shiftjis at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 281,
fh1Sample_questions.text line 38.
-
with the second line repeated six times.
---output--
遨ｺ谺\x{fffd}繧貞沂繧√◆繧翫\x{fffd}∝ｮ悟\x{fffd}ｨ縺ｪ譁\x{fffd}譖ｸ縺ｧ雉ｪ蝠上↓遲斐∴縺溘ｊ縲∽ｸ
\x{fffd}逡ｪ驕ｩ蛻\x{fffd}縺ｫ諤昴≧隗｣遲斐ｒ縲\x{fffd}縺ｧ險倥＠縺溘ｊ縺吶ｋ縲
-
With the 'use open IO = :raw :encoding(Shift_JIS);' uncommented, the
result:
---errors--
Wide character in print at (eval 11) line 85.
\x{fffd} does not map to shiftjis at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 281,
fh1Sample_questions.text line 38.
-
again, with the second line repeated six times.
---output--
遨ｺ谺\x{fffd}繧貞沂繧√◆繧翫\x{fffd}∝ｮ悟\x{fffd}ｨ縺ｪ譁\x{fffd}譖ｸ縺ｧ雉ｪ蝠上↓遲斐∴縺溘ｊ縲∽ｸ
\x{fffd}逡ｪ驕ｩ蛻\x{fffd}縺ｫ諤昴≧隗｣遲斐ｒ縲\x{fffd}縺ｧ險倥＠縺溘ｊ縺吶ｋ縲
-
Almost exactly the same. This is also the results when I use binmode
raw on the file handle for the upload.
With the use open IO ... commented out and the 'binmode( $fh, :raw
:encoding(Shift_JIS) );' uncommented, the results are
---errors--
Wide character in print at (eval 10) line 85.
shiftjis \x84 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x80 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x85 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x87 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x80 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x87 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x87 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x82 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
-
---output--
遨ｺ谺\x84繧貞沂繧√◆繧翫\x80∝ｮ悟\x85ｨ縺ｪ譁\x87譖ｸ縺ｧ雉ｪ蝠上↓遲斐∴縺溘ｊ縲∽ｸ\x80逡ｪ驕ｩ蛻\x87縺ｫ諤昴≧
隗｣遲斐ｒ縲\x87縺ｧ險倥＠縺溘ｊ縺吶ｋ縲\x82
-
And, with both uncommented, the results are
---errors--
Wide character in print at (eval 11) line 85.
shiftjis \x84 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x80 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x85 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x87 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x80 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x87 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x87 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
shiftjis \x82 does not map to Unicode at
/Volumes/web/userweb/joel/cgi/cs/ranbunhyou/withfile2 line 212.
-
---output--
遨ｺ谺\x84繧貞沂繧√◆繧翫\x80∝ｮ悟\x85ｨ縺ｪ譁\x87譖ｸ縺ｧ雉ｪ蝠上↓遲斐∴縺溘ｊ縲∽ｸ\x80逡ｪ驕ｩ蛻\x87縺ｫ諤昴≧
隗｣遲斐ｒ縲\x87縺ｧ險倥＠縺溘ｊ縺吶ｋ縲\x82
-
Looks like I'll need to get a hexdump from perl to get a better handle
on this, or perhaps try the script on a Linux box, but I've got to get
some sleep tonight. Taking the kids to the doc and to piano practice
didn't leave much time to work on it today.
I'm suspecting the problem lies in the incomplete implementation of
locales in Jaguar, or perhaps in my choice of locales when I compiled
perl on this box.
Re: incantation for uploading shift-jis files
At 12:00 am +0900 27/12/05, Joel Rees wrote:
I'll have to tell you a war story or two, sometime.
Unicode is a kludge. It's one of the better kludges, and evidence
that kludges make the world go round...
Just as well it doesn't rely on iso-2022-jp or us-ascii. It's not
Unicode that is the kludge; Unicode is simply the assignment of a
unique character to a large range of numbers rather than the
assignment of an arbitrary number of characters to a range any
American president can conceive of. The present temporary problems
with Unicode arise only from a long anarchic heritage of monumental
kludges.
...The frustrating thing about this is that I've been here before,
about three years back when the perl implementation wasn't quite as
complete, but I can't remember what I did, and I don't have access
to the code I built then anymore.
I have the same problem again and again with a mere hour's interval!
The script below reduces the problem to its simplest. Notice the
deadly caveats. In my experience (and I have war stories too) the
harder one tries with Perl/Unicode the worse the mess you get into.
You can probably forget about locale -- try “use encoding
(:locale)” in the script below and see what you get! -- and lots
of other things. It's certainly a jungle, and it's growing, but it's
getting tidier.
#!/usr/bin/perl
#
# In BBEdit/TextWrangler set this document's
# encoding to Japanese (Shift JIS); always open/reopen
# as Japanese (Shift JIS).
#
# In BBEdit/TextWrangler Preferences/Unix Scripting
# check “use UTF-8” for Unix Script I/O.
#
# When running in Terminal set Window Settings...
# [Display] [Character Set Encoding] to “Unicode (UTF-8)”.
#
### use utf8; # NO !!
# no encoding; # OK, optional
# binmode STDOUT, UTF-8; # OK, optional
### binmode STDOUT, :utf8; ### NO !! Quite different !!
use Encode qw~from_to~;
while (DATA) { /^#/ and next;
from_to ($_, Shift_JIS, utf8);
print
}
__DATA__
# Must not contain non-Shift_JIS characters
空欄を埋めたり、完全な文書で質問に答えたり、
一番適切に思う解答を〇で記したりする。
##
Re: incantation for uploading shift-jis files
The script below reduces the problem to its simplest. Notice the
deadly caveats. In my experience (and I have war stories too) the
harder one tries with Perl/Unicode the worse the mess you get into.
You can probably forget about locale -- try “use encoding (:locale)”
in the script below and see what you get! -- and lots of other things.
It's certainly a jungle, and it's growing, but it's getting tidier.
#!/usr/bin/perl
#
# In BBEdit/TextWrangler set this document's
# encoding to Japanese (Shift JIS); always open/reopen
# as Japanese (Shift JIS).
#
# In BBEdit/TextWrangler Preferences/Unix Scripting
# check “use UTF-8” for Unix Script I/O.
#
# When running in Terminal set Window Settings...
# [Display] [Character Set Encoding] to “Unicode (UTF-8)”.
#
### use utf8; # NO !!
# no encoding; # OK, optional
# binmode STDOUT, UTF-8; # OK, optional
### binmode STDOUT, :utf8; ### NO !! Quite different !!
use Encode qw~from_to~;
while (DATA) { /^#/ and next;
from_to ($_, Shift_JIS, utf8);
print
}
__DATA__
# Must not contain non-Shift_JIS characters
空欄を埋めたり、完全な文書で質問に答えたり、
一番適切に思う解答を〇で記したりする。
##
That's a nice little script to have on the list, for reference.
Now, as far as my little problem goes, I was able to get some success
with the following:
-snippet--
use encoding( 'Shift_JIS' );
...
my $query = new CGI;
...
my $fileToSend = $query-param( 'file-to-send' );
my $FileSent = $query-param( 'FileSent' );
...
elsif ( $FileSent )
{
my $fh;
if ( !defined( $fileToSend ) || length( $fileToSend ) 1 || !( $fh
= $query-upload( 'file-to-send' ) ) )
{ print $query-header(-status=$error),
$query-start_html( 'Bad request' ),
$query-h2( 'Failed to find or open file, maybe bad
file name selected.' ),
$query-strong( Upload request for $fileToSend not
processed. );
exit 0;
}
my $type = $query-uploadInfo( $fileToSend )-{ 'Content-Type' };
if ( $type ne 'text/plain' )
{ print $query-header(-status=$error),
$query-start_html( 'Bad file type' ),
$query-h2( 'File type must be plain text.' ),
$query-strong( 'Request not processed.' );
exit 0;
}
# One line at a time is STILL not safe if length not already
checked.
# Doing this one line at a time to handle the shift JIS problem,
somehow.
my @fileLines = ();
my $line = '';
# binmode( $fh, :raw :encoding(Shift_JIS) );
binmode( $fh, :raw :utf8 ); # As best as I understand, this
should be wrong.
# binmode( $fh, :raw );
while ( $line = $fh )
{
my @hexdump = unpack( 'C256', $line ); # debug
my $hexdumpstring = ''; # debug
foreach my $byte ( @hexdump ) # debug
{ $hexdumpstring .= sprintf( '%02x ', $byte );# debug YUCK!
} # debug
push( @fileLines, $line );
push( @fileLines, $hexdumpstring . \n );# debug
}
@words = @fileLines;
...
---end-snippet
This is in spite of the headers, the XML declaration, and the HTML
header meta declaration all declaring the document to be shift-JIS, and
the source itself declaring use encoding( 'Shift_JIS' );. I should
probably expect that I muffed it when I compiled perl, but I'll need to
push the whole thing onto my Linux/BSD box, bring up apache over there,
and compare notes to have a decent idea what's going on.
In the meantime, Firefox on Linux is no longer uploading the file at
all.
Joel
Re: incantation for uploading shift-jis files
erm I think I forgot to point out what I changed --
Now, as far as my little problem goes, I was able to get some success
with the following:
-snippet--
use encoding( 'Shift_JIS' );
...
my $query = new CGI;
...
my $fileToSend = $query-param( 'file-to-send' );
my $FileSent = $query-param( 'FileSent' );
...
elsif ( $FileSent )
{
my $fh;
if ( !defined( $fileToSend ) || length( $fileToSend ) 1 || !(
$fh = $query-upload( 'file-to-send' ) ) )
{ print $query-header(-status=$error),
$query-start_html( 'Bad request' ),
$query-h2( 'Failed to find or open file, maybe bad
file name selected.' ),
$query-strong( Upload request for $fileToSend not
processed. );
exit 0;
}
my $type = $query-uploadInfo( $fileToSend )-{ 'Content-Type' };
if ( $type ne 'text/plain' )
{ print $query-header(-status=$error),
$query-start_html( 'Bad file type' ),
$query-h2( 'File type must be plain text.' ),
$query-strong( 'Request not processed.' );
exit 0;
}
# One line at a time is STILL not safe if length not already
checked.
# Doing this one line at a time to handle the shift JIS problem,
somehow.
my @fileLines = ();
my $line = '';
# binmode( $fh, :raw :encoding(Shift_JIS) );
This is what seems to get it to upload from the iBook to itself:
binmode( $fh, :raw :utf8 ); # As best as I understand, this
should be wrong.
The debug stuff below didn't really tell me much beyond that it was
already not shift-jis by the time the script was reading it from the
CGI upload function.
# binmode( $fh, :raw );
while ( $line = $fh )
{
my @hexdump = unpack( 'C256', $line ); # debug
my $hexdumpstring = ''; # debug
foreach my $byte ( @hexdump ) # debug
{ $hexdumpstring .= sprintf( '%02x ', $byte );# debug YUCK!
} # debug
push( @fileLines, $line );
push( @fileLines, $hexdumpstring . \n );# debug
}
@words = @fileLines;
...
---end-snippet
This is in spite of the headers, the XML declaration, and the HTML
header meta declaration all declaring the document to be shift-JIS,
and the source itself declaring use encoding( 'Shift_JIS' );. I
should probably expect that I muffed it when I compiled perl, but I'll
need to push the whole thing onto my Linux/BSD box, bring up apache
over there, and compare notes to have a decent idea what's going on.
In the meantime, Firefox on Linux is no longer uploading the file at
all.
Joel
Re: incantation for uploading shift-jis files
On 2005.12.25, at 10:58 AM, John Delacour wrote: At 1:45 am + 25/12/05, John Delacour wrote: The script is here, warts and all: http://bd8.com/temp/upload.pl.txt I note your are running both the script and the HTML in Unicode UTF-8. There is wisdom in that, of course, and I may rethink my choice of running these scripts in shift-JIS. (I like to avoid conversions that require tables and context decisions as much as possible.) I also notice you are saving the file back to disk so you can re-open it as shift-JIS. I want to avoid that, since perl is already saving it once to a temporary directory anyway, and since I hope the guy who wrote that code covered more corners than I have time to think of. (Muttering to self -- can perl open strings as streams like Java?) NB. Safari doesn't treat it as as text file. Didn't mention this before, but if you move the header here doc down far enough (put it in a sub, declare the sub after all the rest of the script), you can usually get Safari to behave itself. It's disappointing to discover this old bug in Safari, and it's disappointing that it's still there in Safari 2. (Didn't really intend that comment to go off-list, particularly since it actually is on topic for this list,) FireFox, Opera, Omniweb display it properly as text. If Omniweb isn't doing this, the problem would not seem to be in the KHTML base. Of course, KDE just passes the view off to the selected text editor, so the code to have the problem probably wouldn't be in the base anyway. (I like Omniweb, but I do have good reasons for using Safari on Jaguar, mostly financial -- no room to put it since I had to put the old 5.6G drive back in, no money for new hardware.) Joel Rees
Re: incantation for uploading shift-jis files
I think that the problem is that I have set the encoding (multi-part) for the post, but not for the file part, and I can't figure out how to set the encoding for the file part. I'm worried that I'm going to have to force a re-conversion within perl from some sort of automatic conversion done when writing to the temporary file. I do not think there is any way to tell the web browser what the encoding of the uploaded file should be. That only works for form fields such as text areas. So it's not just me that thinks so. Thanks. You will get the file in the same encoding (if applicable, this could be a binary file, such as an image, too) that it has on the user's hard disk. That doesn't seem to match some run-time error messages about codes that could not be converted. I'll have to check those messages again, there's probably a clue in there. So you will have to auto-detect the encoding on the server-side or give the user a pulldown to select the file encoding (or support only Shift-JIS, which you might get away with in your case). Do you know of a way to tell perl, or, rather, the CGI module to open the file handle as shift-JIS? Here's where I get the file handle: if ( !defined( $fileToSend ) || length( $fileToSend ) 1 || !( $fh = $query-upload( 'file-to-send' ) ) ) I'm not seeing any room in the syntax here: $fh = $query-upload( 'file-to-send' ), but I'm SUCH a newb. (Seriously. If I weren't bald I should be blonde.) Merry Christmas, Merry late Christmas and happy holidays to all! Joel Thilo
Re: incantation for uploading shift-jis files
At 9:22 am +0900 26/12/05, Joel Rees wrote: Do you know of a way to tell perl, or, rather, the CGI module to open the file handle as shift-JIS? open F, :encoding(shift_jis), $f JD
incantation for uploading shift-jis files
Anyone know? I've looked around on the web, and it looks like I'm playing with edge-of-the-world stuff and rather OS and browser dependent. The source I'm working with: http://reiisi.homedns.org/~joel/cs/ranbunhyou/withfile2.text http://reiisi.homedns.org/~joel/cs/ranbunhyou/requester2.html Both have shift-JIS mixed in, but everything in Japanese is duplicated in English. The form tag: form action=http://reiisi.homedns.org/~joel/cs/ranbunhyou/cgi/withfile2; method=post enctype=multipart/form-data; charset=Shift_JIS accept-charset=Shift_JIS The charset in the enctype seems to make the textarea field work intermittently, the accept-charset makes the textarea field work reliably independent of the charset in the enctype. The charset reported by the debug print always comes out ISO-8859-1 even when it works, so I think that's a red-herring. The test file that contains Japanese to show the problem is Sample_questions.text. The linked file works fine, The contents of the linked file pasted into the textarea works fine. It's just the file upload that does not. (The reason I'm using shift-jis is that the bulk of Japanese documents remain in shift-jis, and most people operate by default in the encoding that the bulk of their files are stored in.) My partial explanation of what the code is doing is at http://reiisi.homedns.org/~joel/cs/ranbunhyou/ (And, yeah, I know I'm using some archaic techniques. It's what I know or can find in my old references, and/or what I hope will run with modules in the standard distribution. Criticism welcome.) joel rees
Re: incantation for uploading shift-jis files
At 7:10 pm +0900 24/12/05, Joel Rees wrote: I've looked around on the web, and it looks like I'm playing with edge-of-the-world stuff and rather OS and browser dependent. The source I'm working with: http://reiisi.homedns.org/~joel/cs/ranbunhyou/withfile2.text http://reiisi.homedns.org/~joel/cs/ranbunhyou/requester2.html Can you reduce the problem to the bare minimum rather than requiring us to plough through the whole thing? When I try to Send File, I get : Failed to find or open file, maybe bad file name selected. : Upload request not processed. I don't think it is edge-of-the-world stuff but I like simplified problems without noise. JD
Re: incantation for uploading shift-jis files
On 2005.12.24, at 09:24 PM, John Delacour wrote: At 7:10 pm +0900 24/12/05, Joel Rees wrote: I've looked around on the web, and it looks like I'm playing with edge-of-the-world stuff and rather OS and browser dependent. The source I'm working with: http://reiisi.homedns.org/~joel/cs/ranbunhyou/withfile2.text http://reiisi.homedns.org/~joel/cs/ranbunhyou/requester2.html Can you reduce the problem to the bare minimum rather than requiring us to plough through the whole thing? I'll see if I can work something simple up. (Now that you mention it, that's what I should have done before posting, since simplification generally reveals simple problems.) But I'm going to take a break for Sunday, try to spend time with the family. When I try to Send File, I get : Failed to find or open file, maybe bad file name selected. : Upload request not processed. Did you select a file before hitting send? I'm new to file uploading, so I have no idea if I'm tripping over something basic here. Maybe something as simple as telling the web browser to send the thing as binary instead of text. (If I can dig that back up.) I think that the problem is that I have set the encoding (multi-part) for the post, but not for the file part, and I can't figure out how to set the encoding for the file part. I'm worried that I'm going to have to force a re-conversion within perl from some sort of automatic conversion done when writing to the temporary file. I don't think it is edge-of-the-world stuff but I like simplified problems without noise. One can always hope. ;-/
Re: incantation for uploading shift-jis files
At 04:17 PM 12/24/2005, Joel Rees wrote: I think that the problem is that I have set the encoding (multi-part) for the post, but not for the file part, and I can't figure out how to set the encoding for the file part. Encoding is set as a MIME header on each MIME part - or can be. -jeff
Re: incantation for uploading shift-jis files
I think that the problem is that I have set the encoding (multi-part) for the post, but not for the file part, and I can't figure out how to set the encoding for the file part. I'm worried that I'm going to have to force a re-conversion within perl from some sort of automatic conversion done when writing to the temporary file. I do not think there is any way to tell the web browser what the encoding of the uploaded file should be. That only works for form fields such as text areas. You will get the file in the same encoding (if applicable, this could be a binary file, such as an image, too) that it has on the user's hard disk. So you will have to auto-detect the encoding on the server-side or give the user a pulldown to select the file encoding (or support only Shift-JIS, which you might get away with in your case). Merry Christmas, Thilo
Re: incantation for uploading shift-jis files
At 10:30 am +0900 25/12/05, Thilo Planz wrote: So you will have to auto-detect the encoding on the server-side or give the user a pulldown to select the file encoding (or support only Shift-JIS, which you might get away with in your case). Here's an example that deals with sjis text files and us-ascii: http://ccgi.bd8.com/cgi-bin/upload.pl The script is here, warts and all: http://bd8.com/temp/upload.pl.txt Merry Christmas, Thilo And to everyone! JD
Re: incantation for uploading shift-jis files
At 1:45 am + 25/12/05, John Delacour wrote: The script is here, warts and all: http://bd8.com/temp/upload.pl.txt NB. Safari doesn't treat it as as text file. FireFox, Opera, Omniweb display it properly as text. JD
