Hi,
I will have to rewrite the core part of a Configuration Management System in
order to make it Unicode aware.
The system must be able to treat plain ASCII (as now), UTF-8 and UTF-16
(Windows NLS) files.
As Perl is using UTF-8 internally the problem is reduced to read data into Perl
and treat everything using UTF-8 and
convert the output to ASCII or UTF-16 only when needed and possible.
So I started to print out and read PERL documentation on : Encode ,
Encode::Unicode, utf8 pragma, Unicode::String
and some Unicode FAQ.
Then I wrote a simple test script to read in a file in plain ASCII and convert
it to UTF-8 and UTF-16 and I ran into the
following problem: The underlying PERLIO does not correctly convert to UTF-16.
I updated ActivePerl from Build 810 to 811, because Encode has been renewed,
but nothing changed.
System : P4 running Windows 2000 SP4, ActiveState Perl 5.8.6 Build 811
Detailed description of the problem:
Plain ASCII text file containing several lines, edited under Windows and stored
having DOS line endings (CRLF).
open / read using PERLIO will eliminate the CR (see test script)
write UTF-8/UTF-16 using PERLIO tries to add CR but fails on UTF-16:
\n is replaced by \x00 \x0d \x0a instead of \x00 \x0d \x00 \0a.
I can't imagine that this is wanted, since it produces illegal UTF-16.
I've converted the text file to several UTF-Formats using my editor (Visual
Slick) with DOS and Unix line endings and the
result seems good. \x0d \x0a is converted to \x00 \x0d \x00 \x0a.
Am I missing something or is this a real bug?
Regards,
Axel Mock
Here' s the test script which reproduces the error:
Prerequisites:
Create a plain text file with CRLF line endings and name it 'master.tx_'.
Run ActivePerl 5.8.6 Build 810 or 811 on Windows.
use Encode;
use strict;
use Unicode::String qw (utf8 latin1 utf16);
sub readFile {
my ($filename, $encoding) = @_;
unless ($encoding) {
$encoding = 'iso-8859-1';
}
open (my $file, "<:encoding($encoding)", $filename);
my (@content) = <$file>;
close ($file);
if (wantarray ()) {
return (@content)
}
else {
return (join ('',@content));
}
}
sub writeFile {
my ($filename, $encoding, $pContent) = @_;
my ($type) = ref ($pContent);
unless ($type =~ /ARRAY|SCALAR/) {
die ("WRITEFILE: No method to encode : $type\n");
}
open (my $file, ">:encoding($encoding)", $filename);
if ($type eq 'ARRAY') {
print $file @{$pContent};
}
else {
print $file $$pContent;
}
close ($file);
}
sub writeFile2 {
my ($filename, $utfobj, $utftype) = @_;
my ($content);
my ($BOM16)= "\xFE"."\xFF";
open (FILE, ">$filename");
binmode (FILE);
Unicode::String->stringify_as($utftype);
if ($utftype eq 'utf16') {
$content = $BOM16;
$content .= $utfobj->as_string();
}
else {
$content = $utfobj->as_string();
}
print FILE $content;
close (FILE);
}
#------------------- M A I N
--------------------------------------------------------
my ($content);
# File master.tx_ is a plain ASCII file consisting of several lines
# File has been editied under Windows and saved with DOS line endings
# (CRLF) \r\n = \x0d\x0a
# This PERL script should be running under Windows
# using ActivePerl 5.8.6 Build 810 or 811
# first read the file using PERLIO
$content = readFile ('master.tx_');
# and dump it to a file: result will be file with UNIX line endings (\n=0x0a
only)
open (DEBUG, '>debug.dmp');
binmode (DEBUG);
print DEBUG $content;
close (DEBUG);
# Now convert and write the content in UTF-8 and UTF-16 using PERLIO
# UTF-8 file will have CRLF line endings and is OK
writeFile ('test.8ut_', 'utf8', \$content);
# UTF-16 file will have erroneous CRLF line endings and is illegal
# \n=\x0a is replaced with \x00\x0d\x0a instead of \x00\x0d\x00\x0a
writeFile ('test.16ut_', 'UTF-16', \$content);
# next try using Unicode::String
my ($u) = utf8 ($content);
# UTF-8 file is OK
writeFile2 ('test2.8ut_', $u, 'utf8');
# UTF-16 file is OK but has UNIX line endings \x00\x0a only
writeFile2 ('test2.16ut_', $u, 'utf16');
__END__
----------------------------------------------------------------------
S y s K o n n e c t G m b H
A Marvell Company
Siemensstr. 23
D-76275 Ettlingen
----------------------------------------------------------------------
Axel Mock
Software Engineer
phone: +49 7243 502 319
fax: +49 7243 502 931
email: [EMAIL PROTECTED]
http://www.syskonnect.de
-----------------------------------------------------------------------
_______________________________________________
ActivePerl mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
