On Wednesday, 5 August 2020 at 17:39:36 UTC, Mike Surette wrote:
In my efforts to learn D I am writing some code to read files
in different UTF encodings with the aim of having them end up
as UTF-8 internally. As a start I have the following code:
import std.stdio;
import std.file;
void main(string[] args)
{
if (args.length == 2)
{
if (args[1].exists && args[1].isFile)
{
auto f = File(args[1]);
writeln(args[1]);
for (auto i = 1; i <= 3; ++i)
write(f.readln);
}
}
}
It works well outputting the file name and first three lines of
the file properly, without any regard to the encoding of the
file. The exception to this is if the file is UTF-16, with both
LE and BE encodings, two characters representing the BOM are
printed.
I assume that write detects the encoding of the string returned
by readln and prints it correctly rather than readln reading in
as a consistent encoding. Is this correct?
Is there a way to remove the BOM from the input buffer and
still know the encoding of the file?
Is there a D idiomatic way to do what I want to do?
Mike
all strings in D are _assumed_ to be UTF-8, so your I/O reading
function needs to check that it is actually UTF-8.
File/File.readln does not do that, so you are actually getting
UTF-16 bytes in your string, not UTF-8 bytes.
What you are seeing through writeln is not fully correct: If you
only test English characters there are null bytes (0) before each
english character byte, which aren't rendered in the console.
You can verify this with this simple code:
auto s = File("test.txt").readln;
writefln("%(%02x %)", s.representation);
result: ff fe 68 00 65 00 6c 00 6c 00 6f 00 20 00 77 00 6f 00 72
00 6c 00 64 00 21 00 0d 00 0a
You can see there is a UTF-16 LE BOM in there and then all the
English characters, which are null bytes and the characters.
Basically what you want to do is writing a function converting
the input encoding to UTF-8 so you can use it in D strings. If
you want to get into this yourself, there is std.encoding which
offers you most of the functionality:
https://dlang.org/phobos/std_encoding.html
You can use the getBOM method to try to determine encoding by BOM
and can then convert it to UTF-8 from the source encoding using
the `transcode` method or manually using the encoding classes.
If you don't have a BOM you need some kind of algorithm to
determine the encoding of your file. If you don't want to do that
and just want to check if it's UTF-8, use the `std.utf :
validate` function which throws if your string is not valid UTF-8
and otherwise does nothing.
If you only support UTF-8, UTF-16 and possibly UTF-32, you can
use just the std.utf methods after determining BOM to lazily
convert without allocating memory (useful if you only go over
your string linearly without going back):
import std;
void main() {
// readln is actually rather unsafe for this! You should use
std.file.read
// or File.rawRead or read byte chunks instead. For chunking you
need to adjust
// the encode API however and probably make a helper struct with
a small buffer.
string s = File("test.txt").readln;
// need to remove the line terminator before encoding
// (it's encoded in UTF-8, potentially after UTF-16)
if (s.length) s = s[0 .. $ - 1];
string e = encodeUTF8(s.representation);
writefln("%s\n(%(%02x %))", s, s.representation);
writefln("%s\n(%(%02x %))", e, e.representation);
}
string encodeUTF8(immutable(ubyte)[] bytes) {
auto bom = getBOM(bytes);
bytes = bytes[bom.sequence.length .. $];
switch (bom.schema) {
// optionally we could validate, but we just trust because there
is a UTF-8 BOM
case BOM.utf8: return cast(string)bytes;
case BOM.utf16le: return convertUTF!wchar(bytes, true);
case BOM.utf16be: return convertUTF!wchar(bytes, false);
case BOM.utf32le: return convertUTF!dchar(bytes, true);
case BOM.utf32be: return convertUTF!dchar(bytes, false);
default: string input = cast(string)bytes; validate(input);
return input;
}
}
private string convertUTF(T)(scope const(ubyte)[] bytes, bool
littleEndian) {
// T.sizeof expecting 2 or 4 (which divided maps to 0 and 1)
enum name = ["UTF-16", "UTF-32"][T.sizeof / 4];
alias Int = AliasSeq!(ushort, uint)[T.sizeof / 4];
if (bytes.length % T.sizeof != 0)
throw new Exception("File is " ~ name ~ ", but got "
~ bytes.length.to!string ~ " bytes, which is not a multiple of
"
~ T.sizeof.stringof ~ "!");
scope Int[] units = (cast(Int*) bytes.ptr)[0 .. bytes.length /
T.sizeof];
// swap mismatching endianness
version (LittleEndian) // CPU is little endian, swap if file is
big endian
bool swap = !littleEndian;
else // CPU is big endian, swap if file is little endian
bool swap = littleEndian;
if (swap) swapAllEndian(units);
scope wstr = cast(const(T)[]) units;
auto ret = wstr.toUTF8;
// because we are operating in-place, we need to revert to keep
memory consistent
// if you don't use the byte data anywhere else, you could omit
this
// (note this could be unsafe though)
if (swap) swapAllEndian(units);
return ret;
}
private void swapAllEndian(T)(T[] data) {
// TODO: could probably optimize this with SIMD instructions
foreach (i; 0 .. data.length)
data[i] = swapEndian(data[i]);
}
Example library if you want to guess encoding without BOM:
https://code.dlang.org/packages/libguess-d
used API docs:
https://dlang.org/phobos/std_bitmanip.html#swapEndian <- swapping
BE/LE for native encoding
https://dlang.org/phobos/std_encoding.html <- BOM detection,
transcoding capabilities for encodings other than UTF
https://dlang.org/phobos/std_utf.html <- low level UTF-8
encoding/decoding, lazy decoding, validation
