Re: Parsing a UTF-16LE file line by line, BUG?

[Nestor via Digitalmars-d-learn]

On Friday, 6 January 2017 at 11:42:17 UTC, Mike Wey wrote:
On 01/06/2017 11:33 AM, pineapple wrote:
On Friday, 6 January 2017 at 06:24:12 UTC, rumbu wrote:

I'm not sure if this works quite as intended, but I was at 
least able
to produce a UTF-16 decode error rather than a UTF-8 decode 
error by
setting the file orientation before reading it.

    import std.stdio;
    import core.stdc.wchar_ : fwide;
    void main(){
        auto file = File("UTF-16LE encoded file.txt");
        fwide(file.getFP(), 1);
        foreach(line; file.byLine){
            writeln(file.readln);
        }
    }

fwide is not implemented in Windows:
https://msdn.microsoft.com/en-us/library/aa985619.aspx

That's odd. It was on Windows 7 64-bit that I put together and 
tested
that example, and calling fwide definitely had an effect on 
program
behavior.

Are you compiling a 32bit binary? Because in that case you 
would be using the digital mars c runtime which might have an 
implementation for fwide.

After some testing I realized that byLine was not the one 
failing, but any string manipulation done to the obtained line. 
Compile the following example with and without -debug and run to 
see what I mean:

import std.stdio, std.string;

enum
  EXIT_SUCCESS = 0,
  EXIT_FAILURE = 1;

int main() {
  version(Windows) {
    import core.sys.windows.wincon;
    SetConsoleOutputCP(65001);
  }
  auto f = File("utf16le.txt", "r");
  foreach (line; f.byLine()) try {
    string s;
    debug s = cast(string)strip(line); // this is the one causing 
problems
    if (1 > s.length) continue;
    writeln(s);
  } catch(Exception e) {
    writefln("Error. %s\nFile \"%s\", line %s.", e.msg, e.file, 
e.line);
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}