Re: How to detect textfile format?

Fri, 28 Oct 2011 09:46:21 -0700
The first three bytes of a UTF-8 file are EF BB BF, which is the UTF-8 encoding of the Unicode Byte Order Mark, U+FEFF. If you're seeing FFFE as the first two bytes, there's some sort of translation being done before you get it. Also, UTF-8 uses one byte for values of 127 or less, not 255, and two or more bytes for larger values. 


The only other encoding I know of (besides the little-endian UTF-16 that 
we all know and love) that would have FF for the first byte is 
little-endian UTF-32, and that's not really seen in the wild in the 
Windows ecosystem.



Note, too, that not all Unicode-encoded files begin with a Byte Order 
Mark. There is no 100% reliable way to determine whether an arbitrary 
file is ASCII or Unicode if it doesn't start with a Byte Order Mark. 
Notepad and Window-Eyes use a best-guess heuristic that is occasionally 
wrong; for more details and a link to an example see 
https://blogs.msdn.com/b/oldnewthing/archive/2004/03/24/95235.aspx




On 10/28/2011 12:25 PM, Christo de Klerk wrote:

Hi Chip


The third flavour of text files is UTF8 which seems to be a kind of 
hybrid between Unicode and ANSI. It seems to use one byte for ANSI 
characters (value of aSCII 255 or less) and two bytes for Unicode 
characters with higher values. The first two bytes of UTF8 files are 
hex FFFE.


Kind regards

Christo

On 2011/10/28 1:17 PM, Chip Orange wrote:

David,
there's a third variation of encoding, whose name escapes me at the
moment (although it may be ANSI), but is read with the "unicode" option.
That's why I only test for the first character being "ff" in my example,
as it covers both variations that way.
Chip

------------------------------------------------------------------------
*From:* David [mailto:[email protected]]
*Sent:* Friday, October 28, 2011 12:03 AM
*To:* [email protected]
*Subject:* Re: How to detect textfile format?

Thanks Jeff.
It somehow got up working now. But I had a bit of a hard time, as it
does differ a bit from your code. See, your code states, that the
Hex-number should be FFFE. But on my system, if I want to get it
working, I have to test for a HEX-number of FF0.
Puzzled why, but that is the way it is. Wonder if my script then will
work on other computers, since there could be the difference in the
testing number. Anyone could shed some light on that one?

    ----- Original Message -----
    *From:* Jeff Weiss <mailto:[email protected]>
    *To:* [email protected] <mailto:[email protected]>
    *Sent:* Friday, October 28, 2011 4:53 AM
    *Subject:* Re: How to detect textfile format?


    Here are some lines from one of my apps which should do what you 
need:

    Const ForReading = 1
    Dim FSO : Set FSO = CreateObject("Scripting.FileSystemObject")
    On Error Resume Next
    ' determine whether the file is Unicode or not.
    Dim ObjScriptFile : Set objScriptFile = FSO.OpenTextFile(Path &
    nString & ".vbs")
    Dim firstByte, secondByte
    firstByte = Hex(AscB(MidB(objScriptFile.Read(1), 1, 1)))
    secondByte = Hex(AscB(MidB(objScriptFile.Read(1), 1, 1)))
    If Err.Number > 0 Then
    Mess =Mess & " " & "Error " & Err.Number & " " & Err.Description &
    VbCrLf
    If Err.Number = 424 Then
    Mess = Mess & "File not Found! "
    End If
    On Error goto 0
    Exit Function
    End If
    If firstByte & secondByte = "FFFE" Then
    ' unicode file
    strContents = SOFileToString(Path & nString & ".vbs", "Unicode")
    Else
    ' not unicode
    strContents = SOFileToString(Path & nString & ".vbs", "")
    End If
    objScriptFile.Close
    hth
    Jeff Weiss

        ----- Original Message -----
        *From:* David <mailto:[email protected]>
        *To:* [email protected] <mailto:[email protected]>
        *Sent:* Thursday, October 27, 2011 9:36 PM
        *Subject:* How to detect textfile format?

        In my script, I want to read the information from a text file.
        As it is going to read from any text file on the disk, I have no
        clue whether the actual text file will be an ASCII, or an
        Unicode file. I am using the CreateTextFile method to open the
        text file. This method does require me to determine if I am
        going to open one or the other of the file formats. Is there any
        good work around for this in VBS? Some way for me to open a text
        file, and have it properly read, no matter which of the two
        formats it has been saved in?
         From my testing here, I get the following results. Try to open
        an ASCII file, having CreateTextFile set to UniCode, only gives
        me one line, and no recognized text.
        Try to open an UniCode file, having CreateTextFile set to ASCII,
        will recognize all the lines, but does not work properly when I
        try to perform textual operations; like Mid, and InStr.
        Yep, even computers have their multi-lingual challenges. Smile.
        Anyone has a good idea here, I would be thankful.


























					Reply via email to