Hi guys, to my understanding, current versions of perl have utf-8 support activated by default to recognize utf-8 files and treat them accordingly. Even variables containing utf-8 encoded characters are internally marked and treated as such in matching operations etc. So I believe reopening the file will not be necessary. There are loads of webpages that actually refer to older perl versions that did not have the current level of utf-8 support and therefore propagate all manner of workarounds no longer necessary.
The BOM is permissible, but superfluous on utf-8 files. That is unless they originated in a different encoding and are being altered and returned to the sender. So unless Gedcom.pm intends to be able to export encodings other than ASCII or utf-8, it should be perfectly safe to simply delete/ignore the BOM when it is ASCII- or utf-8-encoded. I personally wouldn't ignore it when encoded otherwise as that might lead to problems later on in processing the actual data. In those cases an error upon processing the BOM should be just what the doctor ordered. I'm not an authority on utf-8, it's simply how I understand http://www.perlmonks.org/?node_id=599720 https://en.wikipedia.org/wiki/Byte_Order_Mark http://www.unicode.org/faq/utf_bom.html#22 applied to this situation. If I should be wrong I would appreciate being corrected as I will be spending some time on the processing of utf-8 gedcom files using Gedcom.pm in the immediate future. Michael On 07.09.2012 02:20, Ron Savage wrote: > Hi Jose > > On 07/09/12 02:06, Jose Joao Dias de Almeida wrote: >> >> >> On 09/05/2012 11:25 PM, Ron Savage wrote: >>> Hi Jose >>> >>> On 05/09/12 22:07, Jose Joao Dias de Almeida wrote: >>>> Dear Gedcom-ers, >>>> I just star with Gedcom.pm and things are beginning to work! >>>> >>>> But I have problems with files in Unicode. >>>> >>>> When files are in utf8 + BOM --> it returns error in first line (the >>>> BOM) >>>> >>>> If I remove the BOM and try again, apparently it does not pay attention >>>> to "1 CHAR UTF-8" >>>> >>>> Is there any extra thing to say in this cases? >>>> Um abraço >>>> J.Joao >>>> >>>> 0 HEAD >>>> 1 GEDC >>>> 2 VERS 5.5 >>>> 2 FORM LINEAGE-LINKED >>>> >>>> 1 LANG Portuguese >>>> 1 SOUR MYHERITAGE >>>> ... >>> >>> I just checked the source code of Gedcom.pm V 1.16, and the only >>> reference to utf8 is on line 390, where it is writing an XML file. >>> >>> So, you're right, of course. >>> >>> What do you think should happen? >>> >>> Perhaps if the code detects 1 CHAR UTF-8 the input file should be closed >>> and re-opened in utf-8 mode, yes? >> >> I think that would solve the problem. >> >> Probably a simple >> if(/1 CHAR UTF-8/){ binmode(...) } >> would also work. > > I believe binmode does not work after the file has been read. That is, > it must be executed immediately after opening the file, before the 1st > read. > > But see File::BOM. Gedcom needs to use that. > > The string 1 CHAR UTF-8 obviously must be read in before knowing the > file is intended to be utf8, so File::BOM won't solve that problem. > >> One extra thing: >> Some of the unicode files sometime include the initial byte order marker >> (BOM) in order to sign the unicode format used. >> >> Bytes Encoding Form >> 00 00 FE FF UTF-32, big-endian >> FF FE 00 00 UTF-32, little-endian >> FE FF UTF-16, big-endian >> FF FE UTF-16, little-endian >> EF BB BF UTF-8 >> >> it would be nice if we could at least skip/ignore them (for exemple >> Myheritage tools are generating gedcom files with BOMs) >> >> eg: >> >> $ged =~ >> s/^(\x00\x00\xFE\xFF|\xFF\xFE\x00\x00|\xFF\xFE|\xFE\xFF|\xEF\xBB\xBF)//; >> ## remove BOM ! >> >> Um abraço >> J.Joao >>
