-Original Message-
From: Shelley Doljack [mailto:sdolj...@stanford.edu]
Sent: 31 July 2012 20:18
The problem was I wasn't telling perl to output UTF-8. Now that I added
binmode(FILE, ':utf8') to my script, the problem is fixed. However, it sounds
like once I set binmode to UTF-8 everything will be interpreted as such, even
when the record is in MARC-8. Is that right? So this means that I can only use
my script with a file of records where all of them are encoded in UTF-8. If I
want to run the script against a file with all MARC-8 encoding, then I'd need
to remove the binmode line.
It depends how much manipulation of the records you are doing in the script.
One approach is to use
binmode(FILE, ':raw');
for both input and output. Perl will then keep the bytes of the records
exactly as they are. You won't be able to test for exotic characters so
easily, and amending field content would be inadvisable, but if all you are
doing is something like reading in the records and filtering out any that have
no 245 field, or something fairly basic like that, this could be the best
approach.
The MARC::Record module does not seem to care how the records are encoded.
It's only once you start altering field content, testing field content, or
adding fields that the character set being used becomes an issue. Removing
fields would be fine too.
MARC-8 can be very complex, particularly if other code tables like CJK are
invoked, or even just Greek or Cyrillic. If you were manipulating field
content in that kind of way they converting everything to UTF-8 would make
things very much easier.
Matthew
--
Matthew Phillips
Electronic Systems Librarian, Durham University
Durham University Library, Stockton Road, Durham, DH1 3LY
+44 (0)191 334 2941
