On 4/18/2012 12:08 PM, Jonathan Rochkind wrote:
On 4/18/2012 11:09 AM, Doran, Michael D wrote:
I don't believe that is the case. Take UTF-8 out of the picture, and
consider the MARC-8 character set with its escape sequences and
combining characters. A character such as an "n" with a tilde would
consist of two bytes. The Greek small letter alpha, if invoked in
accordance with ANSI X3.41, would consist of five bytes (two bytes
for the initial escape sequence, a byte for the character, and then
two bytes for the escape sequence returning to the default character
set).
ISO 2709 doesn't care how many bytes your characters are. The
directory and offsets and other things count bytes, not characters.
(which was, in my opinion, the _right_ decision, for once with marc!)
How bytes translate into characters is not a concern of ISO 2709.
The majority of non-7-bit-ASCII encodings will have chars that are
more than one byte, either sometimes or always. This is true of MARC8
(some chars), UTF8 (some chars), and UTF16 (all chars), all of them.
(It is not true of Latin-1 though, for instance, I don't think).
ISO 2709 doesn't care what char encodings you use, and there's no
standard ISO 2709 way to determine what char encodings are used for
_data_ in the MARC record. ISO 2709 does say that _structural_
elements like field names, subfield names, the directory itself,
seperator chars, etc, all need to be (essentially, over-simplifying)
7-bit-ASCII. The actual data itself is application dependent, 2709
doesn't care, and 2709 doesn't give any standard cross-2709 way to
determine it.
That is my conclusion at the moment, helped by all of you all in this
thread, thanks!
The conclusion that I came to in the work I have done on marc4j (which
is used heavily by SolrMarc) is that for any significant processing of
Marc records the only solution that makes sense is to translate the
record data into Unicode characters as it is being read in. Of course
as you and others have stated, determining what the data actually is, in
order to correctly translate it to Unicode, is no easy task. The leader
byte that merely indicates "is UTF8" or "is not UTF8" is wrong often
enough in the real world that it is of little value when it indicates
"is UTF-8"and is even less value when it indicates "is not UTF-8"
Significant portions of the code I've added to marc4j deal with trying
to determine what the encoding of that data actually is and trying to
translate the data correctly into Unicode even when the data is incorrect.
You also argued in another message that cataloger entry tools should
give feedback to help the cataloger not create errors. I agree. I
think one possible step towards this would be that the editor must work
in Unicode, irrespective of the data format that the underlying system
expects the data to be. If the underlying system expects MARC8 then the
"save as" process should be able to translate the data into MARC8 on
output.
-Robert Haschart
