Hi everyone,

Just providing an update on this issue. As you may recall, I've
been putting the MARC::Record suite, specifically MARC::File::XML
and MARC::Charset, through some fairly rigourous tests, including
a 'roundtrip' test, which converts the binary MARC-8 records to
MARCXML / UTF-8 and then back to binary MARC but encoded as UTF-8.
This test is available here:


I discovered a number of bugs or issues, not in the MARC::* stuff, but in the
back-end SAX parsers. I'll just summarize my discoveries here for 

1. MARC::File::XML, if it encounteres unmapped encoding in a
MARC-8 encoded binary MARC file (in as_xml()) will drop the entire
subfield where the improper encoding exists. The simple solution is
to always use: MARC::Charset->ignore_errors(1); if you expect your 
data will have improper encoding.

2. the XML::SAX::PurePerl parser cannot properly handle combining
characters. I've reported this bug here:


At the suggestion of several, I tried replacing my default system
parser with expat, which cause another problem:

3. handing valid UTF-8 encoded XML to new_from_xml() sometimes 
causes the entire record to be destroyed when using XML::SAX::Expat
as the parser (with PurePerl these seem to work). It fails with
the error:

not well-formed (invalid token) at line 23, column 43, byte 937 at 
/usr/lib/perl5/XML/Parser.pm line 187

I haven't been able to track the cause of this bug, I eventually
found a workaround that didn't result in the above error, but instead,
silently mangled the resulting binary MARC record on the way out:

4. Using incompatible version of XML::SAX::LibXML and libxml2 will
cause binary MARC records to be mangled when passed through new_from_xml()
in some cases. The solution here is to make sure you're running
compatible versions of XML::SAX::LibXML and libxml2. I run Debian
Sarge and when I just used the package maintainer's versions it
fixed the bug. It's unclear to me why the binary MARC would be 
mangled, this may indicate a problem with MARC::* but I haven't
had time to track it down and since installing compatible versions
of the parser back-end solves the problem I can only assume it's
the fault of the incompatible parsers.

Issues #3 and #4 above can be replicated following batch of records
through the roundtrip.pl script above:


If you want to test #2, try running this record through roundtrip.pl:


BTW: you can change your default SAX parser by editing the .ini file ... 
mine is located in /usr/local/share/perl/5.8.4/XML/SAX/ParserDetails.ini

So the bottom line is, if you want to use MARC::File::XML in any
serious application, you've got to use compatible versions of the
libxml2 parser and XML::SAX::LibXML. Check the README in the perl
package for documentation on which are compatible...

Maybe a note somewhere in the MARC::File::XML documentation to point 
these issues out would be useful. Also, it wouldn't be too bad to have
a few tests to make sure that the system's default SAX parser is capable
of handling these cases. Just my two cents. 


President, Technology       migration, training, maintenance, support
LibLime                                Featuring Koha Open-Source ILS
[EMAIL PROTECTED] |Full Demos at http://liblime.com/koha |1(888)KohaILS

Reply via email to