Re: [OPEN-ILS-GENERAL] Mangled UTF8 characters with imported MARC records in Z39.50

[Mike Rylander]

Jason hit on (almost certainly) the answer: bad records from sources that
don't restrict cataloging to valid character sets.  I'll add a couple
comments below for general clarification, as well...

On Fri, Dec 2, 2016 at 4:52 PM, Brent Mills 
wrote:

> Hello,
>
> I’ve recently noticed some issues with imported MARC records from a
> specific set of Z39.50 servers.
>
> A noticeable amount of records that are imported through
> Prospector/MaineCat targets have mangled characters when diacritics,
> symbols,etc.. are present in the record.
>
> Does anyone have some ideas on what could be causing the character
> encoding problems from these particular targets? Or run into this at their
> own site?
>
> - dgo.conf has marc-8. changing that to usmarc, utf8
> has had no effect
> - xml2marc-yaz.cfg is setup like described in https://wiki.evergreen-ils.
> org/doku.php?id=evergreen-admin:sru_and_z39.50 changing the charset
> options hasn’t had any effect either
>

The reason this doesn't change anything is that it's only used to describe
how Evergreen will server records to /others/ as a z39.50 server.  Those
are not client settings.


> - the encoding/translation problems do not happen with OCLC and Library of
> Congress targets, it seems to mainly affect servers with the INNOPAC db
> type. I’m not sure if that’s related.
>
>
This and the log message below are the smoking guns.  OCLC and LoC are
generally very good about making sure records really are in the character
set they advertise, and that that character set is one of only MARC-8 or
UTF8.

So, Jason nailed it -- there are non-UTF8, non-MARC-8 characters in those
records, as served by the INNOPAC sources.  That's a (remote) cataloging
issue.

HTH,

--Mike

Going through the logs I can see things like:
>
> open-ils.search.z3950.search_class: no mapping found for [0x80] at
> position 56 in Kurt and Joe tangle with the most determined enemy they’ve
> ever encountered when a ruthless powerbroker schemes to build a new
> Egyptian empire as glorious as those of the Pharaohs. Part of his plan
> rests on the manipulation of a newly discovered aquifer beneath the Sahara,
> but an even more devastating weapon at his disposal may threaten the entire
> world: a plant extract known as the black mist, discovered in the City of
> the Dead and rumored to have the power to take life from the living and
> restore it to the dead. With the balance of power in Africa and Europe on
> the verge of tipping, Kurt, Joe, and the rest of the NUMA team will have to
> fight to discover the truth behind the legends—but to do that, they have
> to confront in person the greatest legend of them all: Osiris, the ruler of
> the Egyptian underworld. g0=ASCII_DEFAULT g1=EXTENDED_LATIN at
> /usr/share/perl5/MARC/Charset.pm line 308.
>
>
> So I’m thinking something is happening in the MARC8 to UTF8 conversion?
>
> Attaching a screenshot of what it looks like in the Z39.50 Import screen.
> The 264s have been the most obvious place to see the issue, but it happens
> in any field with special characters.
>
> Been banging my head trying to figure out what’s causing this. Any help
> would be appreciated!
>
> Thank you,
>
> -Brent
>
> -
>
> Brent Mills
> Systems Librarian | Sage Library System
>
> email: br...@hoodriverlibrary.org
> tickets: https://sagelib.org/support
>
>

Re: [OPEN-ILS-GENERAL] Mangled UTF8 characters with imported MARC records in Z39.50

[Jason Stephenson]

Brent,

The records are mostly likely not MARC-8 or UTF-8. The example you
shared looks like a Windows-1252 "smart" quote. I would not be surprised
if the records have characters from multiple character sets in them.
I've seen that before.

I don't have any useful suggestions for you, other than suggesting that
staff not try to import records from those sources.

Jason

On 12/02/2016 04:52 PM, Brent Mills wrote:
> Hello,
> 
> I’ve recently noticed some issues with imported MARC records from a
> specific set of Z39.50 servers.
> 
> A noticeable amount of records that are imported through
> Prospector/MaineCat targets have mangled characters when diacritics,
> symbols,etc.. are present in the record.
> 
> Does anyone have some ideas on what could be causing the character
> encoding problems from these particular targets? Or run into this at
> their own site?
> 
> - dgo.conf has marc-8. changing that to usmarc, utf8
> has had no effect
> - xml2marc-yaz.cfg is setup like described
> in https://wiki.evergreen-ils.org/doku.php?id=evergreen-admin:sru_and_z39.50 
> changing
> the charset options hasn’t had any effect either
> - the encoding/translation problems do not happen with OCLC and Library
> of Congress targets, it seems to mainly affect servers with the INNOPAC
> db type. I’m not sure if that’s related.
> 
> Going through the logs I can see things like:
> 
> open-ils.search.z3950.search_class: no mapping found for [0x80] at
> position 56 in Kurt and Joe tangle with the most
> determined enemy they’ve ever encountered when a ruthless
> powerbroker schemes to build a new Egyptian empire as glorious as
> those of the Pharaohs. Part of his plan rests on the manipulation of
> a newly discovered aquifer beneath the Sahara, but an even
> more devastating weapon at his disposal may threaten the entire
> world: a plant extract known as the black mist, discovered in the
> City of the Dead and rumored to have the power to take life from the
> living and restore it to the dead. With the balance of power
> in Africa and Europe on the verge of tipping, Kurt, Joe, and the
> rest of the NUMA team will have to fight to discover the
> truth behind the legends—but to do that, they have to confront in
> person the greatest legend of them all: Osiris, the ruler of
> the Egyptian underworld. g0=ASCII_DEFAULT g1=EXTENDED_LATIN at
> /usr/share/perl5/MARC/Charset.pm line 308.
> 
> 
> So I’m thinking something is happening in the MARC8 to UTF8 conversion?
> 
> Attaching a screenshot of what it looks like in the Z39.50 Import
> screen. The 264s have been the most obvious place to see the issue, but
> it happens in any field with special characters.
> 
> Been banging my head trying to figure out what’s causing this. Any help
> would be appreciated!
> 
> Thank you,
> 
> -Brent
> 
> -
> 
> Brent Mills
> Systems Librarian | Sage Library System
> 
> email: br...@hoodriverlibrary.org 
> tickets: https://sagelib.org/support
> 

[OPEN-ILS-GENERAL] Mangled UTF8 characters with imported MARC records in Z39.50

[Brent Mills]

Hello,

I’ve recently noticed some issues with imported MARC records from a specific 
set of Z39.50 servers.

A noticeable amount of records that are imported through Prospector/MaineCat 
targets have mangled characters when diacritics, symbols,etc.. are present in 
the record.

Does anyone have some ideas on what could be causing the character encoding 
problems from these particular targets? Or run into this at their own site?

- dgo.conf has marc-8. changing that to usmarc, utf8 has had 
no effect
- xml2marc-yaz.cfg is setup like described in 
https://wiki.evergreen-ils.org/doku.php?id=evergreen-admin:sru_and_z39.50 
 
changing the charset options hasn’t had any effect either
- the encoding/translation problems do not happen with OCLC and Library of 
Congress targets, it seems to mainly affect servers with the INNOPAC db type. 
I’m not sure if that’s related.

Going through the logs I can see things like:

open-ils.search.z3950.search_class: no mapping found for [0x80] at position 56 
in Kurt and Joe tangle with the most determined enemy they’ve ever 
encountered when a ruthless powerbroker schemes to build a new Egyptian empire 
as glorious as those of the Pharaohs. Part of his plan rests on the 
manipulation of a newly discovered aquifer beneath the Sahara, but an even more 
devastating weapon at his disposal may threaten the entire world: a plant 
extract known as the black mist, discovered in the City of the Dead and rumored 
to have the power to take life from the living and restore it to the dead. With 
the balance of power in Africa and Europe on the verge of tipping, Kurt, Joe, 
and the rest of the NUMA team will have to fight to discover the truth behind 
the legends—but to do that, they have to confront in person the greatest 
legend of them all: Osiris, the ruler of the Egyptian underworld. 
g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

So I’m thinking something is happening in the MARC8 to UTF8 conversion?

Attaching a screenshot of what it looks like in the Z39.50 Import screen. The 
264s have been the most obvious place to see the issue, but it happens in any 
field with special characters.

Been banging my head trying to figure out what’s causing this. Any help would 
be appreciated!

Thank you,

-Brent


-

Brent Mills
Systems Librarian | Sage Library System

email: br...@hoodriverlibrary.org
tickets: https://sagelib.org/support

Re: [OPEN-ILS-GENERAL] Fines conundrum

[John Clark]

Thank you Josh, We'll dig in and see if this is the culprit.

Regards,

John Clark


--
Retired librarian, author, freelance editor, book reviewer
https://www.facebook.com/wizardofhartland
http://cmldbookreviews.com/

Re: [OPEN-ILS-GENERAL] Fines conundrum

[Josh Stompro]

John, do you mean that the summary balance under the bills tab shows nothing?



[cid:image001.png@01D24C8F.F0680970]



I’ve seen this happen when I’ve changed values in the money.payments table 
instead of the table the corresponds with the payment type.  There are 
materialized views (a summary table that is kept in sync with triggers instead 
of pulling live data) that can get out of sync with reality.  It could be that 
the triggers that keep the summary tables up to date have been broken.  I think 
you need someone to take a look at the money.* tables and triggers to make sure 
they match up with your releases setting.



The summary tables can be re-created from the systems current state to get the 
values back in line.



You can check on this by choosing a patron that shows the symptoms, and looking 
at a few tables for that patron.  You need sql access for this.

· money.usr_summary should show that users balance totals.

· Money.billable_xact_summary should show a summary for each bill for 
that customer.  If they should have a fine, but it shows a zero balance for 
that bill, then the money.billable_xact_summary is out of sync.

· Money.billings for the same bill should show the fine.





Josh Stompro - LARL IT Director





-Original Message-
From: Open-ils-general 
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of John 
Clark
Sent: Friday, December 02, 2016 9:03 AM
To: Evergreen Discussion Group
Subject: [OPEN-ILS-GENERAL] Fines conundrum



Hi folks,





At least two of our member libraries in the Maine Balsam Consortium are 
perplexed and vexed. I've copied their problem below in hopes someone in the 
Evergreen world might have experienced (and solved) a similar issue.

I thank all in advance for any shared wisdom.



Regards,

John R. Clark-treasurer, Maine Balsam Library consortium





Evergreen no longer adds fines to a patron’s account.  When an item is returned 
and checked in it shows that a fine has accrued, but when we go to that 
patron’s account there is nothing under bills……But click on bills and  overdues 
are there!  It started at first just on movies and now its books as well. How 
can we get accrued fines to show?



--

Retired librarian, author, freelance editor, book reviewer

https://www.facebook.com/wizardofhartland

http://cmldbookreviews.com/

[OPEN-ILS-GENERAL] Fines conundrum

[John Clark]

Hi folks,


At least two of our member libraries in the Maine Balsam Consortium are 
perplexed and vexed. I've copied their problem below in hopes someone in 
the Evergreen world might have experienced (and solved) a similar issue. 
I thank all in advance for any shared wisdom.


Regards,
John R. Clark-treasurer, Maine Balsam Library consortium


Evergreen no longer adds fines to a patron’s account.  When an item is 
returned and checked in it shows that a fine has accrued, but when we go 
to that patron’s account there is nothing under bills……But click on 
bills and  overdues are there!  It started at first just on movies and 
now its books as well. How can we get accrued fines to show?


--
Retired librarian, author, freelance editor, book reviewer
https://www.facebook.com/wizardofhartland
http://cmldbookreviews.com/

Re: [OPEN-ILS-GENERAL] Evergreen International 2017 Pre-conference Proposals

[Garry Collum]

Thanks Jim,

If others have ideas of what type of pre-conference programs they would
like to see, please chime in.   We would love to see what people are
interested in.

The deadline for submitting pre-conference proposals is coming up, Dec 8.
Pre-conference presenters will receive a complimentary conference
registration.  Proposals should be submitted to
eg-conf-progr...@list.evergreen-ils.org.

Garry


On Thu, Dec 1, 2016 at 5:56 PM, Terran McCanna <
tmcca...@georgialibraries.org> wrote:

> That sounds excellent to me, Jim! I would love to attend that
> pre-conference.
>
> Terran McCanna
> PINES Program Manager
> Georgia Public Library Service
> 1800 Century Place, Suite 150
> Atlanta, GA 30345
> 404-235-7138
> tmcca...@georgialibraries.org
>
>
> On Wed, Nov 30, 2016 at 5:54 PM, Jim Taylor  wrote:
>
>> Assuming I am not the only interested party.  If I am, maybe someone
>> would be willing to take a little time and help me sort through some of the
>> process.   In regards to the question though…
>>
>>
>>
>> Basically walking through the process from the beginning.  I am not
>> wanting this to be a programming class but rather one on how to interact
>> with the project.
>>
>>
>>
>> Acquiring credentials ( probably needs very little attention…the WIKI
>> covers this very well.)
>>
>> Recommended tools for working with GitHub.  (Again, not hard to connect
>> but maybe there are better tools than some of us have found)
>>
>> The big thing, at least for me, is how one navigates the project after
>> PULLING it.   So many threads it is a bit overwhelming.
>>
>> An overview of the project so if I wanted to make changes to a particular
>> area I could have a reasonable chance of finding it.
>>
>> Making and Submitting changes.
>>
>>
>>
>> Jim
>>
>>
>>
>>
>>
>>
>>
>> *From:* Open-ils-general [mailto:open-ils-general-bounc
>> e...@list.georgialibraries.org] *On Behalf Of *Rogan Hamby
>> *Sent:* Monday, November 28, 2016 11:51 AM
>> *To:* Evergreen Discussion Group > libraries.org>
>> *Subject:* Re: [OPEN-ILS-GENERAL] Evergreen International 2017
>> Pre-conference Proposals
>>
>>
>>
>> This is something (as a general topic) that was discussed at the
>> Hack-A-Way quite a bit.  Assuming you'd be an interested p party in
>> attending I'm curious what you would see as topics that could be covered
>> versus what skills should be prerequisites.
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Nov 28, 2016 at 9:34 AM, Donald Butterworth <
>> don.butterwo...@asburyseminary.edu> wrote:
>>
>> Any chance we could offer "Evergreen Programming 101" for programmers who
>> want to begin contributing code, but don't want to get in over their head?
>>
>>
>>
>> On Fri, Nov 18, 2016 at 2:01 PM, Garry Collum  wrote:
>>
>> Hi all,
>>
>>
>>
>> The 2017 Evergreen International Conference Committee is seeking
>> proposals for 3 hour long pre-conference workshops to take place on
>> Wednesday April 5th, 2017.
>>
>>
>>
>> A few weeks ago the committee performed a survey to see what types of
>> programs people would be interested in attending.  The results of the
>> survey included the following:
>>
>>
>>
>>  SQL (PostgreSQL)
>>
>>
>>
>>  Reports
>>
>>
>>
>>  Using Acquisitions
>>
>>
>>
>>  Using Serials
>>
>>
>>
>>  Holds configuration.
>>
>>
>>
>>  Cataloging clean-up/deduplication
>>
>>
>>
>>  Linked data
>>
>>
>>
>>  How to use git
>>
>>
>>
>>  How to contribute documentation
>>
>>
>>
>>  Action triggers, how, what, when
>>
>>
>>
>>  Cataloging in a consortia environment
>>
>>
>>
>>  Web client in depth
>>
>>
>>
>>
>>
>> Selected presenters will be offered complimentary conference registration.
>>
>>
>>
>> The original deadline for submission of proposals for pre-conference
>> workshops was November 24, 2016.  Because of my error of posting the notice
>> only to the developer's listserv, I would like to extend that deadline to
>> December 8, 2016.
>>
>>
>>
>> Please contact Garry Collum at (garry.col...@kentonlibrary.org) or Anna
>> Goben at (ago...@library.in.gov) with any questions.
>>
>>
>>
>>
>>
>>
>>
>> Please submit proposals to eg-conf-progr...@list.evergreen-ils.org.
>>
>>
>>
>>
>>
>>
>>
>>  Garry Collum
>>
>>
>>
>>  Kenton County Public Library
>>
>>
>>
>> --
>>
>> Don Butterworth
>> Collection Management Librarian /
>> Faculty Associate
>> B.L. Fisher Library
>> Asbury Theological Seminary
>> don.butterwo...@asburyseminary.edu
>> (859) 858-2227
>>
>>
>>
>>
>>
>> --
>>
>> --
>>
>> Rogan R. Hamby, Data and Project Analyst
>>
>> Equinox - Open Your Library
>>
>> ro...@esilibrary.com
>>
>> 1-877-OPEN-ILS | www.esilibrary.com
>>
>>
>>
>
>