[CODE4LIB] MARC reporting engine
You don't have to build your own indexer. You might use the pymarc parser to pull the records into a flat database like Mongo, then pull reports from there. It really depends on what the service is delivering. This would be much less insanity inducing than regexes in vi. I do agree with Jonathan. If authorities were easy, everyone would be doing them. Cary On Sunday, November 2, 2014, Stuart Yeates wrote: > Do any of these have built-in indexing? 800k records isn't going to fit in > memory and if building my own MARC indexer is 'relatively straightforward' > then you're a better coder than I am. > > cheers > stuart > > -- > I have a new phone number: 04 463 5692 > > > From: Code for Libraries on behalf of Jonathan > Rochkind > Sent: Monday, 3 November 2014 1:24 p.m. > To: CODE4LIB@LISTSERV.ND.EDU > Subject: Re: [CODE4LIB] MARC reporting engine > > If you are, can become, or know, a programmer, that would be relatively > straightforward in any programming language using the open source MARC > processing library for that language. (ruby marc, pymarc, perl marc, > whatever). > > Although you might find more trouble than you expect around authorities, > with them being less standardized in your corpus than you might like. > > From: Code for Libraries [CODE4LIB@LISTSERV.ND.EDU] on behalf of Stuart > Yeates [stuart.yea...@vuw.ac.nz] > Sent: Sunday, November 02, 2014 5:48 PM > To: CODE4LIB@LISTSERV.ND.EDU > Subject: [CODE4LIB] MARC reporting engine > > I have ~800,000 MARC records from an indexing service ( > http://natlib.govt.nz/about-us/open-data/innz-metadata CC-BY). I am > trying to generate: > > (a) a list of person authorities (and sundry metadata), sorted by how many > times they're referenced, in wikimedia syntax > > (b) a view of a person authority, with all the records by which they're > referenced, processed into a wikipedia stub biography > > I have established that this is too much data to process in XSLT or > multi-line regexps in vi. What other MARC engines are there out there? > > The two options I'm aware of are learning multi-line processing in sed or > learning enough koha to write reports in whatever their reporting engine is. > > Any advice? > > cheers > stuart > -- > I have a new phone number: 04 463 5692 > -- Cary Gordon The Cherry Hill Company http://chillco.com
Re: [CODE4LIB] MARC reporting engine
Do any of these have built-in indexing? 800k records isn't going to fit in memory and if building my own MARC indexer is 'relatively straightforward' then you're a better coder than I am. cheers stuart -- I have a new phone number: 04 463 5692 From: Code for Libraries on behalf of Jonathan Rochkind Sent: Monday, 3 November 2014 1:24 p.m. To: CODE4LIB@LISTSERV.ND.EDU Subject: Re: [CODE4LIB] MARC reporting engine If you are, can become, or know, a programmer, that would be relatively straightforward in any programming language using the open source MARC processing library for that language. (ruby marc, pymarc, perl marc, whatever). Although you might find more trouble than you expect around authorities, with them being less standardized in your corpus than you might like. From: Code for Libraries [CODE4LIB@LISTSERV.ND.EDU] on behalf of Stuart Yeates [stuart.yea...@vuw.ac.nz] Sent: Sunday, November 02, 2014 5:48 PM To: CODE4LIB@LISTSERV.ND.EDU Subject: [CODE4LIB] MARC reporting engine I have ~800,000 MARC records from an indexing service (http://natlib.govt.nz/about-us/open-data/innz-metadata CC-BY). I am trying to generate: (a) a list of person authorities (and sundry metadata), sorted by how many times they're referenced, in wikimedia syntax (b) a view of a person authority, with all the records by which they're referenced, processed into a wikipedia stub biography I have established that this is too much data to process in XSLT or multi-line regexps in vi. What other MARC engines are there out there? The two options I'm aware of are learning multi-line processing in sed or learning enough koha to write reports in whatever their reporting engine is. Any advice? cheers stuart -- I have a new phone number: 04 463 5692
Re: [CODE4LIB] MARC reporting engine
It looks like the dataset is available in XML format. Perhaps you can import it into an XML database (eXist - exist-db.org comes to mind), and then generate a report via its query capabilities. Miles Fidelman Jonathan Rochkind wrote: If you are, can become, or know, a programmer, that would be relatively straightforward in any programming language using the open source MARC processing library for that language. (ruby marc, pymarc, perl marc, whatever). Although you might find more trouble than you expect around authorities, with them being less standardized in your corpus than you might like. From: Code for Libraries [CODE4LIB@LISTSERV.ND.EDU] on behalf of Stuart Yeates [stuart.yea...@vuw.ac.nz] Sent: Sunday, November 02, 2014 5:48 PM To: CODE4LIB@LISTSERV.ND.EDU Subject: [CODE4LIB] MARC reporting engine I have ~800,000 MARC records from an indexing service (http://natlib.govt.nz/about-us/open-data/innz-metadata CC-BY). I am trying to generate: (a) a list of person authorities (and sundry metadata), sorted by how many times they're referenced, in wikimedia syntax (b) a view of a person authority, with all the records by which they're referenced, processed into a wikipedia stub biography I have established that this is too much data to process in XSLT or multi-line regexps in vi. What other MARC engines are there out there? The two options I'm aware of are learning multi-line processing in sed or learning enough koha to write reports in whatever their reporting engine is. Any advice? cheers stuart -- I have a new phone number: 04 463 5692 -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra
Re: [CODE4LIB] MARC reporting engine
If you are, can become, or know, a programmer, that would be relatively straightforward in any programming language using the open source MARC processing library for that language. (ruby marc, pymarc, perl marc, whatever). Although you might find more trouble than you expect around authorities, with them being less standardized in your corpus than you might like. From: Code for Libraries [CODE4LIB@LISTSERV.ND.EDU] on behalf of Stuart Yeates [stuart.yea...@vuw.ac.nz] Sent: Sunday, November 02, 2014 5:48 PM To: CODE4LIB@LISTSERV.ND.EDU Subject: [CODE4LIB] MARC reporting engine I have ~800,000 MARC records from an indexing service (http://natlib.govt.nz/about-us/open-data/innz-metadata CC-BY). I am trying to generate: (a) a list of person authorities (and sundry metadata), sorted by how many times they're referenced, in wikimedia syntax (b) a view of a person authority, with all the records by which they're referenced, processed into a wikipedia stub biography I have established that this is too much data to process in XSLT or multi-line regexps in vi. What other MARC engines are there out there? The two options I'm aware of are learning multi-line processing in sed or learning enough koha to write reports in whatever their reporting engine is. Any advice? cheers stuart -- I have a new phone number: 04 463 5692
[CODE4LIB] MARC reporting engine
I have ~800,000 MARC records from an indexing service (http://natlib.govt.nz/about-us/open-data/innz-metadata CC-BY). I am trying to generate: (a) a list of person authorities (and sundry metadata), sorted by how many times they're referenced, in wikimedia syntax (b) a view of a person authority, with all the records by which they're referenced, processed into a wikipedia stub biography I have established that this is too much data to process in XSLT or multi-line regexps in vi. What other MARC engines are there out there? The two options I'm aware of are learning multi-line processing in sed or learning enough koha to write reports in whatever their reporting engine is. Any advice? cheers stuart -- I have a new phone number: 04 463 5692
Re: [CODE4LIB] Q: Summon API Service?
Has anyone constructed Summon API queries in javascript? (assuming that they
can be constructed to use jsonp and avoid cross domain problems …???)
Subject:
Re: Q: Summon API Service?
From:
Doug Chestnut
Reply-To:
Code for Libraries
Date:
Wed, 27 Oct 2010 11:56:04 -0400
Content-Type:
text/plain
Parts/Attachments:
text/plain (45 lines)
Reply
If it helps, here are a few lines in python that I use to make summon queries:
def summonMkHeaders(querystring):
summonAccessID = 'yourIDhere'
summonSecretKey = 'yourSecretHere'
summonAccept = "application/json"
summonThedate = datetime.utcnow().strftime("%a, %d %b %Y
%H:%M:%S GMT")
summonQS = "&".join(sorted(querystring.split('&')))
summonQS = urllib.unquote_plus(summonQS)
summonIdString = summonAccept + "\n" + summonThedate +
"\n" + summonHost + "\n" + summonPath + "\n" + summonQS + "\n"
summonDigest =
base64.encodestring(hmac.new(summonSecretKey, unicode(summonIdString),
hashlib.sha1).digest())
summonAuthstring = "Summon "+summonAccessID+';'+summonDigest
summonAuthstring = summonAuthstring.replace('\n','')
return
{'Accept':summonAccept,'x-summon-date':summonThedate,'Host':summonHost,'Authorization':summonAuthstring}
--Doug
On Tue, Oct 26, 2010 at 6:46 PM, Godmar Back wrote:
> Hi,
>
> Unlike Link/360, Serials Solution's Summon API is extremely cumbersome to
> use - requiring, for instance, that requests be digitally signed. (*)
>
> Has anybody developed a proxy server for Summon that makes its API public
> (e.g. receives requests, signs them, forwards them to Summon, and relays the
> result back to a HTTP client?)
>
> Serials Solutions publishes some PHP5 and Ruby sample code in two API
> libraries (**), but these don't appear to be fully fledged nor
> easy-to-install solutions. (Easy to install here is defined as an average
> systems librarian can download them, provide the API key, and have a running
> solution in less time than it takes to install Wordpress.)
>
> Thanks!
>
> - Godmar
>
> (*) http://api.summon.serialssolutions.com/help/api/authentication
> (**) http://api.summon.serialssolutions.com/help/api/code
>
Re: [CODE4LIB] Terrible Drupal vulnerability
If you can migrate to a maintained service, you could use feeds or migrate to move your content. You could also take that approach on your own new site. Obviously, none of your entities — nodes, menus, users, blocks, taxonomies, etc. — should contain executable code. I suggest that you do not migrate users or menus, unless you have the ability to validate your data. I love the internets, but I have learned that nobody should be running public facing services — open-source or other — unless they are prepared to maintain them, including managing a disaster recovery plan and vigilantly monitoring and acting on security notices. If this is not doable, use a service provider to manage it. The days of running services from a computer under a desk are gone. Cary On Sunday, November 2, 2014, Hickner, Andrew wrote: > I'd be curious to hear how others are proceeding. We had already planned > to migrate our D7 sites to a centralized Drupal instance offered here at > Yale and this has just accelerated the timetable. I imagine there are a > lot of libraries running Drupal though who don't have this kind of option > and might not have pre-October 15 backups to revert to (we don't!) > > > > Andy Hickner > Web Services Librarian > Yale University > Cushing/Whitney Medical Library > http://library.medicine.yale.edu/ > > > From: Code for Libraries [CODE4LIB@LISTSERV.ND.EDU ] on > behalf of Lin, Kun [l...@cua.edu ] > Sent: Friday, October 31, 2014 2:10 PM > To: CODE4LIB@LISTSERV.ND.EDU > Subject: Re: [CODE4LIB] Terrible Drupal vulnerability > > I think so. However, Cloudflare in their blog post claim they have develop > a way to block the attack immediately when the vulnerability was announced. > Whether or not they know the exploit ahead of time or not, it would be good > to know someone is watching out for you for $20 a month. And you will be > mad if you took Oct 15th off without it. I just check, I patched my > instance on Oct 16th. Not sure what's going to happened. > > Kun > > -Original Message- > From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU ] > On Behalf Of Cary Gordon > Sent: Friday, October 31, 2014 1:44 PM > To: CODE4LIB@LISTSERV.ND.EDU > Subject: Re: [CODE4LIB] Terrible Drupal vulnerability > > The vulnerability was discovered in the course of an audit by SektionEins, > a German security firm, and immediately reported to the Drupal Security > Team. Because this was a pretty obscure vulnerability with no reported > exploits, the team decided to wait until the first scheduled release date > after DrupalCon Amsterdam to put out the notice and patch. Obviously, they > knew that once word of the vulnerability was announced, there would > immediately be a wave of exploits, so they imposed a blackout on any > mention of it before October 15th. I think that they stuck to their word. > > Of course, attacks started a few hours after the announcement. > > Cary > > > On Oct 31, 2014, at 9:38 AM, Joe Hourcle > wrote: > > > > On Oct 31, 2014, at 11:46 AM, Lin, Kun wrote: > > > >> Hi Cary, > >> > >> I don't know from whom. But for the heartbeat vulnerability earlier > this year, they as well as some other big providers like Google and Amazon > were notified and patched before it was announced. > > > > If they have an employee who contributes to the project, it's possible > > that this was discussed on development lists before it was sent down > > to user level mailing lists. > > > > Odds are, there's also some network of people who are willing to give > > things a cursory review / beta test in a more controlled manner before > > they're officially released (and might break thousands of websites). > > It would make sense that companies who derive a good deal of their > > profits in supporting software would participate in those programs, as > well. > > > > I could see categorizing either of those as 'ahead of the *general* > > public', which was Kun's assertion. > > > > -Joe > > > > > > > >> -Original Message- > >> From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU > ] On Behalf > >> Of Cary Gordon > >> Sent: Friday, October 31, 2014 11:10 AM > >> To: CODE4LIB@LISTSERV.ND.EDU > >> Subject: Re: [CODE4LIB] Terrible Drupal vulnerability > >> > >> How do they receive vulnerability report ahead of general public? From > whom? > >> > >> Cary > >> > >> On Friday, October 31, 2014, Lin, Kun > > wrote: > >> > >>> If you are using drupal as main website, consider using Cloudflare Pro. > >>> It's just $20 a month and worth it. They'll help block most attacks. > >>> And they usually receive vulnerability report ahead of general public. > >>> > >>> Kun > >>> > >>> -Original Message- > >>> From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU > > >>> ] On Behalf Of Cary Gordon > >>> Sent: Friday, October 31, 2014 9:59 AM > >>> To: CODE4LIB@LISTSERV.ND.EDU > >>> Subject: Re: [CODE4LIB] Terrible Drupal vulnerability > >>> > >>> This is what I
Re: [CODE4LIB] Terrible Drupal vulnerability
I'd be curious to hear how others are proceeding. We had already planned to migrate our D7 sites to a centralized Drupal instance offered here at Yale and this has just accelerated the timetable. I imagine there are a lot of libraries running Drupal though who don't have this kind of option and might not have pre-October 15 backups to revert to (we don't!) Andy Hickner Web Services Librarian Yale University Cushing/Whitney Medical Library http://library.medicine.yale.edu/ From: Code for Libraries [CODE4LIB@LISTSERV.ND.EDU] on behalf of Lin, Kun [l...@cua.edu] Sent: Friday, October 31, 2014 2:10 PM To: CODE4LIB@LISTSERV.ND.EDU Subject: Re: [CODE4LIB] Terrible Drupal vulnerability I think so. However, Cloudflare in their blog post claim they have develop a way to block the attack immediately when the vulnerability was announced. Whether or not they know the exploit ahead of time or not, it would be good to know someone is watching out for you for $20 a month. And you will be mad if you took Oct 15th off without it. I just check, I patched my instance on Oct 16th. Not sure what's going to happened. Kun -Original Message- From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU] On Behalf Of Cary Gordon Sent: Friday, October 31, 2014 1:44 PM To: CODE4LIB@LISTSERV.ND.EDU Subject: Re: [CODE4LIB] Terrible Drupal vulnerability The vulnerability was discovered in the course of an audit by SektionEins, a German security firm, and immediately reported to the Drupal Security Team. Because this was a pretty obscure vulnerability with no reported exploits, the team decided to wait until the first scheduled release date after DrupalCon Amsterdam to put out the notice and patch. Obviously, they knew that once word of the vulnerability was announced, there would immediately be a wave of exploits, so they imposed a blackout on any mention of it before October 15th. I think that they stuck to their word. Of course, attacks started a few hours after the announcement. Cary > On Oct 31, 2014, at 9:38 AM, Joe Hourcle > wrote: > > On Oct 31, 2014, at 11:46 AM, Lin, Kun wrote: > >> Hi Cary, >> >> I don't know from whom. But for the heartbeat vulnerability earlier this >> year, they as well as some other big providers like Google and Amazon were >> notified and patched before it was announced. > > If they have an employee who contributes to the project, it's possible > that this was discussed on development lists before it was sent down > to user level mailing lists. > > Odds are, there's also some network of people who are willing to give > things a cursory review / beta test in a more controlled manner before > they're officially released (and might break thousands of websites). > It would make sense that companies who derive a good deal of their > profits in supporting software would participate in those programs, as well. > > I could see categorizing either of those as 'ahead of the *general* > public', which was Kun's assertion. > > -Joe > > > >> -Original Message- >> From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU] On Behalf >> Of Cary Gordon >> Sent: Friday, October 31, 2014 11:10 AM >> To: CODE4LIB@LISTSERV.ND.EDU >> Subject: Re: [CODE4LIB] Terrible Drupal vulnerability >> >> How do they receive vulnerability report ahead of general public? From whom? >> >> Cary >> >> On Friday, October 31, 2014, Lin, Kun wrote: >> >>> If you are using drupal as main website, consider using Cloudflare Pro. >>> It's just $20 a month and worth it. They'll help block most attacks. >>> And they usually receive vulnerability report ahead of general public. >>> >>> Kun >>> >>> -Original Message- >>> From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU >>> ] On Behalf Of Cary Gordon >>> Sent: Friday, October 31, 2014 9:59 AM >>> To: CODE4LIB@LISTSERV.ND.EDU >>> Subject: Re: [CODE4LIB] Terrible Drupal vulnerability >>> >>> This is what I posted to the Drupal4Lib list: >>> >>> >>> >>> By now, you should have seen https://www.drupal.org/PSA-2014-003 and >>> heard about the "Drupageddon" exploits. and you may be wondering if >>> you were vulnerable or iff you were hit by this, how you can tell >>> and what you should do. Drupageddon affects Drupal 7, Drupal 8 and, >>> if you use the DBTNG module, Drupal 6. >>> >>> The general recommendation is that if you do not know or are unsure >>> of your server's security and you did not either update to Drupal >>> 7.32 or apply the patch within a few hours of the notice, you should >>> assume that your site (and server) was hacked and you should restore >>> everything to a backup from before October 15th or earlier. If your >>> manage your server and you have any doubts about your file security, >>> you should restore that to a pre 10/15 image, as well or do a reinstall of >>> your server software. >>> >>> I know this sounds drastic, and I know that not everyone will do th
[CODE4LIB] Job: Data Curator | Virginia Tech at Virginia Tech
Data Curator | Virginia Tech Virginia Tech Blacksburg US - VA - Blacksburg, Required: Master's degree or Bachelor's degree and significant experience equivalent to an advanced degree. Relevant experience in working with data and metadata for the purposes of resear Brought to you by code4lib jobs: http://jobs.code4lib.org/job/17415/ To post a new job please visit http://jobs.code4lib.org/
[CODE4LIB] Job: Instructor or Assistant Professor - Electronic Resources Librarian (Queensborough Community College, New York) at Queensborough Community College
Instructor or Assistant Professor - Electronic Resources Librarian (Queensborough Community College, New York) Queensborough Community College New York City Instructor or Assistant Professor - Electronic Resources Librarian (Queensborough Community College, New York) Job Title: Instructor or Assistant Professor - Electronic Resources Librarian Job ID: 11469 Location: Queensborough Community College You can view and apply for this position at[https://home.cunyfirst.cuny.edu/psp/cn yepprd/GUEST/HRMS/c/HRS_HRAM.HRS_CE.GBL?Page=HRS_CE_JOB_DTL&Action=A&JobOpenin gId=11469&SiteId=1&PostingSeq=1](https://home.cunyfirst.cuny.edu/psp/cnyepprd/ GUEST/HRMS/c/HRS_HRAM.HRS_CE.GBL?Page=HRS_CE_JOB_DTL&Action=A&JobOpeningId=114 69&SiteId=1&PostingSeq=1). Brought to you by code4lib jobs: http://jobs.code4lib.org/job/17665/ To post a new job please visit http://jobs.code4lib.org/
[CODE4LIB] Job: Instructor or Assistant Professor - Web Services/Digital Content Librarian (Queensborough Community College, New York) at Queensborough Community College
Instructor or Assistant Professor - Web Services/Digital Content Librarian (Queensborough Community College, New York) Queensborough Community College New York City Instructor or Assistant Professor - Web Services/Digital Content Librarian (Queensborough Community College, New York) Job Title:Instructor or Assistant Professor - Web Services/Digital Content Librarian Job ID:Queensborough Community College You can view and apply for this position at[https://home.cunyfirst.cuny.edu/ psp/cnyepprd/GUEST/HRMS/c/HRS_HRAM.HRS_CE.GBL?Page=HRS_CE_JOB_DTL&Action=A&Job OpeningId=11545&SiteId=1&PostingSeq=1](https://home.cunyfirst.cuny.edu/psp/cny epprd/GUEST/HRMS/c/HRS_HRAM.HRS_CE.GBL?Page=HRS_CE_JOB_DTL&Action=A&JobOpening Id=11545&SiteId=1&PostingSeq=1). Brought to you by code4lib jobs: http://jobs.code4lib.org/job/17666/ To post a new job please visit http://jobs.code4lib.org/
[CODE4LIB] Job: Deputy District Director (Fort Vancouver Regional Library, Washington) at Fort Vancouver Regional Library District
Deputy District Director (Fort Vancouver Regional Library, Washington) Fort Vancouver Regional Library District Vancouver, Washington Deputy District Director (Fort Vancouver Regional Library, Washington) Fort Vancouver Regional Library District is seeking an experienced, energetic and innovative leader and administrator for the key position of Deputy District Director. The position reports to the Executive Director and oversees the public services division. This position requires a strong and collaborative leader with a passion for the challenges inherent in serving diverse and ambitious communities that love and support their libraries. This position requires a Master's Degree in Library Science or the ability to obtain certification from the Washington State Library. Preferred candidates will have at least seven years of public library experience and five years of relevant supervisory and project management experience. Track record with successful capital facilities projects desirable. Salary is negotiable from $100,000 annually. We offer an excellent benefits package with a state retirement program. The position will be open until filled. To apply, please complete the FVRL Application for Employment (pdf) and submit a signed copy along with a resume and cover letter to [j...@fvrl.org](mailto:j...@fvrl.org). The deadline to submit applications is 5:00 p.m. on November 28, 2014. If submitting electronically, application signature will be collected at time of interview. Brought to you by code4lib jobs: http://jobs.code4lib.org/job/17970/ To post a new job please visit http://jobs.code4lib.org/
[CODE4LIB] Job: Library Support Assistant (part-time/term-time) at University of Nottingham
Library Support Assistant (part-time/term-time) University of Nottingham Nottingham **University of Nottingham - Customer Services** Salary: Not specified Brought to you by code4lib jobs: http://jobs.code4lib.org/job/17969/ To post a new job please visit http://jobs.code4lib.org/
