[CODE4LIB] Automation tools - session at the "Pi and Mash" unconference

[Owen Stephens]

Dear all,

A month or so ago I asked for recommendations for automation tools that people 
used in libraries to help inform a session I was going to run. The unconference 
event (Pi and Mash) ran this weekend, and I just wanted to share the materials 
I wrote for the session in case they are of any help. The materials consist of 
a slidedeck called "Automated Love Presentation" (available as Keynote, 
Powerpoint and PDF) and some examples and exercises you can work through in a 
document called "Automated Love Examples" (available as Pages, Word doc, PDF 
and ePub). There are also two accompanying files 'ISBNs.xlsx' and 'isbns.csv' 
which are used in the examples/exercises.

All materials are available at http://bit.ly/automatedlovefolder

Thanks to all who made suggestions which contributed towards the session.

Best wishes,

Owen

Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com
Email: o...@ostephens.com
Telephone: 0121 288 6936

Re: [CODE4LIB] 'automation' tools

[Owen Stephens]

Thanks again all,

I love OpenRefine - I've been working on the GOKb project (http://gokb.org) 
where K-Int (a UK based company) have developed an extension for OpenRefine 
which helps with the cleaning of data about electronic resources (esp. 
journals) from publishers and then pushes it into the GOKb database. The 
extension is fully integrated into the GOKb database but if anyone wants a look 
code is at https://github.com/k-int/gokb-phase1/tree/dev/refine. The extension 
checks the data and reports errors as well as offering ways of fixing common 
issues - there's more on the wiki 
https://wiki.kuali.org/display/OLE/OpenRefine+How-Tos

I did pitch an OpenRefine workshop for the same event as a 'data 
wrangling/cleaning' tool but the 'automation' session got the vote in the end - 
although there is definitely overlap. However I am delivering an OpenRefine 
workshop at the British Library next week - and great to see it is getting used 
across libraries.

The Google Doc Spreadsheets is also a great tip - I've run a course at the 
British Library which uses this to introduce the concept of APIs to 
non-techies. I blogged the original tutorial at 
http://www.meanboyfriend.com/overdue_ideas/2013/02/introduction-to-apis/ but a 
change to the BL open data platform means this no longer works :((

Thanks all again - I'll be trying to put stuff from the automation workshop 
online at some point and I'll post here when there is something up.

Best wishes,

Owen


Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com
Email: o...@ostephens.com
Telephone: 0121 288 6936

On 8 Jul 2014, at 03:52, davesgonechina  wrote:

> +1 to OpenRefine. Some extensions, like RDF Refine <http://refine.deri.ie/>,
> currently only work with the old Google Refine (still available here
> <https://code.google.com/p/google-refine/>). There's a good deal of
> interesting projects for OpenRefine on GitHub and GitHub Gist.
> 
> Google Docs Spreadsheets also has a surprising amount of functionality,
> such as importXML if you're willing to get your hands dirty with regular
> expressions.
> 
> Dave
> 
> 
> On Tue, Jul 8, 2014 at 3:12 AM, Tillman, Ruth K. (GSFC-272.0)[CADENCE GROUP
> ASSOC]  wrote:
> 
>> Definite cosign on Open Refine. It's intuitive and spreadsheet-like enough
>> that a lot of people can understand it. You can do anything from
>> standardizing state names you get from a patron form to normalizing
>> metadata keywords for a database, so I think it'd be useful even for
>> non-techies.
>> 
>> Ruth Kitchin Tillman
>> Metadata Librarian, Cadence Group
>> NASA Goddard Space Flight Center Library, Code 272
>> Greenbelt, MD 20771
>> Goddard Library Repository: http://gsfcir.gsfc.nasa.gov/
>> 301.286.6246
>> 
>> 
>> -----Original Message-
>> From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU] On Behalf Of
>> Terry Brady
>> Sent: Monday, July 07, 2014 1:35 PM
>> To: CODE4LIB@LISTSERV.ND.EDU
>> Subject: Re: [CODE4LIB] 'automation' tools
>> 
>> I learned about Open Refine <http://openrefine.org/> at the Code4Lib
>> conference, and it looks like it would be a great tool for normalizing
>> data.  I worked on a few projects in the past in which this would have been
>> very helpful.
>> 

Re: [CODE4LIB] 'automation' tools

[davesgonechina]

+1 to OpenRefine. Some extensions, like RDF Refine <http://refine.deri.ie/>,
currently only work with the old Google Refine (still available here
<https://code.google.com/p/google-refine/>). There's a good deal of
interesting projects for OpenRefine on GitHub and GitHub Gist.

Google Docs Spreadsheets also has a surprising amount of functionality,
such as importXML if you're willing to get your hands dirty with regular
expressions.

Dave


On Tue, Jul 8, 2014 at 3:12 AM, Tillman, Ruth K. (GSFC-272.0)[CADENCE GROUP
ASSOC]  wrote:

> Definite cosign on Open Refine. It's intuitive and spreadsheet-like enough
> that a lot of people can understand it. You can do anything from
> standardizing state names you get from a patron form to normalizing
> metadata keywords for a database, so I think it'd be useful even for
> non-techies.
>
> Ruth Kitchin Tillman
> Metadata Librarian, Cadence Group
> NASA Goddard Space Flight Center Library, Code 272
> Greenbelt, MD 20771
> Goddard Library Repository: http://gsfcir.gsfc.nasa.gov/
> 301.286.6246
>
>
> -Original Message-
> From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU] On Behalf Of
> Terry Brady
> Sent: Monday, July 07, 2014 1:35 PM
> To: CODE4LIB@LISTSERV.ND.EDU
> Subject: Re: [CODE4LIB] 'automation' tools
>
> I learned about Open Refine <http://openrefine.org/> at the Code4Lib
> conference, and it looks like it would be a great tool for normalizing
> data.  I worked on a few projects in the past in which this would have been
> very helpful.
>

Re: [CODE4LIB] 'automation' tools

[Tillman, Ruth K. (GSFC-272.0)[CADENCE GROUP ASSOC]]

Definite cosign on Open Refine. It's intuitive and spreadsheet-like enough that 
a lot of people can understand it. You can do anything from standardizing state 
names you get from a patron form to normalizing metadata keywords for a 
database, so I think it'd be useful even for non-techies.

Ruth Kitchin Tillman
Metadata Librarian, Cadence Group
NASA Goddard Space Flight Center Library, Code 272
Greenbelt, MD 20771
Goddard Library Repository: http://gsfcir.gsfc.nasa.gov/
301.286.6246 


-Original Message-
From: Code for Libraries [mailto:CODE4LIB@LISTSERV.ND.EDU] On Behalf Of Terry 
Brady
Sent: Monday, July 07, 2014 1:35 PM
To: CODE4LIB@LISTSERV.ND.EDU
Subject: Re: [CODE4LIB] 'automation' tools

I learned about Open Refine <http://openrefine.org/> at the Code4Lib 
conference, and it looks like it would be a great tool for normalizing data.  I 
worked on a few projects in the past in which this would have been very helpful.

Re: [CODE4LIB] 'automation' tools

[Terry Brady]

I learned about Open Refine  at the Code4Lib
conference, and it looks like it would be a great tool for normalizing
data.  I worked on a few projects in the past in which this would have been
very helpful.

Bohyun Kim wrote a great article about how to query Google Spreadsheet data
from a web page: http://www.bohyunkim.net/blog/archives/2831.  I have found
this approach very helpful for parsing Google Form data.

I have created an application that has been very useful for our library:
http://georgetown-university-libraries.github.io/File-Analyzer/.  We use
this application within our digitization and ingest workflows.  We have
written some custom code to convert files exported from our ILS.  If you
check out the wiki pages
,
you might find some tasks that would be useful to recommend.

Good luck with your workshop.

Terry


On Fri, Jul 4, 2014 at 1:51 AM, Owen Stephens  wrote:

> I'm doing a workshop in the UK at a library tech unconference-style event
> (Pi and Mash http://piandmash.info) on automating computer based tasks.
> I want to cover tools that are usable by non-programmers and that would
> work in a typical library environment. The types of tools I'm thinking of
> are:
>
> MacroExpress
> AutoHotKey
> iMacros for Firefox
>
> While I'm hoping workshop attendees will bring ideas about tasks they
> would like to automate the type of thing I have in mind are things like:
>
> Filling out a set of standard data on a GUI or Web form (e.g. standard set
> of budget codes for an order)
> Processing a list of item barcodes from a spreadsheet and doing something
> with them on the library system (e.g. change loan status, check for holds)
> Similarly for User IDs
> Navigating to a web page and doing some task
>
> Clearly some of these tasks would be better automated with appropriate
> APIs and scripts, but I want to try to introduce those without programming
> skills to some of the concepts and tools and essentially how they can work
> around problems themselves to some extent.
>
> What tools do you use for this kind of automation task, and what kind of
> tasks do they best deal with?
>
> Thanks,
>
> Owen
>
> Owen Stephens
> Owen Stephens Consulting
> Web: http://www.ostephens.com
> Email: o...@ostephens.com
> Telephone: 0121 288 6936
>



-- 
Terry Brady
Applications Programmer Analyst
Georgetown University Library Information Technology
https://www.library.georgetown.edu/lit/code
425-298-5498

Re: [CODE4LIB] 'automation' tools

[Owen Stephens]

Thanks Riley and Andrew for these pointers - some great stuff in there

Other tools and examples still very welcome :)

Owen

Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com
Email: o...@ostephens.com
Telephone: 0121 288 6936

On 4 Jul 2014, at 15:04, Andrew Weidner  wrote:

> Great idea for a workshop, Owen.
> 
> My staff and I use AutoHotkey every day. We have some apps for data
> cleaning in the CONTENTdm Project Client that I presented on recently:
> http://scholarcommons.sc.edu/cdmusers/cdmusersMay2014/May2014/13/. I'll be
> talking about those in more detail at the Upper Midwest Digital Collections
> Conference <http://www.wils.org/news-events/wilsevents/umdcc/> if anyone is
> interested.
> 
> I did an in-house training session for our ILS and database management
> folks on a simple AHK app that they now use for repetitive data entry:
> https://github.com/metaweidner/AutoType. When I was working with digital
> newspapers I developed a suite of tools for making repetitive quality
> review tasks easier: https://github.com/drewhop/AutoHotkey/wiki/NDNP_QR
> 
> Basic AHK scripts are really great for text wrangling. Just yesterday I
> wrote a script to grab some values from a spreadsheet, remove commas from
> the numbers, and dump them into a tab delimited file in the format that we
> need. That script will become part of our regular workflow. Wrote another
> one-off script to transform labels on our wiki into links. It wrapped the
> labels in the wiki link syntax, and then I copied and pasted the unique
> URLs into the appropriate spots.
> 
> It's also useful for keeping things organized. I have a set of scripts that
> open up frequently used network drive folders and applications, and I
> packaged them as drop down menu choices in a little GUI that's always open
> on the desktop. We have a few search scripts that either grab values from a
> spreadsheet or input box and then run a search for those terms in a web
> database (e.g. id.loc.gov).
> 
> You might check out Selenium IDE for working with web forms:
> http://docs.seleniumhq.org/projects/ide/. The recording feature makes it
> really easy to get started with as an automation tool. I've used it
> extensively for automated metadata editing:
> http://digital.library.unt.edu/ark:/67531/metadc86138/m1/1/
> 
> Cheers!
> 
> Andrew
> 
> 
> On Fri, Jul 4, 2014 at 6:54 AM, Riley Childs  wrote:
> 
>> Don't forget AutoIT (auto IT, pretty clever eh?)
>> http://www.autoitscript.com/site/autoit/
>> 
>> Riley Childs
>> Student
>> Asst. Head of IT Services
>> Charlotte United Christian Academy
>> (704) 497-2086
>> RileyChilds.net
>> Sent from my Windows Phone, please excuse mistakes
>> 
>> -Original Message-
>> From: "Owen Stephens" 
>> Sent: ‎7/‎4/‎2014 4:55 AM
>> To: "CODE4LIB@LISTSERV.ND.EDU" 
>> Subject: [CODE4LIB] 'automation' tools
>> 
>> I'm doing a workshop in the UK at a library tech unconference-style event
>> (Pi and Mash http://piandmash.info) on automating computer based tasks.
>> I want to cover tools that are usable by non-programmers and that would
>> work in a typical library environment. The types of tools I'm thinking of
>> are:
>> 
>> MacroExpress
>> AutoHotKey
>> iMacros for Firefox
>> 
>> While I'm hoping workshop attendees will bring ideas about tasks they
>> would like to automate the type of thing I have in mind are things like:
>> 
>> Filling out a set of standard data on a GUI or Web form (e.g. standard set
>> of budget codes for an order)
>> Processing a list of item barcodes from a spreadsheet and doing something
>> with them on the library system (e.g. change loan status, check for holds)
>> Similarly for User IDs
>> Navigating to a web page and doing some task
>> 
>> Clearly some of these tasks would be better automated with appropriate
>> APIs and scripts, but I want to try to introduce those without programming
>> skills to some of the concepts and tools and essentially how they can work
>> around problems themselves to some extent.
>> 
>> What tools do you use for this kind of automation task, and what kind of
>> tasks do they best deal with?
>> 
>> Thanks,
>> 
>> Owen
>> 
>> Owen Stephens
>> Owen Stephens Consulting
>> Web: http://www.ostephens.com
>> Email: o...@ostephens.com
>> Telephone: 0121 288 6936
>> 

Re: [CODE4LIB] 'automation' tools

[Andrew Weidner]

Great idea for a workshop, Owen.

My staff and I use AutoHotkey every day. We have some apps for data
cleaning in the CONTENTdm Project Client that I presented on recently:
http://scholarcommons.sc.edu/cdmusers/cdmusersMay2014/May2014/13/. I'll be
talking about those in more detail at the Upper Midwest Digital Collections
Conference <http://www.wils.org/news-events/wilsevents/umdcc/> if anyone is
interested.

I did an in-house training session for our ILS and database management
folks on a simple AHK app that they now use for repetitive data entry:
https://github.com/metaweidner/AutoType. When I was working with digital
newspapers I developed a suite of tools for making repetitive quality
review tasks easier: https://github.com/drewhop/AutoHotkey/wiki/NDNP_QR

Basic AHK scripts are really great for text wrangling. Just yesterday I
wrote a script to grab some values from a spreadsheet, remove commas from
the numbers, and dump them into a tab delimited file in the format that we
need. That script will become part of our regular workflow. Wrote another
one-off script to transform labels on our wiki into links. It wrapped the
labels in the wiki link syntax, and then I copied and pasted the unique
URLs into the appropriate spots.

It's also useful for keeping things organized. I have a set of scripts that
open up frequently used network drive folders and applications, and I
packaged them as drop down menu choices in a little GUI that's always open
on the desktop. We have a few search scripts that either grab values from a
spreadsheet or input box and then run a search for those terms in a web
database (e.g. id.loc.gov).

You might check out Selenium IDE for working with web forms:
http://docs.seleniumhq.org/projects/ide/. The recording feature makes it
really easy to get started with as an automation tool. I've used it
extensively for automated metadata editing:
http://digital.library.unt.edu/ark:/67531/metadc86138/m1/1/

Cheers!

Andrew


On Fri, Jul 4, 2014 at 6:54 AM, Riley Childs  wrote:

> Don't forget AutoIT (auto IT, pretty clever eh?)
> http://www.autoitscript.com/site/autoit/
>
> Riley Childs
> Student
> Asst. Head of IT Services
> Charlotte United Christian Academy
> (704) 497-2086
> RileyChilds.net
>  Sent from my Windows Phone, please excuse mistakes
>
> -Original Message-
> From: "Owen Stephens" 
> Sent: ‎7/‎4/‎2014 4:55 AM
> To: "CODE4LIB@LISTSERV.ND.EDU" 
> Subject: [CODE4LIB] 'automation' tools
>
> I'm doing a workshop in the UK at a library tech unconference-style event
> (Pi and Mash http://piandmash.info) on automating computer based tasks.
> I want to cover tools that are usable by non-programmers and that would
> work in a typical library environment. The types of tools I'm thinking of
> are:
>
> MacroExpress
> AutoHotKey
> iMacros for Firefox
>
> While I'm hoping workshop attendees will bring ideas about tasks they
> would like to automate the type of thing I have in mind are things like:
>
> Filling out a set of standard data on a GUI or Web form (e.g. standard set
> of budget codes for an order)
> Processing a list of item barcodes from a spreadsheet and doing something
> with them on the library system (e.g. change loan status, check for holds)
> Similarly for User IDs
> Navigating to a web page and doing some task
>
> Clearly some of these tasks would be better automated with appropriate
> APIs and scripts, but I want to try to introduce those without programming
> skills to some of the concepts and tools and essentially how they can work
> around problems themselves to some extent.
>
> What tools do you use for this kind of automation task, and what kind of
> tasks do they best deal with?
>
> Thanks,
>
> Owen
>
> Owen Stephens
> Owen Stephens Consulting
> Web: http://www.ostephens.com
> Email: o...@ostephens.com
> Telephone: 0121 288 6936
>

Re: [CODE4LIB] 'automation' tools

[Riley Childs]

Don't forget AutoIT (auto IT, pretty clever eh?)
http://www.autoitscript.com/site/autoit/

Riley Childs
Student
Asst. Head of IT Services
Charlotte United Christian Academy
(704) 497-2086
RileyChilds.net
 Sent from my Windows Phone, please excuse mistakes

-Original Message-
From: "Owen Stephens" 
Sent: ‎7/‎4/‎2014 4:55 AM
To: "CODE4LIB@LISTSERV.ND.EDU" 
Subject: [CODE4LIB] 'automation' tools

I'm doing a workshop in the UK at a library tech unconference-style event (Pi 
and Mash http://piandmash.info) on automating computer based tasks.
I want to cover tools that are usable by non-programmers and that would work in 
a typical library environment. The types of tools I'm thinking of are:

MacroExpress
AutoHotKey
iMacros for Firefox

While I'm hoping workshop attendees will bring ideas about tasks they would 
like to automate the type of thing I have in mind are things like:

Filling out a set of standard data on a GUI or Web form (e.g. standard set of 
budget codes for an order)
Processing a list of item barcodes from a spreadsheet and doing something with 
them on the library system (e.g. change loan status, check for holds)
Similarly for User IDs
Navigating to a web page and doing some task 

Clearly some of these tasks would be better automated with appropriate APIs and 
scripts, but I want to try to introduce those without programming skills to 
some of the concepts and tools and essentially how they can work around 
problems themselves to some extent.

What tools do you use for this kind of automation task, and what kind of tasks 
do they best deal with?

Thanks,

Owen

Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com
Email: o...@ostephens.com
Telephone: 0121 288 6936

[CODE4LIB] 'automation' tools

[Owen Stephens]

I'm doing a workshop in the UK at a library tech unconference-style event (Pi 
and Mash http://piandmash.info) on automating computer based tasks.
I want to cover tools that are usable by non-programmers and that would work in 
a typical library environment. The types of tools I'm thinking of are:

MacroExpress
AutoHotKey
iMacros for Firefox

While I'm hoping workshop attendees will bring ideas about tasks they would 
like to automate the type of thing I have in mind are things like:

Filling out a set of standard data on a GUI or Web form (e.g. standard set of 
budget codes for an order)
Processing a list of item barcodes from a spreadsheet and doing something with 
them on the library system (e.g. change loan status, check for holds)
Similarly for User IDs
Navigating to a web page and doing some task 

Clearly some of these tasks would be better automated with appropriate APIs and 
scripts, but I want to try to introduce those without programming skills to 
some of the concepts and tools and essentially how they can work around 
problems themselves to some extent.

What tools do you use for this kind of automation task, and what kind of tasks 
do they best deal with?

Thanks,

Owen

Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com
Email: o...@ostephens.com
Telephone: 0121 288 6936