On Jul 24, 2009, at 2:20 PM, [Chris Stockwell] wrote:
Over the next few years, I am tasked to download 30,000 archival
masters
from Internet Archive into an archive for long-term staff access
that we may
preserve with LOCKSS. These are masters of Montana state
publications. I
have a hierarchy in mind to receive these files. The hierarchy is
state
agency\year\title\pub_date\*.pdf.
I am intending to download the files in batches of 200 - 500 pdfs,
but am
thinking that if I slot them automatically into the archive
hierarchy, misplaced
or missing files could be very hard to find as the total grows. I
will be logging
the downloads, which should give me some control. Are there other
strategies
for ensuring that I can readily correct download errors? I am
looking for
recommendations for the simplest way to maintain reasonable control
over the
download process.
A couple things:
If you already have archive.org identifiers picked out, you can use
something like this python script to download them all from IA:
http://blog.openlibrary.org/2008/11/24/bulk-access-to-ocr-for-1-million-books/
You can use the archive.org advanced search engine to produce xml,
json, or csv file with all identifiers for a particular contributing
institution:
http://www.archive.org/advancedsearch.php
eg. all identifier for Montana State Library (http://www.archive.org/details/MontanaStateLibrary
) as an xml file (change rows=10 to rows=10000 to get them all):
http://www.archive.org/advancedsearch.php?q=collection%3A%22montanastatelib%22&fl%5B%5D=creator&fl%5B%5D=identifier&fl%5B%5D=title&sort%5B%5D=&sort%5B%5D=&sort%5B%5D=&rows=10&fmt=xml&xmlsearch=Search
Also, if you have an archive.org identifier, then you can get the
files.xml that contains md5 and sha1 hashes, so you can verify your
download:
To pull the files.xml, use a /download/id/id_files.xml url. e.g.:
http://www.archive.org/download/librariesoffutur00lickuoft/librariesoffutur00lickuoft_files.xml
-raj
