Gary To be pedantically correct, I should have said the "store_id" field of the "bitstream" table. It seems to be "randomly" generated - its is a key to the bucket where the data is kept. And a store_id number translates into a directory path to the bitstream:
So a store_id of 129683849070764175216956097535764632127 means that the pathname to the object in the assetstore is assetstore/12/96/83/129683849070764175216956097535764632127 That is, the first three tuples of the store number become subdirectory paths in the assetstore And it appears that if you try different times with the same entry, you will get a different store number. I am willing to bet that this is all designed to spread the object around the buckets efficiently to speed access (computer science data structure and storage lectures from uni courses decades ago are being salvaged from deep memory in the brain here....) And so Susan's problems probably stem from ownerships and permissions of the creation of the original three tuples of directory path from previous loads by processes with other ownership and permissions. Happy sleeping tonight ... Gary Browne wrote: > Hang on a minute... > > Tony Kyne wrote: > "From what we can gather, > the bitstream identifier created has some random aspect to it." > > > Are you talking about the sequence ID for the bitstream, or the bundle's > handle? I know I'm obsessed with the sequence ID generation process but my belief was that it is (somehow) sequentially generated (hence the name, I guess). > > Can anyone confirm or deny so I can sleep tonight? > > Cheers > Gary > > > Gary Browne > Development Programmer > Library IT Services > University of Sydney > Australia > ph: 61-2-9351 5946 > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Tony Kyne > Sent: Tuesday, 8 May 2007 9:44 AM > To: susan rector > Cc: [email protected]; George Kozak > Subject: Re: [Dspace-tech] help with import - bitstream error > > Susan and others > > We had that problem with an 11,000+ upload. Drove us spare - we even > thought the > there was a hardware problem given the random nature of the problem: it bombs > out > with no such file error, but immediately on restart it works. Even tried > different > version of Java, operating system patches etc,etc > > However, we ultimately worked it out what the problem was. From what > we can gather, > the bitstream identifier created has some random aspect to it. The lead > numbers of > this identifier give the start of the directory path in the assetstore to > store the > images etc. And so if you upload a file on two separate occassions, you get > different magic numbers and hence different directory paths into the > assetstore. So > the problem is directory paths. And if you have the same problem as we had, > bits of > the some of the directory paths existed before the current upload, and thay > had > owenerships and permissions that the owner of the current upload process did > not have > access to: the "No such file..." error, is really saying "Permissions for the > required directory access not valid ...." [ or from our scouring the web, it > is the > Java catch all error message for any sort of possible file access problems] > > So look at the ownership and permissions of the various bits of your > assetstore. In > our case, the problem was a few test examples we did way back at the start of > setting > up and testing Dspace, and these were out of sync with our current development > environment ownership/permission regime. > > susan rector wrote: >> Thanks George, >> I just got them all in the system (467). It seemed to bomb out when > the >> directory had a lot of images or .mov files. I think for our > production >> version, I'm going to just import by year from etd-db so that I can avoid the continuous restart of of the program >> >> thanks again, >> Susan >> >> George Kozak wrote: >>> Susan: >>> >>> I have run into this problem running big batches. Every so often, > the >>> batch importer comes up with the error (at least for me). If I then batch >>> upload with smaller groups of items (without making other changes) everything runs fine. >>> >>> At 11:38 AM 5/7/2007, susan rector wrote: >>>> Hi all, >>>> I fixed the last error with the ^M - thanks for all your help >>>> >>>> Now, I'm running into another strange import error. I run the dspace >>>> importer in test mode and it works fine and imports 466 distinct directories with metadata and files. >>>> Test mode: /dspace/bin/dsrun org.dspace.app.itemimport.ItemImport > --test >>>> --add [EMAIL PROTECTED] --collection=5 >>>> --source=/usr/local/archive_directory --mapfile mapfile.txt >>>> >>>> When I run the same command just not in test mode, I get the > following >>>> error stating that the contents file isn't there. In fact, it is > there >>>> and has a listing of the files in that directory >>>> Add: /dspace/bin/dsrun org.dspace.app.itemimport.ItemImport --add [EMAIL PROTECTED] --collection=5 >>>> --source=/usr/local/archive_directory --mapfile mapfile.txt >>>> >>>> I've changed all the file permissions thinking that might be it - > didn't >>>> work. It seems to get stuck on the jpeg's. >>>> >>>> Thanks for any suggestions! >>>> Susan Rector >>>> [EMAIL PROTECTED] >>>> >>>> >>>> Error: >>>> Processing contents file: >>>> /usr/local/archive_directory/etd-05152006-125030/contents >>>> Bitstream: benitezjm_image06.jpg >>>> Bitstream: benitezjm_thesis.pdf >>>> Bitstream: benitezjm_image04.jpg >>>> Bitstream: benitezjm_image02.jpg >>>> java.io.IOException: No such file or directory >>>> at java.io.UnixFileSystem.createFileExclusively(Native > Method) >>>> at java.io.File.createNewFile(File.java:883) >>>> at >>>> edu.sdsc.grid.io.local.LocalFile.createNewFile(LocalFile.java:486) >>>> at >>>> > org.dspace.storage.bitstore.BitstreamStorageManager.store(BitstreamStora geManager.java:332) >>>> >>>> at org.dspace.content.Bitstream.create(Bitstream.java:184) at > org.dspace.content.Bundle.createBitstream(Bundle.java:362) >>>> at >>>> > org.dspace.app.itemimport.ItemImport.processContentFileEntry(ItemImport. > java:1103) >>>> >>>> at >>>> > org.dspace.app.itemimport.ItemImport.processContentsFile(ItemImport.java > :1030) >>>> >>>> at >>>> org.dspace.app.itemimport.ItemImport.addItem(ItemImport.java:631) >>>> at >>>> org.dspace.app.itemimport.ItemImport.addItems(ItemImport.java:498) >>>> at >>>> org.dspace.app.itemimport.ItemImport.main(ItemImport.java:407) >>>> java.io.IOException: No such file or directory >>>> >>>> >>>> >>>> Contents file: >>>> benitezjm_image06.jpg >>>> benitezjm_thesis.pdf >>>> benitezjm_image04.jpg >>>> benitezjm_image02.jpg >>>> benitezjm_image05.jpg >>>> benitezjm_image01.jpg >>>> benitezjm_image03.jpg >>>> >>>> >>>> > ------------------------------------------------------------------------ - >>>> >>>> This SF.net email is sponsored by DB2 Express >>>> Download DB2 Express C - the FREE version of DB2 express and take control >>>> of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ >>>> _______________________________________________ >>>> DSpace-tech mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/dspace-tech >>> >>> *************************** >>> George Kozak >>> Coordinator >>> Web Development and Management >>> Digital Media Group >>> 501 Olin Library >>> Cornell University >>> 607-255-8924 >>> *************************** >>> [EMAIL PROTECTED] >> >> >> > ------------------------------------------------------------------------ - >> This SF.net email is sponsored by DB2 Express >> Download DB2 Express C - the FREE version of DB2 express and take control of >> your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ >> _______________________________________________ >> DSpace-tech mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> > > > regards, Tony > > ======================================================================== === > Dr. Tony Kyne, Head, Information Technology Services, > The Walter and Eliza Hall Institute of Medical Research, 1G > Royal Parade, Parkville, Victoria, 3050, Australia. > Phone: International +61-3-9345-2586 FAX: International > +61-3-9347-0852 > National 03-9345-2586 National > 03-9347-0852 > Email: [EMAIL PROTECTED] > ======================================================================== == > > > ------------------------------------------------------------------------ - > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take control of > your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > DSpace-tech mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/dspace-tech > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take control of > your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > DSpace-tech mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/dspace-tech > regards, Tony =========================================================================== Dr. Tony Kyne, Head, Information Technology Services, The Walter and Eliza Hall Institute of Medical Research, 1G Royal Parade, Parkville, Victoria, 3050, Australia. Phone: International +61-3-9345-2586 FAX: International +61-3-9347-0852 National 03-9345-2586 National 03-9347-0852 Email: [EMAIL PROTECTED] ========================================================================== regards, Tony =========================================================================== Dr. Tony Kyne, Head, Information Technology Services, The Walter and Eliza Hall Institute of Medical Research, 1G Royal Parade, Parkville, Victoria, 3050, Australia. Phone: International +61-3-9345-2586 FAX: International +61-3-9347-0852 National 03-9345-2586 National 03-9347-0852 Email: [EMAIL PROTECTED] ========================================================================== ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech
