Hello Peter, First of all, I'm sorry for not replying sooner, I had some rather long (and deserved) vacations and also because I didn't want to get back to this before starting to implement it. Secondly, thank you so much for replying to this as your input as helped me a lot in my work. Now, I've had many difficulties trying to implement this partly due to being very inexperienced in java, never worked with GCS client library before nor have I ever programmed in the DSpace core.
I believe I've made some good progress, I can already write and read files from a bucket in Google Cloud Storage. Everything works as expected in the application and everything is surprisingly fast and consistent. However, I'm struggling to delete objects from the bucket and the main problem is that the remove() (in my GCSBitStoreService.java class) function is never called. Every other method is called and working as expected (put, get and about) and I also implemented the updateTime, hash and Etag. Note that I can remove the object in the application, it gets removed from the record, from the database and the index, just not from the bitstore (Google Cloud Storage). I was expecting the object to remain in the bucket until cleanup is run - either as cron or manually, because when you delete an object from the application it actually doesn't remove it from the storage. After that, I run *cleanup -v* and everything runs smoothly, no error no exceptions nothing, but the object still remains in the bucket. It can't be related to bucket permissions because the application never actually runs the remove() method (which contains the Storage call). After that, I started digging in the cleanup method: https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/BitstreamStorageServiceImpl.java#L220 Noticed the 'recent' validation and made sure the object was deleted more than an hour ago. Still no success. Even commented that out and nothing. Looking at the cleanup code, I can't really tell where the object is supposed to be removed. It seems to me it only calls the remove() method if versioning is enabled and more than one object if found: https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/BitstreamStorageServiceImpl.java#L289 Any insight on this? Also, one last question: If I am to contribute my implementation to the DSpace project how should I go about it? Make a pull request in dspace6.0 branch on github? Sorry for the wall of text. Thank you very much, Pedro Amorim segunda-feira, 19 de Dezembro de 2016 às 13:50:49 UTC-1, Peter Dietz escreveu: > > Hi Pedro, > > Yes, these steps look like what you have to do to add a new > storage/BitStore implementation. And yes, add the Google SDK > to dspace-api/pom.xml instead of dspace/pom.xml. > > One implementation pointer I'd like to suggest is that it would be nice to > support the use case of simple/small objects ~5MB, and also multipart > upload/download so that it can work with large 5GB - 5TB sized objects. > https://cloud.google.com/storage/docs/json_api/v1/how-tos/multipart-upload > > Also, be mindful as to what you want to use to track the checksums. > https://cloud.google.com/storage/docs/hashes-etags > > Lastly, ensure that you close any resources opened when putting or getting > objects to the store. You can run out of resources if you open an HTTP > connection each time, but never close it. > > Good luck. > > > ________________ > Peter Dietz > Longsight > www.longsight.com > [email protected] <javascript:> > p: 740-599-5005 x809 > > On Mon, Dec 19, 2016 at 7:52 AM, Pedro Amorim <[email protected] > <javascript:>> wrote: > >> Just noticed, the GCS Java client in 1) should be added to >> [dspace-source]/dspace-api/pom.xml instead >> of [dspace-source]/dspace/pom.xml as originally mentioned. >> >> segunda-feira, 19 de Dezembro de 2016 às 11:46:10 UTC-1, Pedro Amorim >> escreveu: >>> >>> Hello everyone, >>> >>> I'd like to store DSpace bitstreams in a GCS bucket, much like the way >>> the S3BitStore is implemented. >>> Before I start tinkering and testing, I'd much appreciate some input on >>> the matter from more experienced Java programmers, namely DSpace >>> programmers. >>> >>> So, from what I've gathered so far, I need to: >>> >>> 1) Add Google Cloud Storage Java libraries in >>> [dspace-source]/dspace/pom.xml and perform a rebuild. This will be needed >>> for step 2). These libraries are as provided here: >>> >>> https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-java >>> >>> 2) Create and implement a new GCSBitStoreService.java based on the one >>> created for S3: >>> >>> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/S3BitStoreService.java >>> >>> 3) Add new BitStore in bitstore.xml: >>> >>> https://github.com/DSpace/DSpace/blob/master/dspace/config/spring/api/bitstore.xml >>> >>> 4) Activate the new BitStore in bitstore.xml as documented here: >>> >>> https://wiki.duraspace.org/display/DSDOC6x/Storage+Layer#StorageLayer-ConfiguringAmazonS3Storage >>> >>> And that's about it? >>> >>> Thanks as always, >>> >>> Pedro Amorim >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "DSpace Technical Support" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/dspace-tech. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
