Hey Andrea, Thanks for the information, good to know. I have already requested a JIRA account and can only create the issue after I am accepted.
This bug was detected and solved whilst implementing a new bitstore storage to connect DSpace to Google Cloud Storage. With this bug fix, our implementation is now finished and we want to create a PR for that too ( and a JIRA issue as well? ). Should I create 2 separate issues and 2 separate PRs? Or it can/should be done all in one? Thanks again, Pedro Amorim quarta-feira, 18 de Janeiro de 2017 às 16:48:18 UTC-1, Bollini Andrea escreveu: > > The commit is quite old but it has been included only in the 6.0 release(s) > $ git tag --contains 54222f3c1d3c63bd03af263703a950e9cfad7439 > dspace-6.0 > dspace-6.0-rc1 > dspace-6.0-rc2 > dspace-6.0-rc3 > dspace-6.0-rc4 > > so only very recent DSpace installations (6.0) are affected. > > I'm pretty confident about the fix... it will be nice if you can create a > PR and maybe add some unit test specific for these use cases. Otherwise > feel free to create a JIRA issue for that and assign it to me, I will > create the PR in the next couple of days. > > Andrea > > > > Il 18/01/2017 17:41, Pedro Amorim ha scritto: > > Hey Andrea, > > I implement the change you suggest and it's working perfectly. And before > my last reply I was also suspecting this was a core bug. > However, I'm still a bit skeptical about it, because if this is true, it > means that any DSpace installation after nov 8, 2014 has this problem. If > this is an actual core bug, every installation from nov 8, 2014 has its > assetstore with orphaned objects. > > It would be great to have input from even other developers (more > experienced than me) to at least comment on this. > > Meanwhile, I will do as you suggest and submit the JIRA for it and also > create a pull request on github. > > Thank you. > > Pedro Amorim > > quarta-feira, 18 de Janeiro de 2017 às 12:47:11 UTC-1, Bollini Andrea > escreveu: >> >> Hi Pedro, >> >> yes it looks to me that the line >> >> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/BitstreamStorageServiceImpl.java#L289 >> is buggy, it should be >> >> if(0 == bitstreamService.findDuplicateInternalIdentifier(context, >> bitstream).size()) >> also the history of the sourcecode seems to indicate such solution >> >> https://github.com/DSpace/DSpace/commit/54222f3c1d3c63bd03af263703a950e9cfad7439#diff-4858f5a048b9ccffb27b7072486403bcL642 >> >> as we want to remove the file from the assetstore only if it is not used >> by other bitstreams. >> Can you open (or update if exist) a JIRA issue for that? >> Andrea >> >> >> Il 18/01/2017 13:57, Pedro Amorim ha scritto: >> >> Hello Bollini, >> >> Thank you for your input! >> Yes I am aware of that, as mentioned in my other thread regarding this >> issue: >> >> >> https://groups.google.com/forum/#!searchin/dspace-tech/cleanup$20not$20working%7Csort:relevance/dspace-tech/gz73RSg4Fhs/736BHQhmBgAJ >> >> <https://groups.google.com/forum/#%21searchin/dspace-tech/cleanup$20not$20working%7Csort:relevance/dspace-tech/gz73RSg4Fhs/736BHQhmBgAJ> >> >> The following is my understanding of the code: >> >> The cleanup command runs the cleanup function in >> BitstreamStorageServiceImpl.java: >> >> >> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/BitstreamStorageServiceImpl.java#L220 >> >> However, it will only actually call the remove() method in line 291: >> >> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/BitstreamStorageServiceImpl.java#L291 >> >> Which needs *bitstreamService.findDuplicateInternalIdentifier(context, >> bitstream).size() to return* > 0 and it never will. It always returns 0 >> and thus never actually calls the remove() method. >> >> It will never return > 0 because the function >> *findDuplicateInternalIdentifier* is just a validation for versioning >> and should only return > 0 if theres more than one bitstream ID for a given >> bitstream internal ID, as seen: >> >> >> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/content/dao/impl/BitstreamDAOImpl.java#L51 >> >> The above is my interpretation of the code and the point where I'm stuck >> at the moment. Note that I can't get it to work in local standard storage >> as well. Using DSpace 6.0. >> >> Thanks, >> >> Pedro Amorim >> >> quarta-feira, 18 de Janeiro de 2017 às 11:35:45 UTC-1, Bollini Andrea >> escreveu: >>> >>> Hi Pedro, >>> >>> I could say something obvious but just to check: >>> >>> are you aware that a deletion from the UI is just a "promise"? the >>> bitstream row is flagged with deleted = true, the real deletion is >>> performed by the cleanup script >>> >>> i.e. [dspace-dir]/bin/dspace cleanup >>> >>> >>> https://github.com/DSpace/DSpace/blob/ea642d6c9289d96b37b5de3bb7a4863ec48eaa9c/dspace/config/launcher.xml#L37 >>> >>> >>> https://github.com/DSpace/DSpace/blob/269af71afb602808a14edf822ad658c3895e0a37/dspace-api/src/main/java/org/dspace/storage/bitstore/Cleanup.java >>> >>> Best, >>> >>> Andrea >>> >>> Il 18/01/2017 13:08, Pedro Amorim ha scritto: >>> >>> Hello Jaime, >>> >>> Did you come to any conclusion? >>> i'm having the same problem but with Google Cloud Storage and also local >>> standard storage. You can follow the thread here: >>> >>> https://groups.google.com/forum/#!searchin/dspace-tech/cleanup$20not$20working%7Csort:relevance/dspace-tech/gz73RSg4Fhs/736BHQhmBgAJ >>> >>> <https://groups.google.com/forum/#%21searchin/dspace-tech/cleanup$20not$20working%7Csort:relevance/dspace-tech/gz73RSg4Fhs/736BHQhmBgAJ> >>> >>> Thanks, >>> >>> Pedro Amorim >>> >>> quarta-feira, 11 de Janeiro de 2017 às 20:00:04 UTC-1, Jaime Solorzano >>> escreveu: >>>> >>>> >>>> >>>> I was watching this but it does not >>>> >>>> >>>> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/storage/bitstore/S3BitStoreService.java#L247 >>>> >>>> >>>> Jaime >>>> >>>> >>>> El miércoles, 11 de enero de 2017, 11:59:27 (UTC-5), Jaime Solorzano >>>> escribió: >>>>> >>>>> Hello >>>>> >>>>> I see that cleanup does not delete the s3 files, which method should >>>>> be used? >>>>> >>>>> Someone done this? >>>>> >>>>> >>>>> Jaime Solorzano >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Technical Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/dspace-tech. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> Andrea Bollini >>> Chief Technology and Innovation Officer >>> >>> 4Science, www.4science.it >>> office: Via Edoardo D'Onofrio 304, 00155 Roma, Italy >>> mobile: +39 333 934 1808 >>> skype: a.bollini >>> linkedin: andreabollini >>> orcid: 0000-0002-9029-1854 >>> >>> an Itway Group Company >>> Italy, France, Spain, Portugal, Greece, Turkey, Lebanon, Qatar, U.A.Emirates >>> >>> >> -- >> Andrea Bollini >> Chief Technology and Innovation Officer >> >> 4Science, www.4science.it >> office: Via Edoardo D'Onofrio 304, 00155 Roma, Italy >> mobile: +39 333 934 1808 >> skype: a.bollini >> linkedin: andreabollini >> orcid: 0000-0002-9029-1854 >> >> an Itway Group Company >> Italy, France, Spain, Portugal, Greece, Turkey, Lebanon, Qatar, U.A.Emirates >> >> > -- > Andrea Bollini > Chief Technology and Innovation Officer > > 4Science, www.4science.it > office: Via Edoardo D'Onofrio 304, 00155 Roma, Italy > mobile: +39 333 934 1808 > skype: a.bollini > linkedin: andreabollini > orcid: 0000-0002-9029-1854 > > an Itway Group Company > Italy, France, Spain, Portugal, Greece, Turkey, Lebanon, Qatar, U.A.Emirates > > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
