Hi All,
Sorry to bump a really old thread, but I just ran into this problem today,
and figured I should share my notes:
http://library.osu.edu/blogs/it/merge-two-assetstores-dspace/
I think the best part of the story is that we didn't have any nagios alerts
on this box, I just randomly ssh'ed in, and noticed that the old
assetstores we're nearing 99.9% full. DSpace log didn't yet have any
application errors, since there was still some 500MB of space left...
yikes.
We have inherited a DSpace instance from another service provider, and they
have used multiple assetstores to hold the bitstreams. Our sysadmin can do
disk-magic, and just present large disks, so we are going to merge two
existing assetstores into a single one.
Here’s a simplification of the situation.
/assetstore0 == 200GB
/assetstore1 == 100GB
/consolidatedAssetstore == 400GB
dspace.cfg has:
assetstore.dir = /dspace/assetstore
assetstore.dir.1 = /assetstore1
assetstore.incoming = 0
And /dspace/assetstore is a symlink to /assetstore0
So. To address this, my process is:
##sync contents of the two assetstores to a single assetstore
rsync -a –progress –stats /assetstore0/ /consolidatedAssetstore
rsync -a –progress –stats /assetstore1/ /consolidatedAssetstore
##point the symlink to the new storage location
unlink /dspace/assetstore
ln -s /consolidatedAssetstore /dspace/assetstore
##run some sql to “merge” the two assetstores
psql dspace
update bitstream set store_number=0;
##Trim the dspace.cfg to remove the second assetstore.
assetstore.dir = /dspace/assetstore
assetstore.incoming = 0
Restart tomcat, so that the updated dspace.cfg takes effect.
Then test to make sure your bitstreams still work.
Lastly, here’s a SQL query to find bitstreams in assetstore2, hosted on
GitHub Gist.
https://gist.github.com/peterdietz/7009560
Peter Dietz
On Tue, Jun 14, 2011 at 2:39 PM, Blanco, Jose <[email protected]> wrote:
> Oh, ok I see. I share in your paranoia and will keep this in mind.
>
> Thank you!
> Jose
>
> -----Original Message-----
> From: Brian Freels-Stendel [mailto:[email protected]]
> Sent: Tuesday, June 14, 2011 2:26 PM
> To: [email protected]; Blanco, Jose
> Subject: Re: [Dspace-tech] mergeing assetstores.
>
> Certainly, if they differ, there's no problem. (And, to fess up, I was
> considering the last string as part of the string, not the filename, but it
> leaves us with the same situation.) If they're the same, one would
> overwrite the other. As I say, it's unlikely that the randomly-generated
> numbers would be the same, but it's possible. I'm a little paranoid about
> unlikely-but-possible things, but I think they're worth keeping in mind.
>
> B--
>
> >>> On 6/14/2011 at 12:07 PM, in message
>
> <54fa544e85d3934baa21bf8a84be5cd9171d464...@itcs-ecls-1-vs3.adsroot.itcs.umich.e
> u>, "Blanco, Jose" <[email protected]> wrote:
> > I don't quite see what you're saying.
> >
> > Say I have two files, could they not both be in the same dir, but but
> have
> > different names, like here:
> >
> > /assetstore/10/04/02/10040212345
> > /assetstore/10/04/02/10040209898
> >
> > ?
> >
> > -----Original Message-----
> > From: Brian Freels-Stendel [mailto:[email protected]]
> > Sent: Tuesday, June 14, 2011 2:00 PM
> > To: [email protected]; Blanco, Jose
> > Subject: RE: [Dspace-tech] mergeing assetstores.
> >
> > At that level, they don't have names...only the path and then the file
> as a
> > string of bits. The names and file extensions are handled in the
> database.
> >
> > B--
> >
> >>>> On 6/14/2011 at 11:56 AM, in message
> >
> <54fa544e85d3934baa21bf8a84be5cd9171d464...@itcs-ecls-1-vs3.adsroot.itcs.umich.e
> >
> > u>, "Blanco, Jose" <[email protected]> wrote:
> >> But why would two items having the same path be a problem as long as
> their
> >> name is distinct.
> >>
> >> -Jose
> >>
> >> -----Original Message-----
> >> From: Brian Freels-Stendel [mailto:[email protected]]
> >> Sent: Tuesday, June 14, 2011 1:11 PM
> >> To: [email protected]; Blanco, Jose
> >> Subject: Re: [Dspace-tech] mergeing assetstores.
> >>
> >> Hi Jose,
> >>
> >> This process should work, with one thing to look out for. It's
> unlikely,
> >> but possible, that items might be assigned the same path in both
> > assetstores.
> >> Your merge process should be able to alert you about that.
> >>
> >> B--
> >>
> >>>>> On 6/14/2011 at 9:48 AM, in message
> >>
> <54fa544e85d3934baa21bf8a84be5cd9171d464...@itcs-ecls-1-vs3.adsroot.itcs.umich.e
> >
> >>
> >> u>, "Blanco, Jose" <[email protected]> wrote:
> >>> The intent of this email it to verify that merging two assetstores is
> doable
> >
> >>
> >>> in dspace.
> >>>
> >>> Presently we have two assetstores:
> >>>
> >>> assetstore.dir = /directory1
> >>> assetstore.dir.1 = /directory2
> >>>
> >>> what we want to do is to merge these two onto new storage hardware.
> So what
> >
> >>
> >>> we plan to do is:
> >>>
> >>> 1. Merge onto the new location ( with the instance down towards the
> end of
> >>> the merge )
> >>> 2. Change the dspace.cfg parameter to look like this:
> >>>
> >>> assetstore.dir = /new_location
> >>>
> >>> 3. run a this sql:
> >>>
> >>> Update bitstream set store_number=0;
> >>>
> >>> 4. Bring our instance back up.
> >>>
> >>> I believe it should be this easy, but just wanted to make sure.
> >>>
> >>> Thank you!
> >>> Jose
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> EditLive Enterprise is the world's most technically advanced content
> >>> authoring tool. Experience the power of Track Changes, Inline Image
> >>> Editing and ensure content is compliant with Accessibility Checking.
> >>> http://p.sf.net/sfu/ephox-dev2dev
> >>> _______________________________________________
> >>> DSpace-tech mailing list
> >>> [email protected]
> >>> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> >
> >
> >
> ------------------------------------------------------------------------------
> > EditLive Enterprise is the world's most technically advanced content
> > authoring tool. Experience the power of Track Changes, Inline Image
> > Editing and ensure content is compliant with Accessibility Checking.
> > http://p.sf.net/sfu/ephox-dev2dev
> > _______________________________________________
> > DSpace-tech mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
> ------------------------------------------------------------------------------
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> DSpace-tech mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
