Re: [Dspace-tech] Hardware recommendations
Dear Cory, my apologies for not being clear enough. For the asset store, you're fine with slower disks & tier 3 storage products. But it's indeed more particular the postgres and lucene I/O load that require fast disk access. You're totally right. best regards, Bram Luyten On Thu, May 22, 2008 at 3:14 PM, Cory Snavely <[EMAIL PROTECTED]> wrote: > Our experience is markedly different, and I'm particularly struck by the > comment about 15Krpm disk. > > We use 15Krpm SCSI disk in RAID 1 for Postgres, but we use SATA RAID 6 > for the assetstore and do not see DSpace I/O-bound against it. (FYI > we're working on transitioning our assetstore to a Sun Honeycomb.) > > I'm not sure your units are correct for the assetstore--7GB?--but if so, > sure, a little space on your 15Krpm disk is fine. If you mean 7TB, then > you're talking considerable expense over SATA that I am pretty confident > would *not* give any benefit. I/O against the assetstore you is low > volume sequential read and occasional high-volume sequential write, and > that is 100% consistent with "tier 3" type storage products and what > SATA RAID does best. In fact I would think it would be perfectly > reasonable to put the assetstore on NAS. > > FWIW we are currently deploying a replacement server for DSpace; it's a > dual quad-core Xeon server w/ 8GB RAM with 15Krpm internal SAS disk in > RAID 1 and 9TB SATA RAID 6. Our experience has shown that we need to > handle heavy multiprocessing Postgres load as well as large memory > allocation and that aside from Postgres storage I/O requirements are > relatively light. > > Cory Snavely > University of Michigan Library IT Core Services > > On Thu, 2008-05-22 at 14:54 +0200, Bram Luyten wrote: > > Hello Jim, > > > > we have installed and manage some very large DSpaces, but also a few > > moderate ones. An example of a large installation: > > > > http://lirias.kuleuven.be holds around 130.000 items, of which only > > around 2500 of them contain full-text. This asset store is around 7GB. > > It's mainly academic research output (papers, conference > > presentations, ...). So currently, the average size of a bitstream is > > around 2.8MB. But this will be very different if your repository is > > oriented towards datasets, audio or video. > > > > Concerning processing power and memory: the system currently has > > around 1000 unique visitors daily. There are 4000 e-persons. During > > office hours, we experience an average of 4 concurrent logged-in > > users, performing submissions, etc (= intensive on database and > > indexes). The tomcat has been given 2GB of memory, while the whole > > system has 3.5GB of RAM. This doesn't really cover the total load, as > > we use swapping, but it's enough to keep the system running. A > > recommendation: don't cut down on disk speed, you will need 15.000 rpm > > disks. > > > > The server has 1 physical dual-core processor, with hyper threading > > (so actually 4 virtual processors). In peak times, this is becoming a > > bottle neck. > > > > If you could illustrate the purposes of your installation, or the > > estimated number of users, I could provide you with a specific case, > > that possibly more closly matches what you're looking at. > > > > with kindest regards, > > > > Bram Luyten > > > > -- > > @mire NV > > Romeinse Straat 18 > > 3001 Heverlee > > Belgium > > +32 2 888 29 56 > > > > http://www.atmire.com - Institutional Repository Solutions > > http://www.togather.eu - Before getting together, get [EMAIL PROTECTED] > > > > On Wed, May 21, 2008 at 4:33 PM, Jim Price > > <[EMAIL PROTECTED]> wrote: > > Hello, > > > > We are currently running a test instance on a Sun Enterprise > > 450 running Solaris 10. We are currently looking into new > > hardware for a production environment. > > We would like to find out if anyone is willing to share their > > experiences with hardware? > > > > What platform would you recommend for a school starting with > > dspace? > > What are you using for hardware? > > What do you store? > > How big is your data storage? > > How much growth did you see in your first year of use? > > What kind of loads are you experiencing? > > > > > > Thanks, > > Jim > > > > > > > > > - > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > > ___ > > DSpace-tech mailing list > > DSpace-tech@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/dspace-tech > > > > > > > > - > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studi
Re: [Dspace-tech] Hardware recommendations
Our experience is markedly different, and I'm particularly struck by the comment about 15Krpm disk. We use 15Krpm SCSI disk in RAID 1 for Postgres, but we use SATA RAID 6 for the assetstore and do not see DSpace I/O-bound against it. (FYI we're working on transitioning our assetstore to a Sun Honeycomb.) I'm not sure your units are correct for the assetstore--7GB?--but if so, sure, a little space on your 15Krpm disk is fine. If you mean 7TB, then you're talking considerable expense over SATA that I am pretty confident would *not* give any benefit. I/O against the assetstore you is low volume sequential read and occasional high-volume sequential write, and that is 100% consistent with "tier 3" type storage products and what SATA RAID does best. In fact I would think it would be perfectly reasonable to put the assetstore on NAS. FWIW we are currently deploying a replacement server for DSpace; it's a dual quad-core Xeon server w/ 8GB RAM with 15Krpm internal SAS disk in RAID 1 and 9TB SATA RAID 6. Our experience has shown that we need to handle heavy multiprocessing Postgres load as well as large memory allocation and that aside from Postgres storage I/O requirements are relatively light. Cory Snavely University of Michigan Library IT Core Services On Thu, 2008-05-22 at 14:54 +0200, Bram Luyten wrote: > Hello Jim, > > we have installed and manage some very large DSpaces, but also a few > moderate ones. An example of a large installation: > > http://lirias.kuleuven.be holds around 130.000 items, of which only > around 2500 of them contain full-text. This asset store is around 7GB. > It's mainly academic research output (papers, conference > presentations, ...). So currently, the average size of a bitstream is > around 2.8MB. But this will be very different if your repository is > oriented towards datasets, audio or video. > > Concerning processing power and memory: the system currently has > around 1000 unique visitors daily. There are 4000 e-persons. During > office hours, we experience an average of 4 concurrent logged-in > users, performing submissions, etc (= intensive on database and > indexes). The tomcat has been given 2GB of memory, while the whole > system has 3.5GB of RAM. This doesn't really cover the total load, as > we use swapping, but it's enough to keep the system running. A > recommendation: don't cut down on disk speed, you will need 15.000 rpm > disks. > > The server has 1 physical dual-core processor, with hyper threading > (so actually 4 virtual processors). In peak times, this is becoming a > bottle neck. > > If you could illustrate the purposes of your installation, or the > estimated number of users, I could provide you with a specific case, > that possibly more closly matches what you're looking at. > > with kindest regards, > > Bram Luyten > > -- > @mire NV > Romeinse Straat 18 > 3001 Heverlee > Belgium > +32 2 888 29 56 > > http://www.atmire.com - Institutional Repository Solutions > http://www.togather.eu - Before getting together, get [EMAIL PROTECTED] > > On Wed, May 21, 2008 at 4:33 PM, Jim Price > <[EMAIL PROTECTED]> wrote: > Hello, > > We are currently running a test instance on a Sun Enterprise > 450 running Solaris 10. We are currently looking into new > hardware for a production environment. > We would like to find out if anyone is willing to share their > experiences with hardware? > > What platform would you recommend for a school starting with > dspace? > What are you using for hardware? > What do you store? > How big is your data storage? > How much growth did you see in your first year of use? > What kind of loads are you experiencing? > > > Thanks, > Jim > > > > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > > > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ __
Re: [Dspace-tech] Hardware recommendations
Hello Jim, we have installed and manage some very large DSpaces, but also a few moderate ones. An example of a large installation: http://lirias.kuleuven.be holds around 130.000 items, of which only around 2500 of them contain full-text. This asset store is around 7GB. It's mainly academic research output (papers, conference presentations, ...). So currently, the average size of a bitstream is around 2.8MB. But this will be very different if your repository is oriented towards datasets, audio or video. Concerning processing power and memory: the system currently has around 1000 unique visitors daily. There are 4000 e-persons. During office hours, we experience an average of 4 concurrent logged-in users, performing submissions, etc (= intensive on database and indexes). The tomcat has been given 2GB of memory, while the whole system has 3.5GB of RAM. This doesn't really cover the total load, as we use swapping, but it's enough to keep the system running. A recommendation: don't cut down on disk speed, you will need 15.000 rpm disks. The server has 1 physical dual-core processor, with hyper threading (so actually 4 virtual processors). In peak times, this is becoming a bottle neck. If you could illustrate the purposes of your installation, or the estimated number of users, I could provide you with a specific case, that possibly more closly matches what you're looking at. with kindest regards, Bram Luyten -- @mire NV Romeinse Straat 18 3001 Heverlee Belgium +32 2 888 29 56 http://www.atmire.com - Institutional Repository Solutions http://www.togather.eu - Before getting together, get [EMAIL PROTECTED] On Wed, May 21, 2008 at 4:33 PM, Jim Price <[EMAIL PROTECTED]> wrote: > Hello, > > We are currently running a test instance on a Sun Enterprise 450 running > Solaris 10. We are currently looking into new hardware for a production > environment. > We would like to find out if anyone is willing to share their experiences > with hardware? > > What platform would you recommend for a school starting with dspace? > What are you using for hardware? > What do you store? > How big is your data storage? > How much growth did you see in your first year of use? > What kind of loads are you experiencing? > > > Thanks, > Jim > > > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Hardware recommendations
Hello, We are currently running a test instance on a Sun Enterprise 450 running Solaris 10. We are currently looking into new hardware for a production environment. We would like to find out if anyone is willing to share their experiences with hardware? What platform would you recommend for a school starting with dspace? What are you using for hardware? What do you store? How big is your data storage? How much growth did you see in your first year of use? What kind of loads are you experiencing? Thanks, Jim - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
