FYI: "Ephemeral Storage" is local disk. Provisioned IOPS can gain you a great deal Also check out these instance types:
High I/O Quadruple Extra Large Instance 60.5 GiB of memory 35 EC2 Compute Units (16 virtual cores*) 2 SSD-based volumes each with 1024 GB of instance storage 64-bit platform I/O Performance: Very High (10 Gigabit Ethernet) Storage I/O Performance: Very High** EBS-Optimized Available: No*** API name: hi1.4xlarge High Storage Instances 117 GiB of memory 35 EC2 Compute Units (16 virtual cores*) 24 hard disk drives each with 2 TB of instance storage 64-bit platform I/O Performance: Very High (10 Gigabit Ethernet) Storage I/O Performance: Very High** EBS-Optimized Available: No*** API name: hs1.8xlarge ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected] Phone: +1 812-482-5224 Cell: +1 812-630-7622 www.marklogic.com -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Ron Hitchens Sent: Tuesday, January 08, 2013 8:46 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] MarkLogic in AWS Cloud We tried the EBS Optimized option and that hasn't made much of a difference either. I suppose RAIDing across EBS is a way to go, but I'm afraid that would fall outside the comfort zone of the people administering this stuff. I'll have them look into the Provisioned IOPs thing. What I really want is high-performance local disk to meet the performance targets we have. Thanks for the help. Is anybody out there actually running large-ish production MarkLogic clusters in the cloud? On Jan 8, 2013, at 12:35 PM, David Lee wrote: > Almost certainly as Wayne suggests your bottleneck is IO. > > The default storage is EBS which is a type of network SAN. > Some instance types have "EBS Optimized" which you should try. > This gives a dedicated network channel to EBS. > Then add RAID across the EBS for extra fun. > > Even better as Wayne suggests is instances with "Provisioned IOPS" > or some of the truly amazing DB oriented instances with tons of local storage. > > Also you could consider using Ephemerial Storage, however as the name > suggests it > will not last beyond the instance life. > > > ----------------------------------------------------------------------------- > David Lee > Lead Engineer > MarkLogic Corporation > [email protected] > Phone: +1 812-482-5224 > Cell: +1 812-630-7622 > www.marklogic.com > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Wayne Feick > Sent: Tuesday, January 08, 2013 7:20 AM > To: General Mark Logic Developer Discussion > Subject: Re: [MarkLogic Dev General] MarkLogic in AWS Cloud > > I don't have a lot of experience with it, but EBS volumes have limited > bandwidth. Some people have had success striping across multiple EBS volumes > from within Linux instances. You could also look at the more recent > guaranteed IOPs capability Amazon now offers. > > Wayne > > Ron Hitchens <[email protected]> wrote: > > > Has anyone had any experience configuring and running non-trivial > MarkLogic clusters in the cloud? Specifically Amazon EC2 VMs? > > I've got a test cluster of three nodes setup in AWS and am trying > to figure out the best configuration for it. The system seems to be > quite slow at some things, but reasonably fast at others. Bumping > the VM up to bigger instances (more ram, more cores) doesn't seem to > have a significant impact on speed or throughput. > > I suspect I/O bandwidth may be the culprit, but that's just a > hunch. Does anyone have any experience with tuning EC2 VMs? > > The test environment I'm working with now is three m2.xlarge > instances (32gb RAM, 4 cores, "high" network speed). The OS is > Windows (groan, I don't have a choice there). Production cluster(s) > are likely to be similar, but probably six nodes or so. > > Any advice//war stories/dire warnings greatly appreciated. > > Thanks. > > --- > Ron Hitchens {mailto:[email protected]} Ronsoft Technologies > +44 7879 358 212 (voice) http://www.ronsoft.com > +1 707 924 3878 (fax) Bit Twiddling At Its Finest > "No amount of belief establishes any fact." -Unknown > > > > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general --- Ron Hitchens {mailto:[email protected]} Ronsoft Technologies +44 7879 358 212 (voice) http://www.ronsoft.com +1 707 924 3878 (fax) Bit Twiddling At Its Finest "No amount of belief establishes any fact." -Unknown _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
