Aaron, Thanks very much for all this detailed info. I'll pass it along to the guys who are getting these instances setup. Your info meshes nicely with what the team here is (slowly) learning about what's possible with AWS.
One thing that's becoming very clear is that the generic, out of the box AWS instances are not very well suited as ML cluster nodes. Some user assembly required. On Jan 8, 2013, at 9:41 PM, Aaron Rosenbaum wrote: > >I'll have them look into the Provisioned IOPs thing. What > >I really want is high-performance local disk to meet the > >performance > >targets we have. > > > * Instance types > Use m2.4xl's with pIOPS, hi1.4xlarge with local SSD storage (using HA), pIOPS > or a mixture of both types of storage using the SSD's as the Fast Data > directory (but then you need to configure HA). > hs1.8xlarge looks interesting but we haven't done testing yet and there is > some uncertainty on how they will deal with failed drives. > hi1.4xlarge has shown some really fantastic results for bulk ingest. > cc2.8xlarge will work also but not as fast as hi1.4xlarge so no real > advantage. pIOPS volumes are already RAID so snapshots suffice for DR (if the > application can support some downtime.) > > * Configuration > Use homogenous node types so you can use cluster groups. > Inside a cluster group, 2ms node-to-node seems typical. The rack-to-rack > timing seems fine. We've seen decent performance on relatively short-runs > across AZ's even. With the high and very high network connection, we haven't > seen networking as an issue. It's absolutely an issue with the smaller > instances. > One advantage of the hi1.4xlarge is that there are no "old" nodes. > With m2.4xl's, there is some variance but if it starts good, it'll generally > stay good. > The Provisioned IOPS volumes have a lot more reliability than general EBS so > if the node is bad, detaching and attaching to a new node isn't all that > painful. > > *If using EBS, use Provisioned IOPS > The performance of non-provisioned IOPS and provisioned IOPS volumes are > dramatically different. > It is not unusual to see IOPS in the 10-40 range – with large variance - on > regular EBS. > RAID only improves marginally – the slowest volume still causes problems. > m2.2xl's don't have provisioned IOPS available. > The m2.4xl's have 1000 Mega-bits/second bandwidth to Provisioned IOPS EBS. > You can spread that over 4-5 volumes, each with 200-300 pIOPS, each with a > single forest. > An IO operation on pIOPS volume is 16K so make sure the volumes are formatted > with 16K blocks. If you format with 4K blocks, expect 1/4 the performance – > the drivers won't use the queue that way. > No matter what the bandwidth of the node, 1GB/sec is the limit to the IO > pool. However, on the instance types I've mentioned, it also appears to be > the floor of bandwidth. > > -Aaron > > ------------------------------------------ > Aaron Rosenbaum > Director, Product Management > MarkLogic Corporation > [email protected] > Phone: 650 287 2573 > Cell: 650 703 7517 > > <B73F9E45-0DF4-40A6-92D4-9C744CC8B3E5.png> > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general --- Ron Hitchens {mailto:[email protected]} Ronsoft Technologies +44 7879 358 212 (voice) http://www.ronsoft.com +1 707 924 3878 (fax) Bit Twiddling At Its Finest "No amount of belief establishes any fact." -Unknown _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
