Hi Yves,
On Apr 9, 2010, at 7:49am, Yves Petinot wrote:
Hi,
I'm currently contemplating migrating my crawler cluster to EC2 and
while this appears very tempting (infinite number of nodes), i've
read about some potential limitations in terms of the number of map/
red tasks that can effectively run on any instance. Especially for
the L/XL instances there doesn't seem to be any swap space set up
(by default at least), so that running more than 2 to 4 tasks per
instance may not be feasible (assuming 8/16 G of RAM and ~ 3G per
JVM). As a comparison, my current setup with dedicated blade servers
can easily sustain 5 to 10 map/red task per node. I'm basically
trying to understand whether this lack of swap space will
effectively mean that i need an EC2 cluster with at least 2 to 3
times more instances than i have nodes in my current cluster
Does anyone on the list have some experience in transitioning to EC2
and maybe with respect to this swap issue and/or on how to spec out
and EC2 cluster ?
I've been running a number of crawl jobs in EC2 (though using Bixo).
With the AMIs I use, there is swap space, but I'm using small (m1)
instances. No swap space sounds very strange - I'd check on the AWS
EC2 forum to see if anybody else has reported this with the AMI that
you're using.
-- Ken
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
