I had run a big mapred job (160GB data) on a small cluster of 7 nodes. I had started 15 Memcached server instances on 7 nodes and I noticed that a single memcached server was processing 1 million requests per second (in my case), however it was definitely 3-4 times slower than in-memory approach. I had to increase the limit of open file descriptors for that.
Since memcached was not performing up to my expectations I used Tokyocabinet ( A file based database) and its performance was near to in-memory approach. On Fri, Oct 2, 2009 at 12:40 AM, Andrzej Jan Taramina <andr...@chaeron.com>wrote: > Chandraprakash: > > > I used Memcached with hadoop, so that i can store (read/write) data on > > memcached server. I can refer this data > > from any task by just connecting the memcached server. > > What kind of impact did that have on your network utilization when running > big mapred jobs? Did you saturate your > network links, having all Hadoop tasktracker nodes accessing the same > memcached box? > > Thx! > > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com > > -- Thanks & Regards, Chandra Prakash Bhagtani,
