Sean Knapp wrote:
Raghu,
Apologies for the confusion. I was seeing the problem with any setting
for dfs.datanode.max.xcievers... 1k, 2k and 8k. Likewise, I was also seeing
the problem with different open file settings, all the way up to 32k.
Since I installed the patch, HDFS has been performing much better. The
current settings that work for me are 16k max open files with
dfs.datanode.max.xcievers=8k, though under heavy balancer load I do start to
hit the 16k max.
Thanks. That clarifies many things.
Even with the patch, if you have lot of simultaneous clients, DataNode
does need a lot of file descriptors (something like 6 times the number
of active 'xceivers'). In your case you seem to have lot of simultaneous
clients. I suggest increasing file limit to much higher (something like
64k).
Raghu.
Regards,
Sean
2009/2/13 Raghu Angadi <rang...@yahoo-inc.com>
