jason hadoop wrote:
Yes.
Otherwise the file descriptors will flow away like water.
I also strongly suggest having at least 64k file descriptors as the open
file limit.
On Sun, Jun 21, 2009 at 12:43 PM, Stas Oskin <stas.os...@gmail.com> wrote:
Hi.
Thanks for the advice. So you advice explicitly closing each and every file
handle that I receive from HDFS?
Regards.
I must disagree somewhat
If you use FileSystem.get() to get your client filesystem class, then
that is shared by all threads/classes that use it. Call close() on that
and any other thread or class holding a reference is in trouble. You
have to wait for the finalizers for them to get cleaned up.
If you use FileSystem.newInstance() - which came in fairly recently
(0.20? 0.21?) then you can call close() safely.
So: it depends on how you get your handle.
see: https://issues.apache.org/jira/browse/HADOOP-5933
Also: the too many open files problem can be caused in the NN -you need
to set up the Kernel to have lots more file handles around. Lots.
