Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "TooManyOpenFiles" page has been changed by SteveLoughran: http://wiki.apache.org/hadoop/TooManyOpenFiles Comment: document the too many open files problem and the two likely causes, quotas and limits on linux New page: = Too Many Open Files = You can see this on Linux machines in client-side applications, server code or even in test runs. It is caused by per-process limits on the number of files that a single user/process can have open, which was introduced in [[http://lkml.indiana.edu/hypermail/linux/kernel/0812.0/01183.html|2.6.27]. The default value, 128, was chosen because "that should be enough". In Hadoop, it isn't. To fix this log in/su/ssh as root and edit {{{/etc/sysctl.conf}}} add the line {{{ fs.epoll.max_user_instances = 2048 }}} Then reboot. Different numbers may be chosen. There is an immediate shortcut: {{{echo 2048 > /proc/sys/fs/epoll/max_user_instances}}} . This setting will be lost on the next reboot, but is handy for trying out different values. == limits.conf == Another limit on the number of files open may be the file {{{/etc/security/limits.conf}}} It has a setting on the number of files a user or group may have, {{{nofile}}}. To set this, as root edit {{{/etc/security/limits.conf}}} and add a line such as {{{ * soft nofile 2048 }}} Then restart. To see the current/default limits, run the command {{{ulimit -a}}}. This should print something like {{{ core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited }}} You can dynamically up the limits until the next reboot with the same command. Specifically {{{ ulimit -n 8192 }}} The updated value can then be printed {{{ # ulimit -n 8192 }}}
