Hi all,
I'm continually running into the "Too many open files" error on 18.3:
DataXceiveServer: java.io.IOException: Too many open files
>
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>
at
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:96)
>
at
> org.apache.hadoop.dfs.DataNode$DataXceiveServer.run(DataNode.java:997)
>
at java.lang.Thread.run(Thread.java:619)
>
I'm writing thousands of files in the course of a few minutes, but nothing
that seems too unreasonable, especially given the numbers below. I begin
getting a surge of these warnings right as I hit 1024 files open by the
DataNode:
had...@u10:~$ ps ux | awk '/dfs\.DataNode/ { print $2 }' | xargs -i ls
> /proc/{}/fd | wc -l
>
1023
>
This is a bit unexpected, however, since I've configured my open file limit
to be 16k:
had...@u10:~$ ulimit -a
>
core file size (blocks, -c) 0
>
data seg size (kbytes, -d) unlimited
>
scheduling priority (-e) 0
>
file size (blocks, -f) unlimited
>
pending signals (-i) 268288
>
max locked memory (kbytes, -l) 32
>
max memory size (kbytes, -m) unlimited
>
open files (-n) 16384
>
pipe size (512 bytes, -p) 8
>
POSIX message queues (bytes, -q) 819200
>
real-time priority (-r) 0
>
stack size (kbytes, -s) 8192
>
cpu time (seconds, -t) unlimited
>
max user processes (-u) 268288
>
virtual memory (kbytes, -v) unlimited
>
file locks (-x) unlimited
>
Note, I've also set dfs.datanode.max.xcievers to 8192 in hadoop-site.xml.
Thanks in advance,
Sean
