[
https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074857#comment-17074857
]
Michael Stack edited comment on HBASE-24072 at 4/3/20, 8:10 PM:
----------------------------------------------------------------
Was able to get ulimit and test how many threads I could create. We are bound
to 10k processes/threads which is borne out by out thread creation script (it
fails at 9753...).
{code}
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 192962
max locked memory (kbytes, -l) 16384
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
opendfs: 81
maxfds: 1048576
New thread #1
New thread #2
New thread #3
New thread #4
...
New thread #9752
New thread #9753
Standard Error
New thread #9753
{code}
was (Author: stack):
Was able to get ulimit and test how many threads I could create. We are bound
to 10k processes/threads which is borne out by out thread creation script (it
fails at 9753...).
{code}
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 192962
max locked memory (kbytes, -l) 16384
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
opendfs: 81
maxfds: 1048576
New thread #1
New thread #2
New thread #3
New thread #4
...
New thread #9752
New thread #9753
Standard Error
New thread #9753
> Nightlies reporting OutOfMemoryError: unable to create new native thread
> ------------------------------------------------------------------------
>
> Key: HBASE-24072
> URL: https://issues.apache.org/jira/browse/HBASE-24072
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Michael Stack
> Priority: Major
> Attachments:
> 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch,
> run_ulimit_a_and_see_how_many_threads_can_create.patch
>
>
> Seeing this kind of thing in nightly...
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new
> native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> {code}
> Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing
> our docker container. It is set to 20G (The hosts are 48G). Fork count is
> 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add
> the maven 4G and we could be over the top.
> Play w/ downing the fork size (in earlier study we didn't seem to need this
> much RAM when running a fat long test). Let me also take th ms off the mvn
> allocation to see if that helps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
