[
https://issues.apache.org/jira/browse/PHOENIX-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153091#comment-15153091
]
Thomas D'Silva commented on PHOENIX-2690:
-----------------------------------------
[~mujtabachohan] When I run the query on a table with a large number of
guideposts (>4000) it crashes my local HBase but I wasn't able to repro the
OutOfMemoryError.
After I restart my local HBase any query on the table fail with the following
exception
{code}
Error: Task org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask@df76c5e
rejected from org.apache.phoenix.job.JobManager$1@51f722f1[Running, pool size =
128, active threads = 128, queued tasks = 5000, completed tasks = 5168]
(state=08000,code=101)
{code}
BaseQueryServicesImpl seems to be setting up the executor correctly.
{code}
this.executor = JobManager.createThreadPoolExec(
options.getKeepAliveMs(),
options.getThreadPoolSize(),
options.getQueueSize(),
options.isGlobalMetricsEnabled());
{code}
Can you run ulimit -a on one of the datanodes to see what the process limit is
set to? Was anything else running while you ran your query?
> Exceeding OS limit for native threads with large number of guide posts
> ----------------------------------------------------------------------
>
> Key: PHOENIX-2690
> URL: https://issues.apache.org/jira/browse/PHOENIX-2690
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.7.0
> Reporter: Mujtaba Chohan
> Assignee: Thomas D'Silva
> Priority: Minor
>
> With >32K guidepost count for a table (which equates to a 320 GB table size
> with 10MB guidepost width), following OOM occurs on 64bit Linux client
> {code}
> select * from table_with_32k_guideposts limit 1000;
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
> at
> org.apache.phoenix.job.JobManager$InstrumentedThreadPoolExecutor.execute(JobManager.java:261)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
> at
> org.apache.phoenix.iterate.ParallelIterators.submitWork(ParallelIterators.java:103)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:621)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:591)
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
> at
> org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
> at
> org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
