[
https://issues.apache.org/jira/browse/PHOENIX-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206735#comment-14206735
]
James Taylor commented on PHOENIX-1437:
---------------------------------------
Yes, a thread per "guidepost" on the client-side of Phoenix. Take a look here
for tuning parameters: http://phoenix.apache.org/tuning.html
The three most important are:
1) phoenix.query.threadPoolSize which is the ThreadExecutor thread pool size
2) phoenix.query.queueSize which is the ThreadExecutor queue depth (after which
work will be rejected)
3) phoenix.stats.guidepost.width which is the number of bytes between
guideposts (or an alternate way of specifying that is
phoenix.stats.guidepost.per.region). These determine how many guideposts will
be created for a table.
> java.lang.OutOfMemoryError: unable to create new native thread
> --------------------------------------------------------------
>
> Key: PHOENIX-1437
> URL: https://issues.apache.org/jira/browse/PHOENIX-1437
> Project: Phoenix
> Issue Type: Bug
> Reporter: Taylor Finnell
>
> Getting a java.lang.OutOfMemoryError when using Phoenix on Storm. Here is the
> full stack trace.
> {code}
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_45]
> at java.lang.Thread.start(java/lang/Thread.java:713) ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(java/util/concurrent/ThreadPoolExecutor.java:949)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(java/util/concurrent/ThreadPoolExecutor.java:1360)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.AbstractExecutorService.submit(java/util/concurrent/AbstractExecutorService.java:132)
> ~[na:1.7.0_45]
> at
> org.apache.phoenix.iterate.ParallelIterators.submitWork(org/apache/phoenix/iterate/ParallelIterators.java:356)
> ~[stormjar.jar:na]
> at
> org.apache.phoenix.iterate.ParallelIterators.getIterators(org/apache/phoenix/iterate/ParallelIterators.java:265)
> ~[stormjar.jar:na]
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(org/apache/phoenix/iterate/ConcatResultIterator.java:44)
> ~[stormjar.jar:na]
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(org/apache/phoenix/iterate/ConcatResultIterator.java:66)
> ~[stormjar.jar:na]
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(org/apache/phoenix/iterate/ConcatResultIterator.java:86)
> ~[stormjar.jar:na]
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.next(org/apache/phoenix/jdbc/PhoenixResultSet.java:732)
> ~[stormjar.jar:na]
> at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)
> ~[na:1.7.0_45]
> at
> RUBY.each(file:/mnt/hadoop/storm/supervisor/stormdist/korrelate_match_log_processor_staging_KOR-2325-online_sync_to_hbase_tf_part_three-1-1415715986/stormjar.jar!/lib/korrelate_match_log_processor/cleanroom_online_event_adapter.rb:51)
> ~[na:na]
> at
> RUBY.finish_batch(file:/mnt/hadoop/storm/supervisor/stormdist/korrelate_match_log_processor_staging_KOR-2325-online_sync_to_hbase_tf_part_three-1-1415715986/stormjar.jar!/lib/korrelate_match_log_processor/bolt/abstract_event_reader_bolt.rb:68)
> ~[na:na]
> at
> RUBY.finishBatch(/Users/tfinnell/.rvm/gems/jruby-1.7.11@O2O-jruby/gems/redstorm-0.6.6/lib/red_storm/proxy/batch_bolt.rb:51)
> ~[na:na]
> at
> redstorm.proxy.BatchBolt.finishBatch(redstorm/proxy/BatchBolt.java:149)
> ~[stormjar.jar:na]
> at
> redstorm.storm.jruby.JRubyTransactionalBolt.finishBatch(redstorm/storm/jruby/JRubyTransactionalBolt.java:56)
> ~[stormjar.jar:na]
> at
> backtype.storm.coordination.BatchBoltExecutor.finishedId(backtype/storm/coordination/BatchBoltExecutor.java:76)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.coordination.CoordinatedBolt.checkFinishId(backtype/storm/coordination/CoordinatedBolt.java:259)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.coordination.CoordinatedBolt.execute(backtype/storm/coordination/CoordinatedBolt.java:322)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.daemon.executor$fn__4329$tuple_action_fn__4331.invoke(executor.clj:630)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.daemon.executor$fn__4329$tuple_action_fn__4331.invoke(backtype/storm/daemon/executor.clj:630)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__4252.invoke(executor.clj:398)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__4252.invoke(backtype/storm/daemon/executor.clj:398)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.disruptor$clojure_handler$reify__1747.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.disruptor$clojure_handler$reify__1747.onEvent(backtype/storm/disruptor.clj:58)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(backtype/storm/utils/DisruptorQueue.java:104)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(backtype/storm/utils/DisruptorQueue.java:78)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(backtype/storm/disruptor.clj:77)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.daemon.executor$fn__4329$fn__4341$fn__4388.invoke(executor.clj:745)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.daemon.executor$fn__4329$fn__4341$fn__4388.invoke(backtype/storm/daemon/executor.clj:745)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at backtype.storm.util$async_loop$fn__442.invoke(util.clj:436)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at
> backtype.storm.util$async_loop$fn__442.invoke(backtype/storm/util.clj:436)
> ~[storm-core-0.9.1.2.1.2.0-402.jar:0.9.1.2.1.2.0-402]
> at clojure.lang.AFn.run(clojure/lang/AFn.java:24)
> ~[clojure-1.4.0.jar:na]
> at java.lang.Thread.run(java/lang/Thread.java:744) ~[na:1.7.0_45]
> {code}
> Here is some of the system configuration.
> {code}
> ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 240435
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 240435
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
> None of the tables I am querying against have more than 4 regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
