[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-05-17 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287554#comment-15287554
 ] 

ASF GitHub Bot commented on SOLR-8727:
--

Github user bjoernhaeuser closed the pull request at:

https://github.com/apache/lucene-solr/pull/11


> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-03-01 Thread Mark Miller (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174465#comment-15174465
 ] 

Mark Miller commented on SOLR-8727:
---

bq. But I also would say that consuming as much threads as possible from the OS 
is also not good user behaviour. 

Indeed, and we would like to limit the number of threads and connections used 
much more so than today - work in progress :) Some complications abound. 
SOLR-7344 is close and should be a big step forward though.

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-29 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173376#comment-15173376
 ] 

Björn Häuser commented on SOLR-8727:


Done :) Thank you!

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread Noble Paul (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163064#comment-15163064
 ] 

Noble Paul commented on SOLR-8727:
--

Yeah this could be marked as a duplicate of solr-7344

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163047#comment-15163047
 ] 

Björn Häuser commented on SOLR-8727:


Well, I was never aware of a problem of distributed locks in solr. So I was 
just aiming at the smallest possible fix to ensure that there are no 
threadpools which dont have a limit.

After reading the linked issue I can see the problem which is imposed by 
limiting the threadpools. But I also would say that consuming as much threads 
as possible from the OS is also not good user behaviour. In this scenario (with 
the linked stack trace) I am not able to see the underlying problem. 

Sure, most of the threadpools can be limited through configuration :) 

Should we close this issue here and rather solve 7344?

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread Mark Miller (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163024#comment-15163024
 ] 

Mark Miller commented on SOLR-8727:
---

I think this is probably the right path for trying to deal with how many 
threads we currently have to allow: 
https://issues.apache.org/jira/browse/SOLR-7344

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread Mark Miller (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163020#comment-15163020
 ] 

Mark Miller commented on SOLR-8727:
---

And this deals with distributed deadlock issues how?

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162868#comment-15162868
 ] 

Björn Häuser commented on SOLR-8727:


How should that look like? I can add this 

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread Noble Paul (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162859#comment-15162859
 ] 

Noble Paul commented on SOLR-8727:
--

{{Integer.MAX_VALUE}} is definitely bad. 128 seems like a small value . BTW , 
it should also be possible to control it through a sys prop

> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>Assignee: Noble Paul
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
>   ... 22 more
> {code}
> After digging a little bit through the source code I found several 
> ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we 
> should figure out a better default then this.
> Going to create the corresponding pull reuquest on github for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8727) Limit Threadpools by default

2016-02-24 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162809#comment-15162809
 ] 

ASF GitHub Bot commented on SOLR-8727:
--

GitHub user bjoernhaeuser opened a pull request:

https://github.com/apache/lucene-solr/pull/11

Limit threadpools by default to 128

Integer.MAX_VALUE could kill the VM instead of providing useful error
messages.

128 is a wild guess by myself. Let's discuss a good default value for
this :)

Jira: https://issues.apache.org/jira/browse/SOLR-8727

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rebuy-de/lucene-solr limit-threadpools

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/11.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11


commit cff552d1dc4e7bc7892d9521355c651c2ed2ad06
Author: Björn Häuser 
Date:   2016-02-24T10:44:09Z

Limit threadpools by default to 128

Integer.MAX_VALUE could kill the VM instead of providing useful error
messages.

128 is a wild guess by myself. Let's discuss a good default value for
this :)




> Limit Threadpools by default
> 
>
> Key: SOLR-8727
> URL: https://issues.apache.org/jira/browse/SOLR-8727
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Björn Häuser
>
> Yesterday we had a problem in our prodution cluster, it was running out of 
> native threads:
> {code}
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create 
> new native thread
>   at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>   at org.eclipse.jetty.server.Server.handle(Server.java:497)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>   at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>   at java.lang.Thread.start0(Native Method)
>   at java.lang.Thread.start(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
>   at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
>   at 
> org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>   at