[
https://issues.apache.org/jira/browse/AIRAVATA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Christie updated AIRAVATA-3186:
--------------------------------------
Description:
{noformat}
2019-07-26 16:02:39,871 [pool-24-thread-28] ERROR
o.a.t.server.TThreadPoolServer - Error occurred during processing of message.
org.apache.airavata.common.utils.ThriftClientPool$ThriftClientException: Could
not get a resource from the pool
at
org.apache.airavata.common.utils.ThriftClientPool.getResource(ThriftClientPool.java:144)
at
org.apache.airavata.api.server.handler.AiravataServerHandler.userHasAccess(AiravataServerHandler.java:5370)
at
org.apache.airavata.service.security.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:56)
at
org.apache.airavata.api.Airavata$Processor$userHasAccess.getResult(Airavata.java:22848)
at
org.apache.airavata.api.Airavata$Processor$userHasAccess.getResult(Airavata.java:22832)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at
org.apache.airavata.common.utils.ThriftClientPool.getResource(ThriftClientPool.java:132)
... 10 common frames omitted
{noformat}
h5. TODO
- [x] add properties for configuring abandoned object removal. There's some
overhead so probably don't want this turned on in production.
- [x] send the logs to slf4j? Right now the abandoned logging is going to stdout
- [ ] Resource leak warning here:
https://github.com/apache/airavata/blob/staging/modules/commons/src/main/java/org/apache/airavata/common/utils/ThriftClientPool.java#L111
- [x] merge back to develop
- [x] add property for configuring whether to
[logAbandoned|https://commons.apache.org/proper/commons-pool/api-2.7.0/index.html]?
This is what we would want to turn off in production because of the overhead
involved, but we might wanted to keep abandoned object removal turned on in
production.
- [ ] question: can we use JMX to monitor size/status of connection pools?
- [x] turn off abandoned object removal in production? Assuming we've found and
fixed all leaks, we should be able to do this now.
-- turned this off since there is a performance overhead with abandoned pool
connection removal. My assumption is that with the removal turned on in
develop/staging we'll capture these problems before they make it to production.
was:
{noformat}
2019-07-26 16:02:39,871 [pool-24-thread-28] ERROR
o.a.t.server.TThreadPoolServer - Error occurred during processing of message.
org.apache.airavata.common.utils.ThriftClientPool$ThriftClientException: Could
not get a resource from the pool
at
org.apache.airavata.common.utils.ThriftClientPool.getResource(ThriftClientPool.java:144)
at
org.apache.airavata.api.server.handler.AiravataServerHandler.userHasAccess(AiravataServerHandler.java:5370)
at
org.apache.airavata.service.security.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:56)
at
org.apache.airavata.api.Airavata$Processor$userHasAccess.getResult(Airavata.java:22848)
at
org.apache.airavata.api.Airavata$Processor$userHasAccess.getResult(Airavata.java:22832)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at
org.apache.airavata.common.utils.ThriftClientPool.getResource(ThriftClientPool.java:132)
... 10 common frames omitted
{noformat}
h5. TODO
- [x] add properties for configuring abandoned object removal. There's some
overhead so probably don't want this turned on in production.
- [x] send the logs to slf4j? Right now the abandoned logging is going to stdout
- [ ] Resource leak warning here:
https://github.com/apache/airavata/blob/staging/modules/commons/src/main/java/org/apache/airavata/common/utils/ThriftClientPool.java#L111
- [x] merge back to develop
- [ ] add property for configuring whether to
[logAbandoned|https://commons.apache.org/proper/commons-pool/api-2.7.0/index.html]?
This is what we would want to turn off in production because of the overhead
involved, but we might wanted to keep abandoned object removal turned on in
production.
- [ ] question: can we use JMX to monitor size/status of connection pools?
- [x] turn off abandoned object removal in production? Assuming we've found and
fixed all leaks, we should be able to do this now.
-- turned this off since there is a performance overhead with abandoned pool
connection removal. My assumption is that with the removal turned on in
develop/staging we'll capture these problems before they make it to production.
> ThriftClientPool$ThriftClientException: Could not get a resource from the pool
> ------------------------------------------------------------------------------
>
> Key: AIRAVATA-3186
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3186
> Project: Airavata
> Issue Type: Bug
> Components: Airavata API
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> {noformat}
> 2019-07-26 16:02:39,871 [pool-24-thread-28] ERROR
> o.a.t.server.TThreadPoolServer - Error occurred during processing of message.
> org.apache.airavata.common.utils.ThriftClientPool$ThriftClientException:
> Could not get a resource from the pool
> at
> org.apache.airavata.common.utils.ThriftClientPool.getResource(ThriftClientPool.java:144)
> at
> org.apache.airavata.api.server.handler.AiravataServerHandler.userHasAccess(AiravataServerHandler.java:5370)
> at
> org.apache.airavata.service.security.interceptor.SecurityInterceptor.invoke(SecurityInterceptor.java:56)
> at
> org.apache.airavata.api.Airavata$Processor$userHasAccess.getResult(Airavata.java:22848)
> at
> org.apache.airavata.api.Airavata$Processor$userHasAccess.getResult(Airavata.java:22832)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
> at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
> at
> org.apache.airavata.common.utils.ThriftClientPool.getResource(ThriftClientPool.java:132)
> ... 10 common frames omitted
> {noformat}
> h5. TODO
> - [x] add properties for configuring abandoned object removal. There's some
> overhead so probably don't want this turned on in production.
> - [x] send the logs to slf4j? Right now the abandoned logging is going to
> stdout
> - [ ] Resource leak warning here:
> https://github.com/apache/airavata/blob/staging/modules/commons/src/main/java/org/apache/airavata/common/utils/ThriftClientPool.java#L111
> - [x] merge back to develop
> - [x] add property for configuring whether to
> [logAbandoned|https://commons.apache.org/proper/commons-pool/api-2.7.0/index.html]?
> This is what we would want to turn off in production because of the overhead
> involved, but we might wanted to keep abandoned object removal turned on in
> production.
> - [ ] question: can we use JMX to monitor size/status of connection pools?
> - [x] turn off abandoned object removal in production? Assuming we've found
> and fixed all leaks, we should be able to do this now.
> -- turned this off since there is a performance overhead with abandoned pool
> connection removal. My assumption is that with the removal turned on in
> develop/staging we'll capture these problems before they make it to
> production.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
