This is caused by a large number of connections created from Carbon to LDAP
are placed into TIME_WAIT state. Under certain conditions like high loads,
etc. it is possible for the instance to run out of available ports to create
new connections.
Inside the bindAsUser() of LDAPUserStoreManager, we have not enabled
connection pooling when creating an InitialLdapContext instance. Enabling
connection pooling as follows in this method, fixed the issue.
env.put("com.sun.jndi.ldap.connect.pool", "true");
Thanks,
Thilina
On Thu, Oct 6, 2011 at 3:47 PM, Hasini Gunasinghe <[email protected]> wrote:
> Got to know that about 40 users per second are authenticated in this case..
> This level of scalability should be supported by ApacheDS side as per [1].
> Therefore, will take a look at the ways in which we can optimize the
> connection to LDAP from carbon product's side, to avoid this.
>
> [1] http://osdir.com/ml/users-directory-apache/2011-05/msg00018.html
>
> Thanks,
> Hasini.
>
> On Thu, Oct 6, 2011 at 1:01 PM, Thilina Buddhika <[email protected]>wrote:
>
>> I can reproduce this consistently. This can be reproduced by trying to
>> authenticate users against LDAP with a concurrency of 10 or higher in a high
>> end machine. I am running a load test against IS in a quad core machine with
>> HT.
>>
>> Thanks,
>> Thilina
>>
>>
>> On Fri, May 6, 2011 at 12:17 AM, Amila Jayasekara <[email protected]>wrote:
>>
>>> Hi Danushka,
>>>
>>> I was not able to re-produce this issue in my local machine.
>>> Can you please give us specific steps.
>>>
>>> According to error message, the user store is unable to connect to
>>> LDAP server. This could be due to LDAP server is down.
>>>
>>> Thanks
>>> AmilaJ
>>>
>>> On Thu, May 5, 2011 at 11:26 AM, Danushka Menikkumbura
>>> <[email protected]> wrote:
>>> > It is intermittent. Server runs fine for some time and then starts to
>>> gives
>>> > this error.
>>> >
>>> > Danushka
>>> >
>>> > On Thu, May 5, 2011 at 11:20 AM, Danushka Menikkumbura <
>>> [email protected]>
>>> > wrote:
>>> >>
>>> >> I see the port is open and directory service is on it.
>>> >>
>>> >> Danushka
>>> >>
>>> >> On Thu, May 5, 2011 at 11:17 AM, Danushka Menikkumbura <
>>> [email protected]>
>>> >> wrote:
>>> >>>
>>> >>> I see this in the latest MB pack I just built.
>>> >>>
>>> >>> [2011-05-05 10:59:02,584] ERROR
>>> >>> {org.wso2.carbon.user.core.ldap.LDAPConnectionContext} - Error
>>> obtaining
>>> >>> connection. localhost:10389
>>> >>> {org.wso2.carbon.user.core.ldap.LDAPConnectionContext}
>>> >>> javax.naming.CommunicationException: localhost:10389 [Root exception
>>> is
>>> >>> java.net.NoRouteToHostException: Cannot assign requested address]
>>> >>> at com.sun.jndi.ldap.Connection.<init>(Connection.java:210)
>>> >>> at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:118)
>>> >>> at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1580)
>>> >>> at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2652)
>>> >>> at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:293)
>>> >>> at
>>> >>> com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
>>> >>> at
>>> >>>
>>> com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:193)
>>> >>> at
>>> >>>
>>> com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:136)
>>> >>> at
>>> >>>
>>> com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:66)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.utils.multitenancy.CarbonContextHolder$CarbonInitialJNDIContextFactory.getInitialContext(CarbonContextHolder.java:754)
>>> >>> at
>>> >>>
>>> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
>>> >>> at
>>> >>>
>>> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
>>> >>> at javax.naming.InitialContext.init(InitialContext.java:223)
>>> >>> at javax.naming.InitialContext.<init>(InitialContext.java:197)
>>> >>> at
>>> >>>
>>> javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:82)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPConnectionContext.getContext(LDAPConnectionContext.java:86)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPUserStoreManager.getListOfNames(LDAPUserStoreManager.java:679)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPUserStoreManager.getRoleListOfUser(LDAPUserStoreManager.java:550)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.qpid.authorization.qpid.QpidAuthorizationHandler.isAdminUser(QpidAuthorizationHandler.java:357)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.qpid.authorization.qpid.QpidAuthorizationHandler.handleConsumeQueue(QpidAuthorizationHandler.java:115)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.qpid.authorization.service.qpid.QpidAuthorizationPlugin.authorise(QpidAuthorizationPlugin.java:147)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.security.SecurityManager$5.allowed(SecurityManager.java:321)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.security.SecurityManager.checkAllPlugins(SecurityManager.java:245)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.security.SecurityManager.authoriseConsume(SecurityManager.java:317)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.queue.SimpleAMQQueue.registerSubscription(SimpleAMQQueue.java:407)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.messageSubscribe(ServerSessionDelegate.java:260)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.messageSubscribe(ServerSessionDelegate.java:96)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.MessageSubscribe.dispatch(MessageSubscribe.java:119)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.SessionDelegate.command(SessionDelegate.java:50)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.command(ServerSessionDelegate.java:112)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.command(ServerSessionDelegate.java:96)
>>> >>> at org.apache.qpid.transport.Method.delegate(Method.java:159)
>>> >>> at org.apache.qpid.transport.Session.received(Session.java:500)
>>> >>> at
>>> org.apache.qpid.transport.Connection.dispatch(Connection.java:404)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:64)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:40)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.MethodDelegate.messageSubscribe(MethodDelegate.java:131)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.MessageSubscribe.dispatch(MessageSubscribe.java:119)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:54)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:40)
>>> >>> at org.apache.qpid.transport.Method.delegate(Method.java:159)
>>> >>> at
>>> org.apache.qpid.transport.Connection.received(Connection.java:369)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerConnection.received(ServerConnection.java:196)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerConnection.received(ServerConnection.java:53)
>>> >>> at
>>> >>> org.apache.qpid.transport.network.Assembler.emit(Assembler.java:95)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:196)
>>> >>> at
>>> >>> org.apache.qpid.transport.network.Assembler.frame(Assembler.java:129)
>>> >>> at
>>> org.apache.qpid.transport.network.Frame.delegate(Frame.java:133)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.Assembler.received(Assembler.java:100)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.Assembler.received(Assembler.java:42)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:187)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:103)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:42)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:102)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:36)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.mina.MINANetworkDriver.messageReceived(MINANetworkDriver.java:337)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:703)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain.access$1200(AbstractIoFilterChain.java:54)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
>>> >>> at
>>> >>>
>>> org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:243)
>>> >>> at
>>> >>>
>>> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:305)
>>> >>> at
>>> >>>
>>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
>>> >>> at
>>> >>>
>>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
>>> >>> at java.lang.Thread.run(Thread.java:619)
>>> >>> Caused by: java.net.NoRouteToHostException: Cannot assign requested
>>> >>> address
>>> >>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> >>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>> >>> at
>>> >>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>> >>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>> >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>> >>> at java.net.Socket.connect(Socket.java:529)
>>> >>> at java.net.Socket.connect(Socket.java:478)
>>> >>> at java.net.Socket.<init>(Socket.java:375)
>>> >>> at java.net.Socket.<init>(Socket.java:189)
>>> >>> at com.sun.jndi.ldap.Connection.createSocket(Connection.java:352)
>>> >>> at com.sun.jndi.ldap.Connection.<init>(Connection.java:187)
>>> >>> ... 64 more
>>> >>> [2011-05-05 10:59:02,654] ERROR
>>> >>> {org.wso2.carbon.user.core.ldap.LDAPConnectionContext} - Trying
>>> again to
>>> >>> get connection.
>>> {org.wso2.carbon.user.core.ldap.LDAPConnectionContext}
>>> >>> [2011-05-05 10:59:02,656] ERROR
>>> >>> {org.wso2.carbon.user.core.ldap.LDAPConnectionContext} - Error
>>> obtaining
>>> >>> connection. localhost:10389
>>> >>> {org.wso2.carbon.user.core.ldap.LDAPConnectionContext}
>>> >>> javax.naming.CommunicationException: localhost:10389 [Root exception
>>> is
>>> >>> java.net.NoRouteToHostException: Cannot assign requested address]
>>> >>> at com.sun.jndi.ldap.Connection.<init>(Connection.java:210)
>>> >>> at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:118)
>>> >>> at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1580)
>>> >>> at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2652)
>>> >>> at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:293)
>>> >>> at
>>> >>> com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
>>> >>> at
>>> >>>
>>> com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:193)
>>> >>> at
>>> >>>
>>> com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:136)
>>> >>> at
>>> >>>
>>> com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:66)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.utils.multitenancy.CarbonContextHolder$CarbonInitialJNDIContextFactory.getInitialContext(CarbonContextHolder.java:754)
>>> >>> at
>>> >>>
>>> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
>>> >>> at
>>> >>>
>>> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
>>> >>> at javax.naming.InitialContext.init(InitialContext.java:223)
>>> >>> at javax.naming.InitialContext.<init>(InitialContext.java:197)
>>> >>> at
>>> >>>
>>> javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:82)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPConnectionContext.getContext(LDAPConnectionContext.java:86)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPUserStoreManager.searchForUser(LDAPUserStoreManager.java:397)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPUserStoreManager.getNameInSpaceForUserName(LDAPUserStoreManager.java:603)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.user.core.ldap.LDAPUserStoreManager.getRoleListOfUser(LDAPUserStoreManager.java:543)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.qpid.authorization.qpid.QpidAuthorizationHandler.isAdminUser(QpidAuthorizationHandler.java:357)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.qpid.authorization.qpid.QpidAuthorizationHandler.handlePublishToExchange(QpidAuthorizationHandler.java:230)
>>> >>> at
>>> >>>
>>> org.wso2.carbon.qpid.authorization.service.qpid.QpidAuthorizationPlugin.authorise(QpidAuthorizationPlugin.java:144)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.security.SecurityManager$11.allowed(SecurityManager.java:390)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.security.SecurityManager.checkAllPlugins(SecurityManager.java:245)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.security.SecurityManager.authorisePublish(SecurityManager.java:386)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.messageTransfer(ServerSessionDelegate.java:307)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.messageTransfer(ServerSessionDelegate.java:96)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.MessageTransfer.dispatch(MessageTransfer.java:108)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.SessionDelegate.command(SessionDelegate.java:50)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.command(ServerSessionDelegate.java:112)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerSessionDelegate.command(ServerSessionDelegate.java:96)
>>> >>> at org.apache.qpid.transport.Method.delegate(Method.java:159)
>>> >>> at org.apache.qpid.transport.Session.received(Session.java:500)
>>> >>> at
>>> org.apache.qpid.transport.Connection.dispatch(Connection.java:404)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:64)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:40)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.MethodDelegate.messageTransfer(MethodDelegate.java:113)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.MessageTransfer.dispatch(MessageTransfer.java:108)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:54)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:40)
>>> >>> at org.apache.qpid.transport.Method.delegate(Method.java:159)
>>> >>> at
>>> org.apache.qpid.transport.Connection.received(Connection.java:369)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerConnection.received(ServerConnection.java:196)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.transport.ServerConnection.received(ServerConnection.java:53)
>>> >>> at
>>> >>> org.apache.qpid.transport.network.Assembler.emit(Assembler.java:95)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:217)
>>> >>> at
>>> >>> org.apache.qpid.transport.network.Assembler.frame(Assembler.java:129)
>>> >>> at
>>> org.apache.qpid.transport.network.Frame.delegate(Frame.java:133)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.Assembler.received(Assembler.java:100)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.Assembler.received(Assembler.java:42)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:187)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:103)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:42)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:102)
>>> >>> at
>>> >>>
>>> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:36)
>>> >>> at
>>> >>>
>>> org.apache.qpid.transport.network.mina.MINANetworkDriver.messageReceived(MINANetworkDriver.java:337)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:703)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:362)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain.access$1200(AbstractIoFilterChain.java:54)
>>> >>> at
>>> >>>
>>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:800)
>>> >>> at
>>> >>>
>>> org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:243)
>>> >>> at
>>> >>>
>>> org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:305)
>>> >>> at
>>> >>>
>>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
>>> >>> at
>>> >>>
>>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
>>> >>> at java.lang.Thread.run(Thread.java:619)
>>> >>> Caused by: java.net.NoRouteToHostException: Cannot assign requested
>>> >>> address
>>> >>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> >>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>> >>> at
>>> >>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>> >>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>> >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>> >>> at java.net.Socket.connect(Socket.java:529)
>>> >>> at java.net.Socket.connect(Socket.java:478)
>>> >>> at java.net.Socket.<init>(Socket.java:375)
>>> >>> at java.net.Socket.<init>(Socket.java:189)
>>> >>> at com.sun.jndi.ldap.Connection.createSocket(Connection.java:352)
>>> >>> at com.sun.jndi.ldap.Connection.<init>(Connection.java:187)
>>> >>> ... 64 more
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Carbon-dev mailing list
>>> > [email protected]
>>> > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>>> >
>>> >
>>> _______________________________________________
>>> Carbon-dev mailing list
>>> [email protected]
>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>>>
>>
>>
>>
>> --
>> Thilina Buddhika
>> Associate Technical Lead
>> WSO2 Inc. ; http://wso2.com
>> lean . enterprise . middleware
>>
>> phone : +94 77 44 88 727
>> blog : http://blog.thilinamb.com
>>
>
>
--
Thilina Buddhika
Associate Technical Lead
WSO2 Inc. ; http://wso2.com
lean . enterprise . middleware
phone : +94 77 44 88 727
blog : http://blog.thilinamb.com
_______________________________________________
Carbon-dev mailing list
[email protected]
http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
