I noticed that both the Presenter and the Worker pods spawn in the same Kubernetes node, probably causing the resource limitation. Using nodeSelector on the Replication Controller for the Worker node, I moved the Worker pod to the other node, and CEP started without an issue. It's most likely that the OOM is causing the thread creation to fail, and subsequently throwing an Address already in use Exception.
Regards, Chamila de Alwis Committer and PMC Member - Apache Stratos Software Engineer | WSO2 | +94772207163 Blog: code.chamiladealwis.com On Wed, May 18, 2016 at 8:09 AM, Chamila De Alwis <[email protected]> wrote: > Netstat shows that 9711 is opened by the Java process started by > wso2server.sh. > > > Regards, > Chamila de Alwis > Committer and PMC Member - Apache Stratos > Software Engineer | WSO2 | +94772207163 > Blog: code.chamiladealwis.com > > > > On Wed, May 18, 2016 at 7:40 AM, Chamila De Alwis <[email protected]> > wrote: > >> Hi, >> >> When starting a CEP 4.0.0 Worker node, I'm coming across the following >> issue, where the BinaryDataReceiverServiceComponent doesn't start. The SSL >> port for the BinaryDataReceiver is the default 9711. Please note that this >> instance is a Pod in a Kubernetes Cluster, and at the same time, there is >> an OOM exception spawning from the H2 driver, along with >> "java.lang.OutOfMemoryError: unable to create new native thread". I'm >> assuming this is because of the particular node's resource limitation. >> >> The exception in question is coming from [1] code when it's trying to >> create threads according to the SSL thread pool size for the >> BinaryDataReceiver. Could "Address already in Use" exception arise from >> conditions resulting in OOM? >> >> [2016-05-18 01:53:36,631] ERROR >> {org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiverServiceComponent} >> - Error while starting binary data receiver >> java.net.BindException: Address already in use >> at java.net.PlainSocketImpl.socketBind(Native Method) >> at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) >> at java.net.ServerSocket.bind(ServerSocket.java:376) >> at java.net.ServerSocket.<init>(ServerSocket.java:237) >> at java.net.ServerSocket.<init>(ServerSocket.java:181) >> at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:136) >> at >> sun.security.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:107) >> at >> sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:77) >> at >> org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiver.startSecureTransmission(BinaryDataReceiver.java:93) >> at >> org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiver.start(BinaryDataReceiver.java:64) >> at >> org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiverServiceComponent.activate(BinaryDataReceiverServiceComponent.java:55) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) >> at >> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) >> at >> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) >> at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) >> at >> org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) >> at >> org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) >> at >> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) >> at >> org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) >> at >> org.wso2.carbon.databridge.core.internal.DataBridgeDS.activate(DataBridgeDS.java:132) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) >> at >> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) >> at >> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) >> at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) >> at >> org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) >> at >> org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) >> at >> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) >> at >> org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) >> at >> org.wso2.carbon.identity.authentication.internal.AuthenticationServiceComponent.activate(AuthenticationServiceComponent.java:57) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) >> at >> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) >> at >> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) >> at >> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) >> at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) >> at >> org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) >> at >> org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) >> at >> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) >> at >> org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) >> at >> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) >> at >> org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:69) >> at >> org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702) >> at >> org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683) >> at >> org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381) >> at >> org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390) >> at >> org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176) >> at >> org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559) >> at >> org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544) >> at >> org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457) >> at >> org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243) >> at >> org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438) >> at >> org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1) >> at >> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) >> at >> org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340) >> >> >> [1] - >> https://github.com/wso2/carbon-commons/blob/4.4.x/components/data-bridge/org.wso2.carbon.databridge.receiver.binary/src/main/java/org/wso2/carbon/databridge/receiver/binary/BinaryDataReceiver.java#L93 >> >> >> Regards, >> Chamila de Alwis >> Committer and PMC Member - Apache Stratos >> Software Engineer | WSO2 | +94772207163 >> Blog: code.chamiladealwis.com >> >> >> >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
