Update: ===== Due to the issues we got in setting up the min HA setup, we reverted to using DAS with spark client mode [1]. After deploying the spark 1.6 application, got the error [2] from the DAS side. The reason was having the spark master url as spark://<HOST>:<PORT>. It should be mesos://<HOST>:<PORT>. But in DAS 3.0.1 we are expecting the spark master url to start with spark [3]. This needs to be patched. After patching it, the connection issue from DAS to spark master got resolved. But facing another issue [4]. Seems we need to have a libmesos.so shared object library in one of the relevant locations (/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib). Therefore, to proceed with this appoach, we might have to have mesos installed in the machine (in this case, in the Docker image) in which DAS is running.
[1]. https://docs.wso2.com/display/CLUSTER44x/Spark+Deployment+Patterns+-+DAS+3.0.1#SparkDeploymentPatterns-DAS3.0.1-Clientmode [2]. [2016-07-20 06:39:25,311] INFO {org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor} - Starting SPARK CLIENT pointing to an external Spark Cluster [2016-07-20 06:39:26,073] INFO {Remoting} - Starting remoting [2016-07-20 06:39:26,258] INFO {Remoting} - Remoting started; listening on addresses :[akka.tcp://[email protected]:21543] [2016-07-20 06:39:26,987] WARN {akka.remote.ReliableDeliverySupervisor} - Association with remote system [akka.tcp://[email protected]:5050] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://[email protected]:5050]]. [2016-07-20 06:39:46,878] WARN {akka.remote.ReliableDeliverySupervisor} - Association with remote system [akka.tcp://[email protected]:5050] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://[email protected]:5050]]. [2016-07-20 06:40:06,882] WARN {akka.remote.ReliableDeliverySupervisor} - Association with remote system [akka.tcp://[email protected]:5050] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://[email protected]:5050]]. [2016-07-20 06:40:26,983] ERROR {akka.actor.OneForOneStrategy} - java.lang.NullPointerException at org.apache.spark.deploy.client.AppClient$ClientActor$$anonfun$receiveWithLogging$1.applyOrElse(AppClient.scala:160) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:59) at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:42) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) at org.apache.spark.util.ActorLogReceive$$anon$1.applyOrElse(ActorLogReceive.scala:42) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at org.apache.spark.deploy.client.AppClient$ClientActor.aroundReceive(AppClient.scala:61) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [2016-07-20 06:40:26,999] WARN {akka.remote.ReliableDeliverySupervisor} - Association with remote system [akka.tcp://[email protected]:5050] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://[email protected]:5050]]. [2016-07-20 06:40:27,306] ERROR {org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent} - Error initializing analytics executor: Unable to create analytics client. Cannot call methods on a stopped SparkContext [3]. https://github.com/wso2/carbon-analytics/blob/v1.0.5/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/internal/SparkAnalyticsExecutor.java#L270 [4]. Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib [2016-07-20 09:27:16,151] ERROR {org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent} - Error initializing analytics executor: no mesos in java.library.path java.lang.UnsatisfiedLinkError: no mesos in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886) at java.lang.Runtime.loadLibrary0(Runtime.java:849) at java.lang.System.loadLibrary(System.java:1088) at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:52) at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:64) at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2562) at org.apache.spark.SparkContext.<init>(SparkContext.scala:489) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.initializeSparkContext(SparkAnalyticsExecutor.java:346) at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.initializeClient(SparkAnalyticsExecutor.java:368) at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.initializeSparkServer(SparkAnalyticsExecutor.java:198) at org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent.activate(AnalyticsComponent.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:950) at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceComponent.activate(AnalyticsDataServiceComponent.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.wso2.carbon.ntask.core.internal.TasksDSComponent.activate(TasksDSComponent.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:199) at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:288) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:514) at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219) at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81) at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60) at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40) at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1284) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5262) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5550) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) On Mon, Jul 18, 2016 at 1:30 PM, Niranda Perera <[email protected]> wrote: > Hi Isuru, > > Sorry for replying late, but I have replied to the previous mail you have > sent regarding akka configurations in Spark [1] > > Best > > [1] [DAS] Setting akka Configuration Properties > > On Mon, Jul 18, 2016 at 1:03 PM, Isuru Haththotuwa <[email protected]> > wrote: > >> Hi all, >> >> For the DAS min HA usecase, there is an additional TCP point to point >> connection for spark masters. In the master branch of >> org.wso2.carbon.analytics.spark.core, we can use SPARK_LOCAL_IP property to >> set a different IP address for this [1]. But, in the version 1.0.5 which >> was released with DAS 3.0.1 does not have this fix. I backported the fix >> and tested, it works. >> >> However, only setting this property to the host name does not solve this >> issue since akka system tries to bind to the same host IP, which is not >> visible to the internal network, resulting in error [2]. >> >> The solution would be to define two IP addresses for akka system; one to >> bind (private Ip), another to advertise (routable IP) as explained in akka >> docs [3]. Can we do this from DAS side? Please share your input as this is >> a blocking issue for deploying DAS in mesos. >> >> [1]. >> https://github.com/wso2/carbon-analytics/blob/master/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/internal/AnalyticsComponent.java#L246 >> >> [2]. ERROR {akka.remote.transport.netty.NettyTransport} - failed to bind >> to /x.x.x.x:22093, shutting down Netty transport >> >> [3]. >> http://doc.akka.io/docs/akka/snapshot/additional/faq.html#Why_are_replies_not_received_from_a_remote_actor_ >> >> >> On Thu, Jul 14, 2016 at 10:04 PM, Isuru Haththotuwa <[email protected]> >> wrote: >> >>> Update: >>> DAS, CEP and MB are the three products that currently we have >>> identified, which need point to point TCP connections for HA deployment. >>> >>> *DAS and CEP:* >>> >>> This is required for event synchronization [1,2] using thrift transport. >>> Had a offline chat with Sinthuja on this. As per the ThriftDataReceiver >>> code [3], the thrift event receiver actually picks up the bind hostname >>> from repository/conf/data-bridge/data-bridge-config.xml file. In that case, >>> we can simply use the local IP for Thrift event receiver, and use the >>> mapped host ip and port for the event synchronization configuration. >>> >>> >>> >>> *MB:* >>> Required for slot co-ordination [4]. As the solution, we would need to >>> define a bind ip + port and a separate public ip + port to advertise. Had a >>> discussion regarding this with Ramith. If MB is using the same thrift data >>> receiver [5], we might be able to use data-bridge-config.xml to define a >>> bind ip. @Ramith: Can you please clarify this? >>> >>> [1]. >>> https://docs.wso2.com/display/CLUSTER44x/Minimum+High+Availability+Deployment+-+DAS+3.0.1 >>> >>> [2]. >>> https://docs.wso2.com/display/CLUSTER44x/Clustering+CEP+4.0.0#ClusteringCEP4.0.0-Highavailabilitydeployment >>> >>> [3]. >>> https://github.com/wso2/carbon-analytics-common/blob/master/components/data-bridge/org.wso2.carbon.databridge.receiver.thrift/src/main/java/org/wso2/carbon/databridge/receiver/thrift/conf/ThriftDataReceiverConfiguration.java >>> >>> [4]. >>> https://docs.wso2.com/display/CLUSTER44x/Configuring+the+Broker+Nodes >>> >>> [5]. >>> https://github.com/wso2/carbon-analytics-common/blob/f3ff09d9655bf599d45436f0630b6413fc3d1b60/components/data-bridge/org.wso2.carbon.databridge.receiver.thrift/src/main/java/org/wso2/carbon/databridge/receiver/thrift/ThriftDataReceiver.java >>> >>> >>> On Tue, Jul 12, 2016 at 1:52 PM, Udara Liyanage <[email protected]> wrote: >>> >>>> >>>> >>>> On Tue, Jul 12, 2016 at 12:56 PM, Isuru Haththotuwa <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> This is to discuss about $subject. >>>>> >>>>> By default Mesos-Marathon [1] does not have an overlay network. >>>>> Therefore when deploying wso2 products in distributed/HA manner in mesos, >>>>> the products are unable to create point to point TCP connections directly >>>>> from container to container. >>>>> >>>>> In Carbon products, p2p connections are created by Hazelcast, within >>>>> cluster members. Since this is just for communication purposes, its >>>>> possible to do a workaround by querying the API for the IPs of the host >>>>> machine. This is done in the Mesos membership scheme [2]. >>>>> >>>>> There are few places where we need to create TCP Listeners, for an >>>>> example in CEP HA/Distributed setup [3]. For this, we can't just bind to >>>>> the host node IP since the public IP is not visible to the container; and >>>>> can't use the local container IP since container to container >>>>> communication >>>>> is not possible (due to the lack of an overlay network). >>>>> >>>>> One possible option is such scenarios is to use the HOST networking >>>>> mode of Docker [4]. Then the container will be using the host IP and ports >>>>> directly. One issue with this approach is since we are mapping container >>>>> ports to host ports directly without assigning random ports from the host >>>>> side, only one container can run in one host node. But, in a way this >>>>> defeats the purpose of containerization. >>>>> >>>> Using host mode would be a good alternative. I don't think limiting one >>>> container per Mesos node (Mesos HOST=UNIQUE constrain) in this scenario >>>> since a HA Mesos deployment will have multiple Mesos nodes. >>>> Another advantage of running in host mode is performance efficiency >>>> rather than bridge mode. Network efficiency would be useful in thrift >>>> scenario since it is assumed to be fast. >>>> >>> Agreed. However, IMHO we should consider this as the last option if >>> nothing else works out. >>> >>>> >>>>> Please share your thoughts on this. >>>>> >>>>> [1]. https://github.com/mesosphere/marathon >>>>> >>>>> [2]. >>>>> https://github.com/wso2/mesos-artifacts/tree/master/common/mesos-membership-scheme >>>>> >>>>> [3]. >>>>> https://docs.wso2.com/display/CLUSTER44x/Clustering+CEP+4.0.0#ClusteringCEP4.0.0-Highavailabilitydeployment >>>>> >>>>> [4]. http://www.dasblinkenlichten.com/docker-networking-101-host-mode/ >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> >>>>> Isuru H. >>>>> +94 716 358 048 >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Udara Liyanage >>>> Software Engineer >>>> WSO2, Inc.: http://wso2.com >>>> lean. enterprise. middleware >>>> >>>> web: http://udaraliyanage.wordpress.com >>>> phone: +94 71 443 6897 >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> >>> Isuru H. >>> +94 716 358 048* <http://wso2.com/>* >>> >>> >>> >> >> >> -- >> Thanks and Regards, >> >> Isuru H. >> +94 716 358 048* <http://wso2.com/>* >> >> >> > > > -- > *Niranda Perera* > Software Engineer, WSO2 Inc. > Mobile: +94-71-554-8430 > Twitter: @n1r44 <https://twitter.com/N1R44> > https://pythagoreanscript.wordpress.com/ > -- Thanks and Regards, Isuru H. +94 716 358 048* <http://wso2.com/>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
