Hi all, For the DAS min HA usecase, there is an additional TCP point to point connection for spark masters. In the master branch of org.wso2.carbon.analytics.spark.core, we can use SPARK_LOCAL_IP property to set a different IP address for this [1]. But, in the version 1.0.5 which was released with DAS 3.0.1 does not have this fix. I backported the fix and tested, it works.
However, only setting this property to the host name does not solve this issue since akka system tries to bind to the same host IP, which is not visible to the internal network, resulting in error [2]. The solution would be to define two IP addresses for akka system; one to bind (private Ip), another to advertise (routable IP) as explained in akka docs [3]. Can we do this from DAS side? Please share your input as this is a blocking issue for deploying DAS in mesos. [1]. https://github.com/wso2/carbon-analytics/blob/master/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/internal/AnalyticsComponent.java#L246 [2]. ERROR {akka.remote.transport.netty.NettyTransport} - failed to bind to /x.x.x.x:22093, shutting down Netty transport [3]. http://doc.akka.io/docs/akka/snapshot/additional/faq.html#Why_are_replies_not_received_from_a_remote_actor_ On Thu, Jul 14, 2016 at 10:04 PM, Isuru Haththotuwa <[email protected]> wrote: > Update: > DAS, CEP and MB are the three products that currently we have identified, > which need point to point TCP connections for HA deployment. > > *DAS and CEP:* > > This is required for event synchronization [1,2] using thrift transport. > Had a offline chat with Sinthuja on this. As per the ThriftDataReceiver > code [3], the thrift event receiver actually picks up the bind hostname > from repository/conf/data-bridge/data-bridge-config.xml file. In that case, > we can simply use the local IP for Thrift event receiver, and use the > mapped host ip and port for the event synchronization configuration. > > > > *MB:* > Required for slot co-ordination [4]. As the solution, we would need to > define a bind ip + port and a separate public ip + port to advertise. Had a > discussion regarding this with Ramith. If MB is using the same thrift data > receiver [5], we might be able to use data-bridge-config.xml to define a > bind ip. @Ramith: Can you please clarify this? > > [1]. > https://docs.wso2.com/display/CLUSTER44x/Minimum+High+Availability+Deployment+-+DAS+3.0.1 > > [2]. > https://docs.wso2.com/display/CLUSTER44x/Clustering+CEP+4.0.0#ClusteringCEP4.0.0-Highavailabilitydeployment > > [3]. > https://github.com/wso2/carbon-analytics-common/blob/master/components/data-bridge/org.wso2.carbon.databridge.receiver.thrift/src/main/java/org/wso2/carbon/databridge/receiver/thrift/conf/ThriftDataReceiverConfiguration.java > > [4]. https://docs.wso2.com/display/CLUSTER44x/Configuring+the+Broker+Nodes > > [5]. > https://github.com/wso2/carbon-analytics-common/blob/f3ff09d9655bf599d45436f0630b6413fc3d1b60/components/data-bridge/org.wso2.carbon.databridge.receiver.thrift/src/main/java/org/wso2/carbon/databridge/receiver/thrift/ThriftDataReceiver.java > > > On Tue, Jul 12, 2016 at 1:52 PM, Udara Liyanage <[email protected]> wrote: > >> >> >> On Tue, Jul 12, 2016 at 12:56 PM, Isuru Haththotuwa <[email protected]> >> wrote: >> >>> Hi all, >>> >>> This is to discuss about $subject. >>> >>> By default Mesos-Marathon [1] does not have an overlay network. >>> Therefore when deploying wso2 products in distributed/HA manner in mesos, >>> the products are unable to create point to point TCP connections directly >>> from container to container. >>> >>> In Carbon products, p2p connections are created by Hazelcast, within >>> cluster members. Since this is just for communication purposes, its >>> possible to do a workaround by querying the API for the IPs of the host >>> machine. This is done in the Mesos membership scheme [2]. >>> >>> There are few places where we need to create TCP Listeners, for an >>> example in CEP HA/Distributed setup [3]. For this, we can't just bind to >>> the host node IP since the public IP is not visible to the container; and >>> can't use the local container IP since container to container communication >>> is not possible (due to the lack of an overlay network). >>> >>> One possible option is such scenarios is to use the HOST networking mode >>> of Docker [4]. Then the container will be using the host IP and ports >>> directly. One issue with this approach is since we are mapping container >>> ports to host ports directly without assigning random ports from the host >>> side, only one container can run in one host node. But, in a way this >>> defeats the purpose of containerization. >>> >> Using host mode would be a good alternative. I don't think limiting one >> container per Mesos node (Mesos HOST=UNIQUE constrain) in this scenario >> since a HA Mesos deployment will have multiple Mesos nodes. >> Another advantage of running in host mode is performance efficiency >> rather than bridge mode. Network efficiency would be useful in thrift >> scenario since it is assumed to be fast. >> > Agreed. However, IMHO we should consider this as the last option if > nothing else works out. > >> >>> Please share your thoughts on this. >>> >>> [1]. https://github.com/mesosphere/marathon >>> >>> [2]. >>> https://github.com/wso2/mesos-artifacts/tree/master/common/mesos-membership-scheme >>> >>> [3]. >>> https://docs.wso2.com/display/CLUSTER44x/Clustering+CEP+4.0.0#ClusteringCEP4.0.0-Highavailabilitydeployment >>> >>> [4]. http://www.dasblinkenlichten.com/docker-networking-101-host-mode/ >>> >>> >>> -- >>> Thanks and Regards, >>> >>> Isuru H. >>> +94 716 358 048 >>> >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> >> Udara Liyanage >> Software Engineer >> WSO2, Inc.: http://wso2.com >> lean. enterprise. middleware >> >> web: http://udaraliyanage.wordpress.com >> phone: +94 71 443 6897 >> > > > > -- > Thanks and Regards, > > Isuru H. > +94 716 358 048* <http://wso2.com/>* > > > -- Thanks and Regards, Isuru H. +94 716 358 048* <http://wso2.com/>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
