Thanks Raj for explaining the solution. Currently the load balancer does not list the available kubernetes service proxies for a given stratos service cluster and then load balance requests among them, rather it uses the member/container list to do it. I think we will need to balance the load among the service proxies to optimally balance the load among all the containers in all the hosts. Will implement this for the next milestone.
On Sat, Oct 25, 2014 at 6:29 PM, Rajkumar Rajaratnam <[email protected]> wrote: > Yep :) > > Only thing we did is that we added allocated service port to the member > object (member's port map) so that LB can read it when it receives member > events. > > On Sat, Oct 25, 2014 at 6:08 PM, Nirmal Fernando <[email protected]> > wrote: > >> Thanks Raj! So, yes this is the same thing we had in M2. Each member gets >> a host ip and the service port. >> >> On Sat, Oct 25, 2014 at 2:29 PM, Lakmal Warusawithana <[email protected]> >> wrote: >> >>> Thanks Raj, more clear now :) >>> >>> On Sat, Oct 25, 2014 at 5:57 PM, Rajkumar Rajaratnam <[email protected] >>> > wrote: >>> >>>> Hi, >>>> >>>> Let me explain how it works now. >>>> >>>> Each member object has kubernetes host IP (in which that member is >>>> created) and service proxy port. LB is working as previously where it >>>> distributes requests between members. Since member's IP and port is now >>>> kubernetes host IP and service proxy port, LB actually distributes requests >>>> to the kubernetes service proxies. >>>> >>>> I will attach LB log. You will see that LB is sending request to the >>>> kubernetes service proxies. >>>> >>>> I am accessing via cluster host name and proxy port ( >>>> http://myphp.apachestratos.org:8281/). >>>> >>>> [2014-10-24 16:39:42,901] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Multi-tenancy enabled, scanning URL for tenant... >>>> [2014-10-24 16:39:42,901] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Request URL: / >>>> [2014-10-24 16:39:42,902] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier regex: t/([^/]*)/ >>>> [2014-10-24 16:39:42,902] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier not found in URL >>>> [2014-10-24 16:39:42,902] DEBUG - RoundRobin Searching for next member: >>>> [service] php [cluster]: myphp.php.domain [member-count]: 2 [current-index] >>>> 0 >>>> [2014-10-24 16:39:42,902] DEBUG - LoadBalancerCache Cached property: >>>> [cache] algorithm.context.cache [property] php-myphp.php.domain [value] 1 >>>> [2014-10-24 16:39:42,902] DEBUG - RequestDelegator Next member >>>> identified in 0ms: [service] php [cluster] myphp.php.domain [member] >>>> c6380d6a-5b6d-11e4-816c-08002794b041 >>>> [2014-10-24 16:39:42,902] DEBUG - TenantAwareLoadBalanceEndpoint Using >>>> member IP address: [member] c6380d6a-5b6d-11e4-816c-08002794b041 [ip] >>>> 172.17.8.101 >>>> [2014-10-24 16:39:42,902] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Updating axis2 member port >>>> [2014-10-24 16:39:42,902] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Outgoing request port found: 4503 >>>> [2014-10-24 16:39:42,902] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Sending request to endpoint:http://172.17.8.101:4503/ >>>> [2014-10-24 16:39:42,908] DEBUG - ResponseInterceptor Response >>>> interceptor mediation started >>>> [2014-10-24 16:39:46,671] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Multi-tenancy enabled, scanning URL for tenant... >>>> [2014-10-24 16:39:46,672] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Request URL: / >>>> [2014-10-24 16:39:46,672] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier regex: t/([^/]*)/ >>>> [2014-10-24 16:39:46,672] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier not found in URL >>>> [2014-10-24 16:39:46,672] DEBUG - RoundRobin Searching for next member: >>>> [service] php [cluster]: myphp.php.domain [member-count]: 2 [current-index] >>>> 1 >>>> [2014-10-24 16:39:46,672] DEBUG - LoadBalancerCache Cached property: >>>> [cache] algorithm.context.cache [property] php-myphp.php.domain [value] 0 >>>> [2014-10-24 16:39:46,672] DEBUG - RequestDelegator Next member >>>> identified in 0ms: [service] php [cluster] myphp.php.domain [member] >>>> c6385923-5b6d-11e4-816c-08002794b041 >>>> [2014-10-24 16:39:46,672] DEBUG - TenantAwareLoadBalanceEndpoint Using >>>> member IP address: [member] c6385923-5b6d-11e4-816c-08002794b041 [ip] >>>> 172.17.8.102 >>>> [2014-10-24 16:39:46,672] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Updating axis2 member port >>>> [2014-10-24 16:39:46,672] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Outgoing request port found: 4503 >>>> [2014-10-24 16:39:46,673] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Sending request to endpoint:http://172.17.8.102:4503/ >>>> [2014-10-24 16:39:46,678] DEBUG - ResponseInterceptor Response >>>> interceptor mediation started >>>> [2014-10-24 16:39:49,554] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Multi-tenancy enabled, scanning URL for tenant... >>>> [2014-10-24 16:39:49,555] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Request URL: / >>>> [2014-10-24 16:39:49,555] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier regex: t/([^/]*)/ >>>> [2014-10-24 16:39:49,555] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier not found in URL >>>> [2014-10-24 16:39:49,555] DEBUG - RoundRobin Searching for next member: >>>> [service] php [cluster]: myphp.php.domain [member-count]: 2 [current-index] >>>> 0 >>>> [2014-10-24 16:39:49,556] DEBUG - LoadBalancerCache Cached property: >>>> [cache] algorithm.context.cache [property] php-myphp.php.domain [value] 1 >>>> [2014-10-24 16:39:49,556] DEBUG - RequestDelegator Next member >>>> identified in 1ms: [service] php [cluster] myphp.php.domain [member] >>>> c6380d6a-5b6d-11e4-816c-08002794b041 >>>> [2014-10-24 16:39:49,556] DEBUG - TenantAwareLoadBalanceEndpoint Using >>>> member IP address: [member] c6380d6a-5b6d-11e4-816c-08002794b041 [ip] >>>> 172.17.8.101 >>>> [2014-10-24 16:39:49,556] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Updating axis2 member port >>>> [2014-10-24 16:39:49,557] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Outgoing request port found: 4503 >>>> [2014-10-24 16:39:49,557] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Sending request to endpoint:http://172.17.8.101:4503/ >>>> [2014-10-24 16:39:49,563] DEBUG - ResponseInterceptor Response >>>> interceptor mediation started >>>> [2014-10-24 16:39:52,104] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Multi-tenancy enabled, scanning URL for tenant... >>>> [2014-10-24 16:39:52,104] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Request URL: / >>>> [2014-10-24 16:39:52,104] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier regex: t/([^/]*)/ >>>> [2014-10-24 16:39:52,105] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier not found in URL >>>> [2014-10-24 16:39:52,105] DEBUG - RoundRobin Searching for next member: >>>> [service] php [cluster]: myphp.php.domain [member-count]: 2 [current-index] >>>> 1 >>>> [2014-10-24 16:39:52,105] DEBUG - LoadBalancerCache Cached property: >>>> [cache] algorithm.context.cache [property] php-myphp.php.domain [value] 0 >>>> [2014-10-24 16:39:52,105] DEBUG - RequestDelegator Next member >>>> identified in 0ms: [service] php [cluster] myphp.php.domain [member] >>>> c6385923-5b6d-11e4-816c-08002794b041 >>>> [2014-10-24 16:39:52,105] DEBUG - TenantAwareLoadBalanceEndpoint Using >>>> member IP address: [member] c6385923-5b6d-11e4-816c-08002794b041 [ip] >>>> 172.17.8.102 >>>> [2014-10-24 16:39:52,105] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Updating axis2 member port >>>> [2014-10-24 16:39:52,105] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Outgoing request port found: 4503 >>>> [2014-10-24 16:39:52,105] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Sending request to endpoint:http://172.17.8.102:4503/ >>>> [2014-10-24 16:39:52,109] DEBUG - ResponseInterceptor Response >>>> interceptor mediation started >>>> [2014-10-24 16:40:03,409] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Multi-tenancy enabled, scanning URL for tenant... >>>> [2014-10-24 16:40:03,410] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Request URL: / >>>> [2014-10-24 16:40:03,410] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier regex: t/([^/]*)/ >>>> [2014-10-24 16:40:03,410] DEBUG - TenantAwareLoadBalanceEndpoint Tenant >>>> identifier not found in URL >>>> [2014-10-24 16:40:03,410] DEBUG - RoundRobin Searching for next member: >>>> [service] php [cluster]: myphp.php.domain [member-count]: 2 [current-index] >>>> 0 >>>> [2014-10-24 16:40:03,410] DEBUG - LoadBalancerCache Cached property: >>>> [cache] algorithm.context.cache [property] php-myphp.php.domain [value] 1 >>>> [2014-10-24 16:40:03,411] DEBUG - RequestDelegator Next member >>>> identified in 1ms: [service] php [cluster] myphp.php.domain [member] >>>> c6380d6a-5b6d-11e4-816c-08002794b041 >>>> [2014-10-24 16:40:03,411] DEBUG - TenantAwareLoadBalanceEndpoint Using >>>> member IP address: [member] c6380d6a-5b6d-11e4-816c-08002794b041 [ip] >>>> 172.17.8.101 >>>> [2014-10-24 16:40:03,411] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Updating axis2 member port >>>> [2014-10-24 16:40:03,411] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Outgoing request port found: 4503 >>>> [2014-10-24 16:40:03,411] DEBUG - TenantAwareLoadBalanceEndpoint >>>> Sending request to endpoint:http://172.17.8.101:4503/ >>>> [2014-10-24 16:40:03,414] DEBUG - ResponseInterceptor Response >>>> interceptor mediation started >>>> >>>> >>>> Thanks. >>>> >>>> >>>> >>>> On Sat, Oct 25, 2014 at 5:39 PM, Lakmal Warusawithana <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Sat, Oct 25, 2014 at 5:32 PM, Lakmal Warusawithana <[email protected] >>>>> > wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sat, Oct 25, 2014 at 2:19 PM, Nirmal Fernando < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Oct 25, 2014 at 6:26 AM, Imesh Gunaratne <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> We have now fixed the following issues identified with the load >>>>>>>> balancer: >>>>>>>> >>>>>>>> - Fixed messaging component dependencies in load balancer package. >>>>>>>> - Fixed an issue in messaging component which raised a class cast >>>>>>>> exception in debug mode. >>>>>>>> - Added generated service host port to the member port map. This >>>>>>>> will overwrite the port value given by the cartridge definition with >>>>>>>> the >>>>>>>> generated service host port, if the cluster is kubernetes. >>>>>>>> >>>>>>> >>>>> Here are we talking about kubernetes host and service (pod) port? And >>>>> we are adding them into LB end point? >>>>> >>>>> >>>>>> - Now the load balancer will apply the load balancing algorithm on >>>>>>>> the list of members found in the cluster, not on the list of service >>>>>>>> proxies. >>>>>>>> >>>>>>> >>>>>>> Can you explain this a bit more please? Does this mean we are not >>>>>>> going to use Kubernetes service proxies anymore? >>>>>>> >>>>>> >>>>>> Yes, I also unclear on this. If we are not taking kubernetes service >>>>>> proxies, how do we identified members? >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> We could improve further in a future milestone. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> -- >>>>>>>> Imesh Gunaratne >>>>>>>> >>>>>>>> Technical Lead, WSO2 >>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Nirmal Fernando. >>>>>>> PPMC Member & Committer of Apache Stratos, >>>>>>> Senior Software Engineer, WSO2 Inc. >>>>>>> >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lakmal Warusawithana >>>>>> Vice President, Apache Stratos >>>>>> Director - Cloud Architecture; WSO2 Inc. >>>>>> Mobile : +94714289692 >>>>>> Blog : http://lakmalsview.blogspot.com/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Lakmal Warusawithana >>>>> Vice President, Apache Stratos >>>>> Director - Cloud Architecture; WSO2 Inc. >>>>> Mobile : +94714289692 >>>>> Blog : http://lakmalsview.blogspot.com/ >>>>> >>>>> >>>> >>>> >>>> -- >>>> Rajkumar Rajaratnam >>>> Software Engineer | WSO2, Inc. >>>> Mobile +94777568639 | +94783498120 >>>> >>> >>> >>> >>> -- >>> Lakmal Warusawithana >>> Vice President, Apache Stratos >>> Director - Cloud Architecture; WSO2 Inc. >>> Mobile : +94714289692 >>> Blog : http://lakmalsview.blogspot.com/ >>> >>> >> >> >> -- >> Best Regards, >> Nirmal >> >> Nirmal Fernando. >> PPMC Member & Committer of Apache Stratos, >> Senior Software Engineer, WSO2 Inc. >> >> Blog: http://nirmalfdo.blogspot.com/ >> > > > > -- > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > Mobile +94777568639 | +94783498120 > -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos
