Hi,
I have few other concerns to get clarified and few thoughts.
1. Original design was to spawn EC2 instances after the number of live
instances exceed a threshold value (eg: 25). What if we spawn an EC2
instance only if we failed to spawn a JVM instance in any Agent? As far as
I see, the latter may be bit inefficient but cost effective (since we're
not spawning an EC2 instance when we can spawn a JVM instance).
2. What's the standard way of specifying Agent's working directory? I
thought of adding it to the autoscaler_agent_config.xml. Please see [a].
3. How to specify the path to the "autoscaler_agent_config.xml"? Currently
it is hard coded.
4. How to let AutoscalerService know the EPR of the AgentManagementService?
Currently it is hard coded. Is it good to assume that both autoscaler
service and agent management service are hosted in one location, and
deriving the EPR of the AgentManagementService?
5. Agent should be smart enough to decide whether it can spawn a new JVM
instance, i.e. whether it has the minimum required memory etc. available.
What about using Dileepa's boot-up validator component for this purpose?
[a]
<!--
This file should specify all the Agent related configuration details.
-->
<agentConfig>
<!-- This points to the place where helper services such as
AgentManagementService
and AutoscalerService hosted. -->
<autoscalerHost url="http://localhost:8080/axis2/services/"/>
<!-- Following element should points to the working directory of this
Agent. This is the directory
where Agent will copy the extratcted carbon-home directories when a new
instance is spawned.-->
<workingDir path="/home/nirmal/Documents/autoscaler/work/"/>
<!--
image element contains the details of the image.
Each image element should have two attributes.
1. path: path to the image.
2. domain: domain this particular image belongs to.
-->
<image
path="/home/nirmal/Documents/autoscaler/wso2as-4.1.0-SNAPSHOT.zip"
domain="wso2.as.domain"/>
<image path="/home/nirmal/Desktop/wso2cep-4.1.0-SNAPSHOT.zip"
domain="wso2.cep.domain"/>
</agentConfig>
On Mon, Feb 27, 2012 at 9:56 AM, Nirmal Fernando <nir...@wso2.com> wrote:
> Hi,
>
> Please find the sequence type of diagram of starting instance scenario,
> attached herewith.
>
> On Wed, Feb 22, 2012 at 12:36 PM, Afkham Azeez <az...@wso2.com> wrote:
>
>>
>>
>> On Wed, Feb 22, 2012 at 9:50 AM, Selvaratnam Uthaiyashankar <
>> shan...@wso2.com> wrote:
>>
>>> On Wed, Feb 22, 2012 at 8:27 AM, Nirmal Fernando <nir...@wso2.com>
>>> wrote:
>>> > Hi Shankar,
>>> >
>>> > On Wed, Feb 22, 2012 at 8:12 AM, Selvaratnam Uthaiyashankar
>>> > <shan...@wso2.com> wrote:
>>> >>
>>> >> Hi Nirmal,
>>> >>
>>> >> On Tue, Feb 21, 2012 at 11:04 AM, Nirmal Fernando <nir...@wso2.com>
>>> wrote:
>>> >> >
>>> >> >
>>> >> > On Tue, Feb 21, 2012 at 10:18 AM, Nirmal Fernando <nir...@wso2.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi All,
>>> >> >>
>>> >> >> I need your views on following.
>>> >> >>
>>> >> >> In the Agent side we need to have a configuration file (say
>>> >> >> instances_config.xml) which specifies the paths and names of the
>>> >> >> instances
>>> >> >> belong to domains.
>>> >> >>
>>> >> >> I thought to have following structure:
>>> >> >>
>>> >> > Value of the path attribute should change like this:
>>> >> >
>>> >> >>
>>> >> >> <domain name="wso2.as.domain">
>>> >> >> <instance name="wso2as-4.1.0-SNAPSHOT"
>>> >> >>
>>> path="/home/nirmal/Desktop/wso2as-4.1.0-SNAPSHOT/bin/wso2server.sh"/>
>>> >> >> <instance name="wso2as-4.1.0-SNAPSHOT"
>>> >> >> path="/home/nirmal/Temp/wso2as-4.1.0-SNAPSHOT/bin/wso2server.sh"/>
>>> >> >>
>>> >> >> .....
>>> >> >> </domain>
>>> >> >> <domain name="wso2.bps.domain">
>>> >> >> <instance name="wso2bps-4.1.0-SNAPSHOT"
>>> >> >>
>>> path="/home/nirmal/Desktop/wso2bps-4.1.0-SNAPSHOT/bin/wso2server.sh"/>
>>> >> >> <instance name="wso2bps-4.1.0-SNAPSHOT"
>>> >> >> path="/home/nirmal/Temp/wso2bps-4.1.0-SNAPSHOT/bin/wso2server.sh"/>
>>> >> >>
>>> >> >> .....
>>> >> >> </domain>
>>> >>
>>> >>
>>> >> What are these two instances? Why they should be in the configuration
>>> >> file? If they are the paths of instances that can be started, then if
>>> >> I want to start a third instance what should I do?
>>> >
>>> >
>>> > Agents should know the instances that it can spawn. Thus, each Agent
>>> machine
>>> > has a configuration file which points to those instances. An Agent can
>>> only
>>> > spawn instances it has.
>>> >
>>> > This configuration file will be loaded each time an Agent get
>>> registered. So
>>> > if you want to add a new instance, you can unregister the Agent, edit
>>> the
>>> > configuration file and re-register. Then the Agent Service will locate
>>> the
>>> > newly added instance and list it as an instance that can be spawned.
>>>
>>> This might become a maintanance problem. Suppose I have a machine
>>> which can run 5 instance of stratos service. This service can be ESB,
>>> AppServer, GS, MS, Manager, .... We have 13 service, so I have to give
>>> 13 * 5 configuration.
>>>
>>> I would like to give the location of binary pack location for 13
>>> services, and when an instance is required, agent can copy the pack
>>> into its "working directory" and run it. So, actual instance location
>>> is not needed and will be handled by agent. This is the normal way
>>> most of the software work (e.g. eucalyptus). In this method, the
>>> configuration for all JVM autoscale agent will be same. (can copy the
>>> binary pack in same location, working directory can be same location).
>>> Based on the request, Agent will copy and start the "instace"
>>> dynamically.
>>>
>>
>> Yes, that is how it should be done. It is basically like providing the
>> image ID. External invokers will simply ask for an instance of a particular
>> binary pack to be started. They don't specify the actual instance.
>>
>>>
>>> WDYT?
>>>
>>> Shankar
>>>
>>>
>>> >
>>> > WDYT?
>>> >
>>> >>
>>> >>
>>> >> Shankar
>>> >>
>>> >>
>>> >> >> .....
>>> >> >>
>>> >> >> WDYT?
>>> >> >>
>>> >> >> PS: It is my understanding that the autoscaler component passes the
>>> >> >> 'domain name' to the "Autoscaler Service", when it wants to scale.
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> On Mon, Feb 20, 2012 at 9:31 PM, Selvaratnam Uthaiyashankar
>>> >> >> <shan...@wso2.com> wrote:
>>> >> >>>
>>> >> >>> On Mon, Feb 20, 2012 at 8:17 PM, Nirmal Fernando <nir...@wso2.com
>>> >
>>> >> >>> wrote:
>>> >> >>> > Hi Shankar,
>>> >> >>> >
>>> >> >>> > On Mon, Feb 20, 2012 at 5:34 PM, Selvaratnam Uthaiyashankar
>>> >> >>> > <shan...@wso2.com> wrote:
>>> >> >>> >>
>>> >> >>> >> Hi Nirmal,
>>> >> >>> >>
>>> >> >>> >> Good progress!. we can do a review on the work already done.
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > Thanks, will arrange a review this week.
>>> >> >>> >
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> Did you try the agent in windows?
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > No, I didn't!
>>> >> >>> >
>>> >> >>> >>
>>> >> >>> >> Is it using any native code?
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > It needs to use native code in cases like spawning a new JVM (in
>>> >> >>> > Ubuntu
>>> >> >>> > we
>>> >> >>> > need /bin/sh to run the script 'wso2server.sh' in windows we
>>> need to
>>> >> >>> > run the
>>> >> >>> > batch file.
>>> >> >>> > But I believe this can be handled within the code it self by
>>> >> >>> > checking
>>> >> >>> > the OS
>>> >> >>> > name and execute appropriate commands!
>>> >> >>> >
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> See some comments below.
>>> >> >>> >>
>>> >> >>> >> On Sat, Feb 18, 2012 at 11:10 PM, Nirmal Fernando <
>>> nir...@wso2.com>
>>> >> >>> >> wrote:
>>> >> >>> >> > Hi All,
>>> >> >>> >> >
>>> >> >>> >> > I've started to implement 'new autoscaler architecture' [1]
>>> on
>>> >> >>> >> > February
>>> >> >>> >> > 8th.
>>> >> >>> >> > After having a discussion with Azeez, we came up with three
>>> >> >>> >> > milestones
>>> >> >>> >> > for
>>> >> >>> >> > the first iteration.
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> >> >>> >> >
>>> >> >>> >> > M1: Make a WS call spawns a new JVM instance & make the new
>>> >> >>> >> > instance
>>> >> >>> >> > joins
>>> >> >>> >> > the cluster with hard coded Agents.
>>> >> >>> >> >
>>> >> >>> >> > M2: Finish Agent registration related parts and spawn a new
>>> >> >>> >> > instance
>>> >> >>> >> > in
>>> >> >>> >> > a
>>> >> >>> >> > selected Agent.
>>> >> >>> >> >
>>> >> >>> >> > M3: Let current autoscaler component calls
>>> 'AutoscalerService'
>>> >> >>> >> > and
>>> >> >>> >> > spawns a
>>> >> >>> >> > new JVM instance in a selected Agent. Also applying WS
>>> security.
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> >> >>> >> >
>>> >> >>> >> > I've attached set of images which depicts different stages
>>> of new
>>> >> >>> >> > architecture. Reading order of images is as follows:
>>> >> >>> >> >
>>> >> >>> >> > agent-reg --> autoscaler --> spawn --> cluster
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > M1 is already finished.
>>> >> >>> >> >
>>> >> >>> >> > I've almost done the Agent registration (M2) aspect.
>>> >> >>> >> >
>>> >> >>> >> > What it does now?
>>> >> >>> >> >
>>> >> >>> >> > AgentManagementService and AutoscalerService should be
>>> deployed
>>> >> >>> >> > in
>>> >> >>> >> > the
>>> >> >>> >> > back-end server. (say: IP is 192.168.1.2)
>>> >> >>> >> >
>>> >> >>> >> > AgentService should be deployed in machines (i.e. where new
>>> JVM
>>> >> >>> >> > instances
>>> >> >>> >> > will be spawned). (say: IP is 192.168.1.3)
>>> >> >>> >> >
>>> >> >>> >> > AgentService do a web service call (eg: curl -X GET
>>> >> >>> >> > http://localhost:8080/axis2/services/AgentService/agent) to
>>> >> >>> >> > AgentManagementService and get registered. Then
>>> >> >>> >> > AgentManagementService
>>> >> >>> >> > stores the URL (eg:
>>> >> >>> >> > http://192.168.1.3:8080/axis2/services/AgentService/) of
>>> >> >>> >> > that particular request.
>>> >> >>> >> >
>>> >> >>> >> > When autoscaler API (implementing this is M3) wanted to
>>> spawn a
>>> >> >>> >> > new
>>> >> >>> >> > instance
>>> >> >>> >> > (JVM or EC2 or anything else), it should call
>>> AutoscalerService
>>> >> >>> >> > (eg:
>>> >> >>> >> > curl
>>> >> >>> >> > --data "instanceName=as" -X POST
>>> >> >>> >> >
>>> >> >>> >> >
>>> http://192.168.1.2:8080/axis2/services/AutoscalerService/instance).
>>> >> >>> >> > Then
>>> >> >>> >> > AutoscalerService decides which kind of instance to spawn.
>>> >> >>> >> >
>>> >> >>> >> > If AutoscalerService decides to spawn a JVM instance, it
>>> creates
>>> >> >>> >> > a
>>> >> >>> >> > new
>>> >> >>> >> > JVMAdaptor instance. JVMAdaptor then calls (eg: curl -X GET
>>> >> >>> >> >
>>> >> >>> >> >
>>> http://192.168.1.2:8080/axis2/services/AgentManagementService/agent)
>>> >> >>> >> > AgentManagementService in order to pick an Agent.
>>> >> >>> >> > AgentManagementService
>>> >> >>> >> > picks an Agent in a round robin manner and sends back that
>>> >> >>> >> > Agent's
>>> >> >>> >> > EPR
>>> >> >>> >> > (http://192.168.1.3:8080/axis2/services/AgentService/) to
>>> >> >>> >> > JVMAdaptor.
>>> >> >>> >> > Then
>>> >> >>> >> > JVMAdaptor does a web service call to that particular
>>> >> >>> >> > AgentService
>>> >> >>> >> > and
>>> >> >>> >> > asked
>>> >> >>> >> > it to spawn the requested instance (instance is yet hard
>>> coded).
>>> >> >>> >> >
>>> >> >>> >> > Spawned instance then joins the cluster.
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > Current observations:
>>> >> >>> >> >
>>> >> >>> >> > Currently starting instance aspect is done.
>>> >> >>> >> >
>>> >> >>> >> > I've tested this using 2 machines, and starting instance
>>> works
>>> >> >>> >> > fine.
>>> >> >>> >> >
>>> >> >>> >> > It is observed that the spawned JVM instances get killed
>>> when you
>>> >> >>> >> > issue
>>> >> >>> >> > 'ctrl+c' in the terminal where axis2 running. But when I
>>> killed
>>> >> >>> >> > axis2server.sh process only, the new JAVA process wasn't get
>>> >> >>> >> > killed.
>>> >> >>> >> > To
>>> >> >>> >> > avoid process get killing when issue ctrl+c, we might need to
>>> >> >>> >> > handle
>>> >> >>> >> > SIGINT
>>> >> >>> >> > [2]. Can't we be happy with the fact that it doesn't get
>>> killed
>>> >> >>> >> > after
>>> >> >>> >> > killing axis2server.sh?
>>> >> >>> >>
>>> >> >>> >> JVMs (which were created) should live even if you kill the
>>> agent.
>>> >> >>> >>
>>> >> >>> >
>>> >> >>> > Yes, but the problem is how you kill! If you use kill -9 and
>>> kills
>>> >> >>> > Axis2
>>> >> >>> > related process (Axis2 is where Agent Service is deployed),
>>> spawned
>>> >> >>> > JVM
>>> >> >>> > instances aren't get killed.
>>> >> >>> > Anyway will research a bit more, I understand that spawned
>>> instance
>>> >> >>> > should
>>> >> >>> > never get killed, unless it is asked to!
>>> >> >>> >
>>> >> >>> >>
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > Work to be done:
>>> >> >>> >> >
>>> >> >>> >> > Terminating and finding instance aspects should be addressed.
>>> >> >>> >>
>>> >> >>> >> You have to keep track of which JVMs/VMs created by the
>>> autoscaler.
>>> >> >>> >> Otherwise you will not be able to stop them when you want to
>>> scale
>>> >> >>> >> down. I believe, this information should be stored/kept with
>>> >> >>> >> "autoscaler service".
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > I have started to implement this part today! According to the
>>> design
>>> >> >>> > 'Autoscaler Service' is only responsible for
>>> >> >>> > deciding the type of instance it should be spawned. That is a
>>> JVM
>>> >> >>> > instance
>>> >> >>> > or an EC2 instance or something else. Thus, it keeps track of
>>> >> >>> > the type of instance spawned.
>>> >> >>> >
>>> >> >>> > It is JVM Adaptor which keeps track of instances spawned in each
>>> >> >>> > Agent.
>>> >> >>>
>>> >> >>> +1. It should be JVM adapter for JVM instances and EC2 adapter
>>> for EC2
>>> >> >>> instance, etc.
>>> >> >>>
>>> >> >>> Shankar
>>> >> >>>
>>> >> >>>
>>> >> >>> >
>>> >> >>> > And it is respective Agent Service that kills an instance, upon
>>> a
>>> >> >>> > request!
>>> >> >>> >
>>> >> >>> >
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> > The 'sad side' (when things go wrong) should be addressed.
>>> >> >>> >> > AgentServices should know EPR of AgentManagementService
>>> (this is
>>> >> >>> >> > hard
>>> >> >>> >> > coded
>>> >> >>> >> > now). Should read from a config file?
>>> >> >>> >>
>>> >> >>> >> +1 fore reading from config file.
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > Great!
>>> >> >>> >
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> > JVMAdaptor should keep a map of Agent EPR to AgentService
>>> client,
>>> >> >>> >> > to
>>> >> >>> >> > avoid
>>> >> >>> >> > creating multiple clients to access the same Agent.
>>> >> >>> >> > AgentService should keep a count of spawned instances and get
>>> >> >>> >> > itself
>>> >> >>> >> > unregistered if it can't handle any new instances (threshold
>>> >> >>> >> > value
>>> >> >>> >> > should be
>>> >> >>> >> > decided). And when it feels (another threshold) that it can
>>> >> >>> >> > handle
>>> >> >>> >> > new
>>> >> >>> >> > instances it should register again.
>>> >> >>> >> > Autoscaler API should be designed.
>>> >> >>> >> > Should port EC2 client to EC2Adaptor.
>>> >> >>> >> > Should test the whole scenario.
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > I appreciate your comments/thoughts on above facts.
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > [1]
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> http://mail.wso2.org/mailarchive/architecture/2011-October/006414.html
>>> >> >>> >> >
>>> >> >>> >> > [2] http://www.cons.org/cracauer/sigint.html
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > --
>>> >> >>> >> >
>>> >> >>> >> > Thanks & regards,
>>> >> >>> >> > Nirmal
>>> >> >>> >> >
>>> >> >>> >> > Software Engineer- Platform Technologies Team, WSO2 Inc.
>>> >> >>> >> > Mobile: +94715779733
>>> >> >>> >> > Blog: http://nirmalfdo.blogspot.com/
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> --
>>> >> >>> >> S.Uthaiyashankar
>>> >> >>> >> Senior Architect & Senior Manager
>>> >> >>> >> WSO2 Inc.
>>> >> >>> >> http://wso2.com/ - "lean . enterprise . middleware"
>>> >> >>> >>
>>> >> >>> >> Phone: +94 714897591
>>> >> >>> >
>>> >> >>> >
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > --
>>> >> >>> >
>>> >> >>> > Thanks & regards,
>>> >> >>> > Nirmal
>>> >> >>> >
>>> >> >>> > Software Engineer- Platform Technologies Team, WSO2 Inc.
>>> >> >>> > Mobile: +94715779733
>>> >> >>> > Blog: http://nirmalfdo.blogspot.com/
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> S.Uthaiyashankar
>>> >> >>> Senior Architect & Senior Manager
>>> >> >>> WSO2 Inc.
>>> >> >>> http://wso2.com/ - "lean . enterprise . middleware"
>>> >> >>>
>>> >> >>> Phone: +94 714897591
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >>
>>> >> >> Thanks & regards,
>>> >> >> Nirmal
>>> >> >>
>>> >> >> Software Engineer- Platform Technologies Team, WSO2 Inc.
>>> >> >> Mobile: +94715779733
>>> >> >> Blog: http://nirmalfdo.blogspot.com/
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> > Thanks & regards,
>>> >> > Nirmal
>>> >> >
>>> >> > Software Engineer- Platform Technologies Team, WSO2 Inc.
>>> >> > Mobile: +94715779733
>>> >> > Blog: http://nirmalfdo.blogspot.com/
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> S.Uthaiyashankar
>>> >> Senior Architect & Senior Manager
>>> >> WSO2 Inc.
>>> >> http://wso2.com/ - "lean . enterprise . middleware"
>>> >>
>>> >> Phone: +94 714897591
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Thanks & regards,
>>> > Nirmal
>>> >
>>> > Software Engineer- Platform Technologies Team, WSO2 Inc.
>>> > Mobile: +94715779733
>>> > Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>>> --
>>> S.Uthaiyashankar
>>> Senior Architect & Senior Manager
>>> WSO2 Inc.
>>> http://wso2.com/ - "lean . enterprise . middleware"
>>>
>>> Phone: +94 714897591
>>>
>>
>>
>>
>> --
>> *Afkham Azeez*
>> Director of Architecture; WSO2, Inc.; http://wso2.com
>> Member; Apache Software Foundation; http://www.apache.org/
>> * <http://www.apache.org/>**
>> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
>> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
>> twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>> *
>> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
>> *
>> *
>> *Lean . Enterprise . Middleware*
>>
>>
>
>
> --
>
> Thanks & regards,
> Nirmal
>
> Software Engineer- Platform Technologies Team, WSO2 Inc.
> Mobile: +94715779733
> Blog: http://nirmalfdo.blogspot.com/
>
--
Thanks & regards,
Nirmal
Software Engineer- Platform Technologies Team, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
_______________________________________________
Carbon-dev mailing list
Carbon-dev@wso2.org
http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
