Hi Mentors, This is the progress so far.
*Auto scaling* Azure supports two paradigms of auto scaling, vertical and horizontal. Vertical scaling, also known as scale up and scale down, means increasing or decreasing virtual machine (VM) sizes in response to a workload. As I explained in one of my previous emails, vertical auto scaling is achieved through monitoring rules. We can set it to trigger an alarm when certain conditions are met and also as the action we can set up a runbook to scale up or down. I could successfully get VMs to scale up and down using this feature. Horizontal scaling, also referred to as scale out and scale in, where the number of VMs is altered depending on the workload. Horizontal scaling is achieved through virtual machine scale sets (VMSS). One important thing about VMSS is that the VMs included should be of the same size and have the same OS image. All the VMs in the scale set are added to the load balancer, as a backend pool. Backend pool is a pool of VMs which share the traffic coming via the load balancer. We can add auto scale rules, as to when additional VMs should be added and removed, based on the conditions. I could test some auto scale rules and observed VMs getting added to the backend pool. But one problem is that when these VMs getting added, it’s a whole new VM. I’m trying to add custom made VMs which has a wso2 product installed and configured. Note: only a limited horizontal scaling features are supported in the azure portal yet so I’m using REST API to create and manage auto scaling features. *Centralized logging* I was able to get log to Log Analytics using custom logs. I have collected logs generated from 2 wso2 application servers. You only have to add respective VMs to the Log analytics and set the path where logs are located. This feature was under public preview, which might have been one reason why it didn’t work last time when I tried. Thanks, On Mon, Jul 18, 2016 at 8:21 AM, Osura Rathnayake <[email protected]> wrote: > Hi Isuru, > > Please accept my apologies I have messed up names in my last email. I'm > not going to be available today due to an unavoidable circumstance so can > we please have the meeting on Wednesday? Extremely sorry if it made any > inconvenience. I will update you with a detailed email today for sure. > > Thanks in advance, > > On Fri, Jul 15, 2016 at 4:04 PM, Isuru Haththotuwa <[email protected]> > wrote: > >> Hi Osura, >> >> Can you send a detailed updated on the current status? Shall we have a >> meeting on Monday to review the progress. >> >> On Tue, Jul 12, 2016 at 2:03 PM, Osura Rathnayake <[email protected]> >> wrote: >> >>> Hi Imesh, >>> >>> About dynamically adding members to the load balancer, I will cross >>> check it with auto-scaling. I couldn't look into that from auto-scaling end >>> since I couldn't test it yet. >>> >>> about monitoring, yes we can do a POC on that. >>> >>> thanks, >>> >>> On Tue, Jul 12, 2016 at 12:53 PM, Imesh Gunaratne <[email protected]> >>> wrote: >>> >>>> On Tue, Jul 12, 2016 at 12:09 PM, Osura Rathnayake <[email protected]> >>>> wrote: >>>> >>>>> Hi Imesh, >>>>> >>>>> About centralized logging, I'm trying to get logs to the Log Analytics >>>>> using few methods supported in azure. We can either parse logs in to >>>>> syslogs and send to the Log Analytics or create custom logs specifying the >>>>> logs location. As you said, logs shouldn't be in .txt extension, I got it >>>>> clarified from a azure blog. Will update you soon after I could resolve >>>>> it. >>>>> >>>>> Right, thanks for the update! >>>> >>>> >>>>> No you can't dynamically add VMs to the load balancer. Backend pool, >>>>> where all the VMs reside, should be predefined. >>>>> >>>> >>>> Technically that capability should be there. Otherwise we would not be >>>> able to autoscale a server cluster dynamically. >>>> >>>> >>>> >>>>> you can auto-scale using scale sets(I'm still researching about it), >>>>> that's the equivalent of AWS auto scaling group . also you can scale up or >>>>> down a VM if it exceeds a certain parameter like CPU usage, using >>>>> monitoring rules. >>>>> >>>>> *Monitoring * >>>>> >>>>> >>>>> Azure has a native monitoring tool which involves collecting and >>>>> tracking metrics, analyzing log files, defining custom metrics and logging >>>>> generated by specific applications or workloads running in Virtual >>>>> Machines. Azure represents monitored data in a graphical manner using >>>>> charts. Monitoring also facilitates triggering alarms when certain >>>>> conditions are met and also it can be configured to take actions on the >>>>> met >>>>> conditions. Monitoring is done by the Diagnostic Extension and it has >>>>> following capabilities. >>>>> >>>>> · Collects and uploads the system performance information >>>>> from the Linux VM to the user's storage table, including diagnostic and >>>>> syslog information. >>>>> >>>>> · Enables users to customize the data metrics that will be >>>>> collected and uploaded. >>>>> >>>>> · Enables users to upload specified log files to a designated >>>>> storage table. >>>>> >>>>> Note: Azure storage tables are a non-relational, key-value-pair, >>>>> storage system suitable for storing massive amounts of unstructured data. >>>>> >>>>> >>>>> We can add monitor rules so that when an alert triggers it notifies >>>>> the admins via email. Furthermore we can set to take automated actions. >>>>> Azure automate actions by running runbooks. A runbook is a set of tasks >>>>> that perform some automated process in Azure Automation. We can create our >>>>> own runbooks as well. Available runbooks include, >>>>> >>>>> · Restart VM >>>>> >>>>> · Stop VM >>>>> >>>>> · Remove VM >>>>> >>>>> · Scale up VM >>>>> >>>>> · Scale down VM >>>>> >>>>> When scaling up it sets the virtual machine to the next larger size >>>>> within the size group and when scaling down it sets the virtual machine to >>>>> the next smaller size within the size group. >>>>> >>>>> More about runbooks and automation [1] >>>>> >>>> >>>> Sounds good, will us be able to do a POC on this? >>>> >>>> >>>>> >>>>> *Auto scaling * >>>>> >>>>> >>>>> Auto-scaling is the process of dynamically allocating the resources >>>>> required by an application to match performance requirements. Virtual >>>>> machine scale sets are an Azure Compute resource you can use to deploy and >>>>> manage a set of identical VMs. With all VMs configured the same, VM scale >>>>> sets are designed to support true auto-scale no pre-provisioning of VMs >>>>> is >>>>> required – and as such makes it easier to build large-scale services >>>>> targeting big compute, big data, and containerized workloads [2]. >>>>> >>>>> >>>>> Note: I couldn’t practically do this as my azure free account lets me >>>>> have only 4 cores and I have used all of them on my current deployment. >>>>> I’m >>>>> getting a new azure account from one of my friends in a day so hopefully I >>>>> will do this on it and update you. >>>>> >>>> >>>> Great! Thanks! >>>> >>>> >>>>> >>>>> [1] >>>>> https://azure.microsoft.com/en-us/documentation/articles/automation-intro/ >>>>> >>>>> [2] >>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machine-scale-sets-overview/ >>>>> >>>>> >>>>> thanks, >>>>> >>>>> >>>>> On Mon, Jul 11, 2016 at 9:54 AM, Imesh Gunaratne <[email protected]> >>>>> wrote: >>>>> >>>>>> On Thu, Jul 7, 2016 at 7:11 PM, Osura Rathnayake <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Mentors, >>>>>>> >>>>>>> >>>>>>> In addition to refining the membership scheme code, I looked into >>>>>>> following features of Azure. >>>>>>> >>>>>>> >>>>>> Good findings Osura, please find few questions inline: >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> *Azure dynamic load balancing* >>>>>>> >>>>>>> >>>>>>> Azure has a native load balancer which is very easy to configure. >>>>>>> It’s a layer 4 (TCP, UDP) load balancer which helps to spread traffic >>>>>>> among >>>>>>> healthy virtual machines. Following are some key terms you need to know. >>>>>>> >>>>>>> >>>>>> Can members be dynamically added and removed to/from a load >>>>>> balancer? To check this we may need to explore how autoscaling works. On >>>>>> AWS this is handled with autoscaling groups [3] >>>>>> >>>>>>> >>>>>>> *Capturing Virtual Machine Images as templates* >>>>>>> >>>>>>> >>>>>>> Azure provides the feature of generalizing and capturing virtual >>>>>>> machines so that they can be used as templates. This is very useful and >>>>>>> time saving when the production environment has many instances of the >>>>>>> same >>>>>>> kind of virtual machine. When the virtual machine is being generalized >>>>>>> all >>>>>>> the data in user directories are erased so better to have wso2 product >>>>>>> directory not in "/home/*". More about this can be found here [2]. >>>>>>> >>>>>>> Once the virtual machine is captured, it is stored in the storage >>>>>>> account that is associated with the virtual machine. You can either >>>>>>> download this or use directly by referring to the URI when you want to >>>>>>> make >>>>>>> other virtual machines with this template. What would be awesome is if >>>>>>> we >>>>>>> can fully configure the virtual machine with a given product and make it >>>>>>> available to users. >>>>>>> >>>>>>> >>>>>>> >>>>>> Yes, this is mandatory. Otherwise we would not be able to autoscale >>>>>> a server cluster. >>>>>> >>>>>> I'm sorry I may have missed, how did it go with centralized logging? >>>>>> >>>>>> [3] >>>>>> http://docs.aws.amazon.com/autoscaling/latest/userguide/AutoScalingGroup.html >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>>> [1] >>>>>>> https://azure.microsoft.com/en-us/documentation/articles/load-balancer-overview/ >>>>>>> >>>>>>> [2] >>>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-capture-image/ >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Wed, Jul 6, 2016 at 12:18 PM, Osura Rathnayake < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Akila, >>>>>>>> >>>>>>>> Please refer to the screenshots that I have attached. When I >>>>>>>> updated localMemberPort to 4200, I can see it being reflected in logs >>>>>>>> when >>>>>>>> members are joining. So should I still make modifications in the code? >>>>>>>> >>>>>>>> .gitignore was added. >>>>>>>> >>>>>>>> okay I will write test cases in testNG and update >>>>>>>> >>>>>>>> thanks, >>>>>>>> >>>>>>>> On Wed, Jul 6, 2016 at 9:06 AM, Akila Ravihansa Perera < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On Tue, Jul 5, 2016 at 11:38 AM, Osura Rathnayake < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Akila, >>>>>>>>>> >>>>>>>>>> Please check the modified code. It now takes the value which is >>>>>>>>>> specified as localMemberPort in axis2.xml. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I still don't see any change to the logic of how member address is >>>>>>>>> calculated. Can you double check? >>>>>>>>> >>>>>>>>> Please make sure "target/**" directories are ignored from >>>>>>>>> .gitignore. These shouldn't be in the repo [1]. You might also need to >>>>>>>>> ignore any IDE specific files. Have a look at .gitignore in Kubernetes >>>>>>>>> artifacts [2]. >>>>>>>>> >>>>>>>>> I see that you have committed some test cases based on JUnit. >>>>>>>>> Please note that as a platform we are moving to testng framework so >>>>>>>>> better >>>>>>>>> to use that. >>>>>>>>> @Imesh, Isuru: Please correct me if I'm wrong. >>>>>>>>> >>>>>>>>> Shall we get a repo created under wso2-incubator for this? >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://github.com/osuran/azure-membership-scheme/tree/master/target >>>>>>>>> [2] >>>>>>>>> https://github.com/wso2/kubernetes-artifacts/blob/master/.gitignore >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> -- >>>>>>>>> Akila Ravihansa Perera >>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>> >>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Osura Rathnayake >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Osura Rathnayake >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Imesh Gunaratne* >>>>>> Software Architect >>>>>> WSO2 Inc: http://wso2.com >>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Osura Rathnayake >>>>> >>>> >>>> >>>> >>>> -- >>>> *Imesh Gunaratne* >>>> Software Architect >>>> WSO2 Inc: http://wso2.com >>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>> W: https://medium.com/@imesh TW: @imesh >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Osura Rathnayake >>> >> >> >> >> -- >> Thanks and Regards, >> >> Isuru H. >> +94 716 358 048* <http://wso2.com/>* >> >> >> > > > -- > Regards, > Osura Rathnayake > -- Regards, Osura Rathnayake
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
