Hi Osura, On Mon, Jul 18, 2016 at 4:20 PM, Osura Rathnayake <[email protected]> wrote:
> Hi Mentors, > > This is the progress so far. > > *Auto scaling* > > > Azure supports two paradigms of auto scaling, vertical and horizontal. > Vertical scaling, also known as scale up and scale down, means increasing > or decreasing virtual machine (VM) sizes in response to a workload. As I > explained in one of my previous emails, vertical auto scaling is achieved > through monitoring rules. We can set it to trigger an alarm when certain > conditions are met and also as the action we can set up a runbook to scale > up or down. I could successfully get VMs to scale up and down using this > feature. > We would use horizontal scaling in most scenarios. > > Horizontal scaling, also referred to as scale out and scale in, where the > number of VMs is altered depending on the workload. Horizontal scaling is > achieved through virtual machine scale sets (VMSS). One important thing > about VMSS is that the VMs included should be of the same size and have the > same OS image. All the VMs in the scale set are added to the load balancer, > as a backend pool. Backend pool is a pool of VMs which share the traffic > coming via the load balancer. We can add auto scale rules, as to when > additional VMs should be added and removed, based on the conditions. I > could test some auto scale rules and observed VMs getting added to the > backend pool. But one problem is that when these VMs getting added, it’s a > whole new VM. I’m trying to add custom made VMs which has a wso2 product > installed and configured. > Yes we would need a VM image with required WSO2 product and pre-requisites to test this. At WSO2 we use Puppet for automating the installation process [1]. With Puppet we can use one base VM image and start a VM for any WSO2 product. Puppet does the WSO2 product installation at the VM startup. > > Note: only a limited horizontal scaling features are supported in the > azure portal yet so I’m using REST API to create and manage auto scaling > features. > > > *Centralized logging* > > > I was able to get log to Log Analytics using custom logs. I have collected > logs generated from 2 wso2 application servers. You only have to add > respective VMs to the Log analytics and set the path where logs are > located. This feature was under public preview, which might have been one > reason why it didn’t work last time when I tried. > Was the issue with the log file path/pattern? Did it work once you remove that? [1] https://github.com/wso2/puppet-modules/wiki/Use-WSO2-Puppet-Modules-in-puppet-master-agent-Environment Thanks > > Thanks, > > On Mon, Jul 18, 2016 at 8:21 AM, Osura Rathnayake <[email protected]> > wrote: > >> Hi Isuru, >> >> Please accept my apologies I have messed up names in my last email. I'm >> not going to be available today due to an unavoidable circumstance so can >> we please have the meeting on Wednesday? Extremely sorry if it made any >> inconvenience. I will update you with a detailed email today for sure. >> >> Thanks in advance, >> >> On Fri, Jul 15, 2016 at 4:04 PM, Isuru Haththotuwa <[email protected]> >> wrote: >> >>> Hi Osura, >>> >>> Can you send a detailed updated on the current status? Shall we have a >>> meeting on Monday to review the progress. >>> >>> On Tue, Jul 12, 2016 at 2:03 PM, Osura Rathnayake <[email protected]> >>> wrote: >>> >>>> Hi Imesh, >>>> >>>> About dynamically adding members to the load balancer, I will cross >>>> check it with auto-scaling. I couldn't look into that from auto-scaling end >>>> since I couldn't test it yet. >>>> >>>> about monitoring, yes we can do a POC on that. >>>> >>>> thanks, >>>> >>>> On Tue, Jul 12, 2016 at 12:53 PM, Imesh Gunaratne <[email protected]> >>>> wrote: >>>> >>>>> On Tue, Jul 12, 2016 at 12:09 PM, Osura Rathnayake <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi Imesh, >>>>>> >>>>>> About centralized logging, I'm trying to get logs to the Log >>>>>> Analytics using few methods supported in azure. We can either parse logs >>>>>> in >>>>>> to syslogs and send to the Log Analytics or create custom logs specifying >>>>>> the logs location. As you said, logs shouldn't be in .txt extension, I >>>>>> got >>>>>> it clarified from a azure blog. Will update you soon after I could >>>>>> resolve >>>>>> it. >>>>>> >>>>>> Right, thanks for the update! >>>>> >>>>> >>>>>> No you can't dynamically add VMs to the load balancer. Backend pool, >>>>>> where all the VMs reside, should be predefined. >>>>>> >>>>> >>>>> Technically that capability should be there. Otherwise we would not >>>>> be able to autoscale a server cluster dynamically. >>>>> >>>>> >>>>> >>>>>> you can auto-scale using scale sets(I'm still researching about it), >>>>>> that's the equivalent of AWS auto scaling group . also you can scale up >>>>>> or >>>>>> down a VM if it exceeds a certain parameter like CPU usage, using >>>>>> monitoring rules. >>>>>> >>>>>> *Monitoring * >>>>>> >>>>>> >>>>>> Azure has a native monitoring tool which involves collecting and >>>>>> tracking metrics, analyzing log files, defining custom metrics and >>>>>> logging >>>>>> generated by specific applications or workloads running in Virtual >>>>>> Machines. Azure represents monitored data in a graphical manner using >>>>>> charts. Monitoring also facilitates triggering alarms when certain >>>>>> conditions are met and also it can be configured to take actions on the >>>>>> met >>>>>> conditions. Monitoring is done by the Diagnostic Extension and it has >>>>>> following capabilities. >>>>>> >>>>>> · Collects and uploads the system performance information >>>>>> from the Linux VM to the user's storage table, including diagnostic and >>>>>> syslog information. >>>>>> >>>>>> · Enables users to customize the data metrics that will be >>>>>> collected and uploaded. >>>>>> >>>>>> · Enables users to upload specified log files to a >>>>>> designated storage table. >>>>>> >>>>>> Note: Azure storage tables are a non-relational, key-value-pair, >>>>>> storage system suitable for storing massive amounts of unstructured data. >>>>>> >>>>>> >>>>>> We can add monitor rules so that when an alert triggers it notifies >>>>>> the admins via email. Furthermore we can set to take automated actions. >>>>>> Azure automate actions by running runbooks. A runbook is a set of tasks >>>>>> that perform some automated process in Azure Automation. We can create >>>>>> our >>>>>> own runbooks as well. Available runbooks include, >>>>>> >>>>>> · Restart VM >>>>>> >>>>>> · Stop VM >>>>>> >>>>>> · Remove VM >>>>>> >>>>>> · Scale up VM >>>>>> >>>>>> · Scale down VM >>>>>> >>>>>> When scaling up it sets the virtual machine to the next larger size >>>>>> within the size group and when scaling down it sets the virtual machine >>>>>> to >>>>>> the next smaller size within the size group. >>>>>> >>>>>> More about runbooks and automation [1] >>>>>> >>>>> >>>>> Sounds good, will us be able to do a POC on this? >>>>> >>>>> >>>>>> >>>>>> *Auto scaling * >>>>>> >>>>>> >>>>>> Auto-scaling is the process of dynamically allocating the resources >>>>>> required by an application to match performance requirements. Virtual >>>>>> machine scale sets are an Azure Compute resource you can use to deploy >>>>>> and >>>>>> manage a set of identical VMs. With all VMs configured the same, VM scale >>>>>> sets are designed to support true auto-scale no pre-provisioning of VMs >>>>>> is >>>>>> required – and as such makes it easier to build large-scale services >>>>>> targeting big compute, big data, and containerized workloads [2]. >>>>>> >>>>>> >>>>>> Note: I couldn’t practically do this as my azure free account lets me >>>>>> have only 4 cores and I have used all of them on my current deployment. >>>>>> I’m >>>>>> getting a new azure account from one of my friends in a day so hopefully >>>>>> I >>>>>> will do this on it and update you. >>>>>> >>>>> >>>>> Great! Thanks! >>>>> >>>>> >>>>>> >>>>>> [1] >>>>>> https://azure.microsoft.com/en-us/documentation/articles/automation-intro/ >>>>>> >>>>>> [2] >>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machine-scale-sets-overview/ >>>>>> >>>>>> >>>>>> thanks, >>>>>> >>>>>> >>>>>> On Mon, Jul 11, 2016 at 9:54 AM, Imesh Gunaratne <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> On Thu, Jul 7, 2016 at 7:11 PM, Osura Rathnayake <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Mentors, >>>>>>>> >>>>>>>> >>>>>>>> In addition to refining the membership scheme code, I looked into >>>>>>>> following features of Azure. >>>>>>>> >>>>>>>> >>>>>>> Good findings Osura, please find few questions inline: >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Azure dynamic load balancing* >>>>>>>> >>>>>>>> >>>>>>>> Azure has a native load balancer which is very easy to configure. >>>>>>>> It’s a layer 4 (TCP, UDP) load balancer which helps to spread traffic >>>>>>>> among >>>>>>>> healthy virtual machines. Following are some key terms you need to >>>>>>>> know. >>>>>>>> >>>>>>>> >>>>>>> Can members be dynamically added and removed to/from a load >>>>>>> balancer? To check this we may need to explore how autoscaling works. On >>>>>>> AWS this is handled with autoscaling groups [3] >>>>>>> >>>>>>>> >>>>>>>> *Capturing Virtual Machine Images as templates* >>>>>>>> >>>>>>>> >>>>>>>> Azure provides the feature of generalizing and capturing virtual >>>>>>>> machines so that they can be used as templates. This is very useful and >>>>>>>> time saving when the production environment has many instances of the >>>>>>>> same >>>>>>>> kind of virtual machine. When the virtual machine is being generalized >>>>>>>> all >>>>>>>> the data in user directories are erased so better to have wso2 product >>>>>>>> directory not in "/home/*". More about this can be found here [2]. >>>>>>>> >>>>>>>> Once the virtual machine is captured, it is stored in the storage >>>>>>>> account that is associated with the virtual machine. You can either >>>>>>>> download this or use directly by referring to the URI when you want to >>>>>>>> make >>>>>>>> other virtual machines with this template. What would be awesome is if >>>>>>>> we >>>>>>>> can fully configure the virtual machine with a given product and make >>>>>>>> it >>>>>>>> available to users. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Yes, this is mandatory. Otherwise we would not be able to autoscale >>>>>>> a server cluster. >>>>>>> >>>>>>> I'm sorry I may have missed, how did it go with centralized >>>>>>> logging? >>>>>>> >>>>>>> [3] >>>>>>> http://docs.aws.amazon.com/autoscaling/latest/userguide/AutoScalingGroup.html >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>>> [1] >>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/load-balancer-overview/ >>>>>>>> >>>>>>>> [2] >>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-capture-image/ >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> On Wed, Jul 6, 2016 at 12:18 PM, Osura Rathnayake < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Akila, >>>>>>>>> >>>>>>>>> Please refer to the screenshots that I have attached. When I >>>>>>>>> updated localMemberPort to 4200, I can see it being reflected in logs >>>>>>>>> when >>>>>>>>> members are joining. So should I still make modifications in the code? >>>>>>>>> >>>>>>>>> .gitignore was added. >>>>>>>>> >>>>>>>>> okay I will write test cases in testNG and update >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> >>>>>>>>> On Wed, Jul 6, 2016 at 9:06 AM, Akila Ravihansa Perera < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> On Tue, Jul 5, 2016 at 11:38 AM, Osura Rathnayake < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Akila, >>>>>>>>>>> >>>>>>>>>>> Please check the modified code. It now takes the value which is >>>>>>>>>>> specified as localMemberPort in axis2.xml. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I still don't see any change to the logic of how member address >>>>>>>>>> is calculated. Can you double check? >>>>>>>>>> >>>>>>>>>> Please make sure "target/**" directories are ignored from >>>>>>>>>> .gitignore. These shouldn't be in the repo [1]. You might also need >>>>>>>>>> to >>>>>>>>>> ignore any IDE specific files. Have a look at .gitignore in >>>>>>>>>> Kubernetes >>>>>>>>>> artifacts [2]. >>>>>>>>>> >>>>>>>>>> I see that you have committed some test cases based on JUnit. >>>>>>>>>> Please note that as a platform we are moving to testng framework so >>>>>>>>>> better >>>>>>>>>> to use that. >>>>>>>>>> @Imesh, Isuru: Please correct me if I'm wrong. >>>>>>>>>> >>>>>>>>>> Shall we get a repo created under wso2-incubator for this? >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://github.com/osuran/azure-membership-scheme/tree/master/target >>>>>>>>>> [2] >>>>>>>>>> https://github.com/wso2/kubernetes-artifacts/blob/master/.gitignore >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> -- >>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>> >>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> Osura Rathnayake >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Osura Rathnayake >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Imesh Gunaratne* >>>>>>> Software Architect >>>>>>> WSO2 Inc: http://wso2.com >>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Osura Rathnayake >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Imesh Gunaratne* >>>>> Software Architect >>>>> WSO2 Inc: http://wso2.com >>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>> W: https://medium.com/@imesh TW: @imesh >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Osura Rathnayake >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> >>> Isuru H. >>> +94 716 358 048* <http://wso2.com/>* >>> >>> >>> >> >> >> -- >> Regards, >> Osura Rathnayake >> > > > > -- > Regards, > Osura Rathnayake > -- *Imesh Gunaratne* Software Architect WSO2 Inc: http://wso2.com T: +94 11 214 5345 M: +94 77 374 2057 W: https://medium.com/@imesh TW: @imesh
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
