Hi, We had a review discussion with DevOps on Wednesday Dec 17, 2014.
Following are the further evaluations to be done. - Using puppet meta data service for information exchange - Puppet lookup for DNS and set the IP in configuration file. - Facts to read and fill puppet templates. - Using Puppet DB for Nagios monitoring - Need to evaluate Cacti for monitoring requirements. Thanks, Suhan On Tue, Dec 16, 2014 at 2:47 PM, Krishantha Samaraweera <[email protected] > wrote: > Hi Shankar, > > On Tue, Dec 16, 2014 at 12:55 PM, Selvaratnam Uthaiyashankar < > [email protected]> wrote: >> >> Did you guys discuss with Chamith on this? >> > > We are planning to have a review with DevOps this week. > > Thanks, > Krishantha. > > >> >> Chamith, this is similar to what you guys are doing for cloud? >> >> On Tue, Dec 16, 2014 at 9:59 AM, Suhan Dharmasuriya <[email protected]> >> wrote: >> >>> We have successfully migrated to the new cloud: >>> https://openstackstg.private.wso2.com/ >>> >>> Hypervisor Summary as follows. >>> Memory available 30.8 GB >>> Disk available 1.8 TB >>> VCPUs available 16 >>> >>> Thanks, >>> Suhan >>> >>> On Tue, Dec 16, 2014 at 9:38 AM, Suhan Dharmasuriya <[email protected]> >>> wrote: >>>> >>>> Hi, >>>> >>>> We have introduced a new configuration file deployment.cfg [1]. Python >>>> will load the relevant initial configurations from >>>> this file including server node names. Therefore nodes.txt file is >>>> removed. >>>> >>>> When loading node names in the cluster initially had a problem of >>>> spawning instances in OpenStack in a random order. This issue was solved >>>> using python collections.OrderedDict [3] instead of using default >>>> dictionary (unordered). >>>> >>>> Cluster information is kept in a config.pp file [2]. >>>> >>>> [1] >>>> https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/deployment.cfg >>>> [2] >>>> https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/config.pp >>>> [3] >>>> http://stackoverflow.com/questions/27483096/python-config-parser-array-sort-according-to-file-content-sequence/27483584#27483584 >>>> >>>> Thanks, >>>> Suhan >>>> >>>> On Mon, Dec 15, 2014 at 11:27 AM, Suhan Dharmasuriya <[email protected]> >>>> wrote: >>>>> >>>>> Adding Ramith... >>>>> >>>>> On Fri, Dec 12, 2014 at 11:43 AM, Suhan Dharmasuriya <[email protected]> >>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Dec 12, 2014 at 11:40 AM, Suhan Dharmasuriya <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We are working on developing a deployment automation process which >>>>>>> will be integrated in to the test automation framework to run test >>>>>>> suites >>>>>>> on different product platforms. >>>>>>> >>>>>>> Following is the current process of deployment automation process. >>>>>>> >>>>>>> Nodes information file contains all the names of nodes in the >>>>>>> cluster of which the collective behavior is tested. It will contain >>>>>>> hostnames of the nodes such as appserver-mgr1 appserver-wkr1. >>>>>>> >>>>>>> Configuration data for each node in the cluster will be kept as a >>>>>>> data structure in another configuration file which will be restructured >>>>>>> according to the spawned instances information (in OpenStack) and >>>>>>> reflected >>>>>>> in puppet master. This configuration file was initially created as an >>>>>>> xml >>>>>>> file similar to the product configuration files following the same xml >>>>>>> hierarchical pattern. Following DevOps practices the configuration >>>>>>> xml file is removed and puppet data structure file is introduced. >>>>>>> >>>>>>> Using python novaclient API, equal number of instances are spawned >>>>>>> in the environment (OpenStack). When instances are being created a >>>>>>> preconfigured OpenStack image [1] will be used from an image repository. >>>>>>> This image includes preconfigured puppet agent and ganglia client >>>>>>> services >>>>>>> (aware of puppet master and ganglia monitor locations). >>>>>>> >>>>>>> Once the instances are spawned in OpenStack, the relevant >>>>>>> information of the node is retrieved using facter [2]. This information >>>>>>> will be cross mapped and merged with the configuration file with the >>>>>>> relevant information such as IP addresses, hostnames of the instances to >>>>>>> create a puppet central configuration file. >>>>>>> >>>>>>> Puppet will be taking over its operations once this configuration >>>>>>> file is in place. By mapping the hostname of the incoming puppet agent >>>>>>> request against the configuration file, puppet master performs the >>>>>>> following operations. >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Ensure essential packages such as unzip are present in agent >>>>>>> node prior starting server operations. >>>>>>> - >>>>>>> >>>>>>> Remove old extractions and installations of conflicting product >>>>>>> packs. >>>>>>> - >>>>>>> >>>>>>> Install the defined jdk to the instance. >>>>>>> - >>>>>>> >>>>>>> Extract product pack file. >>>>>>> - >>>>>>> >>>>>>> Retrieve node specific values and update product pack >>>>>>> configuration (axis2.xml, carbon.xml, catalina-server.xml, etc...). >>>>>>> - >>>>>>> >>>>>>> Start the server with given server options. >>>>>>> >>>>>>> >>>>>>> In future deployment automation module will be expanded to support >>>>>>> OpenStack, EC2, Physical networked machines and Local environments. >>>>>>> Currently development started with OpenStack using python novaclient >>>>>>> API. >>>>>>> Issues encountered and findings are logged in [3], [4] and [5]. >>>>>>> >>>>>>> In the next phase Docker will be evaluated to facilitate cross >>>>>>> platform test execution (Multiple DBs, JDKs, User Stores, OS etc..) >>>>>>> >>>>>>> Please share your ideas for improvement. We are planning to have an >>>>>>> architectural review with DevOps team within next week. >>>>>>> >>>>>>> Attached diagram represents the proposed architecture. >>>>>>> >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://docs.google.com/a/wso2.com/document/d/1XVEs_5Aq5I7S1IbgFKjvzMX-_xCKOX--Yo6SXgk6JYY/edit >>>>>>> >>>>>>> [2]http://packages.ubuntu.com/trusty/facter >>>>>>> >>>>>>> [3]http://suhan-opensource.blogspot.com/ >>>>>>> >>>>>>> [4] >>>>>>> https://docs.google.com/a/wso2.com/document/d/16kNIQVoWYTa8mBj1bZhh8qqrh_6EuB6D53eykFUPduk/edit >>>>>>> >>>>>>> [5] stackoverflow >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://stackoverflow.com/questions/26754915/novaclient-create-cloud-server-using-boot-error-badrequest-multiple-pos >>>>>>> 2. >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://stackoverflow.com/questions/27125371/puppet-catalog-run-error-error-400-on-server-not-authorized-to-call-find-on-p >>>>>>> 3. >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://stackoverflow.com/questions/27183036/puppet-fill-templates-with-ip-addresses-of-newly-spawned-instances >>>>>>> 4. >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://stackoverflow.com/questions/27222189/puppet-iteration-with-2-dimensional-array-to-fill-template >>>>>>> 5. >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://stackoverflow.com/questions/27286731/puppet-fill-template-values-using-params-pp-complex-data-structures >>>>>>> 6. >>>>>>> >>>>>>> http://stackoverflow.com/questions/27402037/puppet-site-pp-applying-node-definition-config-for-nodes-with-same-hostname-pref >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Suhan >>>>>>> -- >>>>>>> Suhan Dharmasuriya >>>>>>> Software Engineer - Test Automation >>>>>>> >>>>>>> *WSO2, Inc. * >>>>>>> >>>>>>> lean . enterprise . middleware >>>>>>> Tel: +94 112 145345 >>>>>>> Mob: +94 779 869138 >>>>>>> Blog: http://suhan-opensource.blogspot.com/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Suhan Dharmasuriya >>>>>> Software Engineer - Test Automation >>>>>> >>>>>> *WSO2, Inc. * >>>>>> >>>>>> lean . enterprise . middleware >>>>>> Tel: +94 112 145345 >>>>>> Mob: +94 779 869138 >>>>>> Blog: http://suhan-opensource.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> -- >>>>> Suhan Dharmasuriya >>>>> Software Engineer - Test Automation >>>>> >>>>> *WSO2, Inc. * >>>>> >>>>> lean . enterprise . middleware >>>>> Tel: +94 112 145345 >>>>> Mob: +94 779 869138 >>>>> Blog: http://suhan-opensource.blogspot.com/ >>>>> >>>> >>>> >>>> -- >>>> Suhan Dharmasuriya >>>> Software Engineer - Test Automation >>>> >>>> *WSO2, Inc. * >>>> >>>> lean . enterprise . middleware >>>> Tel: +94 112 145345 >>>> Mob: +94 779 869138 >>>> Blog: http://suhan-opensource.blogspot.com/ >>>> >>> >>> >>> -- >>> Suhan Dharmasuriya >>> Software Engineer - Test Automation >>> >>> *WSO2, Inc. * >>> >>> lean . enterprise . middleware >>> Tel: +94 112 145345 >>> Mob: +94 779 869138 >>> Blog: http://suhan-opensource.blogspot.com/ >>> >> >> >> >> -- >> S.Uthaiyashankar >> VP Engineering >> WSO2 Inc. >> http://wso2.com/ - "lean . enterprise . middleware" >> >> Phone: +94 714897591 >> > > > -- > Krishantha Samaraweera > Senior Technical Lead - Test Automation > Mobile: +94 77 7759918 > WSO2, Inc.; http://wso2.com/ > lean . enterprise . middlewear. > -- Suhan Dharmasuriya Software Engineer - Test Automation *WSO2, Inc. * lean . enterprise . middleware Tel: +94 112 145345 Mob: +94 779 869138 Blog: http://suhan-opensource.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
