Hi,
To control the puppet catalog run on each newly spawned instances as soon
as an instance is spawned, we suspend the instance.
Once all the instances are spawned, we resume instances one by one.
It is during this stage that the product packs are being extracted and
servers are being started via puppet catalog run.
During this process if the order of server starts in the cluster nodes are
not in the expected order, some nodes are unable to join the
cluster.
e.g.: Say in a worker/manager clustering pattern - pattern 2 [1] server
starting order should be as follows.
1. elb
2. manager
3. worker
If the order has changed, this will give rise to random results in each
time we try to spawn the cluster, i.e. some nodes will be unable
to join the cluster.
Up until now we have been using time delays as a control mechanism.
We have just introduced a different mechanism to check whether the server
is running on a given port (currently 9443) and resume
instances sequentially as mentioned in the deployment.cfg file [3].
This has been tested on OpenStack environment in a simple cluster (elb,
appserver manager node, appserver worker node)
and test output as per the attached file [2].
Code:
call("while ! echo exit | nc "+currentNodeIPAddress+" 9443; do sleep 10;
done", shell="True")
print "Server "+vm+" is online on port 9443..."
*Should there be an order of starting servers on a cluster in the first
place?*
[1] https://docs.wso2.com/pages/viewpage.action?pageId=32346071
[2] Attached file - testOutput_portListener.txt
[3]
https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/deployment.cfg
Thanks,
Suhan
On Tue, Jan 6, 2015 at 1:49 PM, Suhan Dharmasuriya <[email protected]> wrote:
> Hi,
>
> We had a review discussion with DevOps on Wednesday Dec 17, 2014.
>
> Following are the further evaluations to be done.
>
> - Using puppet meta data service for information exchange
> - Puppet lookup for DNS and set the IP in configuration file.
> - Facts to read and fill puppet templates.
> - Using Puppet DB for Nagios monitoring
> - Need to evaluate Cacti for monitoring requirements.
>
> Thanks,
> Suhan
>
>
>
> On Tue, Dec 16, 2014 at 2:47 PM, Krishantha Samaraweera <
> [email protected]> wrote:
>
>> Hi Shankar,
>>
>> On Tue, Dec 16, 2014 at 12:55 PM, Selvaratnam Uthaiyashankar <
>> [email protected]> wrote:
>>>
>>> Did you guys discuss with Chamith on this?
>>>
>>
>> We are planning to have a review with DevOps this week.
>>
>> Thanks,
>> Krishantha.
>>
>>
>>>
>>> Chamith, this is similar to what you guys are doing for cloud?
>>>
>>> On Tue, Dec 16, 2014 at 9:59 AM, Suhan Dharmasuriya <[email protected]>
>>> wrote:
>>>
>>>> We have successfully migrated to the new cloud:
>>>> https://openstackstg.private.wso2.com/
>>>>
>>>> Hypervisor Summary as follows.
>>>> Memory available 30.8 GB
>>>> Disk available 1.8 TB
>>>> VCPUs available 16
>>>>
>>>> Thanks,
>>>> Suhan
>>>>
>>>> On Tue, Dec 16, 2014 at 9:38 AM, Suhan Dharmasuriya <[email protected]>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We have introduced a new configuration file deployment.cfg [1]. Python
>>>>> will load the relevant initial configurations from
>>>>> this file including server node names. Therefore nodes.txt file is
>>>>> removed.
>>>>>
>>>>> When loading node names in the cluster initially had a problem of
>>>>> spawning instances in OpenStack in a random order. This issue was solved
>>>>> using python collections.OrderedDict [3] instead of using default
>>>>> dictionary (unordered).
>>>>>
>>>>> Cluster information is kept in a config.pp file [2].
>>>>>
>>>>> [1]
>>>>> https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/deployment.cfg
>>>>> [2]
>>>>> https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/config.pp
>>>>> [3]
>>>>> http://stackoverflow.com/questions/27483096/python-config-parser-array-sort-according-to-file-content-sequence/27483584#27483584
>>>>>
>>>>> Thanks,
>>>>> Suhan
>>>>>
>>>>> On Mon, Dec 15, 2014 at 11:27 AM, Suhan Dharmasuriya <[email protected]>
>>>>> wrote:
>>>>>>
>>>>>> Adding Ramith...
>>>>>>
>>>>>> On Fri, Dec 12, 2014 at 11:43 AM, Suhan Dharmasuriya <[email protected]
>>>>>> > wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 12, 2014 at 11:40 AM, Suhan Dharmasuriya <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We are working on developing a deployment automation process which
>>>>>>>> will be integrated in to the test automation framework to run test
>>>>>>>> suites
>>>>>>>> on different product platforms.
>>>>>>>>
>>>>>>>> Following is the current process of deployment automation process.
>>>>>>>>
>>>>>>>> Nodes information file contains all the names of nodes in the
>>>>>>>> cluster of which the collective behavior is tested. It will contain
>>>>>>>> hostnames of the nodes such as appserver-mgr1 appserver-wkr1.
>>>>>>>>
>>>>>>>> Configuration data for each node in the cluster will be kept as a
>>>>>>>> data structure in another configuration file which will be restructured
>>>>>>>> according to the spawned instances information (in OpenStack) and
>>>>>>>> reflected
>>>>>>>> in puppet master. This configuration file was initially created as an
>>>>>>>> xml
>>>>>>>> file similar to the product configuration files following the same xml
>>>>>>>> hierarchical pattern. Following DevOps practices the configuration
>>>>>>>> xml file is removed and puppet data structure file is introduced.
>>>>>>>>
>>>>>>>> Using python novaclient API, equal number of instances are spawned
>>>>>>>> in the environment (OpenStack). When instances are being created a
>>>>>>>> preconfigured OpenStack image [1] will be used from an image
>>>>>>>> repository.
>>>>>>>> This image includes preconfigured puppet agent and ganglia client
>>>>>>>> services
>>>>>>>> (aware of puppet master and ganglia monitor locations).
>>>>>>>>
>>>>>>>> Once the instances are spawned in OpenStack, the relevant
>>>>>>>> information of the node is retrieved using facter [2]. This information
>>>>>>>> will be cross mapped and merged with the configuration file with the
>>>>>>>> relevant information such as IP addresses, hostnames of the instances
>>>>>>>> to
>>>>>>>> create a puppet central configuration file.
>>>>>>>>
>>>>>>>> Puppet will be taking over its operations once this configuration
>>>>>>>> file is in place. By mapping the hostname of the incoming puppet agent
>>>>>>>> request against the configuration file, puppet master performs the
>>>>>>>> following operations.
>>>>>>>>
>>>>>>>> -
>>>>>>>>
>>>>>>>> Ensure essential packages such as unzip are present in agent
>>>>>>>> node prior starting server operations.
>>>>>>>> -
>>>>>>>>
>>>>>>>> Remove old extractions and installations of conflicting product
>>>>>>>> packs.
>>>>>>>> -
>>>>>>>>
>>>>>>>> Install the defined jdk to the instance.
>>>>>>>> -
>>>>>>>>
>>>>>>>> Extract product pack file.
>>>>>>>> -
>>>>>>>>
>>>>>>>> Retrieve node specific values and update product pack
>>>>>>>> configuration (axis2.xml, carbon.xml, catalina-server.xml, etc...).
>>>>>>>> -
>>>>>>>>
>>>>>>>> Start the server with given server options.
>>>>>>>>
>>>>>>>>
>>>>>>>> In future deployment automation module will be expanded to support
>>>>>>>> OpenStack, EC2, Physical networked machines and Local environments.
>>>>>>>> Currently development started with OpenStack using python novaclient
>>>>>>>> API.
>>>>>>>> Issues encountered and findings are logged in [3], [4] and [5].
>>>>>>>>
>>>>>>>> In the next phase Docker will be evaluated to facilitate cross
>>>>>>>> platform test execution (Multiple DBs, JDKs, User Stores, OS etc..)
>>>>>>>>
>>>>>>>> Please share your ideas for improvement. We are planning to have an
>>>>>>>> architectural review with DevOps team within next week.
>>>>>>>>
>>>>>>>> Attached diagram represents the proposed architecture.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.google.com/a/wso2.com/document/d/1XVEs_5Aq5I7S1IbgFKjvzMX-_xCKOX--Yo6SXgk6JYY/edit
>>>>>>>>
>>>>>>>> [2]http://packages.ubuntu.com/trusty/facter
>>>>>>>>
>>>>>>>> [3]http://suhan-opensource.blogspot.com/
>>>>>>>>
>>>>>>>> [4]
>>>>>>>> https://docs.google.com/a/wso2.com/document/d/16kNIQVoWYTa8mBj1bZhh8qqrh_6EuB6D53eykFUPduk/edit
>>>>>>>>
>>>>>>>> [5] stackoverflow
>>>>>>>>
>>>>>>>> 1.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/26754915/novaclient-create-cloud-server-using-boot-error-badrequest-multiple-pos
>>>>>>>> 2.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/27125371/puppet-catalog-run-error-error-400-on-server-not-authorized-to-call-find-on-p
>>>>>>>> 3.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/27183036/puppet-fill-templates-with-ip-addresses-of-newly-spawned-instances
>>>>>>>> 4.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/27222189/puppet-iteration-with-2-dimensional-array-to-fill-template
>>>>>>>> 5.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/27286731/puppet-fill-template-values-using-params-pp-complex-data-structures
>>>>>>>> 6.
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/27402037/puppet-site-pp-applying-node-definition-config-for-nodes-with-same-hostname-pref
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Suhan
>>>>>>>> --
>>>>>>>> Suhan Dharmasuriya
>>>>>>>> Software Engineer - Test Automation
>>>>>>>>
>>>>>>>> *WSO2, Inc. *
>>>>>>>>
>>>>>>>> lean . enterprise . middleware
>>>>>>>> Tel: +94 112 145345
>>>>>>>> Mob: +94 779 869138
>>>>>>>> Blog: http://suhan-opensource.blogspot.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Suhan Dharmasuriya
>>>>>>> Software Engineer - Test Automation
>>>>>>>
>>>>>>> *WSO2, Inc. *
>>>>>>>
>>>>>>> lean . enterprise . middleware
>>>>>>> Tel: +94 112 145345
>>>>>>> Mob: +94 779 869138
>>>>>>> Blog: http://suhan-opensource.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Suhan Dharmasuriya
>>>>>> Software Engineer - Test Automation
>>>>>>
>>>>>> *WSO2, Inc. *
>>>>>>
>>>>>> lean . enterprise . middleware
>>>>>> Tel: +94 112 145345
>>>>>> Mob: +94 779 869138
>>>>>> Blog: http://suhan-opensource.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Suhan Dharmasuriya
>>>>> Software Engineer - Test Automation
>>>>>
>>>>> *WSO2, Inc. *
>>>>>
>>>>> lean . enterprise . middleware
>>>>> Tel: +94 112 145345
>>>>> Mob: +94 779 869138
>>>>> Blog: http://suhan-opensource.blogspot.com/
>>>>>
>>>>
>>>>
>>>> --
>>>> Suhan Dharmasuriya
>>>> Software Engineer - Test Automation
>>>>
>>>> *WSO2, Inc. *
>>>>
>>>> lean . enterprise . middleware
>>>> Tel: +94 112 145345
>>>> Mob: +94 779 869138
>>>> Blog: http://suhan-opensource.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> S.Uthaiyashankar
>>> VP Engineering
>>> WSO2 Inc.
>>> http://wso2.com/ - "lean . enterprise . middleware"
>>>
>>> Phone: +94 714897591
>>>
>>
>>
>> --
>> Krishantha Samaraweera
>> Senior Technical Lead - Test Automation
>> Mobile: +94 77 7759918
>> WSO2, Inc.; http://wso2.com/
>> lean . enterprise . middlewear.
>>
>
>
>
> --
> Suhan Dharmasuriya
> Software Engineer - Test Automation
>
> *WSO2, Inc. *
>
> lean . enterprise . middleware
> Tel: +94 112 145345
> Mob: +94 779 869138
> Blog: http://suhan-opensource.blogspot.com/
>
--
Suhan Dharmasuriya
Software Engineer - Test Automation
*WSO2, Inc. *
lean . enterprise . middleware
Tel: +94 112 145345
Mob: +94 779 869138
Blog: http://suhan-opensource.blogspot.com/
Finding Environment...
Loading nova configuration and spawning instances...
Loading deployment cluster configuration...
OrderedDict([('elb 1', None), ('appserver-mgr2 2', None), ('appserver-wkr2',
None)])
['elb', '1']
elb
['appserver-mgr2', '2']
appserver-mgr2
['appserver-wkr2']
['elb', 'appserver-mgr2', 'appserver-wkr2']
5831277d-7e08-450c-ba4c-ebc53be57b51
ACTIVE
<FloatingIP fixed_ip=None, id=d6a1bd21-665f-427c-a9b2-cf22f798f7ad,
instance_id=None, ip=192.168.19.47, pool=ext-net>
{u'qaa-net': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:f3:58:0a', u'version': 4,
u'addr': u'10.0.2.137', u'OS-EXT-IPS:type': u'fixed'}]}
10.0.2.137
{'elb-ip': u'10.0.2.137'}
bf51c68a-45d5-4e93-bbba-f0a6ef544181
ACTIVE
<FloatingIP fixed_ip=None, id=22378766-f341-4610-948e-5cfa189015f0,
instance_id=None, ip=192.168.19.48, pool=ext-net>
{u'qaa-net': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:c0:e6:8b', u'version': 4,
u'addr': u'10.0.2.138', u'OS-EXT-IPS:type': u'fixed'}]}
10.0.2.138
{'appserver-mgr2-ip': u'10.0.2.138', 'elb-ip': u'10.0.2.137'}
a9dd8e0f-8aa0-4fee-9381-c6cab2047831
ACTIVE
<FloatingIP fixed_ip=None, id=72f522ac-5893-4d89-8ef8-2c87d0b90ab9,
instance_id=None, ip=192.168.19.49, pool=ext-net>
{u'qaa-net': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:16:bd:03', u'version': 4,
u'addr': u'10.0.2.139', u'OS-EXT-IPS:type': u'fixed'}]}
10.0.2.139
{'appserver-mgr2-ip': u'10.0.2.138', 'elb-ip': u'10.0.2.137',
'appserver-wkr2-ip': u'10.0.2.139'}
---- end of instance spawning ----
10.0.2.137
Server elb is online on port 9443...
10.0.2.138
Server appserver-mgr2 is online on port 9443...
10.0.2.139
Server appserver-wkr2 is online on port 9443..._______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
