On Mon, May 18, 2015 at 6:34 AM, david hbase <[email protected]> wrote:
> I am going to revert the code base to priors now (May, 4). I will update > here. > While you are testing the reverted code base, can you please send us the complete log for the incident where application becomes inactive in 2-3 minutes? We also want the cartridge agent log. Thanks. > > Thanks > -D > > On Sun, May 17, 2015 at 5:59 PM, david hbase <[email protected]> wrote: > >> I just updated to the current code base and got this error, the >> application can not be "activate" now. >> >> I am also trying to find more detail now. >> >> Thanks >> -D >> >> On Sun, May 17, 2015 at 5:57 PM, Imesh Gunaratne <[email protected]> >> wrote: >> >>> Hi David, >>> >>> It looks like there is an issue in PCA in processing the Artifact >>> Updated event: >>> >>> TypeError: coercing to Unicode: need string or buffer, NoneType found >>> >>> Chamila: Do you have any thoughts on this? >>> >>> Thanks >>> >>> On Monday, May 18, 2015, david hbase <[email protected]> wrote: >>> >>>> Hi, Raj >>>> >>>> The cartridge agent log: (*ERROR:Error processing >>>> 'ArtifactUpdatedEvent' event*) >>>> >>>> 2015-05-18 00:35:34,666:DEBUG:Executing handler for event >>>> 'ArtifactUpdatedEvent' >>>> 2015-05-18 00:35:34,667:INFO:Processing Artifact update event: [tenant] >>>> -1234 [cluster] single-cartridge-app01.my-php.php.domain [status] None >>>> 2015-05-18 00:35:34,667:INFO:Executing git checkout >>>> 2015-05-18 00:35:34,667:DEBUG:Repo path returned : None >>>> 2015-05-18 00:35:34,667:DEBUG:Cloning artifacts from >>>> https://github.com/imesh/stratos-php-applications.git for the first >>>> time to None >>>> 2015-05-18 00:35:34,667:DEBUG:Executing git clone: [tenant-id] -1234 >>>> [repo-url] https://github.com/imesh/stratos-php-applications.git, >>>> [repo path] None >>>> 2015-05-18 00:35:34,667:ERROR:Error processing 'ArtifactUpdatedEvent' >>>> event >>>> Traceback (most recent call last): >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/subscriber/eventsubscriber.py", >>>> line 103, in run >>>> handler(event_msg) >>>> File "agent.py", line 193, in on_artifact_updated >>>> self.__event_handler.on_artifact_updated_event(event_obj) >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/event/eventhandler.py", >>>> line 96, in on_artifact_updated_event >>>> subscribe_run, updated = AgentGitHandler.checkout(repo_info) >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/artifactmgt/git/agentgithandler.py", >>>> line 89, in checkout >>>> git_repo = AgentGitHandler.clone(git_repo) >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/artifactmgt/git/agentgithandler.py", >>>> line 185, in clone >>>> if os.path.isdir(git_repo.local_repo_path) and >>>> os.listdir(git_repo.local_repo_path) != []: >>>> File "/usr/lib/python2.7/genericpath.py", line 41, in isdir >>>> st = os.stat(s) >>>> TypeError: coercing to Unicode: need string or buffer, NoneType found >>>> 2015-05-18 00:35:34,669:INFO:MemberStartedEvent Extension executed. >>>> [output]: Mon May 18 00:35:34 UTC 2015: Member Started Event: MEMBER_ID: , >>>> MEMBER_IP: , CLUSTER_ID: , LB_CLUSTER_ID: , NETWORK_PARTITION_ID: , >>>> SERVICE_NAME: , PORTS: , STRATOS_LB_IP: , STRATOS_LB_PUBLIC_IP: , >>>> APPLICATION_PATH: , STRATOS_PARAM_FILE_PATH: >>>> Member List: >>>> Topology: >>>> --------------- >>>> >>>> 2015-05-18 00:35:34,670:INFO:MemberStartedEvent Extension executed. >>>> [output]: Mon May 18 00:35:34 UTC 2015: Member Started Event: MEMBER_ID: , >>>> MEMBER_IP: , CLUSTER_ID: , LB_CLUSTER_ID: , NETWORK_PARTITION_ID: , >>>> SERVICE_NAME: , PORTS: , STRATOS_LB_IP: , STRATOS_LB_PUBLIC_IP: , >>>> APPLICATION_PATH: , STRATOS_PARAM_FILE_PATH: >>>> Member List: >>>> Topology: >>>> --------------- >>>> >>>> 2015-05-18 00:35:34,671:DEBUG:No plugins registered for event >>>> MemberStartedEvent >>>> 2015-05-18 00:36:27,472:DEBUG:Message received: >>>> topology/CompleteTopologyEvent: >>>> >>>> {"topology":{"clusterMap":{},"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"single-cartridge-app01.my-php.php.domain":{"serviceName":"php","clusterId":"single-cartridge-app01.my-php.php.domain","autoscalePolicyName":"autoscaling-policy-1a","deploymentPolicyName":"deployment-policy-1a","hostNames":[" >>>> single-cartridge-app01.my-php.stratos41beta2a.cisco.com >>>> "],"tenantRange":"*","isLbCluster":false,"isKubernetesCluster":false,"memberMap":{"single-cartridge-app01.my-php.php.domaindea3bc9e-e171-4915-8899-83e3cc0148cf":{"serviceName":"php","clusterId":"single-cartridge-app01.my-php.php.domain","memberId":"single-cartridge-app01.my-php.php.domaindea3bc9e-e171-4915-8899-83e3cc0148cf","clusterInstanceId":"single-cartridge-app01-1","networkPartitionId":"network-partition-1a","partitionId":"partition-1","initTime":1431909208794,"portMap":{},"memberPublicIPs":[],"memberPrivateIPs":["192.168.21.72","10.203.51.54"],"defaultPrivateIP":"192.168.21.72","properties":{"PRIMARY":"false","MIN_COUNT":"1"},"memberStateManager":{"stateStack":["Created","Initialized","Starting"],"identifier":"single-cartridge-app01.my-php.php.domaindea3bc9e-e171-4915-8899-83e3cc0148cf"},"loadBalancingIPType":"Private"}},"appId":"single-cartridge-app01","properties":{},"instanceIdToInstanceContextMap":{"single-cartridge-app01-1":{"instanceId":"single-cartridge-app01-1","instanceProperties":{},"lifeCycleStateManager":{"stateStack":["Created"],"identifier":"single-cartridge-app01.my-php.php.domain_single-cartridge-app01-1"},"networkPartitionId":"network-partition-1a"}},"kubernetesServices":[]}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}} >>>> 2015-05-18 00:36:27,472:DEBUG:Executing handler for event >>>> 'CompleteTopologyEvent' >>>> 2015-05-18 00:36:27,473:DEBUG:Complete topology event updating task >>>> disabled >>>> 2015-05-18 00:36:27,488:DEBUG:Message received: >>>> application/signup/CompleteApplicationSignUpsEvent: >>>> >>>> {"applicationSignUps":[{"tenantId":-1234,"applicationId":"single-cartridge-app01","artifactRepositories":[{"cartridgeType":"php","alias":"my-php","privateRepo":false,"repoUrl":" >>>> https://github.com/imesh/stratos-php-applications.git >>>> ","repoUsername":"","repoPassword":""}],"clusterIds":["single-cartridge-app01.my-php.php.domain"]}]} >>>> 2015-05-18 00:36:27,489:DEBUG:Event handler not found for event : >>>> 'CompleteApplicationSignUpsEvent' >>>> 2015-05-18 00:36:27,505:DEBUG:Message received: >>>> tenant/CompleteTenantEvent: >>>> {"tenants":[]} >>>> 2015-05-18 00:36:27,505:DEBUG:Executing handler for event >>>> 'CompleteTenantEvent' >>>> 2015-05-18 00:36:27,506:DEBUG:Complete tenant event received >>>> 2015-05-18 00:36:27,506:DEBUG:Processing Complete tenant event... >>>> 2015-05-18 00:36:27,506:DEBUG:Complete tenants:[] >>>> 2015-05-18 00:36:27,507:DEBUG:Reading property: param.file.path = >>>> /mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/payload/launch-params >>>> 2015-05-18 00:36:27,507:DEBUG:Executing extension for event >>>> [CompleteTenantEvent] >>>> 2015-05-18 00:36:27,508:DEBUG:Running extension for CompleteTenantEvent >>>> 2015-05-18 00:36:27,510:DEBUG:Running extension for CompleteTenantEvent >>>> 2015-05-18 00:36:27,526:INFO:CompleteTenantEvent Extension executed. >>>> [output]: Mon May 18 00:36:27 UTC 2015: Complete Tenant Event: >>>> Member List: >>>> Tenant List: >>>> >>>> 2015-05-18 00:36:27,529:INFO:CompleteTenantEvent Extension executed. >>>> [output]: Mon May 18 00:36:27 UTC 2015: Complete Tenant Event: >>>> Member List: >>>> Tenant List: >>>> >>>> 2015-05-18 00:36:27,529:DEBUG:No plugins registered for event >>>> CompleteTenantEvent >>>> 2015-05-18 00:37:27,467:DEBUG:Message received: >>>> topology/CompleteTopologyEvent: >>>> >>>> {"topology":{"clusterMap":{},"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"single-cartridge-app01.my-php.php.domain":{"serviceName":"php","clusterId":"single-cartridge-app01.my-php.php.domain","autoscalePolicyName":"autoscaling-policy-1a","deploymentPolicyName":"deployment-policy-1a","hostNames":[" >>>> single-cartridge-app01.my-php.stratos41beta2a.cisco.com >>>> "],"tenantRange":"*","isLbCluster":false,"isKubernetesCluster":false,"memberMap":{"single-cartridge-app01.my-php.php.domaindea3bc9e-e171-4915-8899-83e3cc0148cf":{"serviceName":"php","clusterId":"single-cartridge-app01.my-php.php.domain","memberId":"single-cartridge-app01.my-php.php.domaindea3bc9e-e171-4915-8899-83e3cc0148cf","clusterInstanceId":"single-cartridge-app01-1","networkPartitionId":"network-partition-1a","partitionId":"partition-1","initTime":1431909208794,"portMap":{},"memberPublicIPs":[],"memberPrivateIPs":["192.168.21.72","10.203.51.54"],"defaultPrivateIP":"192.168.21.72","properties":{"PRIMARY":"false","MIN_COUNT":"1"},"memberStateManager":{"stateStack":["Created","Initialized","Starting"],"identifier":"single-cartridge-app01.my-php.php.domaindea3bc9e-e171-4915-8899-83e3cc0148cf"},"loadBalancingIPType":"Private"}},"appId":"single-cartridge-app01","properties":{},"instanceIdToInstanceContextMap":{"single-cartridge-app01-1":{"instanceId":"single-cartridge-app01-1","instanceProperties":{},"lifeCycleStateManager":{"stateStack":["Created"],"identifier":"single-cartridge-app01.my-php.php.domain_single-cartridge-app01-1"},"networkPartitionId":"network-partition-1a"}},"kubernetesServices":[]}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}} >>>> 2015-05-18 00:37:27,467:DEBUG:Executing handler for event >>>> 'CompleteTopologyEvent' >>>> 2015-05-18 00:37:27,468:DEBUG:Complete topology event updating task >>>> disabled >>>> 2015-05-18 00:37:27,492:DEBUG:Message received: >>>> application/signup/CompleteApplicationSignUpsEvent: >>>> >>>> {"applicationSignUps":[{"tenantId":-1234,"applicationId":"single-cartridge-app01","artifactRepositories":[{"cartridgeType":"php","alias":"my-php","privateRepo":false,"repoUrl":" >>>> https://github.com/imesh/stratos-php-applications.git >>>> ","repoUsername":"","repoPassword":""}],"clusterIds":["single-cartridge-app01.my-php.php.domain"]}]} >>>> 2015-05-18 00:37:27,492:DEBUG:Event handler not found for event : >>>> 'CompleteApplicationSignUpsEvent' >>>> 2015-05-18 00:37:27,510:DEBUG:Message received: >>>> tenant/CompleteTenantEvent: >>>> {"tenants":[]} >>>> 2015-05-18 00:37:27,510:DEBUG:Executing handler for event >>>> 'CompleteTenantEvent' >>>> >>>> >>>> >>>> On Sun, May 17, 2015 at 4:57 PM, Rajkumar Rajaratnam < >>>> [email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Mon, May 18, 2015 at 4:33 AM, david hbase <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, Raj >>>>>> >>>>>> Thanks for your help, the application can be activated now. >>>>>> >>>>>> I got another issue is: After the application activated about 2~5 >>>>>> minutes, then the application cluster became "inactive", but the instance >>>>>> keep as "active" status. I have checked our JIRA, there are some tickets >>>>>> related. I will dig out more info and update here. >>>>>> >>>>> >>>>> Can you please attach both stratos log and cartridge agent log here? >>>>> Did you put the correct CEP IP in base.pp? Since you are not using >>>>> external >>>>> CEP, your CEP IP is same as Stratos IP. >>>>> >>>>> >>>>> >>>>>> >>>>>> Q: Could you send me an instruction to update the code base to >>>>>> current master with openstack env ? (My source build is May 4, need to be >>>>>> update to current ) >>>>>> >>>>> >>>>> Simply you can pull the latest code from the master (git pull origin >>>>> master) and work on it. >>>>> >>>>> >>>>> Thanks. >>>>> >>>>> >>>>>> >>>>>> Thanks >>>>>> -D >>>>>> >>>>>> On Fri, May 15, 2015 at 11:53 PM, Rajkumar Rajaratnam < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi David, >>>>>>> >>>>>>> Please attach the complete stratos logs for this scenario. >>>>>>> >>>>>>> 1. >>>>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>>>> Failed to allocate an IP address. ( not sure the cause ) >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> On Thu, May 14, 2015 at 10:36 AM, david hbase <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for your quick update. verify with my config now. >>>>>>>> -D >>>>>>>> >>>>>>>> On Wed, May 13, 2015 at 9:48 PM, Udara Liyanage <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi David, >>>>>>>>> >>>>>>>>> I am attached the init script I am using. >>>>>>>>> >>>>>>>>> My Stratos configuration in >>>>>>>>> *repository/conf/cartridge-config.properties* >>>>>>>>> >>>>>>>>> puppet.ip=192.168.30.81 >>>>>>>>> puppet.hostname=puppet.ppaas.puppetmaster.com >>>>>>>>> >>>>>>>>> After executing init script my */etc/hosts* file content is as >>>>>>>>> below. 825114051510 is a random number generated which you get a >>>>>>>>> different >>>>>>>>> value. >>>>>>>>> 192.168.30.81 puppet.ppaas.puppetmaster.com >>>>>>>>> 127.0.0.1 >>>>>>>>> 825114051510.default.appserver-manager.puppet.ppaas.puppetmaster.com >>>>>>>>> 825114051510 >>>>>>>>> >>>>>>>>> Content in* /etc/hostname* >>>>>>>>> 825114051510 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 14, 2015 at 8:01 AM, david hbase <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Additional info: >>>>>>>>>> >>>>>>>>>> 1. During the instance created, successfully assigned an IP, but >>>>>>>>>> Stratos can not get the IP. >>>>>>>>>> 2. The init script have issue with the >>>>>>>>>> PUPPET_HOSTNAME=puppet-hostname >>>>>>>>>> <https://github.com/imesh/stratos-php-applications.git,PORTS=80,PROVIDER=apache,PUPPET_IP=127.0.0.1,PUPPET_HOSTNAME=puppet-hostname> >>>>>>>>>> , the puppet master server can not be assigned >>>>>>>>>> >>>>>>>>>> The above two issues did not triggerd by 4.0, only repeatable >>>>>>>>>> with 4.1 >>>>>>>>>> >>>>>>>>>> Please help. Thanks >>>>>>>>>> -D >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 13, 2015 at 6:03 PM, david hbase < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, Udara >>>>>>>>>>> >>>>>>>>>>> I have tried multiple times, Instance creating and termination >>>>>>>>>>> are working without any error. But got the following errors as: >>>>>>>>>>> >>>>>>>>>>> 1. >>>>>>>>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>>>>>>>> Failed to allocate an IP address. ( not sure the cause ) >>>>>>>>>>> >>>>>>>>>>> 2. puppet-agent[1181]: Could not request certificate: Connection >>>>>>>>>>> refused - connect(2) (cause by payload is wrong data: >>>>>>>>>>> MCUuvbV7,DEPLOYMENT=default,REPO_URL= >>>>>>>>>>> https://github.com/imesh/stratos-php-applications.git,PORTS=80,PROVIDER=apache,PUPPET_IP=127.0.0.1,PUPPET_HOSTNAME=puppet-hostname >>>>>>>>>>> ) >>>>>>>>>>> >>>>>>>>>>> Could you test with current init script ( >>>>>>>>>>> https://github.com/apache/stratos/tree/master/tools/init-scripts) >>>>>>>>>>> and our beta2 code base? >>>>>>>>>>> >>>>>>>>>>> The php cartridge defined as following: >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> "type": "php", >>>>>>>>>>> "provider": "apache", >>>>>>>>>>> "category": "data", >>>>>>>>>>> "host": "stratos41beta2a.abc.com", >>>>>>>>>>> "displayName": "php1a", >>>>>>>>>>> "description": "php Cartridge", >>>>>>>>>>> "version": "7", >>>>>>>>>>> "multiTenant": "false", >>>>>>>>>>> "portMapping": [ >>>>>>>>>>> { >>>>>>>>>>> "protocol": "http", >>>>>>>>>>> "port": "80", >>>>>>>>>>> "proxyPort": "8280" >>>>>>>>>>> } >>>>>>>>>>> ], >>>>>>>>>>> "deployment": { >>>>>>>>>>> }, >>>>>>>>>>> "iaasProvider": [ >>>>>>>>>>> { >>>>>>>>>>> "type": "openstack", >>>>>>>>>>> "imageId": >>>>>>>>>>> "us-internal-1/xxxxxx6-aa6d-4c3c-9b5d-e51cef6ecca0", >>>>>>>>>>> "networkInterfaces": [ >>>>>>>>>>> { >>>>>>>>>>> "name": "public-direct-600", >>>>>>>>>>> "networkUuid": "2d84exxxxxx-4dc8-9897-dd8ef4719f8b" >>>>>>>>>>> } >>>>>>>>>>> ], >>>>>>>>>>> "property": [ >>>>>>>>>>> { >>>>>>>>>>> "name": "instanceType", >>>>>>>>>>> "value": >>>>>>>>>>> "us-internal-1/6xxxxxx-541b-49a5-be4c-f2320ae4a917" >>>>>>>>>>> }, >>>>>>>>>>> { >>>>>>>>>>> "name": "keyPair", >>>>>>>>>>> "value": "david" >>>>>>>>>>> }, >>>>>>>>>>> { >>>>>>>>>>> "name": "securityGroups", >>>>>>>>>>> "value": "default" >>>>>>>>>>> } >>>>>>>>>>> ] >>>>>>>>>>> } >>>>>>>>>>> ] >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> Thanks for your help >>>>>>>>>>> -D >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, May 13, 2015 at 10:33 AM, Udara Liyanage <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi David, >>>>>>>>>>>> >>>>>>>>>>>> I have tested Stratos with Openstack Icehouse and it is working >>>>>>>>>>>> without any issue for the moment. see my exact Openstack version >>>>>>>>>>>> rpm -qa | grep nova-common >>>>>>>>>>>> openstack-nova-common-2014.1.2-1.el6.noarch >>>>>>>>>>>> >>>>>>>>>>>> Instance creating, termination, ip assigning working for me >>>>>>>>>>>> without any issue. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 13, 2015 at 10:51 PM, david hbase < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, Dev Team >>>>>>>>>>>>> >>>>>>>>>>>>> If anyone tested the Stratos 4.1 with Openstack? I am focusing >>>>>>>>>>>>> on the 4.1 with icehouse now. Open this email chain to post the >>>>>>>>>>>>> issues and >>>>>>>>>>>>> solutions. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> -D >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Udara Liyanage >>>>>>>>>>>> Software Engineer >>>>>>>>>>>> WSO2, Inc.: http://wso2.com >>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>> >>>>>>>>>>>> web: http://udaraliyanage.wordpress.com >>>>>>>>>>>> phone: +94 71 443 6897 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Udara Liyanage >>>>>>>>> Software Engineer >>>>>>>>> WSO2, Inc.: http://wso2.com >>>>>>>>> lean. enterprise. middleware >>>>>>>>> >>>>>>>>> web: http://udaraliyanage.wordpress.com >>>>>>>>> phone: +94 71 443 6897 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Rajkumar Rajaratnam >>>>>>> Committer & PMC Member, Apache Stratos >>>>>>> Software Engineer, WSO2 >>>>>>> >>>>>>> Mobile : +94777568639 >>>>>>> Blog : rajkumarr.com >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Rajkumar Rajaratnam >>>>> Committer & PMC Member, Apache Stratos >>>>> Software Engineer, WSO2 >>>>> >>>>> Mobile : +94777568639 >>>>> Blog : rajkumarr.com >>>>> >>>> >>>> >>> >>> -- >>> Imesh Gunaratne >>> >>> Senior Technical Lead, WSO2 >>> Committer & PMC Member, Apache Stratos >>> >>> >> > -- Rajkumar Rajaratnam Committer & PMC Member, Apache Stratos Software Engineer, WSO2 Mobile : +94777568639 Blog : rajkumarr.com
