I have built php docker image locally and found that members are terminated on app undeployment. So the issue is resolved with the fix.
However, members was not activated because port 80 was not open in containers (agent log says Port '80' is not active). Will look into it. Also wee need to build all cartridge-images with latest agent and push these to docker hub, otherwise app undeployment won't work. Thanks. On Mon, Mar 2, 2015 at 3:10 PM, Rajkumar Rajaratnam <[email protected]> wrote: > Thanks Chamila for the quick fix. I have now merged into master branch. > > Thanks. > > On Mon, Mar 2, 2015 at 2:58 PM, Chamila De Alwis <[email protected]> > wrote: > >> Hi Raj, >> >> This has been caused by an incorrect addition of the instanceClusterId >> from the received JSON object. I've submitted the fix at PR[1]. Please >> review and merge. >> >> [1] - https://github.com/apache/stratos/pull/248 >> >> >> Regards, >> Chamila de Alwis >> Software Engineer | WSO2 | +94772207163 >> Blog: code.chamiladealwis.com >> >> >> >> On Mon, Mar 2, 2015 at 2:45 PM, Chamila De Alwis <[email protected]> >> wrote: >> >>> Hi Raj, >>> >>> Yes, this is wrong. I'll correct this and submit a PR soon. >>> >>> >>> Regards, >>> Chamila de Alwis >>> Software Engineer | WSO2 | +94772207163 >>> Blog: code.chamiladealwis.com >>> >>> >>> >>> On Mon, Mar 2, 2015 at 1:47 PM, Rajkumar Rajaratnam <[email protected]> >>> wrote: >>> >>>> Hi Devs, >>>> >>>> I am experiencing $Subject in kubernetes setup with >>>> stratos/php:4.1.0-beta docker image (probably outdated agent). >>>> >>>> Reason for this issue is that member termination cycle is broken at >>>> python agent. >>>> >>>> Agent is failing to process Instance Cleanup Cluster Event, so agent >>>> won't send instance ready to shutdown event CC, CC won't send member ready >>>> shutdown event to AS, AS won't ask CC to terminate the instance. So >>>> monitors and members are running even after we undeploy the application. >>>> >>>> Here is the agent log. >>>> >>>> [2015-03-02 07:51:29,971] DEBUG >>>> {agent.py:on_application_signup_removed} - Application signup removed event >>>> received: '{"applicationId":"single-cartridge-app","tenantId":-1234}' >>>> [2015-03-02 07:51:29,971] ERROR >>>> {agent.py:on_application_signup_removed} - Error processing tenant >>>> unSubscribed event >>>> Traceback (most recent call last): >>>> File "agent.py", line 347, in on_application_signup_removed >>>> >>>> CartridgeAgent.extension_handler.on_application_signup_removal_event(event_obj) >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/extensions/extensionhandler.py", >>>> line 362, in on_application_signup_removal_event >>>> " [application ID] " + >>>> application_signup_removal_event.applicationId >>>> TypeError: cannot concatenate 'str' and 'int' objects >>>> [2015-03-02 07:51:30,002] DEBUG {eventsubscriber.py:on_message} - >>>> Message received: topology/ClusterInstanceTerminatingEvent: >>>> >>>> {"serviceName":"php","clusterId":"my-php.php.domain","appId":"single-cartridge-app","instanceId":"single-cartridge-app-1"} >>>> [2015-03-02 07:51:30,002] DEBUG {eventsubscriber.py:run} - Event >>>> handler not found for event : 'ClusterInstanceTerminatingEvent' >>>> [2015-03-02 07:51:30,013] DEBUG {eventsubscriber.py:on_message} - >>>> Message received: instance/notifier/InstanceCleanupClusterEvent: >>>> >>>> {"clusterId":"my-php.php.domain","clusterInstanceId":"single-cartridge-app-1"} >>>> [2015-03-02 07:51:30,014] DEBUG {eventsubscriber.py:run} - Executing >>>> handler for event 'InstanceCleanupClusterEvent' >>>> [2015-03-02 07:51:30,014] ERROR {eventsubscriber.py:run} - Error >>>> processing 'InstanceCleanupClusterEvent' event >>>> Traceback (most recent call last): >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/subscriber/eventsubscriber.py", >>>> line 103, in run >>>> handler(event_msg) >>>> File "agent.py", line 195, in on_instance_cleanup_cluster >>>> event_obj = >>>> InstanceCleanupClusterEvent.create_from_json(msg.payload) >>>> File >>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/event/instance/notifier/events.py", >>>> line 67, in create_from_json >>>> instance_cluster_id = json_obj["instanceClusterId"] if "clusterId" >>>> in json_obj else None >>>> KeyError: 'instanceClusterId' >>>> >>>> >>>> @Chamila, do you know the reason for this? >>>> >>>> -- >>>> Rajkumar Rajaratnam >>>> Committer & PMC Member, Apache Stratos >>>> Software Engineer, WSO2 >>>> >>>> Mobile : +94777568639 >>>> Blog : rajkumarr.com >>>> >>> >>> >> > > > -- > Rajkumar Rajaratnam > Committer & PMC Member, Apache Stratos > Software Engineer, WSO2 > > Mobile : +94777568639 > Blog : rajkumarr.com > -- Rajkumar Rajaratnam Committer & PMC Member, Apache Stratos Software Engineer, WSO2 Mobile : +94777568639 Blog : rajkumarr.com
