Hi Devs,
I am experiencing $Subject in kubernetes setup with stratos/php:4.1.0-beta
docker image (probably outdated agent).
Reason for this issue is that member termination cycle is broken at python
agent.
Agent is failing to process Instance Cleanup Cluster Event, so agent won't
send instance ready to shutdown event CC, CC won't send member ready
shutdown event to AS, AS won't ask CC to terminate the instance. So
monitors and members are running even after we undeploy the application.
Here is the agent log.
[2015-03-02 07:51:29,971] DEBUG {agent.py:on_application_signup_removed} -
Application signup removed event received:
'{"applicationId":"single-cartridge-app","tenantId":-1234}'
[2015-03-02 07:51:29,971] ERROR {agent.py:on_application_signup_removed} -
Error processing tenant unSubscribed event
Traceback (most recent call last):
File "agent.py", line 347, in on_application_signup_removed
CartridgeAgent.extension_handler.on_application_signup_removal_event(event_obj)
File
"/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/extensions/extensionhandler.py",
line 362, in on_application_signup_removal_event
" [application ID] " + application_signup_removal_event.applicationId
TypeError: cannot concatenate 'str' and 'int' objects
[2015-03-02 07:51:30,002] DEBUG {eventsubscriber.py:on_message} - Message
received: topology/ClusterInstanceTerminatingEvent:
{"serviceName":"php","clusterId":"my-php.php.domain","appId":"single-cartridge-app","instanceId":"single-cartridge-app-1"}
[2015-03-02 07:51:30,002] DEBUG {eventsubscriber.py:run} - Event handler
not found for event : 'ClusterInstanceTerminatingEvent'
[2015-03-02 07:51:30,013] DEBUG {eventsubscriber.py:on_message} - Message
received: instance/notifier/InstanceCleanupClusterEvent:
{"clusterId":"my-php.php.domain","clusterInstanceId":"single-cartridge-app-1"}
[2015-03-02 07:51:30,014] DEBUG {eventsubscriber.py:run} - Executing
handler for event 'InstanceCleanupClusterEvent'
[2015-03-02 07:51:30,014] ERROR {eventsubscriber.py:run} - Error processing
'InstanceCleanupClusterEvent' event
Traceback (most recent call last):
File
"/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/subscriber/eventsubscriber.py",
line 103, in run
handler(event_msg)
File "agent.py", line 195, in on_instance_cleanup_cluster
event_obj = InstanceCleanupClusterEvent.create_from_json(msg.payload)
File
"/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/event/instance/notifier/events.py",
line 67, in create_from_json
instance_cluster_id = json_obj["instanceClusterId"] if "clusterId" in
json_obj else None
KeyError: 'instanceClusterId'
@Chamila, do you know the reason for this?
--
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2
Mobile : +94777568639
Blog : rajkumarr.com
