cluster node went down ---------------------- Key: QPID-3286 URL: https://issues.apache.org/jira/browse/QPID-3286 Project: Qpid Issue Type: Bug Components: C++ Clustering Affects Versions: 0.10 Environment: Two node persistent cluster using openais. Both nodes are CentOS 5.5. Reporter: sujith paily Assignee: Alan Conway Priority: Critical
I have configured qpid 0.10 c++ brocker as 2 node persistent cluster. I was worked without any issue for few hours or sometimes one or two day. But one node went down after some time with following error. --------------------------------------- 2011-05-30 12:55:28 warning Journal "OPC_MESSAGE_QUEUE": Enqueue capacity threshold exceeded on queue "OPC_MESSAGE_QUEUE". 2011-05-30 12:55:28 error Unexpected exception: Enqueue capacity threshold exceeded on queue "OPC_MESSAGE_QUEUE". (JournalImpl.cpp:587) 2011-05-30 12:55:28 error Connection 192.168.1.138:5672-192.168.1.10:58839 closed by error: Enqueue capacity threshold exceeded on queue "OPC_MESSAGE_QUEUE". (JournalImpl.cpp:587)(501) 2011-05-30 12:55:28 critical cluster(192.168.1.138:6321 READY/error) local error 11545 did not occur on member 192.168.1.139:25161: Enqueue capacity threshold exceeded on queue "OPC_MESSAGE_QUEUE". (JournalImpl.cpp:587) 2011-05-30 12:55:28 critical Error delivering frames: local error did not occur on all cluster members : Enqueue capacity threshold exceeded on queue "OPC_MESSAGE_QUEUE". (JournalImpl.cpp:587) (qpid/cluster/ErrorCheck.cpp:89) 2011-05-30 12:55:28 notice cluster(192.168.1.138:6321 LEFT/error) leaving cluster QCLUSTER 2011-05-30 12:55:28 notice Shut down -------------------------------------- But the remaining node was working without any issue.I have again started the cluster with debug log enabled. After some time both the nodes went down with following errors ------------------------------------------------------------------------------------------------------------------------------- 2011-05-31 05:01:03 debug Exception constructed: Error in CPG dispatch: library (2) 2011-05-31 05:01:03 debug SEND raiseEvent (v1) class=org.apache.qpid.broker.clientDisconnect 2011-05-31 05:01:03 debug SEND raiseEvent (v2) class=org.apache.qpid.broker.clientDisconnect 2011-05-31 05:01:05 debug Exception constructed: Cannot mcast to CPG group QCLUSTER: library (2) 2011-05-31 05:01:05 debug DISCONNECTED [192.168.1.138:5672-192.168.1.139:56213] 2011-05-31 05:01:05 debug DISCONNECTED [192.168.1.138:5672-192.168.1.139:56214] 2011-05-31 05:01:05 debug DISCONNECTED [127.0.0.1:5672-127.0.0.1:52930] 2011-05-31 05:01:05 debug SEND raiseEvent (v1) class=org.apache.qpid.broker.clientDisconnect 2011-05-31 05:01:05 debug SEND raiseEvent (v2) class=org.apache.qpid.broker.clientDisconnect 2011-05-31 05:01:05 debug Auto-deleting reply-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [reply-alphonse.perfomixint.com.3139.1] from queue reply-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [reply-alphonse.perfomixint.com.3139.1] from queue reply-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Auto-deleting topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [topic-alphonse.perfomixint.com.3139.1] from queue topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [schema.#] from queue topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [console.obj.*.*.org.apache.qpid.broker.agent] from queue topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [console.event.*.*.org.apache.qpid.broker.agent] from queue topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [console.heartbeat.#] from queue topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [console.obj.*.*.org.apache.qpid.broker.queue.#] from queue topic-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Auto-deleting qmfc-v2-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [qmfc-v2-alphonse.perfomixint.com.3139.1] from queue qmfc-v2-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [qmfc-v2-alphonse.perfomixint.com.3139.1] from queue qmfc-v2-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Auto-deleting qmfc-v2-ui-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [qmfc-v2-ui-alphonse.perfomixint.com.3139.1] from queue qmfc-v2-ui-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [agent.ind.data.org_apache_qpid_broker.queue.#] from queue qmfc-v2-ui-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Auto-deleting qmfc-v2-hb-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbind key [qmfc-v2-hb-alphonse.perfomixint.com.3139.1] from queue qmfc-v2-hb-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Unbound [agent.ind.heartbeat.org_apache.qpidd.#] from queue qmfc-v2-hb-alphonse.perfomixint.com.3139.1 2011-05-31 05:01:05 debug Shutting down CPG 2011-05-31 05:01:05 debug Journal "TplStore": Destroyed 2011-05-31 05:01:05 debug Journal "OPC_MESSAGE_QUEUE": Destroyed ----------------------------------------------------------------------------------------------------------------------------- This is my openais configuration ----------------------------------------------------- totem { version: 2 secauth: off threads: 0 interface { ringnumber: 0 bindnetaddr: 192.168.1.0 mcastaddr: 226.94.1.1 mcastport: 5405 } } logging { to_file: yes debug: on timestamp: on logfile: /var/log/ais.log } -------------------------------------------- openais log -------------------------------------------------- amf { mode: disabled } -------------------------------------------------------------------------------- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org