Duncan Godwin created BROOKLYN-319:
--------------------------------------
Summary: RabbitMQ restart failure
Key: BROOKLYN-319
URL: https://issues.apache.org/jira/browse/BROOKLYN-319
Project: Brooklyn
Issue Type: Bug
Environment: Centos 6 on BlueBox and Google Compute
Reporter: Duncan Godwin
RabbitMQ intermittently fails to come back up after restart of the entity on
some clouds (BlueBox and Google Compute known to fail).
The brooklyn.debug.log shows the following:
{code}
2016-07-28 14:05:12,642 DEBUG brooklyn.SSH
[brooklyn-execmanager-E26Om0Sg-12886]: check-running
RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine
SshMachineLocation[10.104.2.181:[email protected]/10.104.2.181:22(id=t0tvrv6o0h)]
(env
{RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672,
RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
#!/bin/bash -e
; export
INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
; export
RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl -q status
2016-07-28 14:05:12,982 DEBUG o.a.b.u.c.i.w.w.Winrm4jTool
[brooklyn-execmanager-E26Om0Sg-12879]: Finished WinRM exec on
[email protected]:11049 done after 2.01s (connected in 0ms)
2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350446]:
[[email protected]:stdout] Executed
/tmp/brooklyn-20160728-140512642-NDCx-check-running_RabbitBrokerImpl.sh, result
69
2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] Error: unable to connect to node
'd5kw04uk46@qa-scheduler-rabbitmq-server-be5': nodedown
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] DIAGNOSTICS
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] ===========
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] attempted to contact:
['d5kw04uk46@qa-scheduler-rabbitmq-server-be5']
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] d5kw04uk46@qa-scheduler-rabbitmq-server-be5:
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] * connected to epmd (port 4369) on
qa-scheduler-rabbitmq-server-be5
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] * epmd reports: node 'd5kw04uk46' not
running at all
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] no other nodes on
qa-scheduler-rabbitmq-server-be5
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] * suggestion: start the node
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] current node details:
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] - node name:
'rabbitmq-cli-23557@qa-scheduler-rabbitmq-server-be5'
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] - home dir: /home/users/amp
2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
[[email protected]:stderr] - cookie hash: hGw+myYNiff8VA9lwwfh0g==
{code}
the console-out.log contains:
{code}
ERROR: node with name "d5kw04uk46" already running on
"qa-scheduler-rabbitmq-server-be5"
{code}
The RabbitMQ service is not runing on the machine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
