[
https://issues.apache.org/jira/browse/BROOKLYN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407922#comment-15407922
]
ASF GitHub Bot commented on BROOKLYN-319:
-----------------------------------------
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-library/pull/54
> RabbitMQ restart failure
> ------------------------
>
> Key: BROOKLYN-319
> URL: https://issues.apache.org/jira/browse/BROOKLYN-319
> Project: Brooklyn
> Issue Type: Bug
> Environment: Centos 6 on BlueBox and Google Compute
> Reporter: Duncan Godwin
>
> RabbitMQ intermittently fails to come back up after restart of the entity on
> some clouds (BlueBox and Google Compute known to fail).
> The brooklyn.debug.log shows the following for stopping and starting:
> {code}
> 2016-07-28 09:49:58,719 DEBUG brooklyn.SSH
> [brooklyn-execmanager-E26Om0Sg-10422]: stopping
> RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine
> SshMachineLocation[10.104.2.181:[email protected]/10.104.2.181:22(id=t0tvrv6o0h)]
> (env {RABBITMQ_
> HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
>
> RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
> RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=567
> 2,
> RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
>
> RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
> #!/bin/ba
> sh -e
> ; export
> INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
> ; export
> RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
> ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl
> stop
> 2016-07-28 09:49:59,071 DEBUG brooklyn.SSH [Thread-335764]:
> [[email protected]:stdout] Stopping and halting node
> 'd5kw04uk46@qa-scheduler-rabbitmq-server-be5' ...
> 2016-07-28 09:49:59,071 DEBUG brooklyn.SSH [Thread-335764]:
> [[email protected]:stdout] Executed
> /tmp/brooklyn-20160728-094958719-Sndu-stopping_RabbitBrokerImpl_id_d.sh,
> result 0
> 2016-07-28 09:49:59,072 DEBUG brooklyn.SSH
> [brooklyn-execmanager-E26Om0Sg-10422]: stopping
> RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, on machine
> SshMachineLocation[10.104.2.181:[email protected]/10.104.2.181:22(id=t0tvrv6o0h)],
> completed: return status 0
> 2016-07-28 09:49:59,078 DEBUG brooklyn.SSH
> [brooklyn-execmanager-E26Om0Sg-10385]: launching
> RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine
> SshMachineLocation[10.104.2.181:[email protected]/10.104.2.181:22(id=t0tvrv6o0h)]
> (env
> {RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
>
> RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
> RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672,
> RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
>
> RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
> #!/bin/bash -e
> ; export
> INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
> ; export
> RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
> ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; nohup ./sbin/rabbitmq-server >
> console-out.log 2> console-err.log & ; for i in {1..60}
> do
> grep 'Starting broker... completed' console-out.log && exit
> sleep 1
> done ; echo "Couldn't determine if rabbitmq-server is running" ; exit 1
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [Thread-335769]:
> [[email protected]:stdout] Starting broker... completed
> with 0 plugins.
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [Thread-335769]:
> [[email protected]:stdout] Executed
> /tmp/brooklyn-20160728-094959078-mjxF-launching_RabbitBrokerImpl_id_.sh,
> result 0
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH
> [brooklyn-execmanager-E26Om0Sg-10385]: launching
> RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, on machine
> SshMachineLocation[10.104.2.181:[email protected]/10.104.2.181:22(id=t0tvrv6o0h)],
> completed: return status 0
> {code}
> the console-out.log of the RabbitMQ instance contains:
> {code}
> ERROR: node with name "d5kw04uk46" already running on
> "qa-scheduler-rabbitmq-server-be5"
> {code}
> The brooklyn.debug.log then contains the following check running messages
> {code}
> 2016-07-28 14:05:12,642 DEBUG brooklyn.SSH
> [brooklyn-execmanager-E26Om0Sg-12886]: check-running
> RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine
> SshMachineLocation[10.104.2.181:[email protected]/10.104.2.181:22(id=t0tvrv6o0h)]
> (env
> {RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
>
> RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
> RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672,
> RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
>
> RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
> #!/bin/bash -e
> ; export
> INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
> ; export
> RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
> ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl -q status
> 2016-07-28 14:05:12,982 DEBUG o.a.b.u.c.i.w.w.Winrm4jTool
> [brooklyn-execmanager-E26Om0Sg-12879]: Finished WinRM exec on
> [email protected]:11049 done after 2.01s (connected in 0ms)
> 2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350446]:
> [[email protected]:stdout] Executed
> /tmp/brooklyn-20160728-140512642-NDCx-check-running_RabbitBrokerImpl.sh,
> result 69
> 2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] Error: unable to connect to node
> 'd5kw04uk46@qa-scheduler-rabbitmq-server-be5': nodedown
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] DIAGNOSTICS
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] ===========
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] attempted to contact:
> ['d5kw04uk46@qa-scheduler-rabbitmq-server-be5']
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] d5kw04uk46@qa-scheduler-rabbitmq-server-be5:
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] * connected to epmd (port 4369) on
> qa-scheduler-rabbitmq-server-be5
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] * epmd reports: node 'd5kw04uk46' not
> running at all
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] no other nodes on
> qa-scheduler-rabbitmq-server-be5
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] * suggestion: start the node
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] current node details:
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] - node name:
> 'rabbitmq-cli-23557@qa-scheduler-rabbitmq-server-be5'
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] - home dir: /home/users/amp
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]:
> [[email protected]:stderr] - cookie hash: hGw+myYNiff8VA9lwwfh0g==
> {code}
> The RabbitMQ service is not runing on the machine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
