Myroslav Papirkovskyi created AMBARI-15714:
----------------------------------------------
Summary: Express Upgrade hung after FAILED step is retried
Key: AMBARI-15714
URL: https://issues.apache.org/jira/browse/AMBARI-15714
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.2.2
Reporter: Myroslav Papirkovskyi
Assignee: Myroslav Papirkovskyi
Priority: Critical
Fix For: 2.2.2
*Steps:*
# Start Express Upgrade from HDP 2.4.0.0 to 2.4.2.0-130
# Reach till backup the Hive Metastore message and hit Proceed
# Stop Ambari agent on one of the HBase RegionServer host
(os-r7-kwjvku-ambari-eu-4-4.novalocal on the current cluster)
# Wait for EU to report failure (status as HOLDING_TIMEDOUT)
# Start ambari-agent on the RS host and wait 60 secs. for the heartbeat to be
operational
# Retry the failed step in EU wizard
*Result*
EU hangs
ambari-server logs report below:
{code}
05 Apr 2016 08:09:26,526 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:157 -
Heartbeat lost from host os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,563 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component SECONDARY_NAMENODE on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,566 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component HISTORYSERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,568 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component HIVE_METASTORE on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,571 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component WEBHCAT_SERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,574 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component HIVE_SERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,577 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component OOZIE_SERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,580 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component ZOOKEEPER_SERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,583 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component DRPC_SERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,585 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component KAFKA_BROKER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,589 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component SPARK_JOBHISTORYSERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,592 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component METRICS_GRAFANA on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,595 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component DATANODE on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,597 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component NFS_GATEWAY on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,600 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component NODEMANAGER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,603 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component HBASE_REGIONSERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,606 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component SUPERVISOR on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,609 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component FLUME_HANDLER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,611 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component SPARK_THRIFTSERVER on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,615 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component METRICS_MONITOR on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:26,618 WARN [ambari-hearbeat-monitor] HeartbeatMonitor:172 -
Setting component state to UNKNOWN for component HST_AGENT on
os-r7-kwjvku-ambari-eu-4-4.novalocal
05 Apr 2016 08:09:27,571 INFO [ambari-action-scheduler] ActionScheduler:702 -
Host:os-r7-kwjvku-ambari-eu-4-4.novalocal, role:HBASE_REGIONSERVER,
actionId:13-29 timed out
05 Apr 2016 08:09:28,750 INFO [ambari-action-scheduler] ActionScheduler:702 -
Host:os-r7-kwjvku-ambari-eu-4-4.novalocal, role:HBASE_REGIONSERVER,
actionId:13-29 timed out
05 Apr 2016 08:09:28,751 WARN [ambari-action-scheduler] ActionScheduler:704 -
Host:os-r7-kwjvku-ambari-eu-4-4.novalocal, role:HBASE_REGIONSERVER,
actionId:13-29 expired
05 Apr 2016 08:09:28,789 ERROR [ambari-action-scheduler]
ServiceComponentHostImpl:1030 - Can't handle ServiceComponentHostEvent event at
current state, serviceComponentName=HBASE_REGIONSERVER,
hostName=os-r7-kwjvku-ambari-eu-4-4.novalocal, currentState=UNKNOWN,
eventType=HOST_SVCCOMP_OP_FAILED, event=EventType: HOST_SVCCOMP_OP_FAILED
05 Apr 2016 08:09:28,789 WARN [ambari-action-scheduler] ActionScheduler:806 -
Unable to transition to failed state.
org.apache.ambari.server.state.fsm.InvalidStateTransitionException: Invalid
event: HOST_SVCCOMP_OP_FAILED at UNKNOWN
at
org.apache.ambari.server.state.fsm.StateMachineFactory.doTransition(StateMachineFactory.java:297)
at
org.apache.ambari.server.state.fsm.StateMachineFactory.access$300(StateMachineFactory.java:39)
at
org.apache.ambari.server.state.fsm.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:440)
at
org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.handleEvent(ServiceComponentHostImpl.java:1025)
at
org.apache.ambari.server.actionmanager.ActionScheduler.transitionToFailedState(ActionScheduler.java:789)
at
org.apache.ambari.server.actionmanager.ActionScheduler.processInProgressStage(ActionScheduler.java:710)
at
org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:289)
at
org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:196)
at java.lang.Thread.run(Thread.java:745)
05 Apr 2016 08:09:28,790 INFO [ambari-action-scheduler] ActionScheduler:717 -
Removing command from queue, host=os-r7-kwjvku-ambari-eu-4-4.novalocal,
commandId=13-29
05 Apr 2016 08:09:59,571 WARN [qtp-ambari-agent-418] SecurityFilter:103 -
Request https://os-r7-kwjvku-ambari-eu-4-5.novalocal:8440/ca doesn't match any
pattern.
05 Apr 2016 08:09:59,571 WARN [qtp-ambari-agent-418] SecurityFilter:62 - This
request is not allowed on this port:
https://os-r7-kwjvku-ambari-eu-4-5.novalocal:8440/ca
5 Apr 2016 08:10:00,761 INFO [qtp-ambari-agent-418] HeartBeatHandler:400 -
agentOsType = centos7
05 Apr 2016 08:10:00,983 INFO [qtp-ambari-agent-418] HostImpl:285 - Received
host registration,
host=[hostname=os-r7-kwjvku-ambari-eu-4-4,fqdn=os-r7-kwjvku-ambari-eu-4-4.novalocal,domain=novalocal,architecture=x86_64,processorcount=2,physicalprocessorcount=2,osname=centos,osversion=7.0.1406,osfamily=redhat,memory=16269820,uptime_hours=2,mounts=(available=24033216,mountpoint=/,used=2165564,percent=9%,size=26198780,device=/dev/vda1,type=xfs)(available=8119336,mountpoint=/dev,used=0,percent=0%,size=8119336,device=devtmpfs,type=devtmpfs)(available=8134900,mountpoint=/dev/shm,used=8,percent=1%,size=8134908,device=tmpfs,type=tmpfs)(available=8093468,mountpoint=/run,used=41440,percent=1%,size=8134908,device=tmpfs,type=tmpfs)(available=234937192,mountpoint=/grid/0,used=9839132,percent=5%,size=257899908,device=/dev/vdb,type=ext4)]
, registrationTime=1459843800761, agentVersion=2.2.2.0
05 Apr 2016 08:10:00,983 INFO [qtp-ambari-agent-418] TopologyManager:316 -
TopologyManager.onHostRegistered: Entering
05 Apr 2016 08:10:00,984 INFO [qtp-ambari-agent-418] TopologyManager:318 -
TopologyManager.onHostRegistered: host = os-r7-kwjvku-ambari-eu-4-4.novalocal
is already associated with the cluster or is currently being processed
05 Apr 2016 08:10:01,127 INFO [qtp-ambari-agent-418] HeartBeatHandler:467 -
Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6,
windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, disabledComponents=,
enabledComponents=METRICS_COLLECTOR}
05 Apr 2016 08:10:02,289 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:03,489 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:04,685 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:05,911 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:07,124 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:08,336 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:09,570 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:10,857 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:12,066 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:13,295 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:14,284 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component HISTORYSERVER of service
MAPREDUCE2 of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,290 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component WEBHCAT_SERVER of service
HIVE of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,294 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component KAFKA_BROKER of service
KAFKA of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,298 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component SPARK_JOBHISTORYSERVER of
service SPARK of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,303 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component METRICS_GRAFANA of service
AMBARI_METRICS of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,307 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component DATANODE of service HDFS of
cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,313 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component NFS_GATEWAY of service HDFS
of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,316 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component HBASE_REGIONSERVER of
service HBASE of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,321 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component SUPERVISOR of service STORM
of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,325 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component OOZIE_SERVER of service
OOZIE of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,329 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component METRICS_MONITOR of service
AMBARI_METRICS of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,333 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component NODEMANAGER of service YARN
of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,337 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component HIVE_METASTORE of service
HIVE of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,343 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component HIVE_SERVER of service HIVE
of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,348 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component ZOOKEEPER_SERVER of service
ZOOKEEPER of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,351 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component SPARK_THRIFTSERVER of
service SPARK of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,354 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component DRPC_SERVER of service
STORM of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,359 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component SECONDARY_NAMENODE of
service HDFS of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,363 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component FLUME_HANDLER of service
FLUME of cluster cl1 has changed from UNKNOWN to INSTALLED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,366 INFO [ambari-heartbeat-processor-0]
HeartbeatProcessor:605 - State of service component HST_AGENT of service
SMARTSENSE of cluster cl1 has changed from UNKNOWN to STARTED at host
os-r7-kwjvku-ambari-eu-4-4.novalocal according to STATUS_COMMAND report
05 Apr 2016 08:10:14,499 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:15,706 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:16,894 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
05 Apr 2016 08:10:18,151 WARN [ambari-action-scheduler] ActionScheduler:695 -
Detected ambari-agent restart during command execution.The command has been
aborted.Execution command details: host: os-r7-kwjvku-ambari-eu-4-4.novalocal,
role: HBASE_REGIONSERVER, actionId: 13-29
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
