Sumit Mohanty created SLIDER-629:
------------------------------------
Summary: Slider's count of failure threshold may not be accurate
or it could be a logging issue
Key: SLIDER-629
URL: https://issues.apache.org/jira/browse/SLIDER-629
Project: Slider
Issue Type: Bug
Components: appmaster
Affects Versions: Slider 0.50
Reporter: Sumit Mohanty
Assignee: Jonathan Maron
Fix For: Slider 0.60
One of the long running HBase tests failed with the following error:
{noformat}
2014-11-08 01:07:26,407 [AmExecutor-008] ERROR appmaster.SliderAppMaster -
Cluster teardown triggered
org.apache.slider.core.exceptions.TriggerClusterTeardownException: Unstable
Application Instance : - failed with component H BASE_REGIONSERVER
failing 8 times (0 in startup); threshold is 5 - last failure: Failure
container_1415341585168_0005_01_000008 on host onprem-slider23:
http://onprem-slider21:19888/jobhistory/logs/onprem-slider23:45454/contai
ner_1415341585168_0005_01_000008/ctx/hadoop^M
{noformat}
However, there were total of "9" REGION_SERVERs created.
{noformat}
2014-11-07 16:00:35,346 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000002, on onprem-slider25:45454,
2014-11-07 16:00:35,347 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000005, on onprem-slider24:45454,
2014-11-07 16:00:35,347 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000007, on onprem-slider22:45454,
2014-11-07 16:00:35,347 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000008, on onprem-slider23:45454,
2014-11-07 23:51:20,040 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000009, on onprem-slider22:45454,
2014-11-07 23:58:44,810 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000013, on onprem-slider24:45454,
2014-11-08 00:12:17,804 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000015, on onprem-slider22:45454,
2014-11-08 00:15:57,373 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000018, on onprem-slider25:45454,
2014-11-08 01:06:36,771 [AMRM Callback Handler Thread] INFO state.AppState -
Assigning role HBASE_REGIONSERVER to container
container_1415341585168_0005_01_000020, on onprem-slider25:45454,
{noformat}
As the ask was for 4 but 9 were created, obviously there are 5 failures.
Perhaps its a logging issue. Can we also print the Window - e.g. 5 failures in
X minutes or hours.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
