Feb 23 03:28:15 ****************
Feb 23 03:28:15 Overall Results:{'auditfail': 3, 'failure': 1, 'success': 999,
'BadNews': 525}
Feb 23 03:28:15 ****************
Feb 23 03:28:15 Detailed Results
Feb 23 03:28:15 Test Flip: {'elapsed_time': 5666.8282814025879, 'skipped': 0,
'calls': 92, 'success': 92, 'started': 27, 'down->up': 27, 'auditfail': 0,
'failure': 0, 'stopped': 65, 'max_time': 91.738150119781494, 'min_time':
33.669075012207031, 'up->down': 65}
Feb 23 03:28:15 Test Restart: {'elapsed_time': 4730.183242559433, 'skipped':
0, 'calls': 63, 'success': 63, 'min_time': 45.759351015090942, 'node:hadev3':
18, 'node:hadev2': 20, 'node:hadev1': 25, 'auditfail': 0, 'failure': 0,
'max_time': 93.791674852371216, 'WasStopped': 48}
Feb 23 03:28:15 Test Stonithd: {'elapsed_time': 15115.051451444626, 'skipped':
0, 'calls': 81, 'success': 81, 'auditfail': 3, 'failure': 0, 'max_time':
320.50085711479187, 'min_time': 113.38061618804932}
Feb 23 03:28:15 Test StartOnebyOne: {'elapsed_time': 11985.118540287018,
'skipped': 0, 'calls': 60, 'success': 60, 'auditfail': 0, 'failure': 0,
'max_time': 187.17729306221008, 'min_time': 171.58960103988647}
Feb 23 03:28:15 Test SimulStart: {'elapsed_time': 7857.5724384784698,
'skipped': 0, 'calls': 68, 'success': 68, 'auditfail': 0, 'failure': 0,
'max_time': 130.50963091850281, 'min_time': 99.602179050445557}
Feb 23 03:28:15 Test SimulStop: {'elapsed_time': 4646.7682843208313,
'skipped': 0, 'calls': 85, 'success': 84, 'auditfail': 0, 'failure': 1,
'max_time': 318.71122002601624, 'min_time': 23.685976028442383}
Feb 23 03:28:15 Test StopOnebyOne: {'elapsed_time': 6677.8760294914246,
'skipped': 0, 'calls': 84, 'success': 84, 'auditfail': 0, 'failure': 0,
'max_time': 66.564048051834106, 'min_time': 35.026233196258545}
Feb 23 03:28:15 Test RestartOnebyOne: {'elapsed_time': 19591.623623371124,
'skipped': 0, 'calls': 80, 'success': 80, 'auditfail': 0, 'failure': 0,
'max_time': 233.51037788391113, 'min_time': 180.09967494010925}
Feb 23 03:28:15 Test standby2: {'elapsed_time': 8981.5219097137451, 'skipped':
0, 'calls': 80, 'success': 80, 'auditfail': 0, 'failure': 0, 'max_time':
197.96205687522888, 'min_time': 67.44992208480835}
Feb 23 03:28:15 Test Bandwidth: {'elapsed_time': 3935.1037755012512,
'skipped': 20, 'calls': 80, 'success': 60, 'min': 3017.6562654962818, 'max':
31226.993380415472, 'totalbandwidth': 317251.65126632131, 'auditfail': 0,
'failure': 0, 'max_time': 181.26427006721497, 'min_time':
5.8889389038085938e-05}
Feb 23 03:28:15 Test ResourceRecover: {'elapsed_time': 3369.784544467926,
'skipped': 0, 'calls': 71, 'success': 71, 'auditfail': 0, 'failure': 0,
'max_time': 132.40587091445923, 'min_time': 12.435383081436157}
Feb 23 03:28:15 Test SpecialTest1: {'elapsed_time': 14276.373644113541,
'skipped': 0, 'calls': 89, 'success': 0, 'auditfail': 0, 'failure': 0,
'max_time': 173.57451200485229, 'min_time': 139.60692501068115}
Feb 23 03:28:15 Test NearQuorumPoint: {'elapsed_time': 3214.2986478805542,
'skipped': 15, 'calls': 67, 'success': 52, 'auditfail': 0, 'failure': 0,
'max_time': 276.41317415237427, 'min_time': 0.00019407272338867188}
Feb 23 03:28:15 <<<<<<<<<<<<<<<< TESTS COMPLETED
1. Most Badnews,
"crmd: [13510]: WARN: mask(../../../linux-ha/crm/crmd/control.c:register_with_ha):
Node hadev3: no uuid found"
It is known alreay.
2. Any one know about this? it will be reported bugzilla.
"
Feb 22 12:48:28 Running test NearQuorumPoint (hadev1) [588]
Feb 22 12:53:06 BadNews: Feb 22 12:51:32 hadev2 tengine: [24392]: ERROR:
mask(../../../linux-ha/crm/tengine/callbacks.c:tengine_stonith_callback):
Stonith of hadev3 failed (2)... aborting transition.
Feb 22 12:53:06 BadNews: Feb 22 12:53:02 hadev2 tengine: [24392]: ERROR:
mask(../../../linux-ha/crm/tengine/callbacks.c:tengine_stonith_callback):
Stonith of hadev3 failed (2)... aborting transition.
ssh: connect to host hadev3 port 22: No route to host
"
3. [930] and [931] may be related. It looks after the hadev2 came back from
restart the split-brain happened. It will be in bugzilla.
Feb 23 00:56:26 Running test Stonithd (hadev2) [930]
Feb 23 01:01:21 BadNews: Feb 23 01:01:17 hadev1 crmd: [12638]: WARN:
mask(../../../linux-ha/crm/crmd/callbacks.c:crmd_ha_msg_callback): Ignoring HA
message (op=vote) from hadev2: not in our membership list (size=2)
Feb 23 01:01:21 BadNews: Feb 23 01:01:17 hadev3 crmd: [30061]: WARN:
mask(../../../linux-ha/crm/crmd/callbacks.c:crmd_ha_msg_callback): Ignoring HA
message (op=vote) from hadev2: not in our membership list (size=2)
Feb 23 01:01:21 BadNews: Feb 23 01:01:18 hadev3 crmd: [30061]: ERROR:
mask(../../../linux-ha/crm/crmd/join_dc.c:do_dc_join_filter_offer): Node hadev2
is not a member
Feb 23 01:01:21 BadNews: Feb 23 01:01:18 hadev3 crmd: [30061]: ERROR:
mask(../../../linux-ha/crm/crmd/join_dc.c:do_dc_join_filter_offer): join-1:
NACK'ing node hadev2 (ref join_request-crmd-1140627678-6)
Feb 23 01:01:45 Warn: Node hadev3 not stable
Feb 23 01:01:45 Cluster is not stable: 1 (of 3): ['hadev3']
Feb 23 01:01:45 Audit CrmdStateAudit FAILED.
Feb 23 01:01:45 Warn: 2 cluster partitions detected:
Feb 23 01:01:45 hadev3 hadev1
Feb 23 01:01:45 hadev2
Feb 23 01:01:55 Resource {ssh::child_DoFencing:0} served too many times:
['hadev2', 'hadev3']
Feb 23 01:01:56 Incarnation DoFencing has 2 instances(max 3 instances).
Now 3 nodes are up
Feb 23 01:01:56 Audit HAResourceAudit FAILED.
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <diff>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <diff-removed>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <cib num_updates="113">
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <status>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <node_state join="pending"
id="879e65f8-4b38-4c56-9552-4752ad436669"/>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <node_state join="pending"
id="6125a0df-456a-4395-829a-418e9a380d36"/>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <node_state join="pending"
id="190b75b6-5585-42d9-8cde-eb6041843ae3"/>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </status>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </cib>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </diff-removed>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <diff-added>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <cib num_updates="112">
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <status>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <node_state join="member"
id="879e65f8-4b38-4c56-9552-4752ad436669"/>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <node_state join="member"
id="6125a0df-456a-4395-829a-418e9a380d36"/>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: <node_state join="down"
id="190b75b6-5585-42d9-8cde-eb6041843ae3"/>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </status>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </cib>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </diff-added>
Feb 23 01:01:58 CibDiff[hadev3-hadev1]: </diff>
Feb 23 01:01:58 Audit CibAudit FAILED.
Feb 23 01:01:58 Running test SimulStop (hadev1) [931]
Feb 23 01:07:15 Node status for hadev1 is up but we think it should be down:
Status of [EMAIL PROTECTED]: S_NOT_DC (ok)
Feb 23 01:07:15 Node status for hadev3 is up but we think it should be down:
Status of [EMAIL PROTECTED]: S_IDLE (ok)
Feb 23 01:07:15 Test SimulStopLite failed [reason:Active nodes exist:
['hadev1', 'hadev3']]
Feb 23 01:07:15 Test SimulStop failed [reason:Stopall failed]
Feb 23 01:07:15 Test SimulStop (hadev1) [FAILED]
Feb 23 01:07:17 BadNews: Feb 23 01:03:18 hadev3 crmd: [30061]: ERROR:
mask(../../../linux-ha/crm/crmd/utils.c:crm_timer_popped): Integration Timer
(I_INTEGRATED) just popped!
Feb 23 01:07:17 BadNews: Feb 23 01:03:18 hadev3 crmd: [30061]: info:
mask(../../../linux-ha/crm/crmd/fsa.c:do_state_transition): State transition
S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb 23 01:07:17 BadNews: Feb 23 01:03:18 hadev3 crmd: [30061]: info:
mask(../../../linux-ha/crm/crmd/fsa.c:ghash_print_node): Welcome reply not
received from: hadev1 3
Feb 23 01:07:17 BadNews: Feb 23 01:03:18 hadev3 crmd: [30061]: info:
mask(../../../linux-ha/crm/crmd/fsa.c:ghash_print_node): Welcome reply not
received from: hadev3 3
Feb 23 01:07:17 BadNews: Feb 23 01:03:19 hadev3 tengine: [30062]: ERROR:
mask(../../../linux-ha/crm/tengine/callbacks.c:te_graph_trigger): Triggered dev
assert at ../../../linux-ha/crm/tengine/callbacks.c:465 : pending_updates == 0
Feb 23 01:07:17 BadNews: Feb 23 01:03:19 hadev3 tengine: [30062]: ERROR:
mask(../../../linux-ha/crm/tengine/actions.c:notify_crmd): Delaying completion
until all CIB updates complete
Feb 23 01:07:17 BadNews: Feb 23 01:03:19 hadev3 tengine: [30062]: ERROR:
mask(../../../linux-ha/crm/tengine/actions.c:notify_crmd): Delaying completion
until all CIB updates complete
Feb 23 01:07:17 BadNews: Feb 23 01:05:01 hadev3 crmd: [5794]: WARN:
mask(../../../linux-ha/crm/crmd/control.c:register_with_ha): Node hadev2: no
uuid found
Feb 23 01:07:17 BadNews: Feb 23 01:05:01 hadev3 crmd: [5794]: WARN:
mask(../../../linux-ha/crm/crmd/control.c:register_with_ha): Node hadev1: no
uuid found
Feb 23 01:07:17 BadNews: Feb 23 01:06:02 hadev1 crmd: [5423]: WARN:
mask(../../../linux-ha/crm/crmd/control.c:register_with_ha): Node hadev2: no
uuid found
--
Best Regards,
Huang Zhen
Linux Technology Center
IBM China Development Lab, Beijing
Telno: (8610)82782244-2845
_______________________________________________________
Linux-HA-Dev: [email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
