Chun-Hung Hsiao created MESOS-9765: -------------------------------------- Summary: Test `ROOT_CreateDestroyPersistentMountVolumeWithReboot` is flaky. Key: MESOS-9765 URL: https://issues.apache.org/jira/browse/MESOS-9765 Project: Mesos Issue Type: Bug Components: test Reporter: Chun-Hung Hsiao Assignee: Chun-Hung Hsiao
Observed a failure on test {{CSIVersion/StorageLocalResourceProviderTest.ROOT_CreateDestroyPersistentMountVolumeWithReboot/v0}} with the following error: {noformat} unknown file: Failure Unexpected mock function call - returning directly. Function call: statusUpdate(0x7ffea80beb70, @0x7f6027ff6b90 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>) Google Mock tried the following 6 expectations, but none matched: /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2552: tried expectation #0: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_STARTING)))... Expected arg #1: task status state eq TASK_STARTING Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00> Expected: to be called once Actual: called once - saturated and active /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2553: tried expectation #1: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_RUNNING)))... Expected arg #1: task status state eq TASK_RUNNING Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00> Expected: to be called once Actual: called once - saturated and active /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2554: tried expectation #2: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_FINISHED)))... Expected arg #1: task status state eq TASK_FINISHED Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00> Expected: to be called once Actual: called once - saturated and active /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2630: tried expectation #3: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_STARTING)))... Expected arg #1: task status state eq TASK_STARTING Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00> Expected: to be called once Actual: called once - saturated and active /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2631: tried expectation #4: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_RUNNING)))... Expected arg #1: task status state eq TASK_RUNNING Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00> Expected: to be called once Actual: never called - unsatisfied and active /home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2632: tried expectation #5: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_FINISHED)))... Expected arg #1: task status state eq TASK_FINISHED Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00> Expected: to be called once Actual: never called - unsatisfied and active {noformat} The problem is that the task was OOM-killed: {noformat} I0503 01:04:24.410918 24105 memory.cpp:515] OOM detected for container df47d711-3ee8-4430-b90d-b29bcb90d407 I0503 01:04:24.411118 24105 memory.cpp:555] Memory limit exceeded: Requested: 32MB Maximum Used: 32MB MEMORY STATISTICS: cache 471040 rss 33083392 rss_huge 14680064 mapped_file 0 swap 0 pgpgin 4662 pgpgout 47 pgfault 77439 pgmajfault 0 inactive_anon 471040 active_anon 33083392 inactive_file 0 active_file 0 unevictable 0 hierarchical_memory_limit 33554432 hierarchical_memsw_limit 9223372036854771712 total_cache 471040 total_rss 33083392 total_rss_huge 14680064 total_mapped_file 0 total_swap 0 total_pgpgin 4662 total_pgpgout 47 total_pgfault 77439 total_pgmajfault 0 total_inactive_anon 471040 total_active_anon 33083392 total_inactive_file 0 total_active_file 0 total_unevictable 0 I0503 01:04:24.412195 24105 containerizer.cpp:3147] Container df47d711-3ee8-4430-b90d-b29bcb90d407 has reached its limit for resource [{"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}] and will be terminated I0503 01:04:24.412314 24105 containerizer.cpp:2586] Destroying container df47d711-3ee8-4430-b90d-b29bcb90d407 in RUNNING state {noformat} This might happen to other SLRP root tests as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)