I think a report like this is very useful to have real data about which flaky
tests fail more often.
It would be great if a report like this were automatically generated and
updated after each CI execution. In a project I was working before a similar
report was implemented and it was very useful for developers to check if a test
case was failing in the past, and also to identify which were the flaky test
cases that we should try to fix first.
________________________________
De: Mark Hanson <hans...@vmware.com>
Enviado: jueves, 11 de junio de 2020 7:59
Para: dev@geode.apache.org <dev@geode.apache.org>
Asunto: [INFO] Distributed Test runs in bulk results.
Hello All,
I have been doing bulk test runs of DistributedTestOpenJDK8, in this case over
200. Here is a simplified report to kind of help you see what I am seeing and I
think everybody sees with random failures as part of the PR process.
It is very easy to cause failures like this by not knowing what is running
asynchronous and Geode is a complex system or introducing timing constraints
that may not hold up in the system e.g. waiting 5 seconds for a test result
that could take longer unbeknownst you.
All of that said, here are the results. There are tickets already open for most
if not all of these issues.
Please let me know how often you all would like to see these reports…
Thanks,
Mark
***********************************************************************************
Overall build success rate: 84.00000%
The following test methods see failures in more than one class. There may be a
failing *TestBase class
*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
18 failures :
ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures (96.889%
success rate)
ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 11 failures
(95.111% success rate)
*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
4 failures :
SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 3 failures
(98.667% success rate)
SerialWANPersistenceEnabledGatewaySenderDUnitTest: 1 failures (99.556%
success rate)
***********************************************************************************
org.apache.geode.management.MemberMXBeanDistributedTest: 3 failures (98.667%
success rate)
testBucketCount
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3247
testBucketCount
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3241
testBucketCount
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3199
org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
7 failures (96.889% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3335
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3331
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3294
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3180
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3156
org.apache.geode.internal.cache.partitioned.PersistentPartitionedRegionDistributedTest:
1 failures (99.556% success rate)
testCacheCloseDuringBucketMoveDoesntCauseDataLoss
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3267
org.apache.geode.cache.management.MemoryThresholdsOffHeapDUnitTest: 1 failures
(99.556% success rate)
testDistributedRegionClientPutRejection
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3319
org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
3 failures (98.667% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3239
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3148
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3127
org.apache.geode.test.dunit.tests.AsyncInvocationTimeoutDistributedTest: 3
failures (98.667% success rate)
await_callable_timeout_includesStackTraceAsCause
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3288
await_callable_timeout_includesStackTraceAsCause
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3242
await_callable_timeout_includesStackTraceAsCause
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3177
org.apache.geode.internal.cache.tier.sockets.ReliableMessagingDUnitTest: 1
failures (99.556% success rate)
testPeriodicAckSendByClientPrimaryFailover
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3304
org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
11 failures (95.111% success rate)
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3309
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3305
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3279
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3273
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3257
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3210
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3181
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3166
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3158
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3137
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3130
org.apache.geode.management.JMXMBeanReconnectDUnitTest: 7 failures (96.889%
success rate)
serverHasMemberTypeMXBeans
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3151
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3319
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3298
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3193
serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3325
locatorsHaveMemberTypeMXBeansForServer
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3288
org.apache.geode.management.internal.cli.commands.DescribeClientCommandDUnitTest:
1 failures (99.556% success rate)
describeClient
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
1 failures (99.556% success rate)
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3225
org.apache.geode.cache30.DiskDistributedNoAckSyncOverflowRegionDUnitTest: 1
failures (99.556% success rate)
testRegionIdleDestroy
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3310
