Re: [INFO] Distributed Test runs in bulk results.

[Anthony Baker]

Thanks Mark!  Can you highlight any surprises in these results compared to 
previous runs?

Anthony


> On Jun 10, 2020, at 10:59 PM, Mark Hanson  wrote:
> 
> Hello All,
> 
> I have been doing bulk test runs of DistributedTestOpenJDK8, in this case 
> over 200. Here is a simplified report to kind of help you see what I am 
> seeing and I think everybody sees with random failures as part of the PR 
> process.
> 
> It is very easy to cause failures like this by not knowing what is running 
> asynchronous and Geode is a complex system or introducing timing constraints 
> that may not hold up in the system e.g. waiting 5 seconds for a test result 
> that could take longer unbeknownst you.
> 
> All of that said, here are the results. There are tickets already open for 
> most if not all of these issues.
> 
> Please let me know how often you all would like to see these reports…
> 
> Thanks,
> Mark
> 
> 
> ***
> Overall build success rate: 84.0%
> 
> 
> The following test methods see failures in more than one class.  There may be 
> a failing *TestBase class
> 
> *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>   18 failures :
>  ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures (96.889% 
> success rate)
>  ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  11 failures 
> (95.111% success rate)
> 
> *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>   4 failures :
>  SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  3 failures 
> (98.667% success rate)
>  SerialWANPersistenceEnabledGatewaySenderDUnitTest:  1 failures (99.556% 
> success rate)
> 
> ***
> 
> 
> org.apache.geode.management.MemberMXBeanDistributedTest:  3 failures (98.667% 
> success rate)
> 
> testBucketCount   
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3247data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730317409sdata=HRdBnvmthfuddRqVjC8T9EzMufHQ1959MIAcPzJINm0%3Dreserved=0
> testBucketCount   
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3241data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404sdata=DxqiSO%2BbdLgvEND5SdqiWUu8vpKGhNsOvKhWjFydxDA%3Dreserved=0
> testBucketCount   
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3199data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404sdata=ZhrecVfpX9T5NJMFsuxhU0QQbx3LwJx3QjVzK3EvXbo%3Dreserved=0
> 
> org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>   7 failures (96.889% success rate)
> 
> 
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived  
>  
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3335data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404sdata=NzB40k6LwF5cj5D20XL7CyOO7eGAQHuWe0MUzfDKn%2BM%3Dreserved=0
> 
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived  
>  
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3331data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404sdata=%2F4x18mauNSUWf5wng3AU%2B0RNjrMoa0M9X5hfohbf47I%3Dreserved=0
> 
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived  
>  
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3294data=02%7C01%7Cbakera%40vmware.com%7Cfddc96a2b1da4176784508d80dcc9bf6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637274519730327404sdata=O7rOO971KGc2nUS3I6yAFlKDnmcTEJPIiLDmuAw%2B4Q0%3Dreserved=0
> 
> testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived  
>  
> 

RE: [INFO] Distributed Test runs in bulk results.

[Alberto Bustamante Reyes]

I think a report like this is very useful to have real data about which flaky 
tests fail more often.

It would be great if a report like this were automatically generated and 
updated after each CI execution. In a project I was working before a similar 
report was implemented and it was very useful for developers to check if a test 
case was failing in the past, and also to identify which were the flaky test 
cases that we should try to fix first.

De: Mark Hanson 
Enviado: jueves, 11 de junio de 2020 7:59
Para: dev@geode.apache.org 
Asunto: [INFO] Distributed Test runs in bulk results.

Hello All,

I have been doing bulk test runs of DistributedTestOpenJDK8, in this case over 
200. Here is a simplified report to kind of help you see what I am seeing and I 
think everybody sees with random failures as part of the PR process.

It is very easy to cause failures like this by not knowing what is running 
asynchronous and Geode is a complex system or introducing timing constraints 
that may not hold up in the system e.g. waiting 5 seconds for a test result 
that could take longer unbeknownst you.

All of that said, here are the results. There are tickets already open for most 
if not all of these issues.

Please let me know how often you all would like to see these reports…

Thanks,
Mark


***
Overall build success rate: 84.0%


The following test methods see failures in more than one class.  There may be a 
failing *TestBase class

*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: 
 18 failures :
  ParallelWANPersistenceEnabledGatewaySenderDUnitTest:  7 failures (96.889% 
success rate)
  ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  11 failures 
(95.111% success rate)

*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
  4 failures :
  SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:  3 failures 
(98.667% success rate)
  SerialWANPersistenceEnabledGatewaySenderDUnitTest:  1 failures (99.556% 
success rate)

***


org.apache.geode.management.MemberMXBeanDistributedTest:  3 failures (98.667% 
success rate)

 testBucketCount   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3247
 testBucketCount   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3241
 testBucketCount   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3199

org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
  7 failures (96.889% success rate)

 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3335
 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3331
 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3294
 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3285
 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3218
 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3180
 
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3156

org.apache.geode.internal.cache.partitioned.PersistentPartitionedRegionDistributedTest:
  1 failures (99.556% success rate)

 testCacheCloseDuringBucketMoveDoesntCauseDataLoss   
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-mass-test-run-main/jobs/DistributedTestOpenJDK8/builds/3267

org.apache.geode.cache.management.MemoryThresholdsOffHeapDUnitTest:  1 failures 
(99.556% success rate)

 testDistributedRegionClientPutRejection