> On Dec. 19, 2015, 10:18 p.m., Jonathan Hurley wrote: > > I think we need to drop the notion of fast vs non-fast. I know that Nate > > has been working on re-using the same persistence unit between tests (each > > test gets its own cluster, own unique hosts, etc) ... and this has > > dramatically increased performance. > > > > With that said, I think we can still categorize and label our tests. But I > > think we should take a difference approach. > > > > It's cool to group them logically like Alerts, Kerberos, ORM, Upgrade, etc. > > > > But we should also create groupings based on what the tests are doing. > > Something like: > > SmallTest - accesses only mocks, most used for direct unit tests of single > > functional aspects > > MediumTest - works with mocks and DB, sets up cluster and some data for a > > single unit of test > > LargeTest - an end-to-end style test (like testing an upgrade or downgrade) > > > > Even more, we can group them for quick runs from the command by mixing and > > matching above logical groupings. Consider: > > MAT (Minimum Acceptance Tests): Basic functionality that must pass (like > > cluster creation, adding services, pure ORM) > > SRG (Short Regression): A mixture of tests from above logical units > > (Alerts, Kerberos, Upgrade) which don't take too long > > LRG (Long Regression): A mixture of tests from above logical units (Alerts, > > Kerberos, Upgrade) which are known to take a long time > > Nate Cole wrote: > +1 to this. > > Dmitro Lisnichenko wrote: > Are not SmallTest/MediumTest/LargeTest something like > unit/integration/functional tests? I'd say that a big part of our tests > consists of over-detailed functional tests, that tend to test too much and > that's why break on any implementation change like mock method invocation > count. So I'd strongly +1 to clear separation of roles you are proposing. > Behaviour > > Also, it would be great to have some commonly-used mock implementations > like cluster with few hosts and services as a separate class, so we could > skip defining something like createCluster() at every test suite. > > Dmitro Lisnichenko wrote: > * I mean that functional/integration tests should test only high-level > behaviour, and mock out any logic that is covered by unit tests. So tests are > more loosely coupled > > Jayush Luniya wrote: > +1 on the suggestions above.
I like these suggestions. Since performance can always change, I'm ok with renaming them as Unit Test - Very simple mocks, in the order of 100 ms or faster Integration Test - multiple classes being tested together, in the order of 1000 ms or faster Functional Test - heavy lifting that tests features (anything that has to setup a DB, add cluster, services, hosts, perform RUs, etc). Some of these can take less than 1000 ms, but for sure if it takes longer than 1000 ms it is likely in this category. In additional, I will tag them with the feature, e.g., Stack Upgrade, Security (Kerberos), Blueprints, etc. This way, we can use the test runner to be very specific, e.g., Stack Upgrade && Unit Test. - Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41569/#review111389 ----------------------------------------------------------- On Dec. 18, 2015, 10:26 p.m., Alejandro Fernandez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41569/ > ----------------------------------------------------------- > > (Updated Dec. 18, 2015, 10:26 p.m.) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Grinenko, Di Li, Dmitro > Lisnichenko, Jaimin Jetly, Jonathan Hurley, Jayush Luniya, Nate Cole, Robert > Levas, Robert Nettleton, Nahappan Somasundaram, Sumit Mohanty, Srimanth > Gunturi, Swapan Shridhar, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-14439 > https://issues.apache.org/jira/browse/AMBARI-14439 > > > Repository: ambari > > > Description > ------- > > Categorize the unit tests with wither SlowTest, FastTest, and some other > group based on the area. > This will allow us to run a test suite that matches a given group. > > mvn clean test -P ${profile_id} > mvn clean -Drat.ignoreErrors=true test -P FastTest > fast-tests.txt > mvn clean -Drat.ignoreErrors=true test -P NonFastTest > nonfast-tests.txt > > > Diffs > ----- > > ambari-server/pom.xml 56d9e44 > ambari-server/src/test/java/org/apache/ambari/server/AlertTest.java > PRE-CREATION > ambari-server/src/test/java/org/apache/ambari/server/AmbariUpgradeTest.java > PRE-CREATION > ambari-server/src/test/java/org/apache/ambari/server/BlueprintTest.java > PRE-CREATION > ambari-server/src/test/java/org/apache/ambari/server/FastTest.java > PRE-CREATION > ambari-server/src/test/java/org/apache/ambari/server/KerberosTest.java > PRE-CREATION > ambari-server/src/test/java/org/apache/ambari/server/SlowTest.java > PRE-CREATION > ambari-server/src/test/java/org/apache/ambari/server/StackUpgradeTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/checks/ClientRetryPropertyCheckTest.java > 7b8239c > > ambari-server/src/test/java/org/apache/ambari/server/checks/ComponentsInstallationCheckTest.java > 450d74e > > ambari-server/src/test/java/org/apache/ambari/server/checks/ConfigurationMergeCheckTest.java > 68a0522 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HiveDynamicServiceDiscoveryCheckTest.java > cdf13eb > > ambari-server/src/test/java/org/apache/ambari/server/checks/HiveMultipleMetastoreCheckTest.java > 16f383a > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostMaintenanceModeCheckTest.java > 0e14376 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostsHeartbeatCheckTest.java > cc2c276 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostsMasterMaintenanceCheckTest.java > 9fcb319 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheckTest.java > 4529554 > > ambari-server/src/test/java/org/apache/ambari/server/checks/InstallPackagesCheckTest.java > 080ca3a > > ambari-server/src/test/java/org/apache/ambari/server/checks/MapReduce2JobHistoryStatePreservingCheckTest.java > bfe0c3e > > ambari-server/src/test/java/org/apache/ambari/server/checks/RangerPasswordCheckTest.java > afa3789 > > ambari-server/src/test/java/org/apache/ambari/server/checks/SecondaryNamenodeDeletedCheckTest.java > e2617bf > > ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesMaintenanceModeCheckTest.java > a941b7a > > ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesMapReduceDistributedCacheCheckTest.java > 22f2b1b > > ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesNamenodeHighAvailabilityCheckTest.java > abe7abe > > ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesNamenodeTruncateCheckTest.java > 87d4167 > > ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesUpCheckTest.java > 88826a0 > > ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesYarnWorkPreservingCheckTest.java > 98cfb18 > > ambari-server/src/test/java/org/apache/ambari/server/checks/UpgradeCheckOrderTest.java > 7d70311 > > ambari-server/src/test/java/org/apache/ambari/server/checks/YarnTimelineServerStatePreservingCheckTest.java > 7469bbc > > Diff: https://reviews.apache.org/r/41569/diff/ > > > Testing > ------- > > Verified it worked on the handful of tests I annotated. > > > Thanks, > > Alejandro Fernandez > >
