Hey devs,
I have been dealing with sporadic unit test failures since last week. I'd like to share my findings on failing unit tests outlining the patterns causing these. The general advice I was given was to run tests lowering the fork count (the default is 4). This sounds ok to get around the problem. If this is your intent feel free to give a try. However, I wanted to investigate this further, went ahead and set fork count to 8, number of cores testing machine has. The first set of failures manifested around TestUnionAll & TestExampleQueries. After Hakeem's pointer, I noticed that some test cases in TestUnionAll & TestExampleQueries creates & drops views with the same name. In a concurrent settings(given fork count > 0) with no strict ordering among test cases, this sure creates a havoc throwing arbitrary errors at each test run. https://issues.apache.org/jira/browse/DRILL-2684 takes care of this. After applying this fix, now I can cleanly run java-exec tests, faster & consistently. Then my hive tests started failing since multiple test classes use the same metadata folders and clean up after without caring whether other hive tests are still running. I got a patch to unique-ify metadata folders for test class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this however the patch is not public yet. The last problem relates to hive related tests living under jdbc module. I know that there has been an ongoing effort to migrate these tests from jdbc module to where they belong. I am not sure about the timeframe for the migration but I would think the sooner is better. To avoid further failures, I would strongly recommend devs to use view names and external resources that are unique across test cases. One idea is to suffix view a view name with test case/class name for instance prefer using names_view_test_union_all rather than a more generic names_view. Also with increasing number of test cases checked in, it takes more and more time to complete a test run. We should consider concurrent test runs as a legitimate everyday use case and design tests accordingly. I would be interested in hearing other unit test failure patterns so as to be alert. Feel free to share if you discovered any. Regards. -Hanifi
