I have just seen Jacques' comment under the patch. I am not sure whether we should set forks to a fraction of available cores. Any ideas?
On Mon, Apr 6, 2015 at 5:52 PM, Hanifi Gunes <[email protected]> wrote: > @Ted, I actually have a patch pending review for this at > https://issues.apache.org/jira/browse/DRILL-2039. However, @Hakeem you > may still end up in a troubling situation as your mac might have 4 logical > cores =) > > *sysctl hw | grep **cpu* should tell you. > > > -Hanifi > > > On Mon, Apr 6, 2015 at 5:22 PM, Ted Dunning <[email protected]> wrote: > >> Maven allows the forkCount to be set in terms of the number of cores on >> the >> machine by appending a C to the number. >> >> (I think) >> >> >> >> On Mon, Apr 6, 2015 at 5:15 PM, Abdel Hakim Deneche < >> [email protected]> >> wrote: >> >> > From my personal experience, using a forkCount of 4 makes some tests to >> > timeout, not because there are some concurrent problems but just >> because my >> > machine is not powerful enough to run as many tests in parallel. >> > >> > On Mon, Apr 6, 2015 at 5:06 PM, Hanifi Gunes <[email protected]> >> wrote: >> > >> > > Hey devs, >> > > >> > > >> > > I have been dealing with sporadic unit test failures since last week. >> I'd >> > > like to share my findings on failing unit tests outlining the patterns >> > > causing these. >> > > >> > > The general advice I was given was to run tests lowering the fork >> count >> > > (the default is 4). This sounds ok to get around the problem. If this >> is >> > > your intent feel free to give a try. However, I wanted to investigate >> > this >> > > further, went ahead and set fork count to 8, number of cores testing >> > > machine has. >> > > >> > > The first set of failures manifested around TestUnionAll & >> > > TestExampleQueries. After Hakeem's pointer, I noticed that some test >> > cases >> > > in TestUnionAll & TestExampleQueries creates & drops views with the >> same >> > > name. In a concurrent settings(given fork count > 0) with no strict >> > > ordering among test cases, this sure creates a havoc throwing >> arbitrary >> > > errors at each test run. >> > https://issues.apache.org/jira/browse/DRILL-2684 >> > > takes care of this. After applying this fix, now I can cleanly run >> > > java-exec tests, faster & consistently. >> > > >> > > Then my hive tests started failing since multiple test classes use the >> > same >> > > metadata folders and clean up after without caring whether other hive >> > tests >> > > are still running. I got a patch to unique-ify metadata folders for >> test >> > > class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this >> > > however >> > > the patch is not public yet. >> > > >> > > The last problem relates to hive related tests living under jdbc >> module. >> > I >> > > know that there has been an ongoing effort to migrate these tests from >> > jdbc >> > > module to where they belong. I am not sure about the timeframe for the >> > > migration but I would think the sooner is better. >> > > >> > > To avoid further failures, I would strongly recommend devs to use view >> > > names and external resources that are unique across test cases. One >> idea >> > is >> > > to suffix view a view name with test case/class name for instance >> prefer >> > > using names_view_test_union_all rather than a more generic names_view. >> > Also >> > > with increasing number of test cases checked in, it takes more and >> more >> > > time to complete a test run. We should consider concurrent test runs >> as a >> > > legitimate everyday use case and design tests accordingly. >> > > >> > > >> > > I would be interested in hearing other unit test failure patterns so >> as >> > to >> > > be alert. Feel free to share if you discovered any. >> > > >> > > >> > > Regards. >> > > -Hanifi >> > > >> > >> > >> > >> > -- >> > >> > Abdelhakim Deneche >> > >> > Software Engineer >> > >> > <http://www.mapr.com/> >> > >> > >> > Now Available - Free Hadoop On-Demand Training >> > < >> > >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >> > > >> > >> > >
