[
https://issues.apache.org/jira/browse/DRILL-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094589#comment-15094589
]
Chun Chang commented on DRILL-4259:
-----------------------------------
I've looked at this before. The difficulty is that the current test framework
does not have the concept of test suite. Each test definition json file defines
one or a group of tests matching a filename pattern. And these tests are all
collected and shuffled randomly and then executed in parallel. If the
permutation settings are making systemwide changes, the change will interfere
with other tests using the current execution model.
To achieve what we want, and as a work around, I suggest we script our test
runs using what we already have as follows:
./run.sh -s Functional/anytests -g regression -n 10 -f
before_run_iteration1.sql -e after_run_iteration1.sql
./run.sh -s Functional/anytests -g regression -n 10 -f
before_run_iteration2.sql -e after_run_iteration2.sql
...
In each of the iteration?.sql file, we set the parameters for that test.
> Add new functional tests to ensure that failures can be detected independent
> of the testing environment
> -------------------------------------------------------------------------------------------------------
>
> Key: DRILL-4259
> URL: https://issues.apache.org/jira/browse/DRILL-4259
> Project: Apache Drill
> Issue Type: Test
> Reporter: Jason Altekruse
>
> In DRILL-4243 an out of memory issue was fixed after a change to the memory
> allocator made memory limits more strict. While the regression tests had been
> run by the team at Dremio prior to merging the patch, running the tests on a
> cluster with more cores changed the memory limits on the queries and caused
> several tests to fail.
> While changes of this magnitude are not going to be common, we should have a
> test suite that reliably fails independent of the environment it is run
> (assuming that there are sufficient resources for the tests to run).
> It would be good to at least try to reproduce this failure on a few different
> setups (cores, nodes in cluster) by adjusting available configuration options
> and adding tests with those different configurations so that the tests will
> fail in different environments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
