Drill devs use a set of tests that are not available as part of the Apache
distribution. These tests are a pre-requisite for all commits, but are not
available to any contributors outside the current devs.
This thread is to discuss various options to make these tests available.
Assumptions and requirements -
1) A functional test (as opposed to a unit test) needs to be closer to the
end user environment than a development environment. As such, we should be
running functional tests in a cluster environment, connect using zookeeper
etc.
2) Functional test will keep increasing in number, get more complex and
take a longer and longer time to execute as we go along.
3) Some requirements are:
a) We want to be strict in enforcing the pre-commit requirements, but
not penalize the contributor who has a minor fix.
b) All parts of the product (especially various 'certified' storage
plugins like Hive and Hbase should get tested)
c) It should be easy to debug issues when a test fails. Tests should
fail deterministically. If a test fails, it should always fail and always
fail in the same way (easier said than done).
Some suggestions -
1) Tests should be a top-level maven module within the drill project
a) We want the integration tests to run as part of the drill's
maven build process
b) The build step for the integration-tests module would launch an
embedded drillbit and runs tests against it
c) The tests will be a separate target so they need not be run all
the time
2) Tests should be divided into multiple suites that are based on
components. For example a test suite for testing datatypes will contain the
tests for various datatypes including complex types. A contributor or
developer can then run these tests more frequently as an issue is being
addressed and run the entire suite only once before commit.
3) Provide the tests as a hosted service
4) Setup a bot to fire the test on an AWS cluster and post the results to
the JIRA (Hive does this). Or some variant of this idea.
Some questions -
1) What do some other projects do?
2) Are there any technologies we can leverage that will make this easier?
3) How do we make it easier to debug failing tests.
Please feel free to question the assumptions and requirements. Be creative
with your suggestions.
Parth
