Posting it here as the question comes up often. When testing multiple
operators in a DAG, how do I know that it "worked"? Following is the
anti-pattern:
@Test
public void testSomeMethod() throws Exception
{
LocalMode lma = LocalMode.newInstance();
lma.prepareDAG(new Application(), new Configuration(false));
LocalMode.Controller lc = lma.getController();
* lc.run(<millis>);*
}
Something could have gone wrong or maybe the test never terminates and
blocks the entire test suite.
Instead, I can write it to run the DAG (asynchronously) *and* verify the
expected results by polling with a timeout. Examples:
https://github.com/DataTorrent/examples/blob/master/tutorials/exactly-once/src/test/java/com/example/myapexapp/ApplicationTest.java
Writes to DB, uses DB to poll for results.
https://github.com/DataTorrent/examples/blob/master/tutorials/unifiers/src/test/java/com/example/myapexapp/ApplicationTest.java
Collects data in memory and terminates when expected result is found.
You can also use BaseOperator.shutdown within the operator thread and the
operator and all its downstream dependencies will terminate. When that is
done from the (only) input operator, it will terminate the entire DAG and
cause control to return to the unit test thread (when using synchronous
run).
Thomas
