I am also very interested in seeing how this turns out. Even though we don't have a testing framework in-place on the project I am working on, I would very much like to contribute to some general framework for testing DAGs.
As of now we are just implementing dummy tasks that test our actual tasks and verify if the given input produces the expected output. Nothing crazy and certainly not flexible in the long run. On Fri, 5 May 2017 at 22:59, Sam Elamin <[email protected]> wrote: > Haha yes Scott you are in! > On Fri, 5 May 2017 at 20:07, Scott Halgrim <[email protected]> > wrote: > > > Sounds A+ to me. By “both of you” did you include me? My first response > > was just to your email address. > > > > On May 5, 2017, 11:58 AM -0700, Sam Elamin <[email protected]>, > > wrote: > > > Ok sounds great folks > > > > > > Thanks for the detailed response laura! I'll invite both of you to the > > > group if you are happy and we can schedule a call for next week? > > > > > > How does that sound? > > > On Fri, 5 May 2017 at 17:41, Laura Lorenz <[email protected]> > > wrote: > > > > > > > We do! We developed our own little in-house DAG test framework which > we > > > > could share insights on/would love to hear what other folks are up > to. > > > > Basically we use mock a DAG's input data, use the BackfillJob API > > directly > > > > to call a DAG in a test, and compare its outputs to the intended > result > > > > given the inputs. We use docker/docker-compose to manage services, > and > > > > split our dev and test stack locally so that the tests have their own > > > > scheduler and metadata database and so that our CI tool knows how to > > > > construct the test stack as well. > > > > > > > > We co-opted the BackfillJob API for our own purposes here, but it > > seemed > > > > overly complicated and fragile to start and interact with our own > > > > in-test-process executor like we saw in a few of the tests in the > > Airflow > > > > test suite. So I'd be really interested on finding a way to > streamline > > how > > > > to describe a test executor for both the Airflow test suite and > > people's > > > > own DAG testing and make that a first class type of API. > > > > > > > > Laura > > > > > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <[email protected] > > > > wrote: > > > > > > > > > Hi All > > > > > > > > > > A few people in the Spark community are interested in writing a > > testing > > > > > library for Airflow. We would love anyone who uses Airflow heavily > in > > > > > production to be involved > > > > > > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, > > especially if > > > > > you want to run them in a CI server > > > > > > > > > > Is anyone interested in being involved in the discussion? > > > > > > > > > > Kind Regards > > > > > Sam > > > > > > > > > > > >
