Ok sounds great folks Thanks for the detailed response laura! I'll invite both of you to the group if you are happy and we can schedule a call for next week?
How does that sound? On Fri, 5 May 2017 at 17:41, Laura Lorenz <[email protected]> wrote: > We do! We developed our own little in-house DAG test framework which we > could share insights on/would love to hear what other folks are up to. > Basically we use mock a DAG's input data, use the BackfillJob API directly > to call a DAG in a test, and compare its outputs to the intended result > given the inputs. We use docker/docker-compose to manage services, and > split our dev and test stack locally so that the tests have their own > scheduler and metadata database and so that our CI tool knows how to > construct the test stack as well. > > We co-opted the BackfillJob API for our own purposes here, but it seemed > overly complicated and fragile to start and interact with our own > in-test-process executor like we saw in a few of the tests in the Airflow > test suite. So I'd be really interested on finding a way to streamline how > to describe a test executor for both the Airflow test suite and people's > own DAG testing and make that a first class type of API. > > Laura > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <[email protected]> > wrote: > > > Hi All > > > > A few people in the Spark community are interested in writing a testing > > library for Airflow. We would love anyone who uses Airflow heavily in > > production to be involved > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, especially if > > you want to run them in a CI server > > > > Is anyone interested in being involved in the discussion? > > > > Kind Regards > > Sam > > >
