Scratch that. I see the whole video now. On Mon, May 8, 2017 at 3:33 PM Arnie Salazar <[email protected]> wrote:
> Thanks Sam! > > Is there a part 2 to the video? If not, can you post the "next steps" > notes you took whenever you have a chance? > > Cheers, > Arnie > > On Mon, May 8, 2017 at 3:08 PM Sam Elamin <[email protected]> wrote: > >> Hi Folks >> >> For those of you who missed it, you can catch the discussion from the link >> on this tweet <https://twitter.com/samelamin/status/861703888298225670> >> >> Please do share and feel free to get involved as the more feedback we get >> the better the library we create is :) >> >> Regards >> Sam >> >> On Mon, May 8, 2017 at 9:43 PM, Sam Elamin <[email protected]> >> wrote: >> >> > Bit late notice but the call is happening today at 9 15 utc so in about >> > 30 mins or so >> > >> > It will be recorded but if anyone would like to join in on the >> discussion >> > the hangout link is https://hangouts.google.com/hangouts/_/ >> > mbkr6xassnahjjonpuvrirxbnae >> > >> > Regards >> > Sam >> > >> > On Fri, 5 May 2017 at 21:35, Ali Uz <[email protected]> wrote: >> > >> >> I am also very interested in seeing how this turns out. Even though we >> >> don't have a testing framework in-place on the project I am working >> on, I >> >> would very much like to contribute to some general framework for >> testing >> >> DAGs. >> >> >> >> As of now we are just implementing dummy tasks that test our actual >> tasks >> >> and verify if the given input produces the expected output. Nothing >> crazy >> >> and certainly not flexible in the long run. >> >> >> >> >> >> On Fri, 5 May 2017 at 22:59, Sam Elamin <[email protected]> >> wrote: >> >> >> >> > Haha yes Scott you are in! >> >> > On Fri, 5 May 2017 at 20:07, Scott Halgrim <[email protected] >> > >> >> > wrote: >> >> > >> >> > > Sounds A+ to me. By “both of you” did you include me? My first >> >> response >> >> > > was just to your email address. >> >> > > >> >> > > On May 5, 2017, 11:58 AM -0700, Sam Elamin < >> [email protected]>, >> >> > > wrote: >> >> > > > Ok sounds great folks >> >> > > > >> >> > > > Thanks for the detailed response laura! I'll invite both of you >> to >> >> the >> >> > > > group if you are happy and we can schedule a call for next week? >> >> > > > >> >> > > > How does that sound? >> >> > > > On Fri, 5 May 2017 at 17:41, Laura Lorenz < >> [email protected] >> >> > >> >> > > wrote: >> >> > > > >> >> > > > > We do! We developed our own little in-house DAG test framework >> >> which >> >> > we >> >> > > > > could share insights on/would love to hear what other folks >> are up >> >> > to. >> >> > > > > Basically we use mock a DAG's input data, use the BackfillJob >> API >> >> > > directly >> >> > > > > to call a DAG in a test, and compare its outputs to the >> intended >> >> > result >> >> > > > > given the inputs. We use docker/docker-compose to manage >> services, >> >> > and >> >> > > > > split our dev and test stack locally so that the tests have >> their >> >> own >> >> > > > > scheduler and metadata database and so that our CI tool knows >> how >> >> to >> >> > > > > construct the test stack as well. >> >> > > > > >> >> > > > > We co-opted the BackfillJob API for our own purposes here, but >> it >> >> > > seemed >> >> > > > > overly complicated and fragile to start and interact with our >> own >> >> > > > > in-test-process executor like we saw in a few of the tests in >> the >> >> > > Airflow >> >> > > > > test suite. So I'd be really interested on finding a way to >> >> > streamline >> >> > > how >> >> > > > > to describe a test executor for both the Airflow test suite and >> >> > > people's >> >> > > > > own DAG testing and make that a first class type of API. >> >> > > > > >> >> > > > > Laura >> >> > > > > >> >> > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin < >> >> [email protected] >> >> > > > > wrote: >> >> > > > > >> >> > > > > > Hi All >> >> > > > > > >> >> > > > > > A few people in the Spark community are interested in >> writing a >> >> > > testing >> >> > > > > > library for Airflow. We would love anyone who uses Airflow >> >> heavily >> >> > in >> >> > > > > > production to be involved >> >> > > > > > >> >> > > > > > At the moment (AFAIK) testing your DAGs is a bit of a pain, >> >> > > especially if >> >> > > > > > you want to run them in a CI server >> >> > > > > > >> >> > > > > > Is anyone interested in being involved in the discussion? >> >> > > > > > >> >> > > > > > Kind Regards >> >> > > > > > Sam >> >> > > > > > >> >> > > > > >> >> > > >> >> > >> >> >> > >> >
