We can present our internal tool for doing data quality checks for ETL's in Qubole. Iris is an Airflow Operator that currently has built-in functions for count validation, data-profile validation and duplicate data validation.
On Wed, Jan 31, 2018 at 3:10 AM, George Leslie-Waksman < [email protected]> wrote: > If you're still looking for speakers, I have a handful of things I could > pull together to talk about: > > - the interplay between pools, queues, parallelism, dag concurrency, and > other job scheduling levers > - layering additional instrumentation on top of Airflow operators > - integrity testing DAGs in CI before they hit production / unit testing > tasks > - using Airflow metadata to track systemic execution behavior > - how I would have used Airflow if I'd known then what I know now > > > On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <[email protected]> > wrote: > > > +1 on being able to watch a recording, based in Dublin, Ireland but also > > very interested in the cloud side of airflow. > > > > On Thu, 11 Jan 2018, 17:59 Joy Gao, <[email protected]> wrote: > > > > > Great to see that there's a lot of interests! > > > > > > I will go ahead and work on organizing this event. Details TBD. > > > > > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz < > [email protected]> > > > wrote: > > > > > > > I would love to see videos from this as well, as we're east coast. We > > > host > > > > our airflow installation in GCP using GKE, and would love to compare > > > notes. > > > > > > > > Laura > > > > > > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <[email protected]> > > > wrote: > > > > > > > > > If it coincides with qcon.ai (april 10-11) or is close to its > dates > > > > then I > > > > > can join. > > > > > > > > > > B. > > > > > > > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" < > [email protected] > > >: > > > > > > > > > > I can give a talk about all the hacks we did to scale Airflow Local > > > > > Executor and improve the data pipeline on-call experience at Slack > if > > > > folks > > > > > are interested. > > > > > > > > > > Regards, > > > > > Ananth.P, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 8 January 2018 at 15:51, George Leslie-Waksman < > > > > > [email protected]> wrote: > > > > > > > > > > > +1 and would love to hear about other folks running Airflow in > GCP. > > > > > > > > > > > > Would strongly prefer March (but I'm just one person) > > > > > > > > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <[email protected] > > > > > > > wrote: > > > > > > > > > > > > > +1 Joy! > > > > > > > > > > > > > > Would prefer April if possible, I can talk about running > Airflow > > in > > > > GCP > > > > > > if > > > > > > > there's sufficient interest. > > > > > > > > > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > Sounds great Joy! > > > > > > > > > > > > > > > > I've promoted you to *Event Organizer* on the Bay Area Apache > > > > Airflow > > > > > > > > Meetup as per instructions/guidelines on > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups > > > > > > > > > > > > > > > > Go ahead and start setting up the meetup... we recommend 3 > > > speakers > > > > > > with > > > > > > > > 1-2 of them being external to the hosting company. > > > > > > > > > > > > > > > > Once the meetup date is setup, we can add it to > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements > > > > & > > > > > > > tweet > > > > > > > > it or share it over the dev list. > > > > > > > > -s > > > > > > > > > > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <[email protected]> > > wrote: > > > > > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > > > > > I'm Joy from WePay. At the last Airflow meetup in December > > > there > > > > > was > > > > > > a > > > > > > > > > demand for hosting a future meetup to cover topics on > Airflow > > > > > > > > integrations > > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS > > hooks/operators, > > > > best > > > > > > > > > practices on running Airflow in the cloud, managed Airflow, > > > etc.) > > > > > > > > > > > > > > > > > > If there is enough interest, WePay can host the next event > > > > > sometimes > > > > > > in > > > > > > > > > March/April. And if you would like to give a talk or have > > > another > > > > > > topic > > > > > > > > > covered, feel free to suggest them as well. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Joy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
