> > We can present our internal tool for doing data quality checks for ETL's > in Qubole. > Iris is an Airflow Operator that currently has built-in functions for count > validation, data-profile validation and duplicate data validation.
That would be great Devjyoti. folks, I would really love to attend this meetup and meet the Airflow community. I'll be attending dataplatforms <https://www.dataplatforms.com/>conference between 11-13th Apr and would be in the bay area before and after that. On the Airflow meetup page, I see that it's scheduled on 11th Apr, so if possible can we move it to 13th or some other day, it would be great. Thanks, Sumit On Wed, Jan 31, 2018 at 1:52 PM, Devjyoti Patra <[email protected]> wrote: > We can present our internal tool for doing data quality checks for ETL's in > Qubole. > Iris is an Airflow Operator that currently has built-in functions for > count validation, > data-profile validation and duplicate data validation. > > On Wed, Jan 31, 2018 at 3:10 AM, George Leslie-Waksman < > [email protected]> wrote: > > > If you're still looking for speakers, I have a handful of things I could > > pull together to talk about: > > > > - the interplay between pools, queues, parallelism, dag concurrency, > and > > other job scheduling levers > > - layering additional instrumentation on top of Airflow operators > > - integrity testing DAGs in CI before they hit production / unit > testing > > tasks > > - using Airflow metadata to track systemic execution behavior > > - how I would have used Airflow if I'd known then what I know now > > > > > > On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <[email protected]> > > wrote: > > > > > +1 on being able to watch a recording, based in Dublin, Ireland but > also > > > very interested in the cloud side of airflow. > > > > > > On Thu, 11 Jan 2018, 17:59 Joy Gao, <[email protected]> wrote: > > > > > > > Great to see that there's a lot of interests! > > > > > > > > I will go ahead and work on organizing this event. Details TBD. > > > > > > > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz < > > [email protected]> > > > > wrote: > > > > > > > > > I would love to see videos from this as well, as we're east coast. > We > > > > host > > > > > our airflow installation in GCP using GKE, and would love to > compare > > > > notes. > > > > > > > > > > Laura > > > > > > > > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <[email protected]> > > > > wrote: > > > > > > > > > > > If it coincides with qcon.ai (april 10-11) or is close to its > > dates > > > > > then I > > > > > > can join. > > > > > > > > > > > > B. > > > > > > > > > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" < > > [email protected] > > > >: > > > > > > > > > > > > I can give a talk about all the hacks we did to scale Airflow > Local > > > > > > Executor and improve the data pipeline on-call experience at > Slack > > if > > > > > folks > > > > > > are interested. > > > > > > > > > > > > Regards, > > > > > > Ananth.P, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 8 January 2018 at 15:51, George Leslie-Waksman < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > +1 and would love to hear about other folks running Airflow in > > GCP. > > > > > > > > > > > > > > Would strongly prefer March (but I'm just one person) > > > > > > > > > > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu > <[email protected] > > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 Joy! > > > > > > > > > > > > > > > > Would prefer April if possible, I can talk about running > > Airflow > > > in > > > > > GCP > > > > > > > if > > > > > > > > there's sufficient interest. > > > > > > > > > > > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Sounds great Joy! > > > > > > > > > > > > > > > > > > I've promoted you to *Event Organizer* on the Bay Area > Apache > > > > > Airflow > > > > > > > > > Meetup as per instructions/guidelines on > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/ > Meetups > > > > > > > > > > > > > > > > > > Go ahead and start setting up the meetup... we recommend 3 > > > > speakers > > > > > > > with > > > > > > > > > 1-2 of them being external to the hosting company. > > > > > > > > > > > > > > > > > > Once the meetup date is setup, we can add it to > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements > > > > > & > > > > > > > > tweet > > > > > > > > > it or share it over the dev list. > > > > > > > > > -s > > > > > > > > > > > > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > > > > > > > I'm Joy from WePay. At the last Airflow meetup in > December > > > > there > > > > > > was > > > > > > > a > > > > > > > > > > demand for hosting a future meetup to cover topics on > > Airflow > > > > > > > > > integrations > > > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS > > > hooks/operators, > > > > > best > > > > > > > > > > practices on running Airflow in the cloud, managed > Airflow, > > > > etc.) > > > > > > > > > > > > > > > > > > > > If there is enough interest, WePay can host the next > event > > > > > > sometimes > > > > > > > in > > > > > > > > > > March/April. And if you would like to give a talk or have > > > > another > > > > > > > topic > > > > > > > > > > covered, feel free to suggest them as well. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Joy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
