Re: Article: The Rise of the Data Engineer

Maxime, Just wanted to thank you for writing this article - much like the original articles by Jeff Hammerbacher and DJ Patil coining the term "Data Scientist", I feel this article stands as a great explanation of what the title of "Data Engineer" means today.. As someone who has been working in

Re: NYC Meetup?

We at MLB Advanced Media (MLBAM / MLB.com) are just about to get our first few Airflow processes into production, so we'd love to join an NYC-based meetup! -rob On Wed, Dec 21, 2016 at 9:49 AM, Jeremiah Lowin wrote: > It would be wonderful to have an east coast meetup! I

Re: Adjusting DAG Schedules (For Daylight Savings Time, And In General)

r_interval' updates are taken as a delta from the last execution_date. Still would be curious to hear insight from anyone who has had to deal with this! Thanks, Rob On Thu, Mar 9, 2017 at 4:47 PM, Rob Goretsky <robert.goret...@gmail.com> wrote: > With Daylight Savings Time upon

Adjusting DAG Schedules (For Daylight Savings Time, And In General)

With Daylight Savings Time upon us, I was wondering if anyone has had to address this issue -- While I understand that right now Airflow is not timezone-aware, and runs all of its jobs in GMT/UTC time, my team delivers reports to stakeholders that want to consistently see all data reported through

Re: Making Airflow Timezone aware

This will be huge for my team at MLB.com! Really appreciate your work on this, Bolke! We will finally be able to take down the posters we've all hung up at our desks that show the current GMT offset! Let us know how/when we can try it out! -rob > On Nov 15, 2017, at 7:33 PM, George

Re: best way to handle version upgrades of libraries used by tasks

My team has solved for this with Docker. When a developer works on a single project, they freeze their Python library versions via pip freeze > requirements.txt for that project, And then we build one Docker image per project, using something very similar to the official 'onbuild' version of the

Re: Deploy Airflow on Kubernetes using Airflow Operator

Barni, Thank you so much for sharing this! I'm admittedly far from a Kubernetes guru, but I'm just trying to wrap my head around the reasons why we'd need a custom Kubernetes controller to manage Airflow's components, as opposed to the setup here https://github.com/mumoshu/kube-airflow wherein we