Hi all, Thursday the 26/10 my employer Coolblue organized a "Behind the Scenes" event. It is an opportunity for engineers to talk about stuff they work on and usually they provide two presentations.
This event was about BigData and Processing. As (now) team lead of Data Platform, I decided to talk about Apache Airflow, which we are now in the process of migrating to (from Azkaban). Here are the slides: https://www.linkedin.com/feed/update/urn:li:activity:6330346647347875840 It is a technical presentation, aimed at informing people who are new to Airflow what the underlying architecture is and also presenting the why you'd want to use it in the first place. I based the architectural diagrams on AWS on the PoC we did some time. Important takeaway: Airflow is built around some great design principles, which are the result of important insights into data processing. These principles result in a tool, when used correctly according to these principles, to reduce the ETL effort and maintenance and make time to work on higher level intelligent work like Machine Learning, Deep Learning and analysis of your data. It is very similar to the talk I gave at BigData Week London 2017: https://youtu.be/Ch2AQhOhefw Rgds, Gerard
