Hi Airflow folks! I've written up the memory optimization work I did on Airflow 3.x from late last year through the first half of this year, and it's been published on the tech blog of my company, Naver, one of the largest IT companies in Korea.
Part 1 covers how Python task execution is designed and the kinds of Python multiprocessing it relies on. Part 2 walks through how I tracked down and fixed the actual problems I ran into (fork + copy-on-write, and more). Link: https://d2.naver.com/helloworld/4149925 Since the posts dig into Airflow's overall architecture and go fairly deep into the underlying theory, I thought community members might enjoy reading them. There's no English version yet, so I'd appreciate it if you could use Chrome's built-in translation! You can also find the links in LinkedIn post (https://www.linkedin.com/feed/update/urn:li:activity:7513852484356976640) Any reactions or shares there would be a huge help! Thanks Jeongwoo Do
