amoghrajesh commented on issue #50025: URL: https://github.com/apache/airflow/issues/50025#issuecomment-2850846439
Thanks for creating the issue! I have an alternative proposal on this one, we shouldn't require the init container at all. The current flow: - When KE wants to execute a workload in a pod, it first generates the workload with help of the scheduler: https://github.com/apache/airflow/pull/46860/files#diff-7866eac6ce3a122d8e081db95a8793074d87d69bced96280db6c0b187967f8cfR397 - This workload is later put into the file system of our airflow main container: https://github.com/apache/airflow/pull/46860/files#diff-a82432d2e0fd9cf3be813faa6d4361eb102df82c56583caab8ebe30f10a7eb1dR359-R392 - This is picked up by the pod and executed Busybox here is used in an init container that copies the "serialised" workload onto the main container filesystem. I propose an alternative solution here, lets just nuke the init container approach here. Lets instead transfer the workload between executor and the maincontainer using environment variables. @davidsharp7 do you still want to work on this one? Or i can take it up soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
