casassg commented on pull request #12569: URL: https://github.com/apache/airflow/pull/12569#issuecomment-734033894
Okay. Gave it some extra thought. My main worry with CloudPickle is that we would need to have the same version of python on the workers than the one used to serialize. This won't work for the following scenarion: - Store old version of DAGs in DB and want to re-run them - Python version from DAG submitter is not the same as the one from the executor Regarding imports: - We can inherit imports from current context (should have same imports if we execute DAG file). - Recommend users to import inside the function (this is how PySpark also recommends to do it) Not sure what you mean by closure context. How is this worse than executing the function itself? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
