derkuci opened a new pull request #20474: URL: https://github.com/apache/airflow/pull/20474
We recently upgraded from Airflow 1.9 to 2.2.2 (to be more precise, we are still in the process of upgrading, dealing with various issues related to the upgrade). I like the change that the newly refactored SSHHook initializes most of the parameters in `__init__`, making customization easier. However, there is one "heavy" operation related to SSH proxy which I wonder might be better off moving back to where it will be used. I noticed that for every 30+ seconds (`min_file_process_interval=30`), during the run of DAGFileProcessor, a bunch of SSH sessions would be created. This is because a `paramiko.ProxyCommand` is ran for any connection used in tasks, as long as that connection has a "ProxyCommand" in the `.ssh/config` file. This seems too much (also heavy) to me. I wonder if a change in the spirit of the attached pull request is acceptable or not? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
