mik-laj commented on pull request #8651: URL: https://github.com/apache/airflow/pull/8651#issuecomment-632315068
I still don't understand why the additional execution context is configured as a global option. In my opinion, it is worth each task could have its own separate contexts. In other words, the additional context should be a task parameter, not an option in the global file. This will allow the tasks to have a different context and then use SSH tunnel or gcloud authorization. Such contexts will be able to be parameterized and e.g. one task will be able to use SSH tunnnel to connect to server A, and another task will be able to connect to server B If someone wants one global context, it will also be possible by defining the cluster policy. https://airflow.readthedocs.io/en/latest/concepts.html#cluster-policy I propose that the context be defined at the task level. The example context with parameters will look like this. ```python def ssh_tunnel(local_port, remote_port): @contextmanager def ssh_tunnel_context(): from sshtunnel import SSHTunnelForwarder server = SSHTunnelForwarder( 'pahaz.urfuclub.ru', ssh_username="pahaz", ssh_password="secret", remote_bind_address=('127.0.0.1', remote_port), local_bind_address=('127.0.0.1', local_port) ) try: server.start() yield finally: server.stop() return ssh_tunnel_context ``` The example task will look like this. ```python task_a = MySQLExecuteQueryOperator( task_id='execute_query', execution_contexts=[ ssh_tunnel(3306, 3307), ] ) ``` If you want to define a context that will be used by all operators, you can define the cluster policy as follows. ```python def my_task_policy(task): task.execution_contexts.append(my_global_task_context()) ``` What do you think about my proposal? Will it meet your requirements? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
