evgenyshulman commented on pull request #8651: URL: https://github.com/apache/airflow/pull/8651#issuecomment-633530927
> I still don't understand why the additional execution context is configured as a global option. In my opinion, it is worth each task could have its own separate contexts. In other words, the additional context should be a task parameter, not an option in the global file. This will allow the tasks to have a different context and then use SSH tunnel or gcloud authorization. Such contexts will be able to be parameterized and e.g. one task will be able to use SSH tunnnel to connect to server A, and another task will be able to connect to server B > > If someone wants one global context, it will also be possible by defining the cluster policy. > https://airflow.readthedocs.io/en/latest/concepts.html#cluster-policy > > I propose that the context be defined at the task level. > > The example context with parameters will look like this. > > ```python > def ssh_tunnel(local_port, remote_port): > @contextmanager > def ssh_tunnel_context(): > from sshtunnel import SSHTunnelForwarder > > server = SSHTunnelForwarder( > 'pahaz.urfuclub.ru', > ssh_username="pahaz", > ssh_password="secret", > remote_bind_address=('127.0.0.1', remote_port), > local_bind_address=('127.0.0.1', local_port) > ) > try: > server.start() > yield > finally: > server.stop() > return ssh_tunnel_context > ``` > > The example task will look like this. > > ```python > task_a = MySQLExecuteQueryOperator( > task_id='execute_query', > execution_contexts=[ > ssh_tunnel(3306, 3307), > ] > ) > ``` > > If you want to define a context that will be used by all operators, you can define the cluster policy as follows. > > ```python > def my_task_policy(task): > task.execution_contexts.append(my_global_task_context()) > ``` > > What do you think about my proposal? Will it meet your requirements? I would say it would be a nice option to be able to provide execution context per operator as well, however I assume that the idea is to be able to define a context that will affect all operators, instead of changing one by one. So sometimes a user will use context per operator (a new feature to implement), sometimes a user can just apply global context, so every operator will get GCP connection context regardless its type (or with some operator type check) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
