potiuk edited a comment on pull request #20132:
URL: https://github.com/apache/airflow/pull/20132#issuecomment-988820411
> That’s more reason to let the macros stay and improve them, isn’t it
slightly_smiling_face I don’t think deprecating them for pending removal is a
good idea, especially since we probably want to add macros registration to
providers at some point.
That's a good point and it **might** be something that we want to do (but I
am not 100% sure). This is actually very relevant to multi-tenancy. Especially
that providers are loaded in different contexts (workers, webserver,
DAGprocessor, Scheduler, Triggerers) and adding more "capabilities" to
providers should be carefully considered).
* What does it mean that macros are "supported" by providers ?
* Will that make provider's code executed when we just "initialize" macros?
* Where will it happen: Scheduler? Worker? Webserver? Triggerer?
DagProcessor?
Speaking of which. I have a feeling (@ashb ? ) - this also has some "bigger"
implications re: mutti-tenancy that we should think of separating plugins to
three differenet kinds of plugins:
1) Those are add timtetables, trigers. (needed by scheduler and DagProcessor)
2) Those that provide features for workers (for example macros)
3) Those that modify UI
This is not needed in the first stages of mutli-tenancy but in the
long-haul, conceptually those three usages are different kinds of "security
zones" in the most complex setup where we separate those entities completely.
And it makes the administrative side of it a bit more complex. You have to have
"one" entity that allows you to install plugins (basically any package can
become a plugin) for all those security zones, which basically imposes
strictest regulation in high-security deployments i.e. "only the Uber-Admin can
install packages for any component of Airflow". Of course it can be mitigated
(for example different images used by Scheduler, Worker, DAG Processor.
Webserver (with different permissions of installation packages for each of
them), But conceptually Webserver does not need neither macros, nor timetables.
Similarly worker only needs macros and the rest is useless etc.
And it would be great for example to make sure that "workers" only import
the plugins with macros, and no other. This can also be achieved by smart
plugin manager initialization, but it's not entirely fool-proof I think if all
those plugins are in the same "plugin" bag.
WDYT (long term, not now of course)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
