Two comments here as well (just technical rather than whether we
should do it or not) - to clarify maybe what the proposal really is.
Hussein:
Yeah I had exactly the same thoughts and that was even my original
response in the PR
Surprised to hear that it doesn't work with celery. Is that right? I
assumed that this was the main target.
If it's really only a benefit in dag processor, it's surprising that it
provides much benefit because it should be one call per var-file-parse; in
worker it will be once per ti and I
Since the user can implement his own secret backend, I don't see the need to
add a cache layer and maintain it in Airflow regardless of its type either
local (e.g. RocksDB) or shared (Redis), especially since we provide hooks to
communicate with the most of the DB types which makes it easier
