I'm playing around with implementing a custom hook/connection to my
service, and there is some custom service discovery logic I need to perform
on a connection-by-connection basis. Is there any "proper" way to cache the
result (basically just a hostname) of my service discovery, so I don't have
to re-do the discovery on every task that uses my hook? I'd like to be able
to cache it globally if possible, but even on a dagrun-by-dagrun basis
would be ok. These are the options I've considered, in order of decreasing
attractiveness:
1. Variables - simple, clean, but kinda gross that the cached values
will show up in the UI
2. Read/Write values directly to the xcom table in the metadata db -
transparent, but obviously ugly and not exactly future-friendly
3. Provide my own external globally accessible storage (S3 or redis or
something). Probably the most "correct"(?) but also the highest cost both
in terms of infrastructure and dev time/effort...and this direction would
not be practical if this custom hook/connection is ever released to the
public
Is there any other mechanism which would be more appropriate for this use?
