shubham22 commented on PR #30259:
URL: https://github.com/apache/airflow/pull/30259#issuecomment-1583168131
@jedcunningham I missed to address your previous slack comments explicitly
in my write-up. Appreciate you summarizing the earlier conversation here and
converging the discussion.
> I’ve seen enough folks using variables to dictate the structure of their
DAG to know this will be problematic. Having parsing be stale is asking for
runtime problems.
For those cases, what do you think about providing something like
`get("key", skip_cache=True)`? With any caching system, there are always some
edge cases and I think we can build solution around it.
> If we land it, we can’t position it as a “safe” thing to turn on (which is
how I read the current docs changes).
This we should definitely address, we can mark it `experimental`, keep it
default disabled and let people try it out and see if they see the benefits.
> Also with many schedulers, you could get them fighting on what the latest
DAG looks like. Increasing db write traffic.
You're right, but it is something that we can test and evaluate the impact.
@vandonr-amz can help with this, once we have initial agreement that variable
and connection caching is worth going forward with.
> Comes back to the “solution” vs “product” thing
I'd argue that by keeping it disabled by default and introducing as
experimental, it is eligible to be a "product" thing. With configurations and
other options, we can allow users to customize Airflow platform to their needs.
We can follow same approach here, especially as we know that this is beneficial
for most users. Again, this PR is a step towards a more holistic product
offering that provides caching across all components, where applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
