dakshin-k commented on issue #30974: URL: https://github.com/apache/airflow/issues/30974#issuecomment-1926517418
@cmarteepants Yes that would solve the problem at hand, however it's not ideal as re-computing `C` multiple times could cause an issue as there's no guarantee that operation is idempotent. For example, in our case the update to`C` is a bit like appending the newly computed dataset to the existing one. There's a separate pipeline which reads the last 30 `C` datasets and performs a monthly aggregation. So we'd now have to make changes to the way `C` itself works, or perform mitigations like de-duping the outputs. All of that unnecessarily increases the compute and storage cost for `C`, not to mention the additional effort needed to switch to Datasets when the old cron scheduling works just fine 😅 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
