AryaMoghaddam commented on issue #38407: URL: https://github.com/apache/airflow/issues/38407#issuecomment-2028486413
I've read through the issue, since there hasn't been a clear wanted solution I thought I would ask here, or open a discussion post to narrow it. I could think of three possible approaches: Custom Backfill Logic Implementing a custom backfill script or function that programmatically determines which tasks to run based on your specific logic. This script could use Airflow's APIs or database to check task states and decide which tasks to rerun, considering dependencies as needed. Enhancing Airflow Features Proposing changes or enhancements to Airflow's core to support more granular dependency management in backfilling scenarios. This could involve: Introducing a new flag that allows for partial ignoring of dependencies based on more sophisticated rules or patterns. Extending the task_regex functionality to include or exclude specific dependencies. Use of Dynamic Task Generation Utilizing Airflow's dynamic task generation capabilities to create tasks that can decide at runtime whether to execute based on external conditions, previous task states, or custom logic. This approach requires careful design to ensure it doesn't add unnecessary complexity to DAG definitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
