Re: [PR] Add dataset event dataset dag run queue association [airflow]
github-actions[bot] closed pull request #37424: Add dataset event dataset dag run queue association URL: https://github.com/apache/airflow/pull/37424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add dataset event dataset dag run queue association [airflow]
github-actions[bot] commented on PR #37424: URL: https://github.com/apache/airflow/pull/37424#issuecomment-2040804931 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add dataset event dataset dag run queue association [airflow]
uranusjr commented on PR #37424: URL: https://github.com/apache/airflow/pull/37424#issuecomment-1953591955 Notes from talking to Ankit off-thread: 1. I think adding an association table shouldn’t affect `triggering_dataset_events`. SQLA loads relationships lazily (unless we make it; we don’t) so the new relation shouldn’t be loaded at all unless the user accesses it. They shouldn’t (it’s unsupported) but if they do they get an unavoidable performance penalty. 2. Right now we [pass in all triggered events collected by DDRQ during the prior trigger and the current trigger to the downstream timetable](https://github.com/apache/airflow/blob/011cd3debb4bb166908277c764d65eaf5985c7af/airflow/jobs/scheduler_job_runner.py#L1268-L1278), and let it [come up with an appropriate data interval](https://github.com/apache/airflow/blob/011cd3debb4bb166908277c764d65eaf5985c7af/airflow/timetables/simple.py#L178-L192) for the downstream DAG run. The logic is pretty obvious for ALL (default, current logic), but less so for ANY or anything more complicated. We might need a way for users to override that timetable function to generate a more appropriate data interval, but that will be handled in the future when the need comes up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add dataset event dataset dag run queue association [airflow]
sunank200 opened a new pull request, #37424:
URL: https://github.com/apache/airflow/pull/37424
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
