jceresini opened a new issue, #25383: URL: https://github.com/apache/airflow/issues/25383
### What do you see as an issue? The [documentation](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html) for the template variable `data_interval_end` states simply that its the "End of the data interval". Initially I took that to mean the final second/microsecond (depending on precision) of the data interval, but its actually the start of the next interval. For example, given a schedule that runs every 5 minutes, the variables are set as follows (for a simple test I just ran): * **data_interval_start**: 2022-07-28T16:15:00+00:00 * **data_interval_end**: 2022-07-28T16:20:00+00:00 As opposed to: * **data_interval_start**: 2022-07-28T16:15:00+00:00 * **data_interval_end**: 2022-07-28T16:19:59+00:00 It makes sense as implemented, but I'd like to see the documentation state the behavior explicitly. ### Solving the problem The documentation I read when using the template variables is https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html I'm not sure how to word it, but it would be helpful to indicate that `data_interval_end` is effectively the `data_interval_start` of the next interval. Or that the interval the DAG is operating on, using mathematical interval notation, is `[data_interval_start, data_interval_end)` ### Anything else Just an explanation of how we ran into this: We are running DAGS periodically that pull timeseries data from some API. The jobs query an api with filters like this: * timestamp >= data_interval_start * timestamp <= data_interval_end We noticed we were getting some duplicated data, specifically data that happened exactly on the `data_interval_start` value. The simple fix (once we saw the behavior of the variables) was to remove the `=` from the second filter: * timestamp >= data_interval_start * timestamp < data_interval_end ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
