shahar1 opened a new pull request, #74356: URL: https://github.com/apache/airflow/pull/74356
The deferrable `BigQueryValueCheckOperator` and `BigQueryIntervalCheckOperator` in the async BigQuery system test only ever ran trivial `COUNT(*)` queries, so their jobs were `DONE` before the trigger's first poll and the regression fixed in #74305 (#73981: triggers failed with `Job running` whenever the job was still `RUNNING`) could not be observed there. This extends `example_bigquery_queries_async.py` with a view that is slow on purpose and runs both deferrable checks against it: - `create_slow_view` creates a JavaScript UDF with a fixed 5e8-iteration loop and a view over the existing two-row table that evaluates it per row, so a query over the view stays `RUNNING` for tens of seconds. `CURRENT_TIMESTAMP()` in the view keeps BigQuery from serving repeated checks from the query cache. - `check_value_long_running` (`BigQueryValueCheckOperator`, deferrable) and `check_interval_long_running` (`BigQueryIntervalCheckOperator`, deferrable) run against the view. They are wired outside the `howto_operator_*` doc snippets so the documentation examples stay unchanged. Cost: on-demand billing charges for bytes scanned, not for UDF CPU time, and the view only reads the two-row table, so every check job is billed at BigQuery's 10 MB minimum. **Testing** against a real BigQuery project (on-demand, no reservations): - With the trigger from before #74305 (file checked out at `eda8563d98^`), `check_value_long_running` failed all three attempts with `AirflowException: Job running` (the check jobs took 15–45 s), reproducing #73981. In an earlier run with a shorter loop, `check_interval_long_running` also failed with the misreported `Job completed` message from the second bug in that issue. - With the current trigger, both long-running checks passed on the first attempt (check jobs took 8–48 s). - `check_openlineage_events` fails in my local pytest harness for both the modified Dag and the unmodified Dag (`Couldn't find any OpenLineage transport configuration` in task processes), so that task's result is unrelated to this change. The new tasks have no OpenLineage expectations. - Unit tests in `providers/google/tests/unit/google/cloud/triggers/test_bigquery.py` still cover the trigger logic; this PR only touches the system test. related: #73981 related: #74305 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Claude Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
