Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
eladkal merged PR #66962: URL: https://github.com/apache/airflow/pull/66962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on code in PR #66962:
URL: https://github.com/apache/airflow/pull/66962#discussion_r3243617511
##
providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py:
##
@@ -712,11 +712,12 @@ def insert_all(
:param project_id: The name of the project where we have the table
:param dataset_id: The name of the dataset where we have the table
:param table_id: The name of the table
-:param rows: the rows to insert
+:param rows: the rows to insert. Each row is a mapping of column name
to
+value, matching the table schema.
.. code-block:: python
-rows = [{"json": {"a_key": "a_value_0"}}, {"json": {"a_key":
"a_value_1"}}]
Review Comment:
This format seems to have been wrong for quite a while (since 2020)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on code in PR #66962:
URL: https://github.com/apache/airflow/pull/66962#discussion_r3243617511
##
providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py:
##
@@ -712,11 +712,12 @@ def insert_all(
:param project_id: The name of the project where we have the table
:param dataset_id: The name of the dataset where we have the table
:param table_id: The name of the table
-:param rows: the rows to insert
+:param rows: the rows to insert. Each row is a mapping of column name
to
+value, matching the table schema.
.. code-block:: python
-rows = [{"json": {"a_key": "a_value_0"}}, {"json": {"a_key":
"a_value_1"}}]
Review Comment:
This syntax seems to have been wrong for quite a while
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on code in PR #66962: URL: https://github.com/apache/airflow/pull/66962#discussion_r3243619729 ## providers/google/src/airflow/providers/google/cloud/sensors/bigquery.py: ## @@ -332,6 +333,14 @@ class BigQueryStreamingBufferEmptySensor(BaseSensorOperator): ``UPDATE/MERGE/DELETE statement over table ... would affect rows in the streaming buffer`` errors. +.. warning:: +The sensor reads ``table.streaming_buffer`` from BigQuery's table +metadata, which is eventually consistent. For a short window right +after a streaming insert the buffer metadata is still absent, so the +sensor may report the buffer empty before it actually is. Known +limitation tracked at +https://github.com/apache/airflow/issues/66963 Review Comment: Flaw in the existing design, can be fixed later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on PR #66962: URL: https://github.com/apache/airflow/pull/66962#issuecomment-4453834394 CC: @radhwene @VladaZakharova @MaksYermak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 opened a new pull request, #66962: URL: https://github.com/apache/airflow/pull/66962 The `BigQueryStreamingBufferEmptySensor` added in #66652 passed the legacy `project:dataset.table` string (colon separator) to `Client.get_table`. The BigQuery client only accepts a `TableReference` or a standard-SQL `project.dataset.table` string, so against real BigQuery `poke()` always raised `NotFound` and the `example_bigquery_sensors` system test could never get past `check_streaming_buffer_empty`. This builds an explicit `TableReference` instead, mirroring how `BigQueryHook.insert_all` constructs its table reference. The legacy colon form is kept only for log/error messages, where it is the conventional BigQuery display format. The unit test previously asserted the colon string was passed to `get_table`, which masked the bug since the mock accepted anything — it now asserts a proper `TableReference` is used. related: #66652 --- # Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.7) Generated-by: Claude Code (Opus 4.7) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
