Re: [PR] Implement BigQueryStreamingBufferEmptySensor to handle DML operations on streaming tables [airflow]

[via GitHub]

shahar1 commented on PR #61148:
URL: https://github.com/apache/airflow/pull/61148#issuecomment-3876199675

   > ## System Test Results
   > I ran a standalone test script against a live GCP project to validate the 
`BigQueryStreamingBufferEmptySensor` behavior end-to-end.
   > 
   > ### Test Flow
   > 1. Create dataset & table in BigQuery
   > 2. Insert rows via **Streaming API** (`insert_rows_json`)
   > 3. Confirm streaming buffer is **active** (~3 rows, ~400 bytes)
   > 4. Sensor `poke()` polls every 30s until buffer is flushed
   > 5. Run **DML UPDATE** to confirm buffer is clear
   > 6. Cleanup (delete dataset)
   > 
   > ### Screenshot
   > <img alt="image" width="704" height="300" 
src="https://private-user-images.githubusercontent.com/20400181/547578269-88360e20-2c21-4398-93db-692eab42d159.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA3MTMyMTMsIm5iZiI6MTc3MDcxMjkxMywicGF0aCI6Ii8yMDQwMDE4MS81NDc1NzgyNjktODgzNjBlMjAtMmMyMS00Mzk4LTkzZGItNjkyZWFiNDJkMTU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjEwVDA4NDE1M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFmYmI5MWIyYWVkMmE1YjFmNDM2OTIyZTYwMzM0ODY4MWEzYWQ0YjliODdjN2Q4Y2EzMTdhOGU3N2JjYmEzZjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ff9PTo_vZc4N5EoHNmmfvWcJu58budud4FxzmP19T4o";>
   > ### Tested on:
   > * Airflow 2.10.3 / Python 3.12
   > * google-cloud-bigquery SDK
   > * GCP project with BigQuery API enabled
   
   Perfect!
   
   > ## System Test Results
   > I ran a standalone test script against a live GCP project to validate the 
`BigQueryStreamingBufferEmptySensor` behavior end-to-end.
   > 
   > ### Test Flow
   > 1. Create dataset & table in BigQuery
   > 2. Insert rows via **Streaming API** (`insert_rows_json`)
   > 3. Confirm streaming buffer is **active** (~3 rows, ~400 bytes)
   > 4. Sensor `poke()` polls every 30s until buffer is flushed
   > 5. Run **DML UPDATE** to confirm buffer is clear
   > 6. Cleanup (delete dataset)
   > 
   > ### Screenshot
   > <img alt="image" width="704" height="300" 
src="https://private-user-images.githubusercontent.com/20400181/547578269-88360e20-2c21-4398-93db-692eab42d159.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA3MTMyMzQsIm5iZiI6MTc3MDcxMjkzNCwicGF0aCI6Ii8yMDQwMDE4MS81NDc1NzgyNjktODgzNjBlMjAtMmMyMS00Mzk4LTkzZGItNjkyZWFiNDJkMTU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjEwVDA4NDIxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM4NDNjZmE5MGM5MzI0MzMxNmEyODc4MDNmMDYzZGZiMTY5NGU1YWJiNzFjMzBiYWQ4ZGEwNDg1ODQxZTljMTAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.lr7C7apPW1CBJCCD5qkkBEEaXndsO82jvw9-pFVi7_M";>
   > ### Tested on:
   > * Airflow 2.10.3 / Python 3.12
   > * google-cloud-bigquery SDK
   > * GCP project with BigQuery API enabled
   
   Looks great! Failing CI doesn't seem related, I'll merge from `main` and 
rerun.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org