Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]

Fri, 06 Feb 2026 22:42:46 -0800 


shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3863738227

   > Hi @shahar1, sure :) but let me confirm the system tests are 
[example_azure_blob_to_gcs.py](providers/google/tests/system/google/cloud/azure/example_azure_blob_to_gcs.py)
 and 
[example_azure_fileshare_to_gcs.py](https://github.com/apache/airflow/blob/main/providers/google/tests/system/google/cloud/azure/example_azure_fileshare_to_gcs.py),
 right?
   > 
   > If so, here are the results:
   > 
   > # example_azure_blob_to_gcs.py
   > * It works well on Airflow (the first two failures were due to my 
environment misconfigurations, not code issues).
   > 
   > <img alt="example_azure_blob_to_gcs_1_airflow" width="2000" height="608" 
src="https://private-user-images.githubusercontent.com/11390254/546464284-77c49902-6f0d-4fee-8b5c-2627434baba6.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQyODQtNzdjNDk5MDItNmYwZC00ZmVlLThiNWMtMjYyNzQzNGJhYmE2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIxOTI5MTdjYzg3N2E4Yzk3YWNhYzE5NzliY2ZhOWIzZjZjMTA3NDgwNTIwZDc2MDRkNzg1NDNiOWQ2OTA4MDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ZG_P3Xo1UFyC1-xX-vx7mVHcJ_CVlB_fVFSGH61PfcI";>
   > * It reads the file from Azure Blob Storage and transfers it to GCS 
successfully.
   > 
   > <img alt="example_azure_blob_to_gcs_2_azure" width="2000" height="1144" 
src="https://private-user-images.githubusercontent.com/11390254/546464388-01b9d962-a11b-45db-b1b5-285baa1a1b57.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQzODgtMDFiOWQ5NjItYTExYi00NWRiLWIxYjUtMjg1YmFhMWExYjU3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM3MjVkY2EzNGQ3YmUyOTBiODA0OWI4ZTNlYjczNjkyY2YzNjg5ZDA1MDc0YTUzNDQ2Y2M0NDQzY2RiYzFkMjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.J_KBm9GSEU2irTIckKAC4E0Fj-fr8N28T0wQ2ImGMyM";>
 <img alt="example_azure_blob_to_gcs_3_gcp" width="2000" height="538" 
src="https://private-user-images.githubusercontent.com/11390254/54
 
6464405-5f0e9593-079b-4942-a939-59303924b759.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ0MDUtNWYwZTk1OTMtMDc5Yi00OTQyLWE5MzktNTkzMDM5MjRiNzU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkYTBmMjllYTk3Y2FhMzdhZWU2ZTlhZjIwNTBjNTc5NWViODk0MjY5NzQ4MTIwZDZkNTY3NGNjNTc1Yjk0OWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.wYKcht0ifG1Df9Z9e_zxKLRfM-e5xliS6niURiO7Kmc">
   > * It's not a breaking change-- when `unwrap_single` is not set, it returns 
a string as before.
   > 
   > <img alt="example_azure_blob_to_gcs_4_old_return_str" width="2000" 
height="399" 
src="https://private-user-images.githubusercontent.com/11390254/546464592-e0da9b55-9400-4142-a2ca-bdd86346b8d8.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ1OTItZTBkYTliNTUtOTQwMC00MTQyLWEyY2EtYmRkODYzNDZiOGQ4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY5YjQ5MDA5ZTEzNzM1ZTk0ZDEyZjg0ZjRhMjEzNWFkNzJiOGE2YmFlODc0M2M3MDJjNWFhYTQwODk0YWIyZTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.yjoFV-VZWtxdXKBzeJ7BDsDdAnDkpQDNvkmYfkNqMPU";>
   > * After setting `unwrap_single=False`, it returns a list of URIs.
   > 
   > <img alt="example_azure_blob_to_gcs_5_new_return_list" width="2000" 
height="426" 
src="https://private-user-images.githubusercontent.com/11390254/546464672-dee6c181-2726-43c5-bb14-3471f4b5a100.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ2NzItZGVlNmMxODEtMjcyNi00M2M1LWJiMTQtMzQ3MWY0YjVhMTAwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlYmFjODA3OGU3NjgwMTVhOWY1ZjViMjdmNDQ4ZTZlMGM2MDRkNzY4NzVjMGYzOWI2ODMyNjJiZDc0ZDIwMGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.lqBSisJXQtAAIWmRUUcszKScCc-NC-sRF0RMTS07kcg";>
   > # example_azure_fileshare_to_gcs.py
   > * Same for `example_azure_fileshare_to_gcs.py`
   > 
   > <img alt="sync_azure_files_with_gcs" width="2000" height="1351" 
src="https://private-user-images.githubusercontent.com/11390254/546464721-64a1dbe2-c8ae-4a25-b5c8-490c5014ddcc.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ3MjEtNjRhMWRiZTItYzhhZS00YTI1LWI1YzgtNDkwYzUwMTRkZGNjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE2MDNlZTQ3ZWRiMTg4Mzg0ZDA3YzM3ODBkZDk5NzU4MGIxN2QwMjdmMGFlMzcwYjI0ZjAxYjEzM2RjMDY0MjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.DR1jWCarCjBEHkcIfsReQmddZi9QqF2nwCnKm0ywJZ0";>
   > * Also, while running the system test, I noticed `dest_gcs` in 
`example_azure_fileshare_to_gcs.py` was missing the `gs://` prefix, which 
causes `_parse_gcs_url` to fail. I fixed it incidentally.
   > 
   > Please let me know if there's anything else I can help with :)
   
   Awesome! It seems that the system tests are indeed useful for detecting such 
"last minute" bugs.
   Now that everything passes I think that we're all set for this PR to be 
merged.
   If no objections are made, it will be merged by the upcoming release on 
Tuesday.
   Feel free to work on other issues in the meanwhile. Also, if you're able to 
run system tests for other approved PRs related to the original issue, where we 
still don't have any - it would be very helpful.
   
   CC: @VladaZakharova  @MaksYermak


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to