Ajay9704 opened a new pull request, #61347:
URL: https://github.com/apache/airflow/pull/61347
Description
This PR implements consistent GCS destination URI return behavior for
GoogleDriveToGCSOperator and GoogleSheetsToGCSOperator
Changes
GoogleDriveToGCSOperator (transfers/gdrive_to_gcs.py)
Added unwrap_single parameter to control return format (default: True)
Returns full GCS URIs in gs://bucket/object format instead of None
Added deprecation warning for future default behavior change
Maintains backward compatibility with existing XCom functionality
GoogleSheetsToGCSOperator (transfers/sheets_to_gcs.py)
Added unwrap_single parameter to control return format (default: True)
Returns full GCS URIs in gs://bucket/object format instead of object names
only
Added deprecation warning for future default behavior change
Preserves existing XCom push behavior for destination objects
Implementation Details
Both operators now follow the consistent return convention:
When unwrap_single=True (default): Returns single string for one file, list
for multiple files
When unwrap_single=False: Always returns list regardless of file count
Full GCS URI format: All return values include gs:// prefix
Backward compatibility: Existing code continues to work unchanged
Tests
Added comprehensive test coverage for both operators:
test_execute_with_unwrap_single_true - Single file return behavior
test_execute_with_unwrap_single_false - List return behavior
test_execute_with_unwrap_single_default - Default behavior verification
GoogleSheetsToGCSOperator: Additional tests for single vs multiple file
scenarios
Related Issues
Part of issue #11323 - Return files destination URIs in all ToGCS operators
Migration Notes
Current behavior (backward compatible):
python
# Returns single URI string when one file, list when multiple
op = GoogleDriveToGCSOperator(...)
result = op.execute(context) # e.g., "gs://bucket/file.txt" or
["gs://bucket/file1.txt", "gs://bucket/file2.txt"]
Future behavior (prepare now):
python
# Explicitly set unwrap_single to avoid future breaking changes
op = GoogleDriveToGCSOperator(unwrap_single=False, ...)
result = op.execute(context) # Always returns list: ["gs://bucket/file.txt"]
Checklist
Added unwrap_single parameter with proper type hints
Implemented deprecation warning for future default change
Updated return values to full GCS URI format (gs://bucket/object)
Maintained backward compatibility with existing XCom behavior
Added comprehensive unit tests for all scenarios
Updated docstrings with new parameter documentation
Verified system test examples remain compatible
Closes #11323 (partial)
Used AI for resolving conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
