[ https://issues.apache.org/jira/browse/AIRFLOW-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931293#comment-16931293 ]
ASF subversion and git services commented on AIRFLOW-5162: ---------------------------------------------------------- Commit 26aa4ba648697ec582437b1e86ca3191bdcc253c in airflow's branch refs/heads/master from Steven Freed [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=26aa4ba ] [AIRFLOW-5162] GCS Hook Upload Method Improvement (#5770) * [AIRFLOW-5162] add to 'upload' method for gcs file data Allows for local file uploads (parameter 'filename') and file content (parameter 'data') uploads to gcs by using Google's gcs api method 'upload_from_string' which allows for uploading files content as a string or bytes. * [AIRFLOW-5162] add ValueError exceptions and upload tests ValueError exceptions are raised when the filename and data params are provided or neither are provided. This avoids errors of local files and file data. Tests assert file and data upload variants as well as assert exceptions. * [AIRFLOW-5162] fixed unit tests for gcs uploads added seperate unit tests for uploading string content and byte content using the upload_from_string gcp method. Prevents assertion being called twice when tests are seperated as well as organizes testing. * [AIRFLOW-5162] added type annotations and tests modifications Added type annotations for static tests and ease of use for users. Tests modifications get rid of assertions for None responses and add tests for calling upload functions once. * [AIRFLOW-5162] add pylint and flake8 fixes for type annotations type annotation syntax requires a space between '=' sign for default assignment. * [AIRFLOW-5162] added called once and exception tests added called once tests for upload methods for gzip data. This was added to assert that the upload_from_string method was called correctly. Assertions for exceptions confirming the correct exception messages. * [AIRFLOW-5162] added fix to doc build Fixed indentation in docstrings so that the documentation can build properly. Added new lines after descriptions of the methods functionalities. > GCS Uploads for local files and file data (string or bytes) > ----------------------------------------------------------- > > Key: AIRFLOW-5162 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5162 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp > Affects Versions: 1.10.4, 1.10.5 > Reporter: Steven Freed > Priority: Minor > Labels: beginner, features, pull-request-available > > Allows for gcs uploads to be done by specifying a local file name or by using > the file's content. File content can be uploaded as a string or bytes. This > gives developers more alternatives to uploading files where developers may > have constraints on disk space and prefer to upload via file content in > memory. This update also helps airflow update to Google's new > 'upload_from_string' method for gcs. -- This message was sent by Atlassian Jira (v8.3.2#803003)