ferruzzi commented on code in PR #22758:
URL: https://github.com/apache/airflow/pull/22758#discussion_r849805095
##########
airflow/providers/amazon/aws/operators/s3.py:
##########
@@ -318,6 +318,94 @@ def execute(self, context: 'Context'):
)
+class S3CreateObjectOperator(BaseOperator):
+ """
+ Creates a new object from a given string or bytes.
Review Comment:
FWIW, I just ran into a similar situation working with the SageMaker
operators where this would be handy. Use case: Given a URL, fetch the csv
located there and upload it to S3. I can either use the S3Hook().load_string()
as in Niko's examples, or maybe write the data to a tempfile and use the
LocalToS3 transfer; both should work fine, but the operator would be convenient.
```
@task
def load_data_to_S3():
dataset = requests.get(DATA_URL).content
s3_hook = S3Hook()
s3_hook.load_string(dataset, f'{S3_KEY}/input_data.csv', S3_BUCKET,
replace=True)
```
could become
```
load_data_to_S3 = S3CreateObjectOperator(
bucket=S3_BUCKET,
key=f'{S3_KEY}/input_data.csv',
data=requests.get(DATA_URL).content,
replace=True,
)
```
I guess in the end maybe neither is strictly "better" but I'd say it isn't
unreasonable for a user to expect to find an operator to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
