GitHub user potiuk added a comment to the discussion: HttpToS3Operator OOM when downloading large file
Yeah. The fact that the oprator loads teh whole file to memory is a feature, not a bug. But if you would like to contribute operator that could do streaming to S3 @rogalski -> feel free to contribute it. There is no need to have an issue for it or you can easly use any available S3 mechanisms to do so - like rclone. GitHub link: https://github.com/apache/airflow/discussions/46066#discussioncomment-11958191 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
