raphaelauv commented on issue #46008: URL: https://github.com/apache/airflow/issues/46008#issuecomment-2612677645
hi, HttpToS3Operator is an airflow operator doing the data transfer thanks to the code you linked. So data is going through the airflow_worker ( if using celery executor ) you have multiple options: 1) use the kubernetes Executor ( or any other executor that let you custom the RAM context of the task ) for this task and the code of the operator ( but still not doing stream transfer ) will not suffer of the airflow_worker hardware limits or 2) use the KubernetesPodOperator and trigger an efficient specialist tool to execute the transfer , like RCLONE -> https://rclone.org/docs/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
