Hi all,
I've read the docs, I've looked at the source for s3Hook, and I've brute forced
different combinations in my config file but I just can't get this working so
I'm hoping someone can give me some insite.
I need my logs to go into s3 storage, but no matter how I configure it my s3
bucket remains empty, and I never see any errors in my airflow scheduler or
workers regarding issues connection, or that it's even attempted to connect.
Pip install airflow[s3]
Set up a connection in web UI called s3_conn, with type S3, and extra set to
{"aws_access_key_id": "mykey", "aws_secret_access_key": "mykey"}
In airflow config I set the following
remote_base_log_folder = s3://bucket/ (I've tried with and without the
trailing slash)
remote_log_conn_id = s3_conn
encrypt_s3_logs = False
Is there some other step that I'm missing?
Thanks,
Jason Kromm
This email and any attachments may contain confidential and proprietary
information of Blackboard that is for the sole use of the intended recipient.
If you are not the intended recipient, disclosure, copying, re-distribution or
other use of any of this information is strictly prohibited. Please immediately
notify the sender and delete this transmission if you received this email in
error.
