RyanSiu1995 opened a new issue #8780: URL: https://github.com/apache/airflow/issues/8780
**Apache Airflow version**: 1.10.10 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): ``` > kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"} ``` **Environment**: - GKE - Linux - `Linux pod-name-597d575d8-xrmkn 4.14.138+ #1 SMP Tue Sep 3 02:58:08 PDT 2019 x86_64 GNU/Linu` - `pip install` **What happened**: We used Kubernetes executor to execute our pipeline with a GCS remote log storage. We encountered the log was not found in the GCS bucket for some DAGs. It shares the same error like this. ``` *** Unable to read remote log from gs://xxxx/zzz_test_on_hold/on_hold/2020-05-08T07:06:18.991443+00:00/1.log *** 404 GET https://storage.googleapis.com/download/storage/v1/b/xxxx/o/zzz_test_on_hold%2Fon_hold%2F2020-05-08T07%3A06%3A18.991443%2B00%3A00%2F1.log?alt=media: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>) ``` And we have gone into the log on stdout and found that there was error like this. ``` Please make sure that airflow[gcp] is installed and the GCS connection exists. Could not write logs to gs://xxxx/zzz_test_on_hold/on_hold/2020-05-08T07:06:18.991443+00:00/1.log: 'NoneType' object has no attribute 'upload' ``` **What you expected to happen**: I expected the logs can still upload to GCS if the pod was unexpectedly killed by Kubernetes. **How to reproduce it**: Write a DAG with the following function. ```python def on_hold(): """On hold the process""" while True: logger.warn("this is on holding now...") time.sleep(1000) ``` When the task pod is spawned, kill the pod with kubectl. The log then will not be able to be uploaded. **Anything else we need to know**: Nope ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org