RyanSiu1995 opened a new issue #8780:
URL: https://github.com/apache/airflow/issues/8780
**Apache Airflow version**: 1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`):
```
> kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7",
GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean",
BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+",
GitVersion:"v1.14.10-gke.27",
GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean",
BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc",
Platform:"linux/amd64"}
```
**Environment**:
- GKE
- Linux
- `Linux pod-name-597d575d8-xrmkn 4.14.138+ #1 SMP Tue Sep 3 02:58:08 PDT
2019 x86_64 GNU/Linu`
- `pip install`
**What happened**:
We used Kubernetes executor to execute our pipeline with a GCS remote log
storage. We encountered the log was not found in the GCS bucket for some DAGs.
It shares the same error like this.
```
*** Unable to read remote log from
gs://xxxx/zzz_test_on_hold/on_hold/2020-05-08T07:06:18.991443+00:00/1.log
*** 404 GET
https://storage.googleapis.com/download/storage/v1/b/xxxx/o/zzz_test_on_hold%2Fon_hold%2F2020-05-08T07%3A06%3A18.991443%2B00%3A00%2F1.log?alt=media:
('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK:
200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
```
And we have gone into the log on stdout and found that there was error like
this.
```
Please make sure that airflow[gcp] is installed and the GCS connection
exists.
Could not write logs to
gs://xxxx/zzz_test_on_hold/on_hold/2020-05-08T07:06:18.991443+00:00/1.log:
'NoneType' object has no attribute 'upload'
```
**What you expected to happen**:
I expected the logs can still upload to GCS if the pod was unexpectedly
killed by Kubernetes.
**How to reproduce it**:
Write a DAG with the following function.
```python
def on_hold():
"""On hold the process"""
while True:
logger.warn("this is on holding now...")
time.sleep(1000)
```
When the task pod is spawned, kill the pod with kubectl.
The log then will not be able to be uploaded.
**Anything else we need to know**:
Nope
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
