[jira] [Comment Edited] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"

2020-07-24 Thread Dian Fu (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164738#comment-17164738
 ] 

Dian Fu edited comment on FLINK-18122 at 7/25/20, 4:12 AM:
---

Another instance on 1.11: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4901=logs=08866332-78f7-59e4-4f7e-49a56faa3179=3e8647c1-5a28-5917-dd93-bf78594ea994


was (Author: dian.fu):
Another instance on 1.11: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5

> Kubernetes test fails with "error: timed out waiting for the condition on 
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on 
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container 
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read: 
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
>  No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch.  Got 
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open 
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
>  for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383   13191 request.go:621] 
> Throttling request took 1.198606989s, request: 
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234   13191 request.go:621] 
> Throttling request took 4.398107353s, request: 
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209   13191 request.go:621] 
> Throttling request took 1.198449167s, request: 
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320   13191 request.go:621] 
> Throttling request took 4.198274438s, request: 
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258   13191 request.go:621] 
> Throttling request took 1.19871495s, request: 
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276   13191 request.go:621] 
> Throttling request took 4.198369672s, request: 
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY   
> STATUS  RESTARTS   AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67  0/1 
> ErrImageNeverPull   0  67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm   0/1 
> ErrImageNeverPull   0  67m
> 2020-06-04T10:33:30.9185226Z 
> 2020-06-04T10:33:30.9185926Z NAMETYPE
> CLUSTER-IP  EXTERNAL-IP   PORT(S) 
>   AGE
> 2020-06-04T10:33:30.9186832Z service/flink-job-cluster   NodePort
> 10.111.92.199   
> 6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP   67m
> 2020-06-04T10:33:30.9187545Z service/kubernetes  ClusterIP   
> 10.96.0.1   443/TCP 
>   68m
> 2020-06-04T10:33:30.9187976Z 
> 2020-06-04T10:33:30.9188472Z NAME READY   
> UP-TO-DATE   AVAILABLE   AGE
> 2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager   0/1 1   
>  0   67m
> 2020-06-04T10:33:30.9189508Z 
> 2020-06-04T10:33:30.9189815Z NAME   
> DESIRED   CURRENT  

[jira] [Comment Edited] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"

2020-06-04 Thread Yang Wang (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126337#comment-17126337
 ] 

Yang Wang edited comment on FLINK-18122 at 6/5/20, 3:17 AM:


I agree that we should {{exit}} when building the image failed.


was (Author: fly_in_gis):
I agree that we should {{exit}} when build the image failed.

> Kubernetes test fails with "error: timed out waiting for the condition on 
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
>  Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on 
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container 
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read: 
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
>  No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch.  Got 
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open 
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
>  for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383   13191 request.go:621] 
> Throttling request took 1.198606989s, request: 
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234   13191 request.go:621] 
> Throttling request took 4.398107353s, request: 
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209   13191 request.go:621] 
> Throttling request took 1.198449167s, request: 
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320   13191 request.go:621] 
> Throttling request took 4.198274438s, request: 
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258   13191 request.go:621] 
> Throttling request took 1.19871495s, request: 
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276   13191 request.go:621] 
> Throttling request took 4.198369672s, request: 
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY   
> STATUS  RESTARTS   AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67  0/1 
> ErrImageNeverPull   0  67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm   0/1 
> ErrImageNeverPull   0  67m
> 2020-06-04T10:33:30.9185226Z 
> 2020-06-04T10:33:30.9185926Z NAMETYPE
> CLUSTER-IP  EXTERNAL-IP   PORT(S) 
>   AGE
> 2020-06-04T10:33:30.9186832Z service/flink-job-cluster   NodePort
> 10.111.92.199   
> 6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP   67m
> 2020-06-04T10:33:30.9187545Z service/kubernetes  ClusterIP   
> 10.96.0.1   443/TCP 
>   68m
> 2020-06-04T10:33:30.9187976Z 
> 2020-06-04T10:33:30.9188472Z NAME READY   
> UP-TO-DATE   AVAILABLE   AGE
> 2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager   0/1 1   
>  0   67m
> 2020-06-04T10:33:30.9189508Z 
> 2020-06-04T10:33:30.9189815Z NAME   
> DESIRED   CURRENT   READY   AGE
> 2020-06-04T10:33:30.9190418Z replicaset.apps/flink-task-manager-74ccc9bd9   1 
> 1 0   67m
> 2020-06-04T10:33:30.9190662Z 
> 2020-06-04T10:33:30.9190891Z NAME  COMPLETIONS