MirtoBusico opened a new issue #853:
URL: https://github.com/apache/apisix-ingress-controller/issues/853
### Issue description
When creating a route using a yaml file and kubectl the generated upstream
uses the **pod address** instead of the **service address**.
Whenever the cluster is restarted or the pod is redeployed the pod address
changes and the route stop working
### Environment
- your apisix-ingress-controller version (output of
apisix-ingress-controller version --long):
```
apache/apisix-ingress-controller:1.4.0 (taken from the k8s dashboard - btw
where can I issue the "apisix-ingress-controller version --long" command? )
```
- your Kubernetes cluster version (output of kubectl version):
```
sysop@m01serv:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1",
GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean",
BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc",
Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22",
GitVersion:"v1.22.5+k3s1",
GitCommit:"405bf79da97831749733ad99842da638c8ee4802", GitTreeState:"clean",
BuildDate:"2021-12-18T00:43:30Z", GoVersion:"go1.16.10", Compiler:"gc",
Platform:"linux/amd64"}
sysop@m01serv:~$
```
- if you run apisix-ingress-controller in Bare-metal environment, also show
your OS version (uname -a):
```
sysop@m01km:~$ uname -a
Linux m01km 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022
x86_64 x86_64 x86_64 GNU/Linux
sysop@m01km:~$
```
### Minimal test code / Steps to reproduce
1) on 3 virtual machines install a K3S 3 node cluster
2) install Istio with "minimal" profile
3) install the bookinfo demo application with these definitions
```
# Copyright Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##################################################################################################
# This file defines the services, service accounts, and deployments for the
Bookinfo sample.
#
# To apply all 4 Bookinfo services, their corresponding service accounts,
and deployments:
#
# kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
#
# Alternatively, you can deploy any resource separately:
#
# kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -l
service=reviews # reviews Service
# kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -l
account=reviews # reviews ServiceAccount
# kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -l
app=reviews,version=v3 # reviews-v3 Deployment
##################################################################################################
##################################################################################################
# Details service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
service: details
spec:
ports:
- port: 9080
name: http
selector:
app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-details
labels:
account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: details-v1
labels:
app: details
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: details
version: v1
template:
metadata:
labels:
app: details
version: v1
spec:
serviceAccountName: bookinfo-details
containers:
- name: details
image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
securityContext:
runAsUser: 1000
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: ratings
labels:
app: ratings
service: ratings
spec:
ports:
- port: 9080
name: http
selector:
app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-ratings
labels:
account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ratings-v1
labels:
app: ratings
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: ratings
version: v1
template:
metadata:
labels:
app: ratings
version: v1
spec:
serviceAccountName: bookinfo-ratings
containers:
- name: ratings
image: docker.io/istio/examples-bookinfo-ratings-v1:1.16.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
securityContext:
runAsUser: 1000
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
service: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-reviews
labels:
account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reviews-v1
labels:
app: reviews
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: reviews
version: v1
template:
metadata:
labels:
app: reviews
version: v1
spec:
serviceAccountName: bookinfo-reviews
containers:
- name: reviews
image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.2
imagePullPolicy: IfNotPresent
env:
- name: LOG_DIR
value: "/tmp/logs"
ports:
- containerPort: 9080
volumeMounts:
- name: tmp
mountPath: /tmp
- name: wlp-output
mountPath: /opt/ibm/wlp/output
securityContext:
runAsUser: 1000
volumes:
- name: wlp-output
emptyDir: {}
- name: tmp
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reviews-v2
labels:
app: reviews
version: v2
spec:
replicas: 1
selector:
matchLabels:
app: reviews
version: v2
template:
metadata:
labels:
app: reviews
version: v2
spec:
serviceAccountName: bookinfo-reviews
containers:
- name: reviews
image: docker.io/istio/examples-bookinfo-reviews-v2:1.16.2
imagePullPolicy: IfNotPresent
env:
- name: LOG_DIR
value: "/tmp/logs"
ports:
- containerPort: 9080
volumeMounts:
- name: tmp
mountPath: /tmp
- name: wlp-output
mountPath: /opt/ibm/wlp/output
securityContext:
runAsUser: 1000
volumes:
- name: wlp-output
emptyDir: {}
- name: tmp
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reviews-v3
labels:
app: reviews
version: v3
spec:
replicas: 1
selector:
matchLabels:
app: reviews
version: v3
template:
metadata:
labels:
app: reviews
version: v3
spec:
serviceAccountName: bookinfo-reviews
containers:
- name: reviews
image: docker.io/istio/examples-bookinfo-reviews-v3:1.16.2
imagePullPolicy: IfNotPresent
env:
- name: LOG_DIR
value: "/tmp/logs"
ports:
- containerPort: 9080
volumeMounts:
- name: tmp
mountPath: /tmp
- name: wlp-output
mountPath: /opt/ibm/wlp/output
securityContext:
runAsUser: 1000
volumes:
- name: wlp-output
emptyDir: {}
- name: tmp
emptyDir: {}
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: productpage
labels:
app: productpage
service: productpage
spec:
ports:
- port: 9080
name: http
selector:
app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-productpage
labels:
account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: productpage-v1
labels:
app: productpage
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: productpage
version: v1
template:
metadata:
labels:
app: productpage
version: v1
spec:
serviceAccountName: bookinfo-productpage
containers:
- name: productpage
image: docker.io/istio/examples-bookinfo-productpage-v1:1.16.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
volumeMounts:
- name: tmp
mountPath: /tmp
securityContext:
runAsUser: 1000
volumes:
- name: tmp
emptyDir: {}
---
```
4) get the apisix helm chart values.yaml file and change these lines:
```
...
gateway:
type: LoadBalancer
...
tls:
enabled: true
...
stream: # L4 proxy (TCP/UDP) see bug #851
enabled: true
only: false
tcp: # TCP proxy address list
- 9100
- "127.0.0.1:9101"
udp: # UDP proxy address list
- 9200
- "127.0.0.1:9211"
...
dashboard:
enabled: true
ingress-controller:
enabled: true
```
5) inatall apisix with these commands
```
kubectl create ns apisix
kubectl label namespace apisix istio-injection=enabled
helm install apisix apisix/apisix -f apisix-values.yaml \
--set ingress-controller.config.apisix.serviceNamespace=apisix \
--set ingress-controller.config.apisix.serviceName=apisix-admin \
--namespace apisix
```
6) create a file productpage-ar.yaml containing:
```
apiVersion: apisix.apache.org/v2beta2
kind: ApisixRoute
metadata:
name: productpage
spec:
http:
- name: rule1
match:
hosts:
- www2.m01.net
paths:
- /*
backends:
- serviceName: productpage
servicePort: 9080
```
7) create the route (and describe it ) with the commands:
```
sysop@m01serv:~/software/apisisx$ kubectl -n bookinfo apply -f
productpage-ar.yaml
apisixroute.apisix.apache.org/productpage created
sysop@m01serv:~/software/apisisx$ kubectl describe apisixroute productpage
-n bookinfo
Name: productpage
Namespace: bookinfo
Labels: <none>
Annotations: <none>
API Version: apisix.apache.org/v2beta3
Kind: ApisixRoute
Metadata:
Creation Timestamp: 2022-02-06T13:13:31Z
Generation: 1
Resource Version: 171709
UID: 3ccba0af-7622-41f2-90f0-ce2846424ae2
Spec:
Http:
Backends:
Service Name: productpage
Service Port: 9080
Match:
Hosts:
www2.m01.net
Paths:
/*
Name: rule1
Status:
Conditions:
Message: Sync Successfully
Observed Generation: 1
Reason: ResourcesSynced
Status: True
Type: ResourcesAvailable
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ResourcesSynced 50s (x2 over 50s) ApisixIngress ApisixIngress
synced successfully
sysop@m01serv:~/software/apisisx$
```
8) Then verify the route in dashboard
<img width="1706" alt="upstream"
src="https://user-images.githubusercontent.com/11090934/152696999-29b3a2bd-e24d-4fef-8f94-f3bbc24c7e9a.png";>
Note that the "service" in yaml file is translater in the POD address
(10.42.1.188) instead of the SERVICE address (10.43.189.220)
```
sysop@m01serv:~/software/apisisx$ kubectl get pods -n bookinfo -o wide
NAME READY STATUS RESTARTS AGE
IP NODE NOMINATED NODE READINESS GATES
reviews-v3-5cddf46b56-x2hbz 2/2 Running 4 (5h13m ago) 7d8h
10.42.2.159 m01kw2 <none> <none>
ratings-v1-5db74dc944-5z2rz 2/2 Running 4 (5h13m ago) 7d8h
10.42.0.207 m01km <none> <none>
reviews-v2-6fcf6d48b4-bvtzn 2/2 Running 4 (5h13m ago) 7d8h
10.42.0.206 m01km <none> <none>
reviews-v1-d7f5b879b-bf7tz 2/2 Running 4 (5h13m ago) 7d8h
10.42.1.187 m01kw1 <none> <none>
productpage-v1-77748558b4-w2h2h 2/2 Running 4 (5h13m ago) 7d8h
10.42.1.188 m01kw1 <none> <none>
details-v1-65fff9896-fw7qv 2/2 Running 4 (5h13m ago) 7d8h
10.42.1.190 m01kw1 <none> <none>
sysop@m01serv:~/software/apisisx$ kubectl get services -n bookinfo -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
SELECTOR
details ClusterIP 10.43.230.138 <none> 9080/TCP 11d
app=details
ratings ClusterIP 10.43.16.57 <none> 9080/TCP 11d
app=ratings
reviews ClusterIP 10.43.236.52 <none> 9080/TCP 11d
app=reviews
productpage ClusterIP 10.43.189.220 <none> 9080/TCP 11d
app=productpage
sysop@m01serv:~/software/apisisx$
```
9) restart the cluster and try to access the route.You receive the error
```
upstream connect error or disconnect/reset before headers. reset reason:
connection failure, transport failure reason: delayed connect error: 113
```
and looking to the addresses you see that the POD address (10.42.1.211) is
changed and the SERVICE address (10.43.189.220) is the same
```
sysop@m01serv:~$ kubectl get pods -n bookinfo -o wide
NAME READY STATUS RESTARTS AGE IP
NODE NOMINATED NODE READINESS GATES
ratings-v1-5db74dc944-5z2rz 2/2 Running 6 (16m ago) 7d8h
10.42.0.227 m01km <none> <none>
reviews-v3-5cddf46b56-x2hbz 2/2 Running 6 (16m ago) 7d8h
10.42.2.178 m01kw2 <none> <none>
reviews-v2-6fcf6d48b4-bvtzn 2/2 Running 6 (16m ago) 7d8h
10.42.0.228 m01km <none> <none>
reviews-v1-d7f5b879b-bf7tz 2/2 Running 6 (16m ago) 7d8h
10.42.1.214 m01kw1 <none> <none>
details-v1-65fff9896-fw7qv 2/2 Running 6 (16m ago) 7d8h
10.42.1.212 m01kw1 <none> <none>
productpage-v1-77748558b4-w2h2h 2/2 Running 6 (16m ago) 7d8h
10.42.1.211 m01kw1 <none> <none>
sysop@m01serv:~$ kubectl get services -n bookinfo -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
SELECTOR
details ClusterIP 10.43.230.138 <none> 9080/TCP 12d
app=details
ratings ClusterIP 10.43.16.57 <none> 9080/TCP 12d
app=ratings
reviews ClusterIP 10.43.236.52 <none> 9080/TCP 12d
app=reviews
productpage ClusterIP 10.43.189.220 <none> 9080/TCP 12d
app=productpage
sysop@m01serv:~$
```
NOTE: after approx fife minutes the upstream address is updated with the new
value of the POD address and the route works again
### Actual result
```
upstream connect error or disconnect/reset before headers. reset reason:
connection failure, transport failure reason: delayed connect error: 113
```
### Error log
Nothing useful in the logs
### Expected result
```
sysop@m01serv:~/software/apisisx$ curl -v http://www2.m01.net
* Trying 192.168.102.121:80...
* TCP_NODELAY set
* Connected to www2.m01.net (192.168.102.121) port 80 (#0)
> GET / HTTP/1.1
> Host: www2.m01.net
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 1683
< date: Sun, 06 Feb 2022 13:15:42 GMT
< x-envoy-upstream-service-time: 43
< server: istio-envoy
< x-envoy-decorator-operation: apisix-gateway.apisix.svc.cluster.local:80/*
<
<!DOCTYPE html>
<html>
<head>
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://user-images.githubusercontent.com/11090934/152696999-29b3a2bd-e24d-4fef-8f94-f3bbc24c7e9a.png"](https://user-images.githubusercontent.com/11090934/152696999-29b3a2bd-e24d-4fef-8f94-f3bbc24c7e9a.png"){kind=link}