MirtoBusico opened a new issue, #8568: URL: https://github.com/apache/apisix/issues/8568
### Current Behavior I have a keycloak server with certificates signed by a private Certification Authority. Apisix is installed via helm chart in kebernetes The apisix pod is not able to access the keycloak server saying that the issuer is unknown ``` root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration --2022-12-25 11:51:20-- https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration Resolving k6k.h.net (k6k.h.net)... 192.168.100.20 Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected. ERROR: The certificate of 'k6k.h.net' is not trusted. ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer. root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# ``` I created a kubernetes secret: ``` sysop@hserv:~$ cd ~/H/hservcerts sysop@hserv:~/H/hservcerts$ sudo chmod a+r hservca.key sysop@hserv:~/H/hservcerts$ ls -lh hservca.* -rw-r--r-- 1 root root 1,7K ago 15 17:37 hservca.key -rw-r--r-- 1 root root 1,5K ago 15 17:40 hservca.pem sysop@hserv:~/H/hservcerts$ kubectl -n apisix create secret generic hservcacert --from-file=cert=./hservca.pem secret/hservcacert created sysop@hserv:~/H/hservcerts$ kubectl describe secret hservcacert -n apisix Name: hservcacert Namespace: apisix Labels: <none> Annotations: <none> Type: Opaque Data ==== cert: 1464 bytes sysop@hserv:~/H/hservcerts$ ``` and referenced the secret in the tls section of the gateway (in helm chart values.yaml) ``` gateway: type: LoadBalancer ... tls: enabled: true servicePort: 443 containerPort: 9443 existingCASecret: "hservcacert" certCAFilename: "cert" ``` What can I do? ### Expected Behavior In apisix 2.13.0 it worked correctly I created the secret: ``` sysop@m01serv:~/m01certs$ cd ~/m01certs sysop@m01serv:~/m01certs$ ls -lh m01ca.* -rw-r--r-- 1 root root 1,7K gen 17 18:43 m01ca.key -rw-r--r-- 1 root root 1,5K gen 17 18:46 m01ca.pem -rw-r--r-- 1 root root 41 feb 11 14:01 m01ca.srl sysop@m01serv:~/m01certs$ kubectl -n kube-system create secret generic m01cacert --from-file=cert=./m01ca.pem secret/m01cacert created sysop@m01serv:~/m01certs$ kubectl describe secret m01cacert -n kube-system Name: m01cacert Namespace: kube-system Labels: <none> Annotations: <none> Type: Opaque Data ==== cert: 1464 bytes sysop@m01serv:~/m01certs$ ``` Instered the secret in the tls section of gateway (in values.yaml file of the helm chart) ``` gateway: type: LoadBalancer ... tls: enabled: true servicePort: 443 containerPort: 9443 existingCASecret: "m01cacert" certCAFilename: "cert" ``` ### Error Logs Tryin to access the keycloak server gives the issuer unknown error ``` root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration --2022-12-25 11:51:20-- https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration Resolving k6k.h.net (k6k.h.net)... 192.168.100.20 Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected. ERROR: The certificate of 'k6k.h.net' is not trusted. ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer. root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# ``` ### Steps to Reproduce 1. create a kubernetes secret with the private CA pem ``` sysop@hserv:~$ cd ~/H/hservcerts sysop@hserv:~/H/hservcerts$ sudo chmod a+r hservca.key sysop@hserv:~/H/hservcerts$ ls -lh hservca.* -rw-r--r-- 1 root root 1,7K ago 15 17:37 hservca.key -rw-r--r-- 1 root root 1,5K ago 15 17:40 hservca.pem sysop@hserv:~/H/hservcerts$ kubectl -n apisix create secret generic hservcacert --from-file=cert=./hservca.pem secret/hservcacert created sysop@hserv:~/H/hservcerts$ kubectl describe secret hservcacert -n apisix Name: hservcacert Namespace: apisix Labels: <none> Annotations: <none> Type: Opaque Data ==== cert: 1464 bytes sysop@hserv:~/H/hservcerts$ ``` 3. modify the apisix helm chart values.yaml file ``` gateway: type: LoadBalancer ... http: enabled: true servicePort: 80 containerPort: 9080 tls: enabled: true servicePort: 443 containerPort: 9443 existingCASecret: "hservcacert" certCAFilename: "cert" http2: enabled: true sslProtocols: "TLSv1.2 TLSv1.3" ``` 5. install apisix ``` helm install apisix apisix/apisix -f apisix-values.yaml \ --set ingress-controller.config.apisix.serviceNamespace=apisix \ --set ingress-controller.config.apisix.serviceName=apisix-admin \ --set ingress-controller.config.kubernetes.apisixRouteVersion=apisix.apache.org/v2beta3 \ --namespace apisix ``` 7. try to use the openid-connect plugin on a route and gen an error ``` 2022/12/05 08:19:27 [warn] 49#49: *48972 [lua] plugin.lua:934: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET /*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net" 2022/12/05 08:19:27 [alert] 49#49: *48972 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net" 2022/12/05 08:19:27 [alert] 47#47: *48973 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /favicon.ico HTTP/1.0", host: "apisix.h.net", referrer: "https://apisix.h.net/*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756"; 2022/12/05 08:19:27 [error] 48#48: *48980 [lua] openidc.lua:1475: authenticate(): request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net" 2022/12/05 08:19:27 [error] 48#48: *48980 [lua] openid-connect.lua:315: phase_func(): OIDC authentication failed: request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net" 2022/12/05 08:19:27 [warn] 48#48: *48980 [lua] plugin.lua:934: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net" 2022/12/05 08:19:27 [alert] 48#48: *48980 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net" 127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"; 127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /favicon.ico HTTP/1.0" 302 217 0.000 "https://apisix.h.net/*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756"; "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"; 127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"; ``` 9. verify that the keycloak server is not reachable from the apisix pod ``` root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration --2022-12-25 12:11:16-- https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration Resolving k6k.h.net (k6k.h.net)... 192.168.100.20 Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected. ERROR: The certificate of 'k6k.h.net' is not trusted. ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer. root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# ``` ### Environment - APISIX version (run `apisix version`): ``` root@apisix-64fffcfb4c-55vhw:/usr/local/apisix# apisix version /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua version 2.15.1 root@apisix-64fffcfb4c-55vhw:/usr/local/apisix# ``` - Operating system (run `uname -a`): ``` root@apisix-64fffcfb4c-55vhw:/usr/local/apisix# uname -a Linux apisix-64fffcfb4c-55vhw 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 GNU/Linux root@apisix-64fffcfb4c-55vhw:/usr/local/apisix# ``` - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: 2.13.0 - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
