MirtoBusico opened a new issue #6345: URL: https://github.com/apache/apisix/issues/6345
### Issue description I'm trying to use a keycloak server for user authentication (and authorization) in Apisix installed in a kubernetes cluster along with Istio My keycloak server replies to ``` https://k6k.m01.net"; ``` The keycloak certificate is signed by my private Certification Authority (CA) My plugin setup followed these articles: ``` https://www.keycloak.org/2021/12/apisix.html (another article: https://apisix.apache.org/blog/2021/12/10/integrate-keycloak-auth-in-apisix/ ) ``` The definition is correct; but when I try to access a route with openid plugin enabled i receive an 500 internal error Trying to access the keycloak server from the apisix pod shell I see that the CA is not recognized ``` bash-5.1# curl https://k6k.m01.net/auth/realms/apisix_test_realm/.well-known/openid-configuration curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. bash-5.1# ``` ### Environment - apisix version (cmd: `apisix version`): ``` bash-5.1# apisix version /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua version 2.12.0 bash-5.1# ``` - OS (cmd: `uname -a`): ``` bash-5.1# uname -a Linux apisix-dd76474d9-82frr 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022 x86_64 Linux bash-5.1# ``` - OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`): ``` bash-5.1# nginx -V nginx version: openresty/1.19.9.1 built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424) built with OpenSSL 1.1.1g 21 Apr 2020 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.19.9.1.3 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.20 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.10 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-ap i/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../mod_dubbo --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../ngx_multi_upstream_module --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../apisix-nginx-module --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../apisix-nginx-module/src/stream --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../wasm-nginx-module --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../lua-var-nginx-module --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_modul e --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module bash-5.1# openresty -V nginx version: openresty/1.19.9.1 built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424) built with OpenSSL 1.1.1g 21 Apr 2020 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.19.9.1.3 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.20 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.10 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-ap i/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../mod_dubbo --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../ngx_multi_upstream_module --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../apisix-nginx-module --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../apisix-nginx-module/src/stream --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../wasm-nginx-module --add-module=/tmp/tmp.I9j9kfjRTW/openresty-1.19.9.1/../lua-var-nginx-module --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_modul e --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module bash-5.1# ``` - etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API): ``` What pod is erver-info API? From the kubernetes dashboard I see that the image is: docker.io/bitnami/etcd:3.4.16-debian-10-r14 ``` - apisix-dashboard version, if have: ``` dashboard_version 2.10.1 ``` - the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner): ``` Don't know how to get the openid plugin version ``` - luarocks version, if the issue is about installation (cmd: `luarocks --version`): ### Steps to reproduce 1) on 3 virtual machines install a K3S 3 node cluster 2) install Istio with "minimal" profile 3) verify versions of the three virtual machines cluster with K3S ``` sysop@m01serv:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME m01kw1 Ready <none> 28d v1.22.5+k3s1 192.168.102.122 <none> Ubuntu 20.04.3 LTS 5.4.0-99-generic containerd://1.5.8-k3s1 m01kw2 Ready <none> 28d v1.22.5+k3s1 192.168.102.123 <none> Ubuntu 20.04.3 LTS 5.4.0-99-generic containerd://1.5.8-k3s1 m01km Ready control-plane,master 28d v1.22.5+k3s1 192.168.102.121 <none> Ubuntu 20.04.3 LTS 5.4.0-99-generic containerd://1.5.8-k3s1 sysop@m01serv:~$ ``` 4) Istio version 1.12.2 ``` sysop@m01serv:~$ istioctl version client version: 1.12.2 control plane version: 1.12.2 data plane version: 1.12.2 (16 proxies) sysop@m01serv:~$ ``` 5) Helm chart version ``` sysop@m01serv:~$ helm show chart apisix/apisix apiVersion: v2 appVersion: 2.12.0 dependencies: - condition: etcd.enabled name: etcd repository: https://charts.bitnami.com/bitnami version: 6.2.6 - alias: dashboard condition: dashboard.enabled name: apisix-dashboard repository: https://charts.apiseven.com version: 0.4.0 - alias: ingress-controller condition: ingress-controller.enabled name: apisix-ingress-controller repository: https://charts.apiseven.com version: 0.9.0 description: A Helm chart for Apache APISIX icon: https://apache.org/logos/res/apisix/apisix.png maintainers: - name: tao12345666333 name: apisix type: application version: 0.8.2 sysop@m01serv:~$ ``` 6) changes in values.yaml ``` ... gateway: type: LoadBalancer ... tls: enabled: true servicePort: 443 containerPort: 9443 existingCASecret: "m01cacert" certCAFilename: "cert" http2: enabled: true ... discovery: enabled: true registry: dns: servers: - "10.43.0.10:53" ... dashboard: enabled: true ingress-controller: enabled: true ``` 7) Apisix installation: ``` kubectl create ns apisix kubectl label namespace apisix istio-injection=enabled cat m01ca.pem -----BEGIN CERTIFICATE----- MIIEDTCCAvWgAwIBAgIUfUAyqeAGoxCGB6V/5qxOS/ZczrEwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t ZTEVMBMGA1UECgwMQnVzaWNvIE1pcnRvMRMwEQYDVQQLDApMYWJvcmF0b3J5MRUw EwYDVQQDDAxCdXNpY28gTWlydG8xJDAiBgkqhkiG9w0BCQEWFW1pcnRvYnVzaWNv QGdtYWlsLmNvbTAeFw0yMjAxMTcxNzQ2MDZaFw0zMjAxMTUxNzQ2MDZaMIGVMQsw CQYDVQQGEwJJVDEOMAwGA1UECAwFSXRhbHkxDTALBgNVBAcMBFJvbWUxFTATBgNV BAoMDEJ1c2ljbyBNaXJ0bzETMBEGA1UECwwKTGFib3JhdG9yeTEVMBMGA1UEAwwM QnVzaWNvIE1pcnRvMSQwIgYJKoZIhvcNAQkBFhVtaXJ0b2J1c2ljb0BnbWFpbC5j b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7T89OolDaC7YprD0l 3q20y0cchShqovk8Nzo12prDX8CCGxv6zexaHYVKk6qFfSGJJcgHcuxLIHvnflVH Ugx9/LWOxaVz6N0i7z8hjnzxyMb2CXaTOpsbp0CdLeoEHjoZlESzAg6blIL4szPn O2VpWfA10qTGWLC0z/QvcPENOVw3NLBZNJWbCl+WmcsCi+ttyzmSdMDS2ANnk6z0 mTqnFVtiCiRYQCXq0A5dr//Jva2q/h0sGoKHSV9Yt/StMB79uRVCGSpiCJAhalh1 8Zs+O9CPnkk+E2jnKn4jgI8FAY0Cii61nn15+/6BrHMu9h6/SO4oKj8i9UXPyodf NW+3AgMBAAGjUzBRMB0GA1UdDgQWBBSCmgdUoJ6HXR5wMoX47koWO5HNnzAfBgNV HSMEGDAWgBSCmgdUoJ6HXR5wMoX47koWO5HNnzAPBgNVHRMBAf8EBTADAQH/MA0G CSqGSIb3DQEBCwUAA4IBAQBgZsZV87/e/8YauGLLGAen857V+NNvl1fMNPAF58O/ NG+iepahWxBJ5miEyMA6BH8ARUa1Q1fah8HC+/Q1dXEj17+h6d4QFS6PWBKp0a2N MSnq0L4FYMnrUrhYxxyt4buNXDuYvaDit7lchKeHBJLBu/NBXH8WhMo/9g0Fg7YD NRv6xg7wvYJf7YIc3RIg5bjklXKpdcvCZjuF8KVqv70x4eQx2m2zcf4CibvZKDFG g/HY3btrW7fvhz9Ytj5w+SoCVLe3OKR0+koIyoGqsmiej9U4dbPTqVdsl3+XyfUF oTpClYSDqa/kfmlT1o9FXpScRTQMOuHBiMYvEFiDBUGY -----END CERTIFICATE----- kubectl -n apisix create secret generic m01cacert --from-file=cert=./m01ca.pem helm install apisix apisix/apisix -f apisix-values.yaml \ --set ingress-controller.config.apisix.serviceNamespace=apisix \ --set ingress-controller.config.apisix.serviceName=apisix-admin \ --set ingress-controller.config.kubernetes.apisixRouteVersion=apisix.apache.org/v2beta3 \ --namespace apisix ``` 8) install an httpbin demo application with these definitions ``` # Copyright Istio Authors # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ################################################################################################## # httpbin service ################################################################################################## apiVersion: v1 kind: ServiceAccount metadata: name: httpbin --- apiVersion: v1 kind: Service metadata: name: httpbin labels: app: httpbin service: httpbin spec: ports: - name: http port: 8000 targetPort: 80 selector: app: httpbin --- apiVersion: apps/v1 kind: Deployment metadata: name: httpbin spec: replicas: 1 selector: matchLabels: app: httpbin version: v1 template: metadata: labels: app: httpbin version: v1 spec: serviceAccountName: httpbin containers: - image: docker.io/kennethreitz/httpbin imagePullPolicy: IfNotPresent name: httpbin ports: - containerPort: 80 ``` 9) with the dashboard create an SSL resource with key and certificate for www.m02.net 9) Create a route and an upstream for www.m01.net pointing to the httpbin service. Verify that the URL "https://www.m01.net"; works correctly 10) setup the openid connect plugin for the route following thew instructions at ``` https://www.keycloak.org/2021/12/apisix.html ``` 11) try to access "https://www.m01.net"; and receive a "500 Internal Server Error" Looking at the apisix log you see: ``` 2022-02-16T17:24:07.837Z | 127.0.0.6 - - [16/Feb/2022:17:24:05 +0000] www.m01.net "GET /spec.json HTTP/2.0" 200 41019 0.008 "https://www.m01.net/"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0" 10.43.225.202:8000 200 0.004 "http://www.m01.net"; 2022-02-16T17:24:12.043Z | 127.0.0.6 - - [16/Feb/2022:17:24:09 +0000] www.m01.net "GET /headers HTTP/2.0" 499 0 0.000 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0" 10.43.225.202:8000 - 0.000 "http://www.m01.net"; 2022-02-16T17:24:12.043Z | 127.0.0.6 - - [16/Feb/2022:17:24:10 +0000] www.m01.net "GET /headers HTTP/2.0" 200 1116 0.003 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0" 10.43.225.202:8000 200 0.004 "http://www.m01.net"; 2022-02-16T17:38:50.975Z | 2022/02/16 17:38:50 [error] 50#50: *1701601 [lua] openidc.lua:1378: authenticate(): request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET / HTTP/2.0", host: "www.m01.net" 2022-02-16T17:38:50.975Z | 2022/02/16 17:38:50 [error] 50#50: *1701601 [lua] openid-connect.lua:297: phase_func(): OIDC authentication failed: request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET / HTTP/2.0", host: "www.m01.net" 2022-02-16T17:38:50.975Z | 2022/02/16 17:38:50 [warn] 50#50: *1701601 [lua] plugin.lua:724: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET / HTTP/2.0", host: "www.m01.net" 2022-02-16T17:38:53.975Z | 127.0.0.6 - - [16/Feb/2022:17:38:50 +0000] www.m01.net "GET / HTTP/2.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36" - - - "http://www.m01.net"; 2022-02-16T17:38:53.975Z | 127.0.0.6 - - [16/Feb/2022:17:38:51 +0000] www.m01.net "GET /favicon.ico HTTP/2.0" 302 142 0.011 "https://www.m01.net/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36" - - - "http://www.m01.net"; Logs from Feb 16, 2022 to Feb 16, 2022 UTC ``` Trying to access the keycloak server from the apisix pod you see that the CA is not recognized (" SSL certificate problem: unable to get local issuer certificate") ``` bash-5.1# curl https://k6k.m01.net/auth/realms/apisix_test_realm/.well-known/openid-configuration curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. bash-5.1# ``` ### Actual result Trying to access "https://www.m01.net"; and receive a "500 Internal Server Error" ### Error log Looking at the apisix log you see: ``` 2022-02-16T17:24:07.837Z | 127.0.0.6 - - [16/Feb/2022:17:24:05 +0000] www.m01.net "GET /spec.json HTTP/2.0" 200 41019 0.008 "https://www.m01.net/"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0" 10.43.225.202:8000 200 0.004 "http://www.m01.net"; 2022-02-16T17:24:12.043Z | 127.0.0.6 - - [16/Feb/2022:17:24:09 +0000] www.m01.net "GET /headers HTTP/2.0" 499 0 0.000 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0" 10.43.225.202:8000 - 0.000 "http://www.m01.net"; 2022-02-16T17:24:12.043Z | 127.0.0.6 - - [16/Feb/2022:17:24:10 +0000] www.m01.net "GET /headers HTTP/2.0" 200 1116 0.003 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0" 10.43.225.202:8000 200 0.004 "http://www.m01.net"; 2022-02-16T17:38:50.975Z | 2022/02/16 17:38:50 [error] 50#50: *1701601 [lua] openidc.lua:1378: authenticate(): request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET / HTTP/2.0", host: "www.m01.net" 2022-02-16T17:38:50.975Z | 2022/02/16 17:38:50 [error] 50#50: *1701601 [lua] openid-connect.lua:297: phase_func(): OIDC authentication failed: request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET / HTTP/2.0", host: "www.m01.net" 2022-02-16T17:38:50.975Z | 2022/02/16 17:38:50 [warn] 50#50: *1701601 [lua] plugin.lua:724: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET / HTTP/2.0", host: "www.m01.net" 2022-02-16T17:38:53.975Z | 127.0.0.6 - - [16/Feb/2022:17:38:50 +0000] www.m01.net "GET / HTTP/2.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36" - - - "http://www.m01.net"; 2022-02-16T17:38:53.975Z | 127.0.0.6 - - [16/Feb/2022:17:38:51 +0000] www.m01.net "GET /favicon.ico HTTP/2.0" 302 142 0.011 "https://www.m01.net/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36" - - - "http://www.m01.net"; Logs from Feb 16, 2022 to Feb 16, 2022 UTC ``` Trying to access the keycloak server from the apisix pod you see that the CA is not recognized (" SSL certificate problem: unable to get local issuer certificate") ``` bash-5.1# curl https://k6k.m01.net/auth/realms/apisix_test_realm/.well-known/openid-configuration curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. bash-5.1# ``` ### Expected result The keycloak login page should show up and after login the httpbin page is shown -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
