neosis41 opened a new issue, #9610:
URL: https://github.com/apache/apisix/issues/9610
### Current Behavior
hello, i get back to you because im facing an issue to configure my apisix
cluster. Actually i already set 3 node with apisix-dashboard-etcd on each.
everything works fine in http but when i try in https i get weird behaviour. i
already set my ssl certificate. the first time i connect on https, it work
fines, my apisix server has a tcp connection open with the client, as long as
the tcp connection is opened i can make the requests in https. but once the tcp
connection reach the connection timeout and pass to time_wait, i try the same
https request on the client browser and i receive a 500 internal error from
openresty. i can see on the serveur another tcp connection opened but i get
this 500 error code. in the nginx error.log i get a message "lua entry thread
aborted: runtime error: /opt/apisix/apisix/init.lua:332: attempt to index local
'matched_ssl' (a nil value)
stack traceback:
coroutine 0:
/opt/apisix/apisix/init.lua: in function 'verify_https_client'
/opt/apisix/apisix/init.lua:560: in function 'http_access_phase'
access_by_lua(nginx.conf:329):2: in main chunk, client: 10.138.16.8,
server: _, request: "GET /test7212212415527117122562121275915345814351514.html
HTTP/2.0", host:
"[slot_featuretestnico2.mrvol.com](http://slot_featuretestnico2.mrvol.com/)"" .
if i retry some times later like 1h it works again until it fails again. Same
thing if i restart apisix it works again until it fails again.
If i wait the server time_wait connection finish then my https request is
success.
for apisix install , i did the git clone then 'bash
utils/install-dependencies.sh' then install luarock 'curl
https://raw.githubusercontent.com/apache/apisix/master/utils/linux-install-luarocks.sh
-sL | bash -' and '. /root/.cargo/env && LUAROCKS_SERVER=https://luarocks.cn/
make deps' and then 'make install'
for the dahboard : docker run -d --name dashboard --network apisix-network
-p 9000:9000 -v
/opt/apisix/dashboard-conf.yaml:/usr/local/apisix-dashboard/conf/conf.yaml
apache/apisix-dashboard
for etcd : docker run -d --network apisix-network -p 2379:2379 -p 2380:2380
-p 4001:4001 --name etcd
[quay.io/coreos/etcd:v3.4.26](http://quay.io/coreos/etcd:v3.4.26)
/usr/local/bin/etcd --name etcd{{ cluster_id }}
-advertise-client-urls=http://{{ private_ip }}:2379 -listen-client-urls
http://0.0.0.0:2379/,http://0.0.0.0:4001/ -initial-advertise-peer-urls
http://{{ private_ip }}:2380 -listen-peer-urls http://0.0.0.0:2380/
-initial-cluster-token etcd-cluster-1 -initial-cluster {{ cluster_range }}
-initial-cluster-state new
heres below my config.yaml :
apisix:
node_listen: # This style support multiple ports
- 9080
ssl:
enable: true
listen: # APISIX listening port in https.
- port: 443
enable_http2: true
ssl_protocols: TLSv1.2 TLSv1.3
ssl_ciphers:
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl_session_tickets: false # disable ssl_session_tickets
by default for 'ssl_session_tickets' would make Perfect Forward Secrecy useless.
# ref:
https://github.com/mozilla/server-side-tls/issues/135
ssl_session_cache: shared:SSL:1m
key_encrypt_salt: # If not set, will save origin ssl key
into etcd.
- edd1c9f0985e76a2
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
allow_admin:
- 0.0.0.0/0 # Please set it to the subnet address you obtained.
admin_key:
- name: admin
key: edd1c9f034335f136f87ad84b625c8f1 # using fixed API token has
security risk, please update it when you deploy to production environment
role: admin
etcd:
host:
- "http://127.0.0.1:2379";
prefix: "/apisix"
timeout: 30
### Expected Behavior
i expect all my https request return me a code 200 and not 200 the first
time and 500 internal error after.
### Error Logs
first itry https request at 8:19 it works and the backend server give me a
404 :
10.212.135.241 - - [06/Jun/2023:08:19:57 +0200]
[slot_featuretestnico2.mrvol.com](http://slot_featuretestnico2.mrvol.com/) "GET
/test2888854.html HTTP/2.0" 404 1251 0.002 "-" "Mozilla/5.0 (X11; Ubuntu; Linux
x86_64; rv:109.0) Gecko/20100101 Firefox/113.0" 10.10.0.30:80 404 0.002
"http://slot_featuretestnico2.mrvol.com/";
and then i retry at https request at 8:28 and then get my 500 error :
10.212.135.241 - - [06/Jun/2023:08:28:17 +0200]
[slot_featuretestnico2.mrvol.com](http://slot_featuretestnico2.mrvol.com/) "GET
/test28588854.html HTTP/2.0" 500 174 0.000 "-" "Mozilla/5.0 (X11; Ubuntu; Linux
x86_64; rv:109.0) Gecko/20100101 Firefox/113.0" - - -
"http://slot_featuretestnico2.mrvol.com/";
with the message below in error.log :
2023/06/06 08:28:17 [error] 3756#3756: *6713294 lua entry thread aborted:
runtime error: /opt/apisix/apisix/init.lua:332: attempt to index local
'matched_ssl' (a nil value)
stack traceback:
coroutine 0:
/opt/apisix/apisix/init.lua: in function 'verify_https_client'
/opt/apisix/apisix/init.lua:560: in function 'http_access_phase'
access_by_lua(nginx.conf:329):2: in main chunk, client: 10.212.135.241,
server: _, request: "GET /test28588854.html HTTP/2.0", host:
"[slot_featuretestnico2.mrvol.com](http://slot_featuretestnico2.mrvol.com/)"
when it works (in debug mode) :
2023/06/06 14:57:27 [info] 7786#7786: *124362 [lua] radixtree_sni.lua:79:
create_router(): route items: [{"paths":["moc.lovrm.*"],"handler":"function:
0x7f00ae708270"}], context: ssl_certificate_by_lua*, client: 10.212.135.241,
server: 0.0.0.0:443
2023/06/06 14:57:27 [info] 7786#7786: *124362 [lua] radixtree.lua:355:
pre_insert_route(): path: moc.lovrm. operator: <=, context:
ssl_certificate_by_lua*, client: 10.212.135.241, server: 0.0.0.0:443
2023/06/06 14:57:27 [info] 7786#7786: *124362 [lua] radixtree.lua:243:
insert_route(): insert route path: moc.lovrm. dataprt: 1, context:
ssl_certificate_by_lua*, client: 10.212.135.241, server: 0.0.0.0:443
2023/06/06 14:57:27 [debug] 7786#7786: *124362 [lua] radixtree_sni.lua:169:
match_and_set(): sni: slot_featuretestnico2.mrvol.com
2023/06/06 14:57:27 [info] 7786#7786: *124362 [lua] radixtree_sni.lua:209:
match_and_set(): debug - matched:
{"key":"/apisix/ssls/1","createdIndex":1193,"clean_handlers":{},"modifiedIndex":1193,"value":{"create_time":1685969984,"status":1,"key":"Y...
2023/06/06 14:57:27 [info] 7786#7786: *124362 [lua] secret.lua:233:
fetch_secrets(): retrieve secrets refs, context: ssl_certificate_by_lua*,
client: 10.212.135.241, server: 0.0.0.0:443
2023/06/06 14:57:27 [debug] 7786#7786: *124362 [lua] ssl.lua:202:
create_obj_fun(): parsing cert for sni: slot_featuretestnico2.mrvol.com
2023/06/06 14:57:27 [debug] 7786#7786: *124362 [lua] ssl.lua:220:
create_obj_fun(): parsing priv key for sni: slot_featuretestnico2.mrvol.com
2023/06/06 14:57:27 [debug] 7786#7786: *124361 [lua] radixtree_sni.lua:169:
match_and_set(): sni: slot_featuretestnico2.mrvol.com
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] radixtree_sni.lua:209:
match_and_set(): debug - matched:
{"key":"/apisix/ssls/1","createdIndex":1193,"clean_handlers":{},"modifiedIndex":1193,"value":{"create_time":1685969984,"status":1,"key":"...
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] radixtree.lua:355:
pre_insert_route(): path: / operator: <=, client: 10.212.135.241, server: _,
request: "GET /test2858526188365541.html HTTP/2.0", host:
"slot_featuretestnico2.mrvol.com"
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] radixtree.lua:243:
insert_route(): insert route path: / dataprt: 1, client: 10.212.135.241,
server: _, request: "GET /test2858526188365541.html HTTP/2.0", host:
"slot_featuretestnico2.mrvol.com"
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] radixtree.lua:355:
pre_insert_route(): path: moc.lovrm.2ocintseterutaef_tols operator: =, client:
10.212.135.241, server: _, request: "GET /test2858526188365541.html HTTP/2.0",
host: "slot_featuretestnico2.mrvol.com"
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua]
radixtree_host_uri.lua:163: match(): route match mode: radixtree_host_uri,
client: 10.212.135.241, server: _, request: "GET /test2858526188365541.html
HTTP/2.0", host: "slot_featuretestnico2.mrvol.com"
2023/06/06 14:57:27 [debug] 7786#7786: *124361 [lua] radixtree.lua:497:
compare_param(): pcre pat: \/((.|\n)*)
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] init.lua:607:
http_access_phase(): matched route:
{"value":{"uri":"/*","status":1,"create_time":1685455621,"upstream":{"nodes":[{"port":80,"host":"10.10.0.30","weight":1}],"timeout":{"read":10,"send":10,"connect":15},"hash_on":"vars","pass_host":"pass","scheme":"http","type":"roundrobin","keepalive_pool":{"requests":1000,"idle_timeout":60,"size":320},"parent":{"value":"table:
0x7f00b03bb6a8","update_count":0,"key":"/apisix/routes/462927046608159422","createdIndex":431,"orig_modifiedIndex":1065,"clean_handlers":"table:
0x7f00ae664f78","modifiedIndex":1065,"has_domain":false}},"host":"slot_featuretestnico2.mrvol.com","id":"462927046608159422","name":"test_slot_featuretestnico.mrvol.com","plugins":{"authz-keycloak":{"keepalive_pool":5,"grant_type":"urn:ietf:params:oauth:grant-type:uma-ticket","client_id":"Client
ID","timeout":3000,"access_token_expires_in":300,"lazy_load_paths":false,"access_token_expires_leeway":0,"ssl_verify":true
,"_meta":{"disable":true},"refresh_token_expires_in":3600,"refresh_token_expires_leeway":0,"http_method_as_scope":false,"permissions":["resource
name#scope
name"],"keepalive_timeout":60000,"token_endpoint":"http://127.0.0.1:8090/auth/realms/${realm}/protocol/openid-connect/token","policy_enforcement_mode":"ENFORCING","cache_ttl_seconds":86400,"keepalive":true},"openid-connect":{"unauth_action":"auth","bearer_only":false,"use_pkce":false,"access_token_in_authorization_header":false,"introspection_endpoint_auth_method":"client_secret_basic","realm":"Digitrips","redirect_uri":"https://slot_featuretestnico2.mrvol.com/","set_id_token_header":true,"client_id":"apisix","timeout":3,"set_refresh_token_header":false,"ssl_verify":false,"_meta":{"disable":true},"discovery":"https://sso.digitrips.com/realms/Digitrips/.well-known/openid-configuration","session":{"secret":"abcdefghabcdefghabcdefgh"},"set_userinfo_header":true,"logout_path":"/logout","set_access_token_header":true,"client_secret":";
iGRPFw5Y16NJHmAnbn7syWgMSeOGMRDQ","scope":"openid"}},"update_time":1685965267,"priority":0},"update_count":0,"key":"/apisix/routes/462927046608159422","createdIndex":431,"orig_modifiedIndex":1065,"clean_handlers":{},"modifiedIndex":1065,"has_domain":false},
client: 10.212.135.241, server: _, request: "GET /test2858526188365541.html
HTTP/2.0", host: "slot_featuretestnico2.mrvol.com"
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] balancer.lua:195:
pick_server(): route:
{"value":{"uri":"/*","status":1,"create_time":1685455621,"upstream":{"timeout":{"read":10,"send":10,"connect":15},"hash_on":"vars","scheme":"http","nodes_ref":[{"port":80,"host":"10.10.0.30","weight":1,"priority":0}],"pass_host":"pass","nodes":"table:
0x7f00ae5a3cd0","original_nodes":[{"port":80,"host":"10.10.0.30","weight":1}],"type":"roundrobin","keepalive_pool":{"requests":1000,"idle_timeout":60,"size":320},"parent":{"value":"table:
0x7f00b03bb6a8","update_count":0,"key":"/apisix/routes/462927046608159422","createdIndex":431,"orig_modifiedIndex":1065,"clean_handlers":"table:
0x7f00ae664f78","modifiedIndex":1065,"has_domain":false}},"host":"slot_featuretestnico2.mrvol.com","id":"462927046608159422","name":"test_slot_featuretestnico.mrvol.com","plugins":{"authz-keycloak":{"keepalive_pool":5,"grant_type":"urn:ietf:params:oauth:grant-type:uma-ticket","client_id":"Client
ID","timeout":3000,"a
ccess_token_expires_in":300,"lazy_load_paths":false,"access_token_expires_leeway":0,"ssl_verify":true,"_meta":{"disable":true},"refresh_token_expires_in":3600,"refresh_token_expires_leeway":0,"http_method_as_scope":false,"permissions":["resource
name#scope
name"],"keepalive_timeout":60000,"token_endpoint":"http://127.0.0.1:8090/auth/realms/${realm}/protocol/openid-connect/token","policy_enforcement_mode":"ENFORCING","cache_ttl_seconds":86400,"keepalive":true},"openid-connect":{"unauth_action":"auth","bearer_only":false,"use_pkce":false,"access_token_in_authorization_header":false,"introspection_endpoint_auth_method":"client_secret_basic","realm":"Digitrips","redirect_uri":"https://slot_featuretestnico2.mrvol.com/","set_id_token_header":true,"client_id":"apisix","timeout":3,"set_refresh_token_header":false,"ssl_verify":false,"_meta":{"disable":true},"discovery":"https://sso.digitrips.com/realms/Digitrips/.well-known/openid-configuration","session":{"secret":"abcdefghabcdefghabcdefgh";
},"set_userinfo_header":true,"logout_path":"/logout","set_access_token_header":true,"client_secret":"iGRPFw5Y16NJHmAnbn7syWgMSeOGMRDQ","scope":"openid"}},"update_time":1685965267,"priority":0},"update_count":0,"key":"/apisix/routes/462927046608159422","createdIndex":431,"orig_modifiedIndex":1065,"clean_handlers":{},"modifiedIndex":1065,"has_domain":false},
client: 10.212.135.241, server: _, request: "GET /test2858526188365541.html
HTTP/2.0", host: "slot_featuretestnico2.mrvol.com"
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] balancer.lua:196:
pick_server(): ctx: {"matched_sni":["moc.lovrm.*"],"upstream_conf":"table:
0x7f00b02d20e0","upstream_version":"1065#table:
0x7f00b02d20e0","upstream_key":"462927046608159422","matched_ssl":"table:
0x7f00b035aeb8","global_rules":" etcd key:
/apisix/global_rules","curr_req_matched":"table:
0x7f00ae39a8e0","conf_type":"route","upstream_scheme":"http","var":{"_ctx":{"matched_sni":"table:
0x7f00ae708d40","upstream_conf":{"timeout":{"read":10,"send":10,"connect":15},"hash_on":"vars","scheme":"http","nodes_ref":[{"port":80,"host":"10.10.0.30","weight":1,"priority":0}],"pass_host":"pass","nodes":"table:
0x7f00ae5a3cd0","original_nodes":[{"port":80,"host":"10.10.0.30","weight":1}],"type":"roundrobin","keepalive_pool":{"requests":1000,"idle_timeout":60,"size":320},"parent":{"value":{"uri":"/*","status":1,"create_time":1685455621,"upstream":"table:
0x7f00b02d20e0","host":"slot_featuretestnico2.mrvol.com","id":"462927046608
159422","name":"test_slot_featuretestnico.mrvol.com","plugins":{"authz-keycloak":{"keepalive_pool":5,"grant_type":"urn:ietf:params:oauth:grant-type:uma-ticket","client_id":"Client
ID","timeout":3000,"access_token_expires_in":300,"lazy_load_paths":false,"access_token_expires_leeway":0,"ssl_verify":true,"_meta":{"disable":true},"refresh_token_expires_in":3600,"refresh_token_expires_leeway":0,"http_method_as_scope":false,"permissions":["resource
name#scope
name"],"keepalive_timeout":60000,"token_endpoint":"http://127.0.0.1:8090/auth/realms/${realm}/protocol/openid-connect/token","policy_enforcement_mode":"ENFORCING","cache_ttl_seconds":86400,"keepalive":true},"openid-connect":{"unauth_action":"auth","bearer_only":false,"use_pkce":false,"access_token_in_authorization_header":false,"introspection_endpoint_auth_method":"client_secret_basic","realm":"Digitrips","redirect_uri":"https://slot_featuretestnico2.mrvol.com/","set_id_token_header":true,"client_id":"apisix","timeout":3,"set_refresh
_token_header":false,"ssl_verify":false,"_meta":{"disable":true},"discovery":"https://sso.digitrips.com/realms/Digitrips/.well-known/openid-configuration","session":{"secret":"abcdefghabcdefghabcdefgh"},"set_userinfo_header":true,"logout_path":"/logout","set_access_token_header":true,"client_secret":"iGRPFw5Y16NJHmAnbn7syWgMSeOGMRDQ","scope":"openid"}},"update_time":1685965267,"priority":0},"update_count":0,"key":"/apisix/routes/462927046608159422","createdIndex":431,"orig_modifiedIndex":1065,"clean_handlers":{},"modifiedIndex":1065,"has_domain":false}},"upstream_version":"1065#table:
0x7f00b02d20e0","upstream_key":"462927046608159422","matched_ssl":{"key":"/apisix/ssls/1","createdIndex":1193,"clean_handlers":{},"modifiedIndex":1193,"value":{"create_time":1685969984,"status":1,"key":"YnwwDKc5vNzo0OU4StTRQbwgCnTZ3dmYiBFm8aGnvTwyAh+Rz4EWXpJif7colyUhItBIvm962vfX5Me9G9WPJ7DuLfMov4Fb4Vrc1tmSL5fr6N895OAiHGOTuoRt+rH7cNWiWXmgp53bOC+mGcArvKEkvS7YPicjbFhsG8zlyvJh9Tx6G1z/OYaKcHtYe0uFjQZFFNB3d1
d7u2YZKhGTAO7MAbQKDSX8k7U1APLFo5zNkST96MTJbHDYm1v9lrsui7fRM35syM6XIld5HlZKQJG7BXpJQkKGZ2vvTaEvsq3HWOwj+MMiO2J+3V7YasxfgPr3WTthgBBqAL5q2QEZHMD78U8JBiOEllbySWky1uIGEWN563EcWMeMsqpcCAnSZRg5eU0sJnM08B5N337qsb2KrjwdZGicFOV9RAuprN4D8KUt0mVc8bjwHE8wgV0Q7YmjGrBZOPSBmjOhlhRxjdXdhKQv2FhOyGF3qzm3DiIpvAUgNrRMXP2rdUc5jy1V4vg95Bj5Ven7MmKDWE8zwEG3MirTu2A0xRwG0ZZYsBLGJnrpn7We7y8iY2tCQAg+KxWbBuWEPLDB//w9T/FNTPhbxglFsJz14PdpgZl31VSg9I0yIy5Q8xfi/E+UWHhP7tk1Ea9xSj3dXcdAex6c/Letco2jpX0wSPKSGkfQ/0fBfMlcK4iqZrZjnSU9jq8T3+p7mz116lqq+5EVrZOrxehO3SerOLT7AlJEVC+yDRcwqgzbB90cqG/v/ByZCnc1xk6ZBFqF2EEXij2nvnCE0MayC0WD6jqu6YMozuLjHKxA6GZ8i6cMrGwtdbMV57WDwt75f+6IxNcsVFj7JzTnj1o2PiEDz8xednXgTBCZjv9ZIKSSWoQ+jrVbdT8mZ3Gx+eWmn2/uA8Rd/bfNozJnYwpsea21C3qgK5WJvlH71yHO+U0gNSIYidvQ5mlLdHNEMyN2xOv5hHibf09ncrO4/vOlwS28I1TvkLXHEiS5cpR1fO5L1S5gU1/XsymmBCbSPW3Az4diuKODYhX/OioHeIwRfWvJQ9Ke47ec7i5kcQG6JZUtpb+DQMN/5e3MN2bLHDOyEwWPdilpZ2mOP3oi65C76ourrokFXQUOrtUH1TYsBj7Iub6niY10zLlU412W69+Qop+KdC/cgJKA1eRCurEOMSUO4G6HrRjo1pLZCYw2v1Z
jsFghF+PIZhAbsthacsQmdowfteFOAghKEuajJau1diZL9X/ruZK8EfUk6jF7TB7Z4lS/ZR23EV0rK1T/pwbDRCF7bK1fJ7n8xee5ImrI5Ca+
2023/06/06 14:57:27 [info] 7786#7786: *124361 [lua] balancer.lua:384: run():
proxy request to 10.10.0.30:80 while connecting to upstream, client:
10.212.135.241, server: _, request: "GET /test2858526188365541.html HTTP/2.0",
host: "slot_featuretestnico2.mrvol.com"
when it fails :
2023/06/06 14:56:17 [error] 7790#7790: *112661 lua entry thread aborted:
runtime error: /opt/apisix/apisix/init.lua:332: attempt to index local
'matched_ssl' (a nil value)
stack traceback:
coroutine 0:
/opt/apisix/apisix/init.lua: in function 'verify_https_client'
/opt/apisix/apisix/init.lua:560: in function 'http_access_phase'
access_by_lua(nginx.conf:331):2: in main chunk, client: 10.212.135.241,
server: _, request: "GET
/test7212212629523483713991313639659953285668397322511633661532592517216556232655625512485275915553245814351514.html
HTTP/2.0", host: "slot_featuretestnico2.mrvol.com"
### Steps to Reproduce
start etcd docker container
start apisix which is directly install on host
start dashboard docker container
request https://slot_featuretestnico2.mrvol.com/test.html (its an A record
in our local dns server) on a browser in my computer
wait the connectiontimeout setttings if its configured on 60sec then wait
60s and do the same https request and now it fails.
I have still the same weird behaviour on chrome and mozilla firefox. i didnt
test another one.
### Environment
- APISIX version (run `apisix version`): 3.3.0
- Operating system (run `uname -a`): ubuntu 20
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
openresty/1.21.4.1
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`): v3.4.26
- APISIX Dashboard version, if relevant: latest
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
/usr/local/bin/luarocks 3.8.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
