hansedong commented on issue #9336:
URL: https://github.com/apache/apisix/issues/9336#issuecomment-1517161630
@tao12345666333 @monkeyDluffy6017
I want to say that maybe there are problems with both gRPC and HTTP when
connecting APISIX version 3.2.0 to Etcd.
The phenomenon is as follows:
1. I have two sets of APISIX clusters, and I upgraded a part of the nodes in
each cluster to version 3.2.0.
2. Only the control plane was upgraded (I referred to the documentation for
upgrading APISIX 3.x.x), and after the upgrade, APISIX forwarding was normal (I
confirmed this because I also upgraded two nodes in my production environment).
3. Yesterday, I reported that there were problems with gRPC when connecting
APISIX version 3.2.0 to Etcd, but later on, I changed the connection method
from gRPC to HTTP and it worked normally.
4. Today at dawn, I found that there was abnormal traffic on Etcd used by
APISIX, then discovered a large number of continuous error logs.
The error message of the error log is as follows:
```
2023/04/19 10:27:56 [error] 25793#25793: *5597452 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key: /apisix/routes,
context: ngx.timer
2023/04/19 10:27:56 [error] 25788#25788: *5597471 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key:
/apisix/upstreams, context: ngx.timer
2023/04/19 10:27:57 [error] 25787#25787: *5597542 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key:
/apisix/upstreams, context: ngx.timer
2023/04/19 10:27:58 [error] 25789#25789: *5597586 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key: /apisix/routes,
context: ngx.timer
2023/04/19 10:27:58 [error] 25790#25790: *5597612 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key:
/apisix/upstreams, context: ngx.timer
2023/04/19 10:28:00 [error] 25788#25788: *5597711 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key: /apisix/routes,
context: ngx.timer
2023/04/19 10:28:00 [error] 25787#25787: *5597717 [lua] config_etcd.lua:659:
failed to fetch data from etcd:
/usr/local/apisix/apisix/core/config_util.lua:99: attempt to index local 'item'
(a boolean value)
stack traceback:
/usr/local/apisix/apisix/core/config_util.lua:99: in function
'fire_all_clean_handlers'
/usr/local/apisix/apisix/core/config_etcd.lua:376: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:606: in function
</usr/local/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:596: in function
</usr/local/apisix/apisix/core/config_etcd.lua:574>, etcd key: /apisix/routes,
context: ngx.timer
```
> You can see that when APISIX 3.2.0 encounters an error connecting to Etcd,
regardless of whether the connection method is gRPC or HTTP, the error message
is basically the same.
This issue is reproducible and has been reproduced in different APISIX
clusters of mine. As long as APISIX runs for a period of time (in both my
clusters, it did not exceed 24 hours), it will start to report errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
