[GitHub] [apisix] gy09535 opened a new issue #2899: bug: Etcd jwt refresh error can cause gateway cluster crash.

[GitBox]

gy09535 opened a new issue #2899:
URL: https://github.com/apache/apisix/issues/2899


   ### Issue description
   
   Today I find some auth error from etcd, I try to connect etcd to find the 
error, I find so many auth request cause etcd api timeout, this is my etcd logs:
   ```
   2020-11-30 19:55:01.150181 W | auth: invalid auth token: 
SDZBJcAaVedJqUFY.116415
   2020-11-30 19:55:01.226868 W | auth: invalid auth token: 
MNtikXEJuXkIOWVt.114382
   2020-11-30 19:55:01.238848 W | auth: invalid auth token: 
ArWxwPXGQRclPdZp.114301
   2020-11-30 19:55:01.254654 W | auth: invalid auth token: 
WuqYtmtIDCGftGwr.114315
   2020-11-30 19:55:01.283579 W | auth: invalid auth token: 
CqdVqeYChuiEWgWh.114324
   2020-11-30 19:55:01.320021 W | auth: invalid auth token: 
YHlQCFrmBrGsHXXV.116397
   2020-11-30 19:55:01.320712 W | auth: invalid auth token: 
YyOUXXpgpqQyNryf.114384
   2020-11-30 19:55:01.359229 W | auth: invalid auth token: 
YXSKKPuaGObcCrFZ.114346
   2020-11-30 19:55:01.367728 W | auth: invalid auth token: 
HFqBwCvzPfTRVxih.116421
   2020-11-30 19:55:01.389121 W | auth: invalid auth token: 
KGVyAgnXWOgFDFYl.114375
   2020-11-30 19:55:01.408058 W | auth: invalid auth token: 
EGSajOVclqMhLIqS.114317
   2020-11-30 19:55:01.424942 W | auth: invalid auth token: 
qtXATtyhLZJSGIfB.116433
   2020-11-30 19:55:01.480119 W | auth: invalid auth token: 
tqHxDVbjSKBVCDaS.114331
   2020-11-30 19:55:01.485725 W | auth: invalid auth token: 
siQhqOvdtRUXsHiW.116425
   2020-11-30 19:55:01.521545 W | auth: invalid auth token: 
xVPFwHcEMXScHhGZ.114338
   2020-11-30 19:55:01.529845 W | auth: invalid auth token: 
vqtBsVIGKtNCjUpb.114336
   2020-11-30 19:55:01.541498 W | auth: invalid auth token: 
XFqNUGjGYYsHhpWI.114345
   2020-11-30 19:55:01.580206 W | auth: invalid auth token: 
CiBhQUGpwiUPczNo.114314
   2020-11-30 19:55:01.580919 W | auth: invalid auth token: 
JVaWKqKTDxnhwZig.114351
   2020-11-30 19:55:01.600957 W | auth: invalid auth token: 
OIdYBazTuoTeRyhV.114320
   2020-11-30 19:55:01.628087 W | auth: invalid auth token: 
bSuTMTUaydeOSFPw.114311
   2020-11-30 19:55:01.650066 W | auth: invalid auth token: 
CUImlKqMxLllTDAG.116422
   2020-11-30 19:55:01.671630 W | auth: invalid auth token: 
MckpSvnXCslOBZzN.114350
   2020-11-30 19:55:01.730228 W | auth: invalid auth token: 
ZoCzVsZVnxKkXgkh.114323
   2020-11-30 19:55:01.809137 W | auth: invalid auth token: 
MNtikXEJuXkIOWVt.114382
   2020-11-30 19:55:01.840882 W | auth: invalid auth token: 
eLrJyzLIfDCCtLgS.114352
   2020-11-30 19:55:01.892522 W | auth: invalid auth token: 
mprbhPNbjByDExCW.116398
   2020-11-30 19:55:01.973658 W | auth: invalid auth token: 
MMKVaKJYLCPjtYra.114335
   2020-11-30 19:55:02.055462 W | auth: invalid auth token: 
eLrJyzLIfDCCtLgS.114352
   2020-11-30 19:55:02.313522 W | auth: invalid auth token: 
NcKCHbtvAvQTJWap.116409
   2020-11-30 19:55:02.348031 W | auth: invalid auth token: 
iQkxDrTcVsWzzBwz.114330
   2020-11-30 19:55:02.588947 W | auth: invalid auth token: 
THlTKQvjLFueVrsC.116408
   2020-11-30 19:55:02.620312 W | auth: invalid auth token: 
zEoNFjjLmZLFLujv.114328
   2020-11-30 19:55:02.637622 W | auth: invalid auth token: 
jmyLhQNCVaHCuESg.114312
   2020-11-30 19:55:02.760728 W | auth: invalid auth token: 
bloWjYzYVchKqYCb.114376
   ```
   
   I get package from  etcd serve and find this  packages:
   
![image](https://user-images.githubusercontent.com/15153469/100607748-57481080-3346-11eb-9067-731603410649.png)
   
   After I check the apisix code, I find this code can cause dead loop auth.
   ```
   function refresh_jwt_token(self)
       -- token exist and not expire
       -- default is 5min, we use 3min
       -- https://github.com/etcd-io/etcd/issues/8287
       if self.jwt_token and now() - self.last_auth_time < 60 * 3 then
           return true, nil
       end
   
       local opts = {
           body = {
               name         = self.user,
               password     = self.password,
           }
       }
       local res, err = _request_uri(self, 'POST',
                           choose_endpoint(self).full_prefix .. 
"/auth/authenticate",
                           opts, 5, true)    -- default authenticate timeout 5 
second
       if err then
           return nil, err
       end
   
       if not res or not res.body or not res.body.token then
           return nil, 'authenticate refresh token fail'
       end
   
       self.jwt_token = res.body.token
       self.last_auth_time = now()
   
       return true, nil
   end
   ```
   ref: 
https://github.com/api7/lua-resty-etcd/blob/master/lib/resty/etcd/v3.lua#L221
   ### Environment
   
   * apisix version (cmd: `apisix version`):
   * OS:
   
   ### Minimal test code / Steps to reproduce the issue
   1.
   2.
   3.
   
   ### What's the actual result? (including assertion message & call stack if 
applicable)
   
   
   
   ### What's the expected result?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org