RedemptionC opened a new issue #2045: URL: https://github.com/apache/apisix-dashboard/issues/2045
## context this is a subtask of a project from open summer 2021,[here](https://summer.iscas.ac.cn/#/org/prodetail/210050252?lang=en) for full detail. dashboard related: >Add a new command line tool to the Apache APISIX Dashboard. When the tool is executed, the latest data will be obtained from the ETCD in real time, and compared with the local cache item by item to verify whether there is data out of sync. ## what we really need to do for this task(IMO) I used the dashboard and read the source code,I found that what we need to do is to check whether items in cache(sync.Map) are the same as those in ETCD And we need to add a cobra command to do such things ## possible solution **use the migrate/export API** this API will return all items in cache,we can get those info from it,and then traverse them,compare them to corresponding items in ETCD and here is a draft implementation(which only checks the consumers): ```go package store import ( "context" "encoding/json" "fmt" "github.com/apisix/manager-api/internal/conf" "github.com/apisix/manager-api/internal/core/entity" "github.com/apisix/manager-api/internal/core/storage" "github.com/apisix/manager-api/internal/log" "io/ioutil" "net/http" ) type DataSet struct { Counsumers []*entity.Consumer `json:"Counsumers,omitempty"` Routes []*entity.Route `json:"Routes,omitempty"` Services []*entity.Service `json:"Services,omitempty"` SSLs []*entity.SSL `json:"SSLs,omitempty"` Upstreams []*entity.Upstream `json:"Upstreams,omitempty"` Scripts []*entity.Script `json:"Scripts,omitempty"` GlobalPlugins []*entity.GlobalPlugins `json:"GlobalPlugins,omitempty"` PluginConfigs []*entity.PluginConfig `json:"PluginConfigs,omitempty"` } func AreEqualJSON(s1, s2 string) (bool, error) { fmt.Printf("compare:\ncache: %s\nand\netcd: %s\n", s1, s2) var o1 interface{} var o2 interface{} var err error err = json.Unmarshal([]byte(s1), &o1) if err != nil { return false, fmt.Errorf("Error mashalling string 1 :: %s", err.Error()) } err = json.Unmarshal([]byte(s2), &o2) if err != nil { return false, fmt.Errorf("Error mashalling string 2 :: %s", err.Error()) } return reflect.DeepEqual(o1, o2), nil } func CacheVerify() { //fmt.Println("in func cacheVerify") rsp, err := http.Get("http://localhost:9000/apisix/admin/migrate/export";) if err != nil { fmt.Println("get result from migrate/export failed") return } data, err := ioutil.ReadAll(rsp.Body) if err != nil { fmt.Println(err) } fmt.Printf("len of data is %d\n", len(data)) var responseObj DataSet err = json.Unmarshal(data[:len(data)-4], &responseObj) if err != nil { fmt.Println(err) return } //fmt.Println(responseObj) conf.InitConf() consumers := responseObj.Counsumers for _, v := range consumers { s, err := json.Marshal(v) if err != nil { fmt.Printf("json marsharl failed : %v\n", err) continue } cacheValue := string(s) key := fmt.Sprintf("/apisix/consumers/%s", v.Username) if err := storage.InitETCDClient(conf.ETCDConfig); err != nil { log.Errorf("init etcd client fail: %w", err) return } val, err := storage.GenEtcdStorage().Get(context.TODO(), key) if err != nil { fmt.Printf("etcd get failed %v \n", err) continue } etcdValue := val cmp, err := AreEqualJSON(cacheValue, etcdValue) if err != nil { fmt.Printf("compare json failed %v\n", err) } fmt.Printf("cache value and etcd value for key(%s) are the same : %v\n", key, cmp) } } ``` here is the output from cli: **it seems to be working well** ``` ❯ go run main.go cache-verify len of data is 1735 compare: cache: {"username":"dashboard","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627746151,"update_time":1627746151} and etcd: {"username":"dashboard","plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627746151,"create_time":1627746151} cache value and etcd value for key(/apisix/consumers/dashboard) are the same : true compare: cache: {"username":"dumb","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627906781,"update_time":1627906781} and etcd: {"plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627906781,"create_time":1627906781,"username":"dumb"} cache value and etcd value for key(/apisix/consumers/dumb) are the same : true compare: cache: {"username":"john","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627731085,"update_time":1627738858} and etcd: {"username":"john","plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627738858,"create_time":1627731085} cache value and etcd value for key(/apisix/consumers/john) are the same : true compare: cache: {"username":"red","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627738877,"update_time":1627738933} and etcd: {"username":"red","plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627738933,"create_time":1627738877} cache value and etcd value for key(/apisix/consumers/red) are the same : true compare: cache: {"username":"test","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627908039,"update_time":1627908039} and etcd: {"plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627908039,"create_time":1627908039,"username":"test"} cache value and etcd value for key(/apisix/consumers/test) are the same : true compare: cache: {"username":"andy","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627739045,"update_time":1627744978} and etcd: {"username":"andy","plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627744978,"create_time":1627739045} cache value and etcd value for key(/apisix/consumers/andy) are the same : true compare: cache: {"username":"apisix","plugins":{"key-auth":{"key":"key-of-john"}},"create_time":1627746172,"update_time":1627746172} and etcd: {"username":"apisix","plugins":{"key-auth":{"key":"key-of-john"}},"update_time":1627746172,"create_time":1627746172} cache value and etcd value for key(/apisix/consumers/apisix) are the same : true ``` ## thoughts on another "possible" solution I once thought another solution also reasonable: we don't rely on running dashboard,we just initialize everything like function manageapi() did then we traverse items in cache,compare them to corresponding items in ETCD **but I recently found this one not ressonable: when we init everything,we read items from etcd to cache,if we compare them to items in ETCD,how could them be different?** ## things confuse me * I'm not really familiar with this tool's use case,can it rely on a existing running dashboard(manager API),i.e.,first we run manager API,then we run this tool? * is the proposed solution reasonable ? * when I code the draft implementation,I found that I need to traverse the data of a struct(the dataSet),is there a way to do this,instead of hard-coded style? * is there better way to do this? Any advice will be greatly appreciated! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
