Hi, there Below is the proposal about how to support multiple data centers for Apache APISIX. Please review it and your advice will be appreciated.
Background So far, APISIX only supports fetching configurations like Route, Upstream from ETCD, although it’s capable of integrating a couple of service registries (e.g. Consul, Nacos) to discover backend service endpoints (aka upstream nodes) dynamically. For some users, ETCD is not their optimal solution as they may already have another similar data center or they don’t master the skill for ETCD cluster maintenance. In such a case, introducing APISIX to their platform will be stumbling, even letting them give up on using it. So from this perspective, making APISIX decouple from ETCD is important, it also enriches the ecosystem of APISIX. Self Adjustment The most intuitive way is evolving APISIX itself, letting it as the client of the target data center, so no extra components will be deployed, what administrators need to do is configuring the data center in the configuration of APISIX, including address, authentication and so on. That’s good but the overheads to change APISIX is huge, firstly, the protocol used in each data center might vary, considering the weak ecosystem of OpenResty/Nginx, we may need to implement the data center SDK manually, and it will be tracky for some multiplexing protocols (due to the connection pool model of OpenResty); On the other hand, The APISIX internal is tightly coupled with ETCD, it’s not easy to decouple them from the code level, what’s more, these part of codes now are very stable, changing them might influence the stability of APISIX. Agent Mode > All Problems in Computer Science can be solved by another level of > indirection. Since it’s not easy to support this from APISIX itself, why not introducing another component, it supports to communicate with any data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that it can pretend itself as the ETCD cluster from APISIX’s point of view. With that a component, there is nothing that should be done in APISIX itself (You can see the architecture figure in the attachments). Agents can be deployed in any place as the networking is connected for both the APISIX and data centers sides. It should be designed as stateless, so APISIX can configure multiple agent instances and the high availability can be sure. It can be even deployed for each APISIX instance, that’s the famous sidecar mode. Problems need to be addressed The agent component has to solve a series of problems so it can achieve its mission. Firstly, it should implement the server side of ETCD V3 APIs. Fortunately, only minor parts of features are used by APISIX, so the implementation won’t be too complicated, also, since APISIX relis on the ETCD gRPC gateway, the agent should also support it, in general, that’s not too difficult as the gRPC ecosystem is strong. Secondly, the internal design of the agent should be modular as soon as possible, so once we need to support a new data center, the development would be easy. Of course that’s another topic and will not be detailed in this post. Best regards Chao Zhang https://github.com/tokers
