I am glad to hear that there is another option. We can use flatbuffers as the codec instead of capnproto.
> as you are not using a standard RPC protocol, no possibility of internal tracing The protocol is extendable and tracing support can be added. Currently, I focus on performance most because performance is a good selling point. People will say, "APISIX + Java sidecar is faster than Spring Cloud Gateway, let's adopt it!". It is possible as Nginx is faster than Spring Cloud Gateway. People won't say, "APISIX + Java sidecar has better observability than Spring Cloud Gateway, let's adopt it!". As there are two processes than one, it is impossible to have better observability. If people don't even buy, there is no chance to compare observability. There is a trade-off between performance & observability. I want to improve the performance as much as we can, attract people to use it, then we can have a chance to improve the observability. Sheng Wu <wu.sheng.841...@gmail.com> 于2021年4月16日周五 下午8:22写道： > > I want to remind you, as you are not using a standard RPC protocol, no > possibility of internal tracing. So how should we collect > metrics/performance-data of this cross-processes communication? > And it is Java lib(maybe others have the same issue), > https://github.com/capnproto/capnproto-java/graphs/contributors, has > very little activities. I am not sure how healthy their community is. > > Have you taken a look at this? https://google.github.io/flatbuffers/ > It is used in Istio, and SkyWalking used this in mesh scenarios. This > protocol has a very good performance especially when some fields are > not required to deserialize, which from my understanding, is a very > case when running the sidecar process. > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > Zexuan Luo <spacewan...@apache.org> 于2021年4月16日周五 下午6:19写道： > > > > It will hold long connections. > > > > Need to clarify that the "long connection" doesn't conflict with the > > "request / response mode". For example, gRPC's unary mode can work on > > a long connection. > > > > bzp2010 <bzp2...@apache.org> 于2021年4月16日周五 下午6:04写道： > > > > > > Oh, I get it. > > > Will it use a long connection based on UNIX socket to transfer data? Or is > > > it a request / response mode similar to HTTP? > > > > > > Zexuan Luo <spacewan...@apache.org> 于2021年4月16日周五 下午5:53写道： > > > > > > > This is why we choose capnproto rather than protobuf. The > > > > encode/decode performance is important, so we choose capnproto which > > > > has a better performance than protobuf. > > > > > > > > bzp2010 <bzp2...@apache.org> 于2021年4月16日周五 下午4:44写道： > > > > > > > > > > Hi, > > > > > I have some questions about this proposal. > > > > > > > > > > 1. If using similar C/S mode, whether the encoding and decoding > > > > > process > > > > > will affect the performance. > > > > > > > > > > 2.Can we use protobuf, a more popular encoding and decoding toolkit? > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > *发件人: *Zexuan Luo <spacewan...@apache.org> > > > > > *发送时间: *2021年4月16日 14:52 > > > > > *收件人: *dev@apisix.apache.org > > > > > *主题: *[Proposal] support using other languages to write plugin for > > > > > APISIX > > > > > > > > > > > > > > > > > > > > 1. Background > > > > > > > > > > > > > > > > > > > > APISIX currently only supports writing plugins in Lua. If other > > > > > > > > > > languages are supported, it will greatly broaden the APISIX ecosystem > > > > > > > > > > and user base. > > > > > > > > > > > > > > > > > > > > 2. Solution > > > > > > > > > > > > > > > > > > > > Since WASM is not yet mature, we consider implementing it through > > > > > local > > > > IPC. > > > > > > > > > > > > > > > > > > > > For the sake of discussion, the following will assume that the plugin > > > > > > > > > > is written in Java. However, in practice, our solution can be > > > > > > > > > > interfaced with other languages. > > > > > > > > > > > > > > > > > > > > 2.1 Terminology > > > > > > > > > > > > > > > > > > > > Several terms are defined here. > > > > > > > > > > > > > > > > > > > > Plugin Runner: The service that runs the plugin, written in the same > > > > > > > > > > language as the plugin. In the first version, we assume that there > > > > > > > > > > will be only one Plugin Runner. > > > > > > > > > > > > > > > > > > > > 2.2 Plugin Runner lifecycle > > > > > > > > > > > > > > > > > > > > To simplify user operation and reduce the difficulty of upgrading, > > > > > > > > > > Plugin Runner is managed by APISIX. > > > > > > > > > > > > > > > > > > > > APISIX starts the Plugin Runner when it starts and ends it when it > > > > > > > > > > ends. if the Plugin Runner quits in the middle, APISIX will restart it > > > > > > > > > > automatically. > > > > > > > > > > > > > > > > > > > > 2.3 Timing of APISIX communication with Plugin Runner > > > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > Router ----> Global Plugin (written in Lua) ---> Ext Plugin > > > > > > > > > > (ext-plugin-pre-req) ----> Lua Plugin (Router) > > > > > > > > > > ---> Ext plugin (ext-plugin-post-req) ---> Upstream > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > Running the Ext Plugin in the Global Plugin is not supported at this > > > > > > > > > > time, as the global logic can be executed uniformly in the Plugin > > > > > > > > > > Runner. > > > > > > > > > > > > > > > > > > > > Running Ext Plugin after getting an upstream response is not supported > > > > > > > > > > at this time. We can support it later with a buffering response. > > > > > > > > > > > > > > > > > > > > ext-plugin-pre runs before all non-global Lua plugins, and > > > > > > > > > > ext-plugin-post runs after all non-global Lua plugins. > > > > > > > > > > > > > > > > > > > > 2.4 How APISIX communicates with Plugin Runner > > > > > > > > > > > > > > > > > > > > APISIX communicates with Plugin Runner through a unix socket. The > > > > > > > > > > communication protocol is as follows. > > > > > > > > > > > > > > > > > > > > 2.4.1 Communication format > > > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > 1 byte of type + 3 bytes of length + data > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > The type can be 0 ~ 7, and the length can be [0, 8M). data length is > > > > > > > > > > determined by length. > > > > > > > > > > Since Ext Plugin usually does not exchange too much data, 8M should be > > > > > > > > > > enough. The reason for taking 4 bytes is to keep the header small > > > > > > > > > > enough to be read efficiently. > > > > > > > > > > > > > > > > > > > > The current type takes the following values. > > > > > > > > > > > > > > > > > > > > 0 means error > > > > > > > > > > 1 means prepare_conf > > > > > > > > > > 2 means http_req_call > > > > > > > > > > > > > > > > > > > > The data is serialized in capnproto, a binary serialization format. > > > > > > > > > > > > > > > > > > > > capnproto is supported by many programming languages: > > > > > > > > > > https://capnproto.org/otherlang.html > > > > > > > > > > > > > > > > > > > > The advantages of choosing capnproto are. > > > > > > > > > > 1. focus on serialization performance > > > > > > > > > > 2. partial deserialization support, so that decode can be done only > > > > > > > > > > when it is needed > > > > > > > > > > > > > > > > > > > > 2.4.2 Communication steps > > > > > > > > > > > > > > > > > > > > Each ext plugin will have the following configuration. > > > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > { > > > > > > > > > > "conf": [ > > > > > > > > > > { > > > > > > > > > > "name": "configuration name", > > > > > > > > > > "value": "configuration value" > > > > > > > > > > } > > > > > > > > > > ], > > > > > > > > > > "extra_info": [ > > > > > > > > > > ... > > > > > > > > > > ] > > > > > > > > > > } > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > conf can be used to set the execution configuration of the > > > > > > > > > > plugin-related requests inside Plugin Runner. > > > > > > > > > > > > > > > > > > > > The default data sent to Plugin Runner is only the most common > > > > > > > > > > information. If you want additional information, you need to declare > > > > > > > > > > it in extra_info beforehand. > > > > > > > > > > > > > > > > > > > > To save communication costs, conf is sent separately. > > > > > > > > > > > > > > > > > > > > 1. APISIX will check if conf has a corresponding token in the local > > > > cache. > > > > > > > > > > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner > > > > > > > > > > to cache the conf and return a token. > > > > > > > > > > (Note that Plugin Runner's cache time needs to be longer than APISIX's > > > > > > > > > > cache time.) > > > > > > > > > > 3. APISIX sends an http_req_call request to Plugin Runner. > > > > > > > > > > 4. Plugin Runner executes the request and returns a response to > > > > > APISIX. > > > > > > > > > > 5. APISIX processes the request based on the response > > > > > > > > > > > > > > > > > > > > 2.4.3 proto > > > > > > > > > > > > > > > > > > > > Refer to https://capnproto.org/language.html > > > > > > > > > > > > > > > > > > > > The following is the proto for error > > > > > > > > > > > > > > > > > > > > response > > > > > > > > > > ``` > > > > > > > > > > enum ErrorCode { > > > > > > > > > > BAD_REQUEST @0; # Plugin Runner can't understand APISIX > > > > > > > > > > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request > > > > > > > > > > CONF_TOKEN_NOT_FOUND @2; > > > > > > > > > > } > > > > > > > > > > struct ErrorResp { > > > > > > > > > > Code @0 :ErrorCode; > > > > > > > > > > } > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > The following is the proto of prepare_conf > > > > > > > > > > > > > > > > > > > > request > > > > > > > > > > ``` > > > > > > > > > > struct PrepareConfReq { > > > > > > > > > > conf @0 :List(Pair); > > > > > > > > > > } > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > Response > > > > > > > > > > ``` > > > > > > > > > > struct PrepareConfResp { > > > > > > > > > > conf_token @0 :UInt32; > > > > > > > > > > } > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > Here is the proto for http_req_call > > > > > > > > > > > > > > > > > > > > request > > > > > > > > > > ``` > > > > > > > > > > struct Pair { > > > > > > > > > > name @0 :Text; > > > > > > > > > > value @1 :Text; > > > > > > > > > > } > > > > > > > > > > struct PairData { > > > > > > > > > > name @0 :Text; > > > > > > > > > > value @1 :Data; > > > > > > > > > > } > > > > > > > > > > enum Method { > > > > > > > > > > GET @0; > > > > > > > > > > ... > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > struct HTTPReqCallReq { > > > > > > > > > > id @0 :UInt32; > > > > > > > > > > src_ip @1 :Data; > > > > > > > > > > > > > > > > > > > > method @2 :Method; > > > > > > > > > > > > > > > > > > > > path @3 :Text; > > > > > > > > > > args @4 :List(Pair); > > > > > > > > > > headers @5 :List(Pair); > > > > > > > > > > > > > > > > > > > > conf_token @6 :UInt32; > > > > > > > > > > > > > > > > > > > > extra_info @7 :List(PairData); > > > > > > > > > > } > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > Response > > > > > > > > > > ``` > > > > > > > > > > struct HTTPReqCallResp { > > > > > > > > > > id @0 :UInt32; > > > > > > > > > > > > > > > > > > > > struct Stop { > > > > > > > > > > status @0 :UInt16; > > > > > > > > > > headers @1 :List(Pair); > > > > > > > > > > body @2 :Data; > > > > > > > > > > } > > > > > > > > > > struct Rewrite { > > > > > > > > > > path @0 :Text; > > > > > > > > > > headers @1 :List(Pair); > > > > > > > > > > # Note that args are modified in full. > > > > > > > > > > # Either empty, meaning no args need to be moved > > > > > > > > > > # or the entire modified args, not the incrementally changed > > > > parts > > > > > > > > > > args @2 :List(Pair); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > # What needs to be done when the response is received action > > > > > > > > > > action :union { > > > > > > > > > > # Do nothing > > > > > > > > > > continue @1 :Void; > > > > > > > > > > # Equivalent to core.response.exit(status, body), allowing > > > > > > > > > > additional headers to be set > > > > > > > > > > stop @2 :Stop; > > > > > > > > > > # Rewrite the request > > > > > > > > > > rewrite @3 :Rewrite; > > > > > > > > > > } > > > > > > > > > > } > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > 2.4.4 Error handling > > > > > > > > > > > > > > > > > > > > Logging and returning 503 error codes > > > > > > > > > > > > > > > > > > > > 2.4.5 Environment variables > > > > > > > > > > > > > > > > > > > > APISIX configures the Plugin Runner with a number of environment > > > > > > > > > > variables when it is started. > > > > > > > > > > > > > > > > > > > > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to > > > > listen to > > > > > > > > > > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer > > > > than > > > > > this > > > >
