bzp2010 commented on code in PR #7028: URL: https://github.com/apache/apisix/pull/7028#discussion_r870915666
########## docs/en/latest/pubsub.md: ########## @@ -0,0 +1,244 @@ +--- +title: PubSub +keywords: + - APISIX + - Pub-Sub +description: This document contains information about the Apache APISIX pubsub framework. +--- + +<!-- +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--> + +## What is Pub-Sub + +Publish-subscribe is a messaging paradigm in which message producers do not send messages directly to message consumers, but are relayed by a specific broker that caches messages sent by producers and then actively pushes them to subscribed consumers or pulls them by consumers. This pattern is often used in system architectures for system decoupling or to handle high traffic scenarios. + +In Apache APISIX, the most common scenario is for handling north-south traffic from the server to the client. If we can combine it with a publish-subscribe scenario, we can achieve more powerful features, such as real-time collaboration on online documents, online games, etc. + +## Architecture + + + +Currently, Apache APISIX supports WebSocket communication with the client, which can be any application that supports WebSocket, with Protocol Buffer as the serialization mechanism, see the [protocol definition](../../../apisix/pubsub.proto). + +## Supported messaging systems + +- [Aapche Kafka](pubsub/kafka.md) + +## How to support other messaging systems + +An extensible pubsub module is implemented in Apache APISIX, which is responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and through which new messaging system support can be simply added. + +### Basic Steps + +- Add new commands and response body definitions to `pubsub.proto` +- Add a new option to the `scheme` configuration item in upstream +- Add a new `scheme` judgment branch to `http_access_phase` +- Implement the required message system instruction processing functions +- Optional: Create plugins to support advanced configurations of this messaging system + +### the example of Apache Kafka + +#### Add new commands and response body definitions to `pubsub.proto` + +The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`. + +First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`. + +```protobuf +message CmdKafkaFetch { + string topic = 1; + int32 partition = 2; + int64 offset = 3; +} + +message PubSubReq { + int64 sequence = 1; + oneof req { + CmdKafkaFetch cmd_kafka_fetch = 31; + // more commands + }; +} +``` + +Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`. + +```protobuf +message KafkaFetchResp { + repeated KafkaMessage messages = 1; +} + +message PubSubResp { + int64 sequence = 1; + oneof resp { + ErrorResp error_resp = 31; + KafkaFetchResp kafka_fetch_resp = 32; + // more responses + }; +} +``` + +#### Add a new option to the `scheme` configuration item in upstream + +Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`. + +```lua +scheme = { + enum = {"grpc", "grpcs", "http", "https", "tcp", "tls", "udp", "kafka"}, + -- other +} +``` + +#### Add a new `scheme` judgment branch to `http_access_phase` + +Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function. Review Comment: fixed ########## docs/en/latest/pubsub.md: ########## @@ -0,0 +1,244 @@ +--- +title: PubSub +keywords: + - APISIX + - Pub-Sub +description: This document contains information about the Apache APISIX pubsub framework. +--- + +<!-- +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--> + +## What is Pub-Sub + +Publish-subscribe is a messaging paradigm in which message producers do not send messages directly to message consumers, but are relayed by a specific broker that caches messages sent by producers and then actively pushes them to subscribed consumers or pulls them by consumers. This pattern is often used in system architectures for system decoupling or to handle high traffic scenarios. + +In Apache APISIX, the most common scenario is for handling north-south traffic from the server to the client. If we can combine it with a publish-subscribe scenario, we can achieve more powerful features, such as real-time collaboration on online documents, online games, etc. + +## Architecture + + + +Currently, Apache APISIX supports WebSocket communication with the client, which can be any application that supports WebSocket, with Protocol Buffer as the serialization mechanism, see the [protocol definition](../../../apisix/pubsub.proto). + +## Supported messaging systems + +- [Aapche Kafka](pubsub/kafka.md) + +## How to support other messaging systems + +An extensible pubsub module is implemented in Apache APISIX, which is responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and through which new messaging system support can be simply added. Review Comment: fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
