AuroraTwinkle commented on code in PR #8: URL: https://github.com/apache/pulsar-java-contrib/pull/8#discussion_r1798809360
########## pcip/pcip-1.md: ########## @@ -0,0 +1,556 @@ +# PCIP-1: Distributed RPC framework implemented by the Pulsar client + +# Background knowledge + +## Request-Reply Synchronize Model + +In this model, the client sends a request and waits for a response from the server. The server receives the request, processes it, and sends back a response. This interaction pattern is fundamental in client-server communication and is crucial for synchronous operations where the client needs a response before proceeding. + +## Remote Procedure Call (RPC) + +RPC allows functions or procedures to be executed on a different machine from the client making the call, as if they were local. This method abstracts the complexities of network communication, allowing developers to focus on the business logic rather than the underlying network details. + +The implementation of RPC is usually based on the Request-Reply model. In this case: + +- The RPC client plays the role of the requester, calling a remote procedure as if it were sending a request message. +- The RPC server plays the role of the responder, receiving the request, executing the procedure, and returning the result as a reply message. + +## Current behavior of sending messages in Pulsar + +The current sending behavior of Pulsar is when the message is successfully published, that is, successfully persisted to the storage layer. The MessageId assigned to the published message by the broker is returned. + +## Analogies of message flow in RPC and Pulsar + +- In Pulsar, producer is equivalent to RPC Client. +- The RPC Client initiates a request like sending a message producer. +- The RPC Server side receives this request as if the consumer receives the message and then carries out customized processing, and finally ACKs the message. +- If this ACK request contains the result returned by the "server side" and is sent to the original producer(RPC Client). +- After receiving the results returned by the consumer(RPC Server), the producer(RPC Client) directly returns the content of the results. + +# Motivation + +As we known，Pulsar's current asynchronous publish-subscribe model serves well for decoupled message distribution, but it lacks a native mechanism for handling synchronous interactions typical of Remote Procedure Calls (RPC). +This request-reply model can greatly enhance the utility of Pulsar. We can then use Pulsar as a distributed RPC Framework. This also improves the fault tolerance of RPC calls. + +PIP-371 is also committed to building such a model. + +(https://github.com/apache/pulsar/pull/23143 and https://github.com/apache/pulsar/pull/23194) + +But we need to implement this distributed RPC framework in a way that does not intrude into the pulsar core library. +Therefore, we need to use two topics, one is the request topic and the other is the reply topic. The client side sends RPC requests to the request topic, the server side receives request message and performs customized processing, and finally sends them to the reply topic. The client receives the reply message and returns. + +### Why would we use Pulsar for this RPC call? + +Implement RPC using Apache Pulsar. Requests can be sent through a client, received by one or more servers and processed in parallel. Finally, the server returns all processing results after processing, and the client can perform summary and other operations after receiving them. +This proposal to achieve the function is request. Request and existing send function of pulsar can be mixed to same topic. This means that the user can choose, and the call to the server side (consumer) can be asynchronous or synchronous, which is controlled by the user flexibly. +You can directly use Pulsar's own delaying messages, that is, you can execute RPC regularly. +You can directly use Pulsar's own load balancing mechanism. +You can directly use Pulsar's own message consumption throttling mechanism. +You can directly use Pulsar's own expansion and contraction mechanism. +You can directly use Pulsar's own message call tracking, monitoring, and logging mechanisms. + +# Goals + +## In Scope + +- To implement a pulsar-rpc-client similar to pulsar-client, we can encapsulate request request as a `request message` Review Comment: This is a typo? ```suggestion - To implement a pulsar-rpc-client similar to pulsar-client, we can encapsulate request as a `request message` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
