GitHub user spetz created a discussion: Bootstrap server - get metadata request to support future clustering
In the near future, we will start working on the clustering feature, which will also require changes in SDKs. One of the first requirements will be to allow the clients to discover the cluster nodes so that the actual leader can be resolved and connected to. The overall idea and design are rather simple and could be already added transparently to the existing Iggy server (and its SDKs) without having the clustering in place yet. Let's say that we could introduce a new command, e.g., `GetMetadata`, which would return as the response the cluster details, for example, some unique ID, and most importantly, the list of all nodes (IP address, hostname, etc.). Of course, for the time being, there'd always be just a single node (leader). When the client connects to the Iggy server, once the connection is established and authentication is completed, it should send `GetMetadata` and then connect to the returned leader node to perform all the further actions, such as producing/consuming the messages and so on. If for any reason, the leader becomes unavailable, the client should, once again, invoke `GetMetadata` and connect to the next leader node (of course, with clustering in place, such internal cluster updates will also be notified via particular events) - this should be done internally by the SDK (as part of the existing reconnection mechanism). >From now on, the client should accept the list of Iggy servers to connect to >(not just a single server), and similarly, when using the connection string >approach, the servers could be comma-separated (which is pretty much an >industry standard). By default, each Iggy server also becomes the bootstrap server (since it's going to be aware of the whole cluster and all its nodes), yet from the client perspective, it only needs to specify 2 or 3 servers (even if there are tens of nodes), as `GetMetadata` will return the cluster details anyway. What we could currently implement in both server and client is the following: - Introduce a new `GetMetadata` command (and response) along with the schema and (de)serialization - Implement the handler on the server side to return the cluster details (just a list of a single node for now) - Add a new `get_metadata()` command to the SDK (in case someone would like to invoke it manually) - When connecting the client to the server, first authenticate with the valid credentials, and then invoke `GetMetadata` - Pick the leader node and connect to it (if it's already connected to, ignore it - which will be the case for now) - In case of reconnection, automatically fetch the latest metadata and connect to the new leader - Ensure that the client accepts a list of servers to connect to (instead of a single record) As mentioned before, this will be a fully transparent change - there's no requirement (for now) to even use `GetMetadata` and the described flow. Server and all the existing SDKs can function without any changes to them, since we still operate on a single node, however, I think that we could already work on it, and add it to the next release (in the coming weeks). GitHub link: https://github.com/apache/iggy/discussions/2133 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
