My in-process "messaging" library now has all the basic features I wanted, so it's time to add cluster communication.
I would have liked to use something like zmq or nanomsg, but it seems my use-case is not covered by those. I've decribed it lower down; please correct me if I'm wrong. So, I checked into the [nativesockets](https://nim-lang.org/docs/nativesockets.html) module, but this one does not seem to support "accept", which I need for a "server socket" (a socket accepting client connections). Than, I also looked at the [net](https://nim-lang.org/docs/net.html) module, which has "listen" and "accept", but no "select", which I need for scalability reason (I don't want to use the one-thread-per-client model). As this state of affairs doesn't make sense to me (in particular, nativesockets supporting "listen", but not "accept"), I must assume that I'm missing something... Maybe you can't use "select" together with SSL, and that is why it's not available in the net module? _So, can someone tell me how to either use "select" in "net" or "accept" in "nativesockets"?_ * * * Here is my _use-case_: 1. Each cluster node will have a fixed-size pool of worker threads; presumably pinned to CPU cores. 2. "Entities" are assigned to a specific thread of a specific node, using the thread-local heap. 3. Entities "exchange messages", both across threads, and across the cluster. 4. Each message contains the node-ID, thread-ID and entity-ID of the sender and reciever. 5. Worker threads do not ever do any blocking actions; in particular, no IO. 6. IO must therefore be handled by separate threads from the worker pool. 7. But the threads doing the IO cannot "process" the messages; that's the worker thread's job. 8. Therefore, it must be possible to read new incoming messages, without answering previous messages. 9. Each cluster node has one "server socket", accepting connections from other nodes. 10. When a node boots up, it (somehow?) queries the running nodes, and connects to each of them. 11. So, we have a fully-connected node graph. I'm aiming for 10 to 100 nodes. 12. Once connected, the "direction" of the connection becomes meaningless, since communication is "symmetric". 13. I'm assuming a relatively "stable" cluster (hardware cluster nodes, not per-hour VMs), so new "connections" should be rare. 14. Code should run (at least) on Linux, and each cluster node is assumed to be "all of the same hardware/OS". Given all of that, I'm looking for a model with two networking threads: 1. Does "selectRead"s, transfer incoming messages to the appropriate worker threads, and occasionally checks for new connections ("accept"). 2. Fetches (queued) outgoing messages, and sends them ("selectWrite"). I'm open to any solution that allows me to do this (efficiently) in Nim. Ideal solution would also support encrypted communication, since the cluster is going to be "in the cloud". #8 means I cannot use "pairs" from nanomsg or zmq, for example. FYI, I've written the networking code, and load-balancer, for a 20 node cluster that has been productive for over 10 years, so I have experience in sockets (and all the nasty networking issues that come with them) but, that was all in Java. Since I know how much of a pain it is to cover all corner cases, I was hoping to use a "higher level" solution.
