Hi Rob,

On 20 March 2015 at 05:54, Robert Norris <r...@fastmail.fm> wrote:
> What I want to do is distribute users for a single domain across multiple
> Prosody instances while having them look like a single server on the
> outside. The reasons for this are partly to do with load - I'm preparing for
> a time where there are thousands of users online - but also to fit neatly
> into our existing infrastructure, where we spread email users across
> multiple servers. I like the idea of mirroring our email architecture and
> doing redundancy the same way.

Makes sense.

> Note that I'm expressly not talking about real live clustering, where a
> server can fail and the user remains connected. A Prosody instance (or its
> underlying machine) can fail and the sessions will be disconnected. That's
> totally fine with me - most clients will quietly reconnect without issue.

Agreed. I should note that for our initial clustering implementation
we're not aiming for live migration of users between nodes either, I
don't think many people need this feature compared to the number of
people who just want availability and load balancing.

> It seems that all that's necessary to make this happen is to hook into the
> stanza delivery code. If we have a session for a user on the current server,
> we deliver it to them. If not, we do a lookup (via an external daemon) to
> see where the "home" server for the user is. If we find one, we pass the
> stanza over to that server and let it deliver it. If not, we continue as
> normal.

Yes, this is the right approach.

> So I have a very rough prototype in this three plugins:
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard.lua
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard_client.lua
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard_server.lua

I understand the code is rough, so I haven't really "reviewed" it -
but you're definitely going in the right direction. Comments below...

> I'm assuming a very controlled environment here - secure networks (no auth
> required between shards), a user->shard lookup daemon, common storage and
> auth and a proxy that can get incoming user connections to the right shard
> (in my case that's nginx-xmpp). I'm expecting s2s to be covered by a par of
> Prosody instances that accept stanzas and pass them to the right shard. I
> should be able to run multiple of those because they're basically stateless.

Right, what we can and can't assume drastically affects the complexity
of clustering, so this is where things get interesting.

Auth between nodes: easily done, but fine if you don't need it.

The shard lookup daemon. This would be a problem with the current
design. XMPP requires in-order processing and delivery of stanzas. If
you have to delay a stanza based on a call to an out-of-process
third-party, you're going to lose this unless you block the whole
session until you receive the result. There isn't currently a way to
do this in Prosody 0.9 (but there is in 0.10, with the async work).

Alternatively you could assign users to shards based on a hash or some
other means that can be determined immediately without an external
lookup. If you wanted to get fancy you could use "consistent hashing"
to minimize disruption when adding/removing nodes to the cluster.

For the s2s nodes, make sure you set dialback_secret to some secret
static string (the same on both nodes).

> So that's the whole description. Hopefully it all made sense! What I'm
> looking for is some feedback on whether the way I've implemented this makes
> sense (half-arsed prototype code notwithstanding) and any thoughts about
> what user-visible differences in XMPP it might present (stanza delivery
> order comes to mind). And any other thoughts you might have!

Hopefully the above is a good start.

There are some other things to consider if you haven't already. For
example, off the top of my head... what shard will be responsible for
offline users? Various things happen with offline users, including
handling roster state changes (e.g. in response to subscription
authorization) and the storage of offline messages. If every user has
a fixed "home" shard, this is easy enough to handle.


You received this message because you are subscribed to the Google Groups 
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.

Reply via email to