On 03/20/2015 06:54 AM, Robert Norris wrote:
> Hi all,
> I've got a prototype for a Prosody feature that I want/need. I'm keen to
> get your input on it.
> What I want to do is distribute users for a single domain across
> multiple Prosody instances while having them look like a single server
> on the outside. The reasons for this are partly to do with load - I'm
> preparing for a time where there are thousands of users online - but
> also to fit neatly into our existing infrastructure, where we spread
> email users across multiple servers. I like the idea of mirroring our
> email architecture and doing redundancy the same way.
> Note that I'm expressly not talking about real live clustering, where a
> server can fail and the user remains connected. A Prosody instance (or
> its underlying machine) can fail and the sessions will be disconnected.
> That's totally fine with me - most clients will quietly reconnect
> without issue.

Great. I'm was working on that already so maybe we can share ideas. Be
aware that this work was done several months ago for 0.9.4 and I had to
stop working on this due to work load.

> It seems that all that's necessary to make this happen is to hook into
> the stanza delivery code. If we have a session for a user on the current
> server, we deliver it to them. If not, we do a lookup (via an external
> daemon) to see where the "home" server for the user is. If we find one,
> we pass the stanza over to that server and let it deliver it. If not, we
> continue as normal.
> So I have a very rough prototype in this three plugins:
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard.lua
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard_client.lua
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard_server.lua
> mod_shard is the shard lookup hook. mod_shard_client and
> mod_shard_server are the bits that pass stanzas between servers. They're
> not especially interesting, and are mostly hacked from mod_component and
> mod_component_client.

I'd implemented a module to store the info of what jid is where over
redis here:


You just have to ask redis_sessions.get(jid) to know where to send the

> I'm assuming a very controlled environment here - secure networks (no
> auth required between shards), a user->shard lookup daemon, common
> storage and auth and a proxy that can get incoming user connections to
> the right shard (in my case that's nginx-xmpp). I'm expecting s2s to be
> covered by a par of Prosody instances that accept stanzas and pass them
> to the right shard. I should be able to run multiple of those because
> they're basically stateless.

But did you try that?

My approach was to extend mod_s2s as mod_s2sc to connect the nodes:


And this module is the one that hooks and decides if the stanza needs to be
routed somewhere else:


> So that's the whole description. Hopefully it all made sense! What I'm
> looking for is some feedback on whether the way I've implemented this
> makes sense (half-arsed prototype code notwithstanding) and any thoughts
> about what user-visible differences in XMPP it might present (stanza
> delivery order comes to mind). And any other thoughts you might have!

So I think we are trying to do the same thing. I have to point that I didn't
finish to polish this and for sure it could not work as it is right now. But
you can get the idea.

Looking forward your comments
Victor Seva

You received this message because you are subscribed to the Google Groups 
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to