On 03/20/2015 06:54 AM, Robert Norris wrote:
> Hi all,
>  
> I've got a prototype for a Prosody feature that I want/need. I'm keen to
> get your input on it.
>  
> What I want to do is distribute users for a single domain across
> multiple Prosody instances while having them look like a single server
> on the outside. The reasons for this are partly to do with load - I'm
> preparing for a time where there are thousands of users online - but
> also to fit neatly into our existing infrastructure, where we spread
> email users across multiple servers. I like the idea of mirroring our
> email architecture and doing redundancy the same way.
>  
> Note that I'm expressly not talking about real live clustering, where a
> server can fail and the user remains connected. A Prosody instance (or
> its underlying machine) can fail and the sessions will be disconnected.
> That's totally fine with me - most clients will quietly reconnect
> without issue.

Great. I'm was working on that already so maybe we can share ideas. Be
aware that this work was done several months ago for 0.9.4 and I had to
stop working on this due to work load.

> It seems that all that's necessary to make this happen is to hook into
> the stanza delivery code. If we have a session for a user on the current
> server, we deliver it to them. If not, we do a lookup (via an external
> daemon) to see where the "home" server for the user is. If we find one,
> we pass the stanza over to that server and let it deliver it. If not, we
> continue as normal.
>  
> So I have a very rough prototype in this three plugins:
>  
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard.lua
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard_client.lua
> https://github.com/robn/prosody/blob/shard/plugins/mod_shard_server.lua
>  
> mod_shard is the shard lookup hook. mod_shard_client and
> mod_shard_server are the bits that pass stanzas between servers. They're
> not especially interesting, and are mostly hacked from mod_component and
> mod_component_client.

I'd implemented a module to store the info of what jid is where over
redis here:

https://github.com/sipwise/prosody/blob/master/plugins/mod_sipwise_redis_sessions.lua

You just have to ask redis_sessions.get(jid) to know where to send the
stanza

> I'm assuming a very controlled environment here - secure networks (no
> auth required between shards), a user->shard lookup daemon, common
> storage and auth and a proxy that can get incoming user connections to
> the right shard (in my case that's nginx-xmpp). I'm expecting s2s to be
> covered by a par of Prosody instances that accept stanzas and pass them
> to the right shard. I should be able to run multiple of those because
> they're basically stateless.

But did you try that?

My approach was to extend mod_s2s as mod_s2sc to connect the nodes:

https://github.com/sipwise/prosody/blob/master/core/s2scmanager.lua
https://github.com/sipwise/prosody/tree/master/plugins/mod_s2s_cluster
https://github.com/sipwise/prosody/blob/master/plugins/mod_s2sc_dialback.lua

And this module is the one that hooks and decides if the stanza needs to be
routed somewhere else:

https://github.com/sipwise/prosody/blob/master/plugins/mod_sipwise_cluster.lua

> So that's the whole description. Hopefully it all made sense! What I'm
> looking for is some feedback on whether the way I've implemented this
> makes sense (half-arsed prototype code notwithstanding) and any thoughts
> about what user-visible differences in XMPP it might present (stanza
> delivery order comes to mind). And any other thoughts you might have!
>  

So I think we are trying to do the same thing. I have to point that I didn't
finish to polish this and for sure it could not work as it is right now. But
you can get the idea.

Looking forward your comments
Victor Seva

-- 
You received this message because you are subscribed to the Google Groups 
"prosody-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prosody-dev+unsubscr...@googlegroups.com.
To post to this group, send email to prosody-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/prosody-dev.
For more options, visit https://groups.google.com/d/optout.

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to