On Apr 3, 2008, at 9:48 AM, Dave Cridland wrote:
On Thu Apr 3 08:23:36 2008, Pedro Melo wrote:
A) Compression
It's crucial on two counts:
1) It's much simpler to implement, and2) Given that we are (or
should be) encrypting every S2S connection, then TLS is giving
us compression anyway, and moreover, it's cheaper to compress
than not to compress.
We've been down the road before where on some admittedly fairly
simple figures I did a while back - and repeated for Peter's
scalability draft in the SIMPLE WG, I think - the existing
presence "splurges" compress really well, often better than the
"optimized" alternatives.
Smart Presence was one proposal that beat pure compression, as I
recall, but it has security implications which I'm not going to
accept.
Bandwidth is not the only concern. If I have a very popular
pubsub topic (like a Atom node for some high-profile blog or
social site) with hundreds of thousands of subscribers, the cost
of expanding in the source is very high, in terms of CPU also.
Well, yes and no - XML parsing appears to hit a lot of servers
hard, but routing is routing, and repeaters will increase the cost
there, I suspect. If you're having to expand routers, that
expansion has to happen somewhere - it's a database lookup, or a
filesystem read, or a hash table in memory - but it's certainly
additional activity.
This cost might, in some implementations, be outweighed by the
saving in XML parsing. I don't know, and I don't pretend to know,
either. I'm pretty sure it isn't in our implementation.
If you ignore list changes for a minute, routing costs will be the
same on remote nodes (the same amount of stanzas will be routed + 1,
actually) with or without repeaters. The biggest savings are in the
source service + source router which will have much less stanzas
transmitted and routed.
So globally, if you look at the cost of delivering a single
notification to thousands of subscribers, the cost to the XMPP
network as whole is lower with repeaters, and given the the fan-out
processing is pushed to the edges (and therefore the costs of lookup
are distributed amongst interested parties), I would be willing to
bet that the latency from source to destination will also be lower, a
small bonus.
So I do not agree that repeaters will increase the routing cost here.
Maybe a little on the remote hosts, but will save a lot on the source
router.
As for XML processing, well, if/when that becomes a problem, we can
also use a xep-0138 profile for some more efficient representation
format for XMPP stanzas. For example, two ejabberd servers could use
native erlang message payloads when talking to each other... All is
fair you pre-declare.
The mailing list people moved to sub-lists to deal with this.
Repeaters are just like sub-lists. Please note that I'm talking
of the *public* pubsub node use-case.
Right, almost. Repeaters are like sub-lists controlled by the
sender - that's a quite strinking difference.
No, repeaters are sub-lists controlled by the service/the list. If I,
the sender, publish a node to some pub-sub service, I don't know nor
I even care if the pubsub service is using repeaters.
The sender has no control on the repeaters.
B) RTT delays
The introduction of Stanza Repeaters gives us RTT delays whenever
a list needs changing. That just strikes me as worrying - I
don't think anyone has clear figures on how much delay would be
introduced, but I'm sure you'll agree that additional latency
does not a performance improvement make.
It may be possible to alter the protocol to remove these.
This is important if and only if list changes are the most
frequent operation.
If sending messages through the repeaters is the most common
operation, then the lower RTT of distribution that Stanza
Repeaters give us, will over-shadow the delays with list changing.
Right, kind of - for any given repeater, you need a list-change per
event to make it happen every time. And I'm not sure where a
suitable breakpoint is - I don't know how many stanza-fanouts
outweigh an RTT.
I don't feel we know anything like enough about habits and metrics
right now. It could be that even when we have an RTT delay due to a
list rewrite on every event, it's not a big deal because the line
may still be efficiently utilised, but introducing new RTT delays
just doesn't sit right with me.
Do you agree that in the case of "sending a message" though a
repeater does not increase the RTT delay?
If yes, then this RTT delay is only present if I want to modify my
remote list before sending a new message. Right?
Now, I'm reasonably confident in suggesting that (without figures
to back this up), the list-change to stanza-fanout ratio has to be
pretty good before you beat simple compression.
Again, I think you are only thinking about bandwidth when you talk
about compression. My concern is not bandwidth but how to deploy
pubsub for hundreds of thousand subscribers without having my s2s
links shutdown by big domains because I'm a spammer.
And the only way to do that is to allow the remote server to fan-out.
As for the list-change costs, and as suggested before by me with a
pubsub use case, we might need to think about end-users being able to
"add" themselves to a local repeater. This could be an option at
repeater creation.
For example, if I'm a pubsub service at saramago.lit and I create a
repeater at pessoa.lit for a specific node N, I can include a
<allow_local_subscription /> to allow the list to be modified
directly by users trusted by the repeater, or
<allow_local_subscription><domain>pessoa.lit</domain></
allow_local_subscription> to restrict the ability of direct list-
changes to a specific set of domains.
if this is allowed then, when people disco#info my node N, I can tell
them that they can subscribe to me via a local repeater at their
domain, and give the repeater jid for the list.
In this scenario, list changes are not from the pubsub service to the
repeater, but by local users to his local repeater.
Security-wise, this is no worse than the previous version, because
this capability is still controlled by the list creator, who has to
opt-in to this.
RTT-wise if would be on par with a scenario without repeaters.
And we still get the benefits of remote fan-out.
Mind you that I see this possibility of end-users "adding" themselves
to repeaters as a viable possibility for public pubsub nodes and
auditorium-style MUC rooms only.
I'm also of a much more nebulous gut feeling - assuming gut
feelings *can* be nebulous - that for typical large public pubsub
nodes - like, say, a popular blog - it's quite likely that the list
will often change between publication events - which in turn
suggests that this might be completely the wrong solution for that
use-case.
See my example of using a local pubsub service as a proxy, or the
example above of letting repeaters list to be modified by user list.
I agree with you that list-change needs to be more streamlined, but I
think we can do it inside the repeaters spec.
The cost of updating this list is exactly the same as a normal
subscribe in pubsub. A second user will only see steps 1, 2, and
5, same as a normal subscription. The local pubsub service might
signal the remote pubsub service, but thats optional, can be done
out-of- sync with the user flow, and its only required if the
central node wants to have knowledge of the number of subscribers
globally.
Right, or control them, in a relatively trusted environment.
Even in a untrusted environment if you are talking about public
information, like blog posts.
Trust, like, "I want to know if the notification is really from my
friend Peter" is a problem for PKI and digital signatures, not the
"multi-cast" system on top of XMPP. IMHO.
A similar mechanism works for MUC, too - a sub-MUC would send
subscriptions to the master-MUC, which would send only a single
copy of the messages back to the sub-MUC for further routing. If a
user tries to subscribe to the sub-MUC and the subscription is
rejected by master-MUC, the rejection is passed back.
Exactly.
Again, I only see this as useful for auditorium-style MUC rooms. For
normal MUC rooms, then you are implementing IRC on top of XMPP and
multicasting message will be the least of your concerns (think
netsplits, and making sure your nick is unique).
We talked about this in Brussels a bit, and I think that it's an
interesting area to persue.
My opinion, is that without remote fan-out for pubsub, XMPP will not
be a long term option as a notification network. Yes, it beats
pooling, but source servers will be overwhelmed with the costs of
sending the same notification to large numbers of subscribers.
Plus as a server-operator, its wasteful for me seeing thousands of
the same message, with a different 'to' attribute.
FWIW, pubsub and MUC - at least the public ones - are very much
simpler than presence handling, making this style of fan-out much
more practical. Indeed, having chains of sub-MUCs becomes
practical, since either the MUC is public, or potentially a chain
can be set up with bilateral agreements, which is effectively how
IRC and PSYC operate.
Yes, and I think that even if there is no consensus about repeaters
as a general solution, we should have something to repeat public
information via pubsub and certain types of MUC rooms.
For presence distribution, I like remote fan-out based on a
reverse- roster better, but it doesn't interact properly with
privacy lists. Stanza repeaters deal better with privacy lists,
because you can update the remote list based on the privacy list
changes. But I don't know if the complexity of all this privacy-
list interaction will be worth it.
Right - I think it's just going to get exceedingly complex, very
quickly, and the net result will be more bugs, and a failure to
produce scalable specialized support for fan-out in PubSub/MUC.
yep. I must admit that personally I would gladly sacrifice a remote
fan-out solution for presence distribution if I could have one sooner
for pubsub.
I say this because (if we exclude Peter) rosters above 1000 contacts
are not that common (I'll try and see the distribution at SAPO) and
the ones we found where mostly caused by bugs in the interaction with
pyMSNt ;), but pubsub nodes with over 1000 contacts will become quite
common if the social workgroup starts doing their thing.
I'd like to know more about the levels of data we'd save for
presence fanout broadcasts, really, prior to going forward with
this, but I have to admit I'd be surprised if this is worthwhile
for just that.
In the end, a generic stanza repeater implementation is something
useful to have in theory, but in practice all of the services that
it is trying to replace already have the subscriber list inside,
and they could leverage that to add multi-cast capabilities.
Exactly.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: [EMAIL PROTECTED]
Use XMPP!
