Re: [Openstack] Push vs Polling (from Versioning Thread)
When you look at the scalability issue solely from the perspective of the cloud provider, requiring polling is the lazier, but not really more scalable solution. Especially if you go nuts with caching. Then it might be even a bit more scalable. But when you look at the distributed systems use cases, it's terrible for the system as a whole. People are polling your API because they want to know whether or not the state of the system is different from what they remember it to be. The more critical it is for them to rapidly know about changes, the more often they poll. Yes, there are ways to determine when changes are more likely to occur and thus optimize the polling interval. Some clients may be, in your eyes as the provider, overly aggressive. But who the hell are you to judge their use case and throttle them? But here's the bottom line: The vast majority of work is completely wasted when polling is the change propagation method. Push notifications don't make your core system any more complex. You push the change to a message queue and rely on another system to do the work. The other system is scalable. It has no need to be stateless and can be run in an on-demand format using agents to handle the growing/shrinking notification needs. Bryan brings up the point that some of these subscription endpoints may go away. That's a total red-herring. You have mechanisms in place to detect failed deliveries and unsubscribe after a time (among other strategies). The bottom line: When you push changes, the vast majority of your work is meaningful work when pushing is the change propagation method. Let's do the math. Let's say I am interested in any change in VM state so I can auto-scale/auto-recover. Let's use a fairly simplistic polling strategy, but do it efficiently (and assume the API enables me to make a single API call to get state for all VMs). Let's pick 1 query/minute (in reality, you wouldn't pick a flat polling rate like this, but it is useful for this thought experiment). Now multiply that times 1,000 customers. Or 100,000. Or 1,000,000. Now let's say that the client is going through a cloud management service. And that service is serving 20% of your customer base. They are likely making queries across a wide range of resources, not just VMs. And they have to scale the polling from their end. Both sides are thus engaged in trying to figure out a way to scale work that is almost entirely pointless work. There's a reason you see the cloud management tools pushing push. We've seen this IaaS polling across a bunch of clouds. It sucks. -George -- George Reese - Chief Technology Officer, enStratus e: george.re...@enstratus.comt: @GeorgeReesep: +1.207.956.0217f: +1.612.338.5041 enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - http://www.enstratus.com To schedule a meeting with me: http://tungle.me/GeorgeReese smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
On Oct 28, 2011, at 8:11 AM, George Reese wrote: Push notifications don't make your core system any more complex. You push the change to a message queue and rely on another system to do the work. The other system is scalable. It has no need to be stateless and can be run in an on-demand format using agents to handle the growing/shrinking notification needs. Bryan brings up the point that some of these subscription endpoints may go away. That's a total red-herring. You have mechanisms in place to detect failed deliveries and unsubscribe after a time (among other strategies). I think what Bryan is saying is this. Someone, on another system, lets call it a hub, has to do the work of tracking what messages have been received by a particular client. The failure scenarios there can cause a lot of head aches. You can try to scale hubs out horizontally, but each hub will be handling a different set of clients at a particular point in time. So that data needs to be tracked. The best you can do is to have a central data store tracking when a client has received and acknowledged a particular message. If there are a lot of clients that's a lot of data to sort through and partition. If you don't have a central store then a particular hub will be responsible for a certain set of clients. And in this case, how many clients should be tracked by a hub? 100? 1000? 100,000? The more clients a hub handles the more memory it needs to use to track those clients. If a hub is at capacity but you're monitoring system is starting to detect disk failures, how do you migrate those clients to another hub? Do you split the clients up among existing hubs, if so what's the algorithm there? Or do you have to stand up a new hub? As for the other failure states, the issue isn't just about detecting failed deliveries, it's about tracking down successful deliveries too. Say after immediately sending a message to client A, that hub goes down. There's no record in the system that the message was sent to client A. How do we detect that that happened? If we do detect it should we resend the message here? Keep in mind, the client may have received it but may or may not have acknowledged it. If we do resend the message, will that mess up the client? Does the client even care? There's a whole lot of inefficiencies to. Consider that there are some cases where the client also needs to track what messages have been received. Both the client and the hub are tracking the state in this scenario and that's pretty inefficient. I would argue far more inefficient than the polling scenario because it involves memory and potentially storage space. If the client doesn't really care to track state we are tracking it at the hub for no reason. Say we have a client that's tracking sate, maybe saving it to the datastore. (We have a lot of customers that do this.) The client receives a message, but before it can save it, it goes down. Upon coming up again, it has no awareness of the lost message, will it be delivered again? How? How does the client inform the hub of it's state? Other questions arise: How long should you track clients before you unsubscribe them? etc...etc... There's just so many similar scenarios that add a lot of complexity and I would argue, at cloud scale, far greater inefficiencies into the system. With the polling scenario, the work is split between the server and the client. The server keeps track of the messages. The client keeps track of it's own state (what was the last message received? etc). It's scalable and, I would argue more efficient, because it allows the client to track state if it wants to, when it wants to, how it wants to. On the server end statelessness means that each pubsub node is a carbon copy of another -- if one goes down another can replace it with no problem -- no need to transfer sate. What's more, the memory usage of the node is constant, no matter how many clients are hitting it. That's not to say that polling is always the right choice. As Mark said, there are a lot of factors to consider. In cases where there are a large number of messages latencies may increase dramatically. It's just that when we're talking web scale, it is *usually* a good choice. -jOrGe W. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
You are describing an all-purpose system, not one that supports the narrow needs of IaaS state notifications. There's no reason in this scenario to guarantee message delivery. Sent from my iPhone On Oct 28, 2011, at 10:09, Jorge Williams jorge.willi...@rackspace.com wrote: On Oct 28, 2011, at 8:11 AM, George Reese wrote: Push notifications don't make your core system any more complex. You push the change to a message queue and rely on another system to do the work. The other system is scalable. It has no need to be stateless and can be run in an on-demand format using agents to handle the growing/shrinking notification needs. Bryan brings up the point that some of these subscription endpoints may go away. That's a total red-herring. You have mechanisms in place to detect failed deliveries and unsubscribe after a time (among other strategies). I think what Bryan is saying is this. Someone, on another system, lets call it a hub, has to do the work of tracking what messages have been received by a particular client. The failure scenarios there can cause a lot of head aches. You can try to scale hubs out horizontally, but each hub will be handling a different set of clients at a particular point in time. So that data needs to be tracked. The best you can do is to have a central data store tracking when a client has received and acknowledged a particular message. If there are a lot of clients that's a lot of data to sort through and partition. If you don't have a central store then a particular hub will be responsible for a certain set of clients. And in this case, how many clients should be tracked by a hub? 100? 1000? 100,000? The more clients a hub handles the more memory it needs to use to track those clients. If a hub is at capacity but you're monitoring system is starting to detect disk failures, how do you migrate those clients to another hub? Do you split the clients up among existing hubs, if so what's the algorithm there? Or do you have to stand up a new hub? As for the other failure states, the issue isn't just about detecting failed deliveries, it's about tracking down successful deliveries too. Say after immediately sending a message to client A, that hub goes down. There's no record in the system that the message was sent to client A. How do we detect that that happened? If we do detect it should we resend the message here? Keep in mind, the client may have received it but may or may not have acknowledged it. If we do resend the message, will that mess up the client? Does the client even care? There's a whole lot of inefficiencies to. Consider that there are some cases where the client also needs to track what messages have been received. Both the client and the hub are tracking the state in this scenario and that's pretty inefficient. I would argue far more inefficient than the polling scenario because it involves memory and potentially storage space. If the client doesn't really care to track state we are tracking it at the hub for no reason. Say we have a client that's tracking sate, maybe saving it to the datastore. (We have a lot of customers that do this.) The client receives a message, but before it can save it, it goes down. Upon coming up again, it has no awareness of the lost message, will it be delivered again? How? How does the client inform the hub of it's state? Other questions arise: How long should you track clients before you unsubscribe them? etc...etc... There's just so many similar scenarios that add a lot of complexity and I would argue, at cloud scale, far greater inefficiencies into the system. With the polling scenario, the work is split between the server and the client. The server keeps track of the messages. The client keeps track of it's own state (what was the last message received? etc). It's scalable and, I would argue more efficient, because it allows the client to track state if it wants to, when it wants to, how it wants to. On the server end statelessness means that each pubsub node is a carbon copy of another -- if one goes down another can replace it with no problem -- no need to transfer sate. What's more, the memory usage of the node is constant, no matter how many clients are hitting it. That's not to say that polling is always the right choice. As Mark said, there are a lot of factors to consider. In cases where there are a large number of messages latencies may increase dramatically. It's just that when we're talking web scale, it is *usually* a good choice. -jOrGe W. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
Jorge Williams wrote: Push notifications don't make your core system any more complex. You push the change to a message queue and rely on another system to do the work. That is only true if the messaging system and the core system are largely independent, which could have some implications that would probably be fine for most human users but could be quite problematic for applications. Can the push notification system block the core system? If not the push notifications ultimately become unreliable. A human who is not notified that a given update once in 10,000 times is probably just going to shrug it off. But an application that needs to know it is looking at the most recent version of a document before it modifies it is ultimately going to have to rely on polling, or have the notification be built into the core system complete with throttling of updates when absolutely necessary to ensure that notifications are sent. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
There are ways around that without guaranteed message delivery. Sent from my iPhone On Oct 28, 2011, at 11:41, Bryan Taylor btay...@rackspace.com wrote: On 10/28/2011 10:33 AM, George Reese wrote: There's no reason in this scenario to guarantee message delivery. Usage, billing, support all require guaranteed message delivery. Oops, sorry, we don't bill sometimes because we lose messages just doesn't fly with executives, shareholders, and the SEC. With customers, lost messages = credit memos and lost customers. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
On Oct 28, 2011, at 10:33 AM, George Reese wrote: You are describing an all-purpose system, not one that supports the narrow needs of IaaS state notifications. There's no reason in this scenario to guarantee message delivery. Like I said, there are a lot of factors to consider. And guarantee delivery may or may not be a requirement based on your use case. For example, wouldn't we want to monitor based on state changes? Missing a message could mean not sending an alert, right? How do you compensate for that if you don't guarantee delivery? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
Huh? I didn't write that. George did. On Oct 28, 2011, at 11:35 AM, Caitlin Bestler wrote: Jorge Williams wrote: Push notifications don't make your core system any more complex. You push the change to a message queue and rely on another system to do the work. That is only true if the messaging system and the core system are largely independent, which could have some implications that would probably be fine for most human users but could be quite problematic for applications. Can the push notification system block the core system? If not the push notifications ultimately become unreliable. A human who is not notified that a given update once in 10,000 times is probably just going to shrug it off. But an application that needs to know it is looking at the most recent version of a document before it modifies it is ultimately going to have to rely on polling, or have the notification be built into the core system complete with throttling of updates when absolutely necessary to ensure that notifications are sent. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
On Oct 27, 2011, at 10:20 PM, Bryan Taylor wrote: Just to be clear we are talking about APIs fit for customer consumption here, not internal integrations where both ends are under our control. On 10/27/2011 11:38 AM, George Reese wrote: I disagree. The web was designed specifically to solve the distributed scaling problem and it's based on HTTP polling. It scales pretty well. The argument against polling not scaling inevitably neglects using caching properly. The web was not designed to deal with a bunch of clients needing to know about infrastructure changes the instant they happen. Neither physics nor math were designed for that either. The CAP theorem simply doesn't allow a distributed system with an uptime guarantee to communicate changes the instant they happen. Once you realize the best your clients can hope for is eventual consistency, the sooner you'll realize that polling is just fine. In theory, your theory describes the situation . In practice it's different. :- Yes, clients cant get truly instant notifications. What they can get is ASAP notifications. Which is what they really need/want in some situations. Now, this could be implemented by polling some sort of changes since: X resource, as in the Fielding article, or it could be done via a push system. Which one is appropriate depends on the relative importance of a number of things like acceptable latency of notifications since the event, frequency of notifications, requited reliability, etc to the client, and the relative cost of retrieving certain information to the serving system. In the case of Nova, we have use cases for both methods. BTW, here's Roy Fielding's article on this subject of poll vs push. http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons And API data should not be cached. The Rackspace API used to do that, and it created a mess. I'm not sure what you are referring to, but this is a classic strawman. Somebody implemented a mess using caching, so caching is bad!? You didn't say what the mess was, so there's no way to even evaluate your statement. Push doesn't scaled because it requires the server to know about every client and track conversational state with them. No, it doesn't. You push changes as they occur to a message queue. A separate system tracks subscribers and sends them out. There is no conversational state if done right. A separate system? That's why you think it's simple -- you push the hard part outside of your box and claim victory. It's not a separate system, it's all one big cloud. If there are N interested clients the process you described requires O(N) resources. Moving it to another tier means it's somebody else's O(N) resources. You are illustrating Fielding's point in the article above: People need to understand that general-purpose PubSub is not a solution to scalability problems — it simply moves the problem somewhere else, and usually to a place that is inversely supported by the economics. How exactly does this separate system know where to send them out to? Each client has to tell it and you have to store it and look it up on a per outbound message basis. And keep it accurate. Customers just love keeping you informed of where they want to send their messages. Do you know what happens when they forget to tell you they moved and they don't get the message? They blame you and ask for a credit memo. And do you know what happens when you tell them no. They go to your competitor. If there is no conversational state, then you aren't waiting for an acknowledgement from the other side for each message and you can't prove that it was delivered or even try again. It is a separate system, if you can make use of systems someone else has already designed tested for you. The easiest work is that someone else has already done for you. (For example, we don't worry about making use of the worlds largest/oldest push notification system (email) where needed. Postfix has already been designed for us. ) Hubs (and other pubsub/queueing) systems already exist. And the reliability issue is a red herring. push/messaging systems are often of less than guaranteed reliability, which is fine if communicated. There are many use cases where that is good enough. In any case the order of resources needed is closer to O(N * Fp) for a polling system, where N is the number of clients, and Fp is the polling frequency (i.e. the inverse of the minimum acceptable latency). The order for a push system is roughly O(N * Fe) where N is the number of clients, and Fe is the frequency of events you need to notify clients about. Which is appropriate depends on if Fe Fp. If your clients need notification at low latency of infrequent events, they will pound your API servers thru the floor with their polling when a push system will be hardly doing any work. On the other hand, if the client needs a huge
Re: [Openstack] Push vs Polling (from Versioning Thread)
On Oct 27, 2011, at 11:38 AM, George Reese wrote: Sent from my iPhone On Oct 27, 2011, at 11:26, Bryan Taylor btay...@rackspace.com wrote: On 10/27/2011 10:36 AM, George Reese wrote: #3 Push scales a hell of a lot better than having tools polling a cloud constantly. It doesn't matter whether it is polling the API, polling a feed, or polling a message queue. Polling is one of the most unscalable things you can do in any distributed systems scenario. Calling it a feed doesn't magically solve the problem. Actually, it's quite hard on its own in an IaaS scenario and has scaling issues independent of the polling issue. I disagree. The web was designed specifically to solve the distributed scaling problem and it's based on HTTP polling. It scales pretty well. The argument against polling not scaling inevitably neglects using caching properly. The web was not designed to deal with a bunch of clients needing to know about infrastructure changes the instant they happen. True. This whole issue is the reason Nova's existing notification system is designed as a push system. Currently it's used to push error notifications and usage info, but there is no reason it could not eventually also provide notifications to end users. After watching demos of large cloud users where they were polling apis to see when their instances were ready (and often spinning up new ones when that didn't happen fast enough) I kept that use case in mind when coding the notifications system. And API data should not be cached. The Rackspace API used to do that, and it created a mess. Push doesn't scaled because it requires the server to know about every client and track conversational state with them. No, it doesn't. You push changes as they occur to a message queue. A separate system tracks subscribers and sends them out. There is no conversational state if done right. Indeed. this is how the notifications system is/can work right now. If you turn notifications on, nova pushes them to a rabbit queue. A separate app, namely a hub using the standard PubSubHubbub protocol, (plus an external rabbit queue - feed generator app we wrote, called Yagi) manages the subscriptions and pings the subscribers. If you need reliability, this requires persisting that conversational state. In order to allow this to happen you have to have some kind of registration protocol for clients. If some fraction of those clients are flaky, the conversational state tracking will kill you because each client consumes resources and so flaky clients = resource leak. Existing PSH hubs are designed to handle large numbers of potentially unreliable clients. Push wins when you need very low latency delivery, high message throughput to individual consumers, or server side guarantees of delivery to individual consumers, but not for scaling to a large number of clients in a climate of an elastic infrastructure. Push notifications are the only mechanism for solving the scaling issue. You push any changes to a message queue. Agents pick up the changes and send them on to subscriber endpoints. Not that hard. Not that hard with a few fairly reliable clients. Very hard with a web scale set of unreliable clients while I simultaneously need to scale the back end. Actually, we are already implementing this at 200+node scale in nova, since this is how we are handling the collection of usage data for billing. At the moment is seems to be working reasonably well. We are not using the PSH hubs atm, since we are pushing to a few internal consumers via AtomPub, and don't need the complex subscription management, but from our point of view, pinging a hub is no different from what we already do, and the hub handles the subscription details. It would be nice to one day support notifications of end-users, as I think it would be of great benefit to them. There's work that would need to be done around hub/Yagi auth, and I think there is some bigger fish to fry, nova functionality-wise at the moment, but it is something to keep in mind. -- Monsyne M. Dragon OpenStack/Nova cell 210-441-0965 work x 5014190 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Push vs Polling (from Versioning Thread)
+1 Dragon From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Monsyne Dragon [mdra...@rackspace.com] Sent: Thursday, October 27, 2011 4:14 PM To: George Reese Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Push vs Polling (from Versioning Thread) On Oct 27, 2011, at 11:38 AM, George Reese wrote: Sent from my iPhone On Oct 27, 2011, at 11:26, Bryan Taylor btay...@rackspace.com wrote: On 10/27/2011 10:36 AM, George Reese wrote: #3 Push scales a hell of a lot better than having tools polling a cloud constantly. It doesn't matter whether it is polling the API, polling a feed, or polling a message queue. Polling is one of the most unscalable things you can do in any distributed systems scenario. Calling it a feed doesn't magically solve the problem. Actually, it's quite hard on its own in an IaaS scenario and has scaling issues independent of the polling issue. I disagree. The web was designed specifically to solve the distributed scaling problem and it's based on HTTP polling. It scales pretty well. The argument against polling not scaling inevitably neglects using caching properly. The web was not designed to deal with a bunch of clients needing to know about infrastructure changes the instant they happen. True. This whole issue is the reason Nova's existing notification system is designed as a push system. Currently it's used to push error notifications and usage info, but there is no reason it could not eventually also provide notifications to end users. After watching demos of large cloud users where they were polling apis to see when their instances were ready (and often spinning up new ones when that didn't happen fast enough) I kept that use case in mind when coding the notifications system. And API data should not be cached. The Rackspace API used to do that, and it created a mess. Push doesn't scaled because it requires the server to know about every client and track conversational state with them. No, it doesn't. You push changes as they occur to a message queue. A separate system tracks subscribers and sends them out. There is no conversational state if done right. Indeed. this is how the notifications system is/can work right now. If you turn notifications on, nova pushes them to a rabbit queue. A separate app, namely a hub using the standard PubSubHubbub protocol, (plus an external rabbit queue - feed generator app we wrote, called Yagi) manages the subscriptions and pings the subscribers. If you need reliability, this requires persisting that conversational state. In order to allow this to happen you have to have some kind of registration protocol for clients. If some fraction of those clients are flaky, the conversational state tracking will kill you because each client consumes resources and so flaky clients = resource leak. Existing PSH hubs are designed to handle large numbers of potentially unreliable clients. Push wins when you need very low latency delivery, high message throughput to individual consumers, or server side guarantees of delivery to individual consumers, but not for scaling to a large number of clients in a climate of an elastic infrastructure. Push notifications are the only mechanism for solving the scaling issue. You push any changes to a message queue. Agents pick up the changes and send them on to subscriber endpoints. Not that hard. Not that hard with a few fairly reliable clients. Very hard with a web scale set of unreliable clients while I simultaneously need to scale the back end. Actually, we are already implementing this at 200+node scale in nova, since this is how we are handling the collection of usage data for billing. At the moment is seems to be working reasonably well. We are not using the PSH hubs atm, since we are pushing to a few internal consumers via AtomPub, and don't need the complex subscription management, but from our point of view, pinging a hub is no different from what we already do, and the hub handles the subscription details. It would be nice to one day support notifications of end-users, as I think it would be of great benefit to them. There's work that would need to be done around hub/Yagi auth, and I think there is some bigger fish to fry, nova functionality-wise at the moment, but it is something to keep in mind. -- Monsyne M. Dragon OpenStack/Nova cell 210-441-0965 work x 5014190 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net
Re: [Openstack] Push vs Polling (from Versioning Thread)
Just to be clear we are talking about APIs fit for customer consumption here, not internal integrations where both ends are under our control. On 10/27/2011 11:38 AM, George Reese wrote: I disagree. The web was designed specifically to solve the distributed scaling problem and it's based on HTTP polling. It scales pretty well. The argument against polling not scaling inevitably neglects using caching properly. The web was not designed to deal with a bunch of clients needing to know about infrastructure changes the instant they happen. Neither physics nor math were designed for that either. The CAP theorem simply doesn't allow a distributed system with an uptime guarantee to communicate changes the instant they happen. Once you realize the best your clients can hope for is eventual consistency, the sooner you'll realize that polling is just fine. BTW, here's Roy Fielding's article on this subject of poll vs push. http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons And API data should not be cached. The Rackspace API used to do that, and it created a mess. I'm not sure what you are referring to, but this is a classic strawman. Somebody implemented a mess using caching, so caching is bad!? You didn't say what the mess was, so there's no way to even evaluate your statement. Push doesn't scaled because it requires the server to know about every client and track conversational state with them. No, it doesn't. You push changes as they occur to a message queue. A separate system tracks subscribers and sends them out. There is no conversational state if done right. A separate system? That's why you think it's simple -- you push the hard part outside of your box and claim victory. It's not a separate system, it's all one big cloud. If there are N interested clients the process you described requires O(N) resources. Moving it to another tier means it's somebody else's O(N) resources. You are illustrating Fielding's point in the article above: People need to understand that general-purpose PubSub is not a solution to scalability problems — it simply moves the problem somewhere else, and usually to a place that is inversely supported by the economics. How exactly does this separate system know where to send them out to? Each client has to tell it and you have to store it and look it up on a per outbound message basis. And keep it accurate. Customers just love keeping you informed of where they want to send their messages. Do you know what happens when they forget to tell you they moved and they don't get the message? They blame you and ask for a credit memo. And do you know what happens when you tell them no. They go to your competitor. If there is no conversational state, then you aren't waiting for an acknowledgement from the other side for each message and you can't prove that it was delivered or even try again. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
