On Sat, Oct 24, 2009 at 10:21 PM, Eran Sandler <[email protected]> wrote: > I see. > Thanks for the answer. > How well can Causes handle load spikes? I assume its probably dependent on > the amount of RAM the beanstalk server had, pattern of usage and the way > jobs where used in Causes.
Causes does really well here. I don't know the usage patterns these days, as I haven't been at the company for a while. I can only reiterate things I've said before in some form. The body of a typical causes job is less than 100 bytes. Workers generally do a good job of keeping up with offered load from interactive-type stuff. The queue almost never grows to more than a tiny, tiny fraction of available ram. Batch stuff can get pretty sizable. For example, facebook's API forces updates to the "profile box" to be pushed to facebook ahead-of-time. Once upon a time, a change to the content of this box meant making one or more HTTP connections to facebook for each user to push the new content[1]. Getting this done in a timely manner meant making many millions of jobs, roughly one for each user[2]. We were doing this routinely long before beanstalkd had persistence code at all. By contrast, a pageview spike of, say, 100x normal traffic sustained for twelve hours[3] would not even put a dent in the capacity. > I think the flow-to-disk scenario is very important for real fault tolerance > in peak scenarios. As you say, this depends on usage patterns (including job size). I agree that flow-to-disk can be important, for this and other reasons. I'd love to continue discussing possible designs for the feature. kr [1] I think facebook has since made API changes that drastically reduce this problem, but that's beside the point here. [2] I don't know the exact number of users, but it was more than 15 million a year ago. [3] This is far beyond a realistic scenario. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
