Thanks for sharing the data. One thing before I jump and offer some ideas on the flow-to-disk implementation, I want to make sure I understand how the regular binlog works.
When a job enters, it goes into memory and binlog. When the job is marked as finished, it is removed from both memory and binlog, correct? Eran On Sun, Oct 25, 2009 at 9:49 PM, Keith Rarick <[email protected]> wrote: > > On Sat, Oct 24, 2009 at 10:21 PM, Eran Sandler <[email protected]> > wrote: > > I see. > > Thanks for the answer. > > How well can Causes handle load spikes? I assume its probably dependent > on > > the amount of RAM the beanstalk server had, pattern of usage and the way > > jobs where used in Causes. > > Causes does really well here. I don't know the usage patterns these > days, as I haven't been at the company for a while. I can only > reiterate things I've said before in some form. > > The body of a typical causes job is less than 100 bytes. Workers > generally do a good job of keeping up with offered load from > interactive-type stuff. The queue almost never grows to more than a > tiny, tiny fraction of available ram. > > Batch stuff can get pretty sizable. For example, facebook's API forces > updates to the "profile box" to be pushed to facebook ahead-of-time. > Once upon a time, a change to the content of this box meant making one > or more HTTP connections to facebook for each user to push the new > content[1]. Getting this done in a timely manner meant making many > millions of jobs, roughly one for each user[2]. We were doing this > routinely long before beanstalkd had persistence code at all. > > By contrast, a pageview spike of, say, 100x normal traffic sustained > for twelve hours[3] would not even put a dent in the capacity. > > > I think the flow-to-disk scenario is very important for real fault > tolerance > > in peak scenarios. > > As you say, this depends on usage patterns (including job size). > > I agree that flow-to-disk can be important, for this and other > reasons. I'd love to continue discussing possible designs for the > feature. > > kr > > [1] I think facebook has since made API changes that drastically > reduce this problem, but that's beside the point here. > [2] I don't know the exact number of users, but it was more than 15 > million a year ago. > [3] This is far beyond a realistic scenario. > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
