[twitter-dev] Re: large user base push notification solutions?
Great ideas. thanks a lot. Based on your data: 40,000 users, pull every 15 minutes, that means 44.5 requests/ second. Do you pull the data for everyone , or are there any short cuts that we can save server load? Also, did you do DMs as well? if so then you will need to deal 90 requests/second. Can a single VPS do that? which VPS provider? I want to buy that VPS :) On Aug 21, 8:28 pm, Paul Kinlan wrote: > When I developed Twe2, here are some of the things I have learnt > > - 2 minute delay is pretty short - users don't even notice it that much - > at one point on Twe2 we changed it to a 15 minute delay an no one really > complained. If users are getting pushed notifications they are normally > away from a main terminal and thus are not watching twitter through > TweetDeck; in short you don't need realtime to be that realtime > - Also 99.99% of people don't get that many notifications a day, polling > too often is a waste of time. > - We supported about 40,000 users off 1 small VPS. > - To get DM's you will need to use the users credentials (oauth or > otherwise), a 2 minute interval means that you will use 30 of the users > requests per hour (this might have changed) and as such they might get > annoyed. > - 500,000 users is pretty optimistic I wouldn't even worry about that > scale just yet, just get your stuff working for now. > - User since_id everywhere you can. > - We launched with the ability to have quiet periods, that is no > notifications while I am sleeping - people will thank you for this. > > Based on new developments of Twitter you can use something like follow, > shadow and birddog - it offers a migration plan too, start with follow to > get all the tweets from a user and to a user (200 users is good to test > your API works), then when you launch request twitter to allow you to use > shadow (50,000 users is a lot and will probably suit your app for a long > time). Then as soon as you see a tweet on the stream you know it is for > some of your users and you can fire it straight to them. the only issue is > that these API's only get proper replies and not mentions. > > Currently none of the Streaming API's will help you for DM's (AFAIK). > > Paul Kinlan,http://www.Twollo.com > > 2009/8/21 ke...@nibirutech.com > > > > > > > Hi > > > I am a developer , trying to figure out a way to develop a push > > notification solution for iPhone users. > > > The easy way to do the push work, is to have a cron-job to check > > users' new mentions and DMs. It should work for small number of > > users. What if we have a large user base, say , 500, 000 users at > > least? How can we use a proper solution to get a 2-minutes delay push > > for any user's mentions and DMs? (we can't afford the server cost for > > half million requests every 2 minutes) > > > I know there are a few Twitter push clients for iPhone , but none of > > them can work on a scaled user base, am I right? > > > Is there a twitter tech support here? could you please give some > > suggestions?
[twitter-dev] Re: large user base push notification solutions?
Apple push notification for iPhone works when the app is closed. In that case the iPhone client is not able to pull data. Meanwhile users want any mentions or DMs to pushed to them when the iPhone client is closed. So we must use a server to check DM & Mentions for users. On Aug 21, 8:10 pm, Andrew Badera wrote: > If you're already talking about a 2-minute delay, then why "push" and > not "pull" ?? Polling clients will give you greater scalability with > that range of latency easily achievable for little investment.Pushis > meant for immediate notifications. Truepushto 500k+ clients is > costly -- but you don't need millisecond latency here. Pull is much, > much cheaper. > > ∞ Andy Badera > ∞ This email is: [ ] bloggable [x] ask first [ ] private > ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera) > > > > On Fri, Aug 21, 2009 at 8:07 AM, Dewald Pretorius wrote: > > > In addition, your database will have to cope with 8,300 writes per > > second. And then you need to take into account the latency of the > > ApplePushNotificationservice. > > > Dewald > > > On Aug 21, 8:57 am, Dewald Pretorius wrote: > >> On Aug 21, 12:06 am, "ke...@nibirutech.com" > >> wrote: > > >> > What if we have a large user base, say , 500, 000 users at > >> > least? How can we use a proper solution to get a 2-minutes delaypush > >> > for any user's mentions and DMs? (we can't afford the server cost for > >> > half million requests every 2 minutes) > > >> You are actually talking about one million API calls every 2 minutes > >> (1 for mentions, one for DMs). > > >> That's 8,300 API calls per second. > > >> My rough estimate is that you are going to need around 200 servers to > >> cope with that workload. > > >> Dewald
[twitter-dev] Re: large user base push notification solutions?
When I developed Twe2, here are some of the things I have learnt - 2 minute delay is pretty short - users don't even notice it that much - at one point on Twe2 we changed it to a 15 minute delay an no one really complained. If users are getting pushed notifications they are normally away from a main terminal and thus are not watching twitter through TweetDeck; in short you don't need realtime to be that realtime - Also 99.99% of people don't get that many notifications a day, polling too often is a waste of time. - We supported about 40,000 users off 1 small VPS. - To get DM's you will need to use the users credentials (oauth or otherwise), a 2 minute interval means that you will use 30 of the users requests per hour (this might have changed) and as such they might get annoyed. - 500,000 users is pretty optimistic I wouldn't even worry about that scale just yet, just get your stuff working for now. - User since_id everywhere you can. - We launched with the ability to have quiet periods, that is no notifications while I am sleeping - people will thank you for this. Based on new developments of Twitter you can use something like follow, shadow and birddog - it offers a migration plan too, start with follow to get all the tweets from a user and to a user (200 users is good to test your API works), then when you launch request twitter to allow you to use shadow (50,000 users is a lot and will probably suit your app for a long time). Then as soon as you see a tweet on the stream you know it is for some of your users and you can fire it straight to them. the only issue is that these API's only get proper replies and not mentions. Currently none of the Streaming API's will help you for DM's (AFAIK). Paul Kinlan, http://www.Twollo.com 2009/8/21 ke...@nibirutech.com > > Hi > > I am a developer , trying to figure out a way to develop a push > notification solution for iPhone users. > > The easy way to do the push work, is to have a cron-job to check > users' new mentions and DMs. It should work for small number of > users. What if we have a large user base, say , 500, 000 users at > least? How can we use a proper solution to get a 2-minutes delay push > for any user's mentions and DMs? (we can't afford the server cost for > half million requests every 2 minutes) > > I know there are a few Twitter push clients for iPhone , but none of > them can work on a scaled user base, am I right? > > Is there a twitter tech support here? could you please give some > suggestions? >
[twitter-dev] Re: large user base push notification solutions?
On Aug 21, 12:06 am, "ke...@nibirutech.com" wrote: > What if we have a large user base, say , 500, 000 users at > least? How can we use a proper solution to get a 2-minutes delay push > for any user's mentions and DMs? (we can't afford the server cost for > half million requests every 2 minutes) You are actually talking about one million API calls every 2 minutes (1 for mentions, one for DMs). That's 8,300 API calls per second. My rough estimate is that you are going to need around 200 servers to cope with that workload. Dewald
[twitter-dev] Re: large user base push notification solutions?
If you're already talking about a 2-minute delay, then why "push" and not "pull" ?? Polling clients will give you greater scalability with that range of latency easily achievable for little investment. Push is meant for immediate notifications. True push to 500k+ clients is costly -- but you don't need millisecond latency here. Pull is much, much cheaper. ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me: http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera) On Fri, Aug 21, 2009 at 8:07 AM, Dewald Pretorius wrote: > > In addition, your database will have to cope with 8,300 writes per > second. And then you need to take into account the latency of the > Apple Push Notification service. > > Dewald > > On Aug 21, 8:57 am, Dewald Pretorius wrote: >> On Aug 21, 12:06 am, "ke...@nibirutech.com" >> wrote: >> >> > What if we have a large user base, say , 500, 000 users at >> > least? How can we use a proper solution to get a 2-minutes delay push >> > for any user's mentions and DMs? (we can't afford the server cost for >> > half million requests every 2 minutes) >> >> You are actually talking about one million API calls every 2 minutes >> (1 for mentions, one for DMs). >> >> That's 8,300 API calls per second. >> >> My rough estimate is that you are going to need around 200 servers to >> cope with that workload. >> >> Dewald >
[twitter-dev] Re: large user base push notification solutions?
In addition, your database will have to cope with 8,300 writes per second. And then you need to take into account the latency of the Apple Push Notification service. Dewald On Aug 21, 8:57 am, Dewald Pretorius wrote: > On Aug 21, 12:06 am, "ke...@nibirutech.com" > wrote: > > > What if we have a large user base, say , 500, 000 users at > > least? How can we use a proper solution to get a 2-minutes delay push > > for any user's mentions and DMs? (we can't afford the server cost for > > half million requests every 2 minutes) > > You are actually talking about one million API calls every 2 minutes > (1 for mentions, one for DMs). > > That's 8,300 API calls per second. > > My rough estimate is that you are going to need around 200 servers to > cope with that workload. > > Dewald
