[twitter-dev] newbie question

[kual...@gmail.com]

I am seeking a file that has a record by twitter user.

the record would contain minimally the lat-lon coordinates of the
twitter user location.

preferred addl fields would be date and number of tweets; ideally this
would be a file generated daily.

does this exist in a downloadable form?

I seek no personal or contact data; only looking to develop an app
(global to neighborhood map) that shows level of twitter activity.

thank you.

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk

Re: [twitter-dev] newbie question on rate limits

[M. Edward (Ed) Borasky]

Quoting Vijay :


OK Tom, will queue it and spread it, that should give me more room.

But that still doesn't answer the question - how do the big companies
manage?
Do they have multiple IPs?


The *really* big companies - like Google and Microsoft - just connect  
to the Streaming API "Firehose" with one IP address, and maybe a  
backup IP address or two, and index every tweet the Firehose sends  
them. ;-)


--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb

"A mathematician is a device for turning coffee into theorems." - Paul Erdos


--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk?hl=en

Re: [twitter-dev] newbie question on rate limits

[Vijay]

Right, that makes sense.

On Mon, Sep 20, 2010 at 11:52 AM, Tom van der Woerdt  wrote:

> Of course they do :-) Each server gets an IP and gets 2 requests per
> hour. With 20 servers that means 40 requests per hour. Besides, I'd
> assume that these services spread their requests properly and don't
> update every hour. Just because you have 2 requests per hour doesn't
> mean you can use them as much as you want. (API Terms of Service, I think.)
>
> Tom
>
>
> On 9/20/10 5:44 PM, Vijay wrote:
> > OK Tom, will queue it and spread it, that should give me more room.
> >
> > But that still doesn't answer the question - how do the big companies
> > manage?
> > Do they have multiple IPs?
> >
> > On Mon, Sep 20, 2010 at 11:41 AM, Tom van der Woerdt  > > wrote:
> >
> > Just don't and you'll be fine :-) Just queue the requests and spread
> > them over a few hours. It may also help to setup a Twitter account,
> as
> > it will allow you to make 350 requests per hour instead.
> >
> > Tom
> >
> >
> > On 9/20/10 5:38 PM, Vijay wrote:
> > >
> > > "You really shouldn't be planning for the case where every single
> > Twitter
> > > user uses your application"
> > >
> > > Haha, I agree.  Its just that I took a few thousand users
> > randomly, and
> > > was playing
> > > with their publicly available stats, tweets etc and quickly hit
> > the rate
> > > limit.  Hence
> > > this question.
> > >
> > > On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt  > 
> > > >> wrote:
> > >
> > > You really shouldn't be planning for the case where every
> > single Twitter
> > > user uses your application ;-) When you get to 100 users, you
> > request
> > > whitelisting. When you get to 2 users, your server will
> have a
> > > problem handling all the information and you'll need to get a
> > second
> > > server anyway.
> > >
> > > Tom
> > >
> > >
> > > On 9/20/10 5:17 PM, Vijay wrote:
> > > > I am not sure about 20k, but 150 is miniscule.
> > > > If I am collecting stats on a bunch of users every hour for
> > example, I
> > > > can only
> > > > collect on 150 users, which is tiny, compared to 140 million
> > users
> > > > Twitter has.
> > > >
> > > > How do the big companies manage?  For example, twittercounter
> > > claims to
> > > > have stats on 10 million users...that is a LOT of data.
> > > >
> > > > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt
> > mailto:i...@tvdw.eu>
> > > >
> > > > 
> >  > > >
> > > > 20k per IP is a lot. It means that a server should make
> more
> > > than 5
> > > > requests per second to hit the limit, which is a lot.
> > > >
> > > > About your options: no idea. Just make sure to use the
> > proper
> > > functions
> > > > and try not to hit the limits? :-) 150 is a lot as well,
> > most
> > > Desktop
> > > > clients don't hit it (didn't, until Lists came ^^).
> > > >
> > > > Tom
> > > >
> > > >
> > > > On 9/20/10 4:56 PM, Vijay wrote:
> > > > > Hi,
> > > > >
> > > > > Newbie question, so please bear with me.
> > > > >
> > > > > I am experimenting with twitter API, but quickly found
> > > myself hitting
> > > > > the rate limit
> > > > > (150 without authentication, correct?)
> > > > >
> > > > > How do the big sites get over the rate limit
> (tweetstats,
> > > > twittercounter
> > > > > etc)?
> > > > > Twitter's documentation says they can white list IPs,
> but
> > > would still
> > > > > allow only
> > > > > 20k requests per hour.  Also, they would only white
> list
> > > apps that are
> > > > > already in
> > > > > production.  So if I just want to experiment / learn,
> this
> > > > wouldn't work.
> > > > >
> > > > > What are my options?  There is no way I can have
> > multiple IPs.
> > > > >
> > > > > Vijay.
> > > > >
> > > > > --
> > > > > Twitter developer documentation and resources:
> > > > http://dev.twitter.com/doc
> > > > > API updates via Twitter: http://twitter.com/twitterapi
> > > > > Issues/Enhancements Tracker:
> > > > > http://code.google.com/p/twitter-api/issues/list
>

Re: [twitter-dev] newbie question on rate limits

[John Kalucki]

Large companies, and anyone who wants sustained access to high-volume
Twitter data, use the Streaming API. The REST API is primarily for client
applications and very-small-scale integrations. Note that REST whitelisting
is unlikely for use-cases that are covered by the Streaming API.

-John Kalucki
http://twitter.com/jkalucki
Twitter, Inc.


On Mon, Sep 20, 2010 at 8:44 AM, Vijay  wrote:

> OK Tom, will queue it and spread it, that should give me more room.
>
> But that still doesn't answer the question - how do the big companies
> manage?
> Do they have multiple IPs?
>
> On Mon, Sep 20, 2010 at 11:41 AM, Tom van der Woerdt  wrote:
>
>> Just don't and you'll be fine :-) Just queue the requests and spread
>> them over a few hours. It may also help to setup a Twitter account, as
>> it will allow you to make 350 requests per hour instead.
>>
>> Tom
>>
>>
>> On 9/20/10 5:38 PM, Vijay wrote:
>> >
>> > "You really shouldn't be planning for the case where every single
>> Twitter
>> > user uses your application"
>> >
>> > Haha, I agree.  Its just that I took a few thousand users randomly, and
>> > was playing
>> > with their publicly available stats, tweets etc and quickly hit the rate
>> > limit.  Hence
>> > this question.
>> >
>> > On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt > > > wrote:
>> >
>> > You really shouldn't be planning for the case where every single
>> Twitter
>> > user uses your application ;-) When you get to 100 users, you
>> request
>> > whitelisting. When you get to 2 users, your server will have a
>> > problem handling all the information and you'll need to get a second
>> > server anyway.
>> >
>> > Tom
>> >
>> >
>> > On 9/20/10 5:17 PM, Vijay wrote:
>> > > I am not sure about 20k, but 150 is miniscule.
>> > > If I am collecting stats on a bunch of users every hour for
>> example, I
>> > > can only
>> > > collect on 150 users, which is tiny, compared to 140 million users
>> > > Twitter has.
>> > >
>> > > How do the big companies manage?  For example, twittercounter
>> > claims to
>> > > have stats on 10 million users...that is a LOT of data.
>> > >
>> > > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt <
>> i...@tvdw.eu
>> > 
>> > > >> wrote:
>> > >
>> > > 20k per IP is a lot. It means that a server should make more
>> > than 5
>> > > requests per second to hit the limit, which is a lot.
>> > >
>> > > About your options: no idea. Just make sure to use the proper
>> > functions
>> > > and try not to hit the limits? :-) 150 is a lot as well, most
>> > Desktop
>> > > clients don't hit it (didn't, until Lists came ^^).
>> > >
>> > > Tom
>> > >
>> > >
>> > > On 9/20/10 4:56 PM, Vijay wrote:
>> > > > Hi,
>> > > >
>> > > > Newbie question, so please bear with me.
>> > > >
>> > > > I am experimenting with twitter API, but quickly found
>> > myself hitting
>> > > > the rate limit
>> > > > (150 without authentication, correct?)
>> > > >
>> > > > How do the big sites get over the rate limit (tweetstats,
>> > > twittercounter
>> > > > etc)?
>> > > > Twitter's documentation says they can white list IPs, but
>> > would still
>> > > > allow only
>> > > > 20k requests per hour.  Also, they would only white list
>> > apps that are
>> > > > already in
>> > > > production.  So if I just want to experiment / learn, this
>> > > wouldn't work.
>> > > >
>> > > > What are my options?  There is no way I can have multiple
>> IPs.
>> > > >
>> > > > Vijay.
>> > > >
>> > > > --
>> > > > Twitter developer documentation and resources:
>> > > http://dev.twitter.com/doc
>> > > > API updates via Twitter: http://twitter.com/twitterapi
>> > > > Issues/Enhancements Tracker:
>> > > > http://code.google.com/p/twitter-api/issues/list
>> > > > Change your membership to this group:
>> > > >
>> http://groups.google.com/group/twitter-development-talk?hl=en
>> > >
>> > > --
>> > > Twitter developer documentation and resources:
>> > > http://dev.twitter.com/doc
>> > > API updates via Twitter: http://twitter.com/twitterapi
>> > > Issues/Enhancements Tracker:
>> > > http://code.google.com/p/twitter-api/issues/list
>> > > Change your membership to this group:
>> > > http://groups.google.com/group/twitter-development-talk?hl=en
>> > >
>> > >
>> > > --
>> > > Twitter developer documentation and resources:
>> > http://dev.twitter.com/doc
>> > > API updates via Twitter: http://twitter.com/twitterapi
>> > > Issues/Enhancements Tracker:
>> > 

Re: [twitter-dev] newbie question on rate limits

[Tom van der Woerdt]

Of course they do :-) Each server gets an IP and gets 2 requests per
hour. With 20 servers that means 40 requests per hour. Besides, I'd
assume that these services spread their requests properly and don't
update every hour. Just because you have 2 requests per hour doesn't
mean you can use them as much as you want. (API Terms of Service, I think.)

Tom


On 9/20/10 5:44 PM, Vijay wrote:
> OK Tom, will queue it and spread it, that should give me more room.
> 
> But that still doesn't answer the question - how do the big companies
> manage?
> Do they have multiple IPs? 
> 
> On Mon, Sep 20, 2010 at 11:41 AM, Tom van der Woerdt  > wrote:
> 
> Just don't and you'll be fine :-) Just queue the requests and spread
> them over a few hours. It may also help to setup a Twitter account, as
> it will allow you to make 350 requests per hour instead.
> 
> Tom
> 
> 
> On 9/20/10 5:38 PM, Vijay wrote:
> >
> > "You really shouldn't be planning for the case where every single
> Twitter
> > user uses your application"
> >
> > Haha, I agree.  Its just that I took a few thousand users
> randomly, and
> > was playing
> > with their publicly available stats, tweets etc and quickly hit
> the rate
> > limit.  Hence
> > this question.
> >
> > On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt  
> > >> wrote:
> >
> > You really shouldn't be planning for the case where every
> single Twitter
> > user uses your application ;-) When you get to 100 users, you
> request
> > whitelisting. When you get to 2 users, your server will have a
> > problem handling all the information and you'll need to get a
> second
> > server anyway.
> >
> > Tom
> >
> >
> > On 9/20/10 5:17 PM, Vijay wrote:
> > > I am not sure about 20k, but 150 is miniscule.
> > > If I am collecting stats on a bunch of users every hour for
> example, I
> > > can only
> > > collect on 150 users, which is tiny, compared to 140 million
> users
> > > Twitter has.
> > >
> > > How do the big companies manage?  For example, twittercounter
> > claims to
> > > have stats on 10 million users...that is a LOT of data.
> > >
> > > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt
> mailto:i...@tvdw.eu>
> > >
> > > 
>  > >
> > > 20k per IP is a lot. It means that a server should make more
> > than 5
> > > requests per second to hit the limit, which is a lot.
> > >
> > > About your options: no idea. Just make sure to use the
> proper
> > functions
> > > and try not to hit the limits? :-) 150 is a lot as well,
> most
> > Desktop
> > > clients don't hit it (didn't, until Lists came ^^).
> > >
> > > Tom
> > >
> > >
> > > On 9/20/10 4:56 PM, Vijay wrote:
> > > > Hi,
> > > >
> > > > Newbie question, so please bear with me.
> > > >
> > > > I am experimenting with twitter API, but quickly found
> > myself hitting
> > > > the rate limit
> > > > (150 without authentication, correct?)
> > > >
> > > > How do the big sites get over the rate limit (tweetstats,
> > > twittercounter
> > > > etc)?
> > > > Twitter's documentation says they can white list IPs, but
> > would still
> > > > allow only
> > > > 20k requests per hour.  Also, they would only white list
> > apps that are
> > > > already in
> > > > production.  So if I just want to experiment / learn, this
> > > wouldn't work.
> > > >
> > > > What are my options?  There is no way I can have
> multiple IPs.
> > > >
> > > > Vijay.
> > > >
> > > > --
> > > > Twitter developer documentation and resources:
> > > http://dev.twitter.com/doc
> > > > API updates via Twitter: http://twitter.com/twitterapi
> > > > Issues/Enhancements Tracker:
> > > > http://code.google.com/p/twitter-api/issues/list
> > > > Change your membership to this group:
> > > >
> http://groups.google.com/group/twitter-development-talk?hl=en
> > >
> > > --
> > > Twitter developer documentation and resources:
> > > http://dev.twitter.com/doc
> > > API updates via Twitter: http:

Re: [twitter-dev] newbie question on rate limits

[Vijay]

OK Tom, will queue it and spread it, that should give me more room.

But that still doesn't answer the question - how do the big companies
manage?
Do they have multiple IPs?

On Mon, Sep 20, 2010 at 11:41 AM, Tom van der Woerdt  wrote:

> Just don't and you'll be fine :-) Just queue the requests and spread
> them over a few hours. It may also help to setup a Twitter account, as
> it will allow you to make 350 requests per hour instead.
>
> Tom
>
>
> On 9/20/10 5:38 PM, Vijay wrote:
> >
> > "You really shouldn't be planning for the case where every single Twitter
> > user uses your application"
> >
> > Haha, I agree.  Its just that I took a few thousand users randomly, and
> > was playing
> > with their publicly available stats, tweets etc and quickly hit the rate
> > limit.  Hence
> > this question.
> >
> > On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt  > > wrote:
> >
> > You really shouldn't be planning for the case where every single
> Twitter
> > user uses your application ;-) When you get to 100 users, you request
> > whitelisting. When you get to 2 users, your server will have a
> > problem handling all the information and you'll need to get a second
> > server anyway.
> >
> > Tom
> >
> >
> > On 9/20/10 5:17 PM, Vijay wrote:
> > > I am not sure about 20k, but 150 is miniscule.
> > > If I am collecting stats on a bunch of users every hour for
> example, I
> > > can only
> > > collect on 150 users, which is tiny, compared to 140 million users
> > > Twitter has.
> > >
> > > How do the big companies manage?  For example, twittercounter
> > claims to
> > > have stats on 10 million users...that is a LOT of data.
> > >
> > > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt  > 
> > > >> wrote:
> > >
> > > 20k per IP is a lot. It means that a server should make more
> > than 5
> > > requests per second to hit the limit, which is a lot.
> > >
> > > About your options: no idea. Just make sure to use the proper
> > functions
> > > and try not to hit the limits? :-) 150 is a lot as well, most
> > Desktop
> > > clients don't hit it (didn't, until Lists came ^^).
> > >
> > > Tom
> > >
> > >
> > > On 9/20/10 4:56 PM, Vijay wrote:
> > > > Hi,
> > > >
> > > > Newbie question, so please bear with me.
> > > >
> > > > I am experimenting with twitter API, but quickly found
> > myself hitting
> > > > the rate limit
> > > > (150 without authentication, correct?)
> > > >
> > > > How do the big sites get over the rate limit (tweetstats,
> > > twittercounter
> > > > etc)?
> > > > Twitter's documentation says they can white list IPs, but
> > would still
> > > > allow only
> > > > 20k requests per hour.  Also, they would only white list
> > apps that are
> > > > already in
> > > > production.  So if I just want to experiment / learn, this
> > > wouldn't work.
> > > >
> > > > What are my options?  There is no way I can have multiple
> IPs.
> > > >
> > > > Vijay.
> > > >
> > > > --
> > > > Twitter developer documentation and resources:
> > > http://dev.twitter.com/doc
> > > > API updates via Twitter: http://twitter.com/twitterapi
> > > > Issues/Enhancements Tracker:
> > > > http://code.google.com/p/twitter-api/issues/list
> > > > Change your membership to this group:
> > > >
> http://groups.google.com/group/twitter-development-talk?hl=en
> > >
> > > --
> > > Twitter developer documentation and resources:
> > > http://dev.twitter.com/doc
> > > API updates via Twitter: http://twitter.com/twitterapi
> > > Issues/Enhancements Tracker:
> > > http://code.google.com/p/twitter-api/issues/list
> > > Change your membership to this group:
> > > http://groups.google.com/group/twitter-development-talk?hl=en
> > >
> > >
> > > --
> > > Twitter developer documentation and resources:
> > http://dev.twitter.com/doc
> > > API updates via Twitter: http://twitter.com/twitterapi
> > > Issues/Enhancements Tracker:
> > > http://code.google.com/p/twitter-api/issues/list
> > > Change your membership to this group:
> > > http://groups.google.com/group/twitter-development-talk?hl=en
> >
> > --
> > Twitter developer documentation and resources:
> > http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-t

Re: [twitter-dev] newbie question on rate limits

[Tom van der Woerdt]

Just don't and you'll be fine :-) Just queue the requests and spread
them over a few hours. It may also help to setup a Twitter account, as
it will allow you to make 350 requests per hour instead.

Tom


On 9/20/10 5:38 PM, Vijay wrote:
> 
> "You really shouldn't be planning for the case where every single Twitter
> user uses your application"
> 
> Haha, I agree.  Its just that I took a few thousand users randomly, and
> was playing
> with their publicly available stats, tweets etc and quickly hit the rate
> limit.  Hence
> this question.
> 
> On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt  > wrote:
> 
> You really shouldn't be planning for the case where every single Twitter
> user uses your application ;-) When you get to 100 users, you request
> whitelisting. When you get to 2 users, your server will have a
> problem handling all the information and you'll need to get a second
> server anyway.
> 
> Tom
> 
> 
> On 9/20/10 5:17 PM, Vijay wrote:
> > I am not sure about 20k, but 150 is miniscule.
> > If I am collecting stats on a bunch of users every hour for example, I
> > can only
> > collect on 150 users, which is tiny, compared to 140 million users
> > Twitter has.
> >
> > How do the big companies manage?  For example, twittercounter
> claims to
> > have stats on 10 million users...that is a LOT of data.
> >
> > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt  
> > >> wrote:
> >
> > 20k per IP is a lot. It means that a server should make more
> than 5
> > requests per second to hit the limit, which is a lot.
> >
> > About your options: no idea. Just make sure to use the proper
> functions
> > and try not to hit the limits? :-) 150 is a lot as well, most
> Desktop
> > clients don't hit it (didn't, until Lists came ^^).
> >
> > Tom
> >
> >
> > On 9/20/10 4:56 PM, Vijay wrote:
> > > Hi,
> > >
> > > Newbie question, so please bear with me.
> > >
> > > I am experimenting with twitter API, but quickly found
> myself hitting
> > > the rate limit
> > > (150 without authentication, correct?)
> > >
> > > How do the big sites get over the rate limit (tweetstats,
> > twittercounter
> > > etc)?
> > > Twitter's documentation says they can white list IPs, but
> would still
> > > allow only
> > > 20k requests per hour.  Also, they would only white list
> apps that are
> > > already in
> > > production.  So if I just want to experiment / learn, this
> > wouldn't work.
> > >
> > > What are my options?  There is no way I can have multiple IPs.
> > >
> > > Vijay.
> > >
> > > --
> > > Twitter developer documentation and resources:
> > http://dev.twitter.com/doc
> > > API updates via Twitter: http://twitter.com/twitterapi
> > > Issues/Enhancements Tracker:
> > > http://code.google.com/p/twitter-api/issues/list
> > > Change your membership to this group:
> > > http://groups.google.com/group/twitter-development-talk?hl=en
> >
> > --
> > Twitter developer documentation and resources:
> > http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-talk?hl=en
> >
> >
> > --
> > Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-talk?hl=en
> 
> --
> Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en
> 
> 
> -- 
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi

Re: [twitter-dev] newbie question on rate limits

[Vijay]

"You really shouldn't be planning for the case where every single Twitter
user uses your application"

Haha, I agree.  Its just that I took a few thousand users randomly, and was
playing
with their publicly available stats, tweets etc and quickly hit the rate
limit.  Hence
this question.

On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt  wrote:

> You really shouldn't be planning for the case where every single Twitter
> user uses your application ;-) When you get to 100 users, you request
> whitelisting. When you get to 2 users, your server will have a
> problem handling all the information and you'll need to get a second
> server anyway.
>
> Tom
>
>
> On 9/20/10 5:17 PM, Vijay wrote:
> > I am not sure about 20k, but 150 is miniscule.
> > If I am collecting stats on a bunch of users every hour for example, I
> > can only
> > collect on 150 users, which is tiny, compared to 140 million users
> > Twitter has.
> >
> > How do the big companies manage?  For example, twittercounter claims to
> > have stats on 10 million users...that is a LOT of data.
> >
> > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt  > > wrote:
> >
> > 20k per IP is a lot. It means that a server should make more than 5
> > requests per second to hit the limit, which is a lot.
> >
> > About your options: no idea. Just make sure to use the proper
> functions
> > and try not to hit the limits? :-) 150 is a lot as well, most Desktop
> > clients don't hit it (didn't, until Lists came ^^).
> >
> > Tom
> >
> >
> > On 9/20/10 4:56 PM, Vijay wrote:
> > > Hi,
> > >
> > > Newbie question, so please bear with me.
> > >
> > > I am experimenting with twitter API, but quickly found myself
> hitting
> > > the rate limit
> > > (150 without authentication, correct?)
> > >
> > > How do the big sites get over the rate limit (tweetstats,
> > twittercounter
> > > etc)?
> > > Twitter's documentation says they can white list IPs, but would
> still
> > > allow only
> > > 20k requests per hour.  Also, they would only white list apps that
> are
> > > already in
> > > production.  So if I just want to experiment / learn, this
> > wouldn't work.
> > >
> > > What are my options?  There is no way I can have multiple IPs.
> > >
> > > Vijay.
> > >
> > > --
> > > Twitter developer documentation and resources:
> > http://dev.twitter.com/doc
> > > API updates via Twitter: http://twitter.com/twitterapi
> > > Issues/Enhancements Tracker:
> > > http://code.google.com/p/twitter-api/issues/list
> > > Change your membership to this group:
> > > http://groups.google.com/group/twitter-development-talk?hl=en
> >
> > --
> > Twitter developer documentation and resources:
> > http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-talk?hl=en
> >
> >
> > --
> > Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-talk?hl=en
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en
>

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk?hl=en

Re: [twitter-dev] newbie question on rate limits

[Tom van der Woerdt]

You really shouldn't be planning for the case where every single Twitter
user uses your application ;-) When you get to 100 users, you request
whitelisting. When you get to 2 users, your server will have a
problem handling all the information and you'll need to get a second
server anyway.

Tom


On 9/20/10 5:17 PM, Vijay wrote:
> I am not sure about 20k, but 150 is miniscule.
> If I am collecting stats on a bunch of users every hour for example, I
> can only
> collect on 150 users, which is tiny, compared to 140 million users
> Twitter has.
> 
> How do the big companies manage?  For example, twittercounter claims to
> have stats on 10 million users...that is a LOT of data.
> 
> On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt  > wrote:
> 
> 20k per IP is a lot. It means that a server should make more than 5
> requests per second to hit the limit, which is a lot.
> 
> About your options: no idea. Just make sure to use the proper functions
> and try not to hit the limits? :-) 150 is a lot as well, most Desktop
> clients don't hit it (didn't, until Lists came ^^).
> 
> Tom
> 
> 
> On 9/20/10 4:56 PM, Vijay wrote:
> > Hi,
> >
> > Newbie question, so please bear with me.
> >
> > I am experimenting with twitter API, but quickly found myself hitting
> > the rate limit
> > (150 without authentication, correct?)
> >
> > How do the big sites get over the rate limit (tweetstats,
> twittercounter
> > etc)?
> > Twitter's documentation says they can white list IPs, but would still
> > allow only
> > 20k requests per hour.  Also, they would only white list apps that are
> > already in
> > production.  So if I just want to experiment / learn, this
> wouldn't work.
> >
> > What are my options?  There is no way I can have multiple IPs.
> >
> > Vijay.
> >
> > --
> > Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-talk?hl=en
> 
> --
> Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en
> 
> 
> -- 
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk?hl=en

Re: [twitter-dev] newbie question on rate limits

[Vijay]

I am not sure about 20k, but 150 is miniscule.
If I am collecting stats on a bunch of users every hour for example, I can
only
collect on 150 users, which is tiny, compared to 140 million users Twitter
has.

How do the big companies manage?  For example, twittercounter claims to
have stats on 10 million users...that is a LOT of data.

On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt  wrote:

> 20k per IP is a lot. It means that a server should make more than 5
> requests per second to hit the limit, which is a lot.
>
> About your options: no idea. Just make sure to use the proper functions
> and try not to hit the limits? :-) 150 is a lot as well, most Desktop
> clients don't hit it (didn't, until Lists came ^^).
>
> Tom
>
>
> On 9/20/10 4:56 PM, Vijay wrote:
> > Hi,
> >
> > Newbie question, so please bear with me.
> >
> > I am experimenting with twitter API, but quickly found myself hitting
> > the rate limit
> > (150 without authentication, correct?)
> >
> > How do the big sites get over the rate limit (tweetstats, twittercounter
> > etc)?
> > Twitter's documentation says they can white list IPs, but would still
> > allow only
> > 20k requests per hour.  Also, they would only white list apps that are
> > already in
> > production.  So if I just want to experiment / learn, this wouldn't work.
> >
> > What are my options?  There is no way I can have multiple IPs.
> >
> > Vijay.
> >
> > --
> > Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> > API updates via Twitter: http://twitter.com/twitterapi
> > Issues/Enhancements Tracker:
> > http://code.google.com/p/twitter-api/issues/list
> > Change your membership to this group:
> > http://groups.google.com/group/twitter-development-talk?hl=en
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en
>

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk?hl=en

Re: [twitter-dev] newbie question on rate limits

[Tom van der Woerdt]

20k per IP is a lot. It means that a server should make more than 5
requests per second to hit the limit, which is a lot.

About your options: no idea. Just make sure to use the proper functions
and try not to hit the limits? :-) 150 is a lot as well, most Desktop
clients don't hit it (didn't, until Lists came ^^).

Tom


On 9/20/10 4:56 PM, Vijay wrote:
> Hi,
> 
> Newbie question, so please bear with me.
> 
> I am experimenting with twitter API, but quickly found myself hitting
> the rate limit
> (150 without authentication, correct?)
> 
> How do the big sites get over the rate limit (tweetstats, twittercounter
> etc)?
> Twitter's documentation says they can white list IPs, but would still
> allow only
> 20k requests per hour.  Also, they would only white list apps that are
> already in
> production.  So if I just want to experiment / learn, this wouldn't work. 
> 
> What are my options?  There is no way I can have multiple IPs.
> 
> Vijay.
> 
> -- 
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk?hl=en

[twitter-dev] newbie question on rate limits

[Vijay]

Hi,

Newbie question, so please bear with me.

I am experimenting with twitter API, but quickly found myself hitting the
rate limit
(150 without authentication, correct?)

How do the big sites get over the rate limit (tweetstats, twittercounter
etc)?
Twitter's documentation says they can white list IPs, but would still allow
only
20k requests per hour.  Also, they would only white list apps that are
already in
production.  So if I just want to experiment / learn, this wouldn't work.

What are my options?  There is no way I can have multiple IPs.

Vijay.

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk?hl=en

Re: [twitter-dev] Newbie question - please help.

[Abraham Williams]

Try using the popular feature of the search method:
http://dev.twitter.com/doc/get/search

Abraham

On Thu, May 20, 2010 at 15:28, gus  wrote:

> Hello,
>
> I want to develop a web-based application that will search for the
> most favorited tweets for a given hashtag. For example, a user may
> type "cerati" in a textbox and the application will return the tweets
> that contain the hashtag #cerati and have been favorited more than x
> (3, 4, 5) times.
>
> My questions are:
> - Is this possible?
> - If so, what method should I use?
>
> It appears to me that if I use the search API I don't get a
> favourite_count among the results. And I can't specify something like
> "minimum amount of times tweet was favorited" as a parameter.
>
> Or maybe I can use the search API to get the tweets that contain the
> hashtag and then iterate thorugh the list to obtain the number of
> times each tweet was favorited?
>
> Another alternative: instead of doing this with the public timeline
> (ideal solution) do it for all tweets by the users a person is
> following. Can this be done?
>
> Thanks for your help.
>
> Gustavo
>



-- 
Abraham Williams | Developer for hire | http://abrah.am
@abraham | http://projects.abrah.am | http://blog.abrah.am
This email is: [ ] shareable [x] ask first [ ] private.

[twitter-dev] Newbie question - please help.

[gus]

Hello,

I want to develop a web-based application that will search for the
most favorited tweets for a given hashtag. For example, a user may
type "cerati" in a textbox and the application will return the tweets
that contain the hashtag #cerati and have been favorited more than x
(3, 4, 5) times.

My questions are:
- Is this possible?
- If so, what method should I use?

It appears to me that if I use the search API I don't get a
favourite_count among the results. And I can't specify something like
"minimum amount of times tweet was favorited" as a parameter.

Or maybe I can use the search API to get the tweets that contain the
hashtag and then iterate thorugh the list to obtain the number of
times each tweet was favorited?

Another alternative: instead of doing this with the public timeline
(ideal solution) do it for all tweets by the users a person is
following. Can this be done?

Thanks for your help.

Gustavo

[twitter-dev] Newbie Question about signing requests

[CB]

I am testing the Twitter API for a Firefox extension I am writing, and
trying out the OAuth flow for desktop clients, as described on
http://twitterapi.pbworks.com/Authentication.

I get stuck in the very first step "The application uses oauth/
request_token to obtain a request token from twitter.com." and have a
newbie question about the signing.

I construct a GET request which looks similar to this:

http://twitter.com/oauth/request_token?oauth_consumer_key=&oauth_signature_method=HMAC-SHA1&oauth_timestamp=&oauth_nonce=?oauth_signature=

My question: What string is actually to be signed with HMAC-SHA1 and
included as value for oauth_signature? (I am using the sha1 javascript
implementation from http://pajhome.org.uk/crypt/md5/)

I figure I should encode my client's consumer secret with my client's
consumer key, but I only get "Failed to validate oauth signature and
token".

Sorry for this dumb question, but I hope someone can point me in the
right direction.

Thanks.

[twitter-dev] Newbie question, how to modify the settings of Twitter app that I registered ?

[CG]

I have registered a Twitter App at http://twitter.com/oauth_clients/new,
and I was provided a consumer key and secret.

However, I found that I have wrongly set the app as Client rather than
web apps, how can I alter that via twitter UI ?

Or I need to register a new apps instead ?