Re: [twitter-dev] Re: No of statuses extracted by statuses/filter
The only way to get limited is to specify a too broad predicate and go
beyond the allowed proportion of tweets. If you specify too many keywords,
you aren't limited, your connection is rejected. This is all documented on
http://dev.twitter.com/pages/streaming_api, specifically:
http://dev.twitter.com/pages/streaming_api_concepts#parsing-responses
-John Kalucki
http://twitter.com/jkalucki
Twitter, Inc.
On Thu, Oct 14, 2010 at 12:29 PM, AA wrote:
> Thanks a lot!
> This is very helpful.
>
> John:
>
> You said:
> "If you don't receive a limit message, you know that you've received
> all
> possible tweets for the predicate"
>
> But:
> -The only way to get limited in status/filter is using more keywords
> or more users id than is allowed according to access level?
> Is there any other way?
>
> -The limit message contains some kind of sum info? (Additionnally,
> where can I find the "data spec" for this limit message and for data
> returned by status/filter in general?)
>
> Thanks in advance.
> Alejandro.
>
>
>
> On Oct 12, 7:17 pm, John Kalucki wrote:
> > Sorry. Gmail fail / Groups fail.
> >
> >
> >
> > On Tue, Oct 12, 2010 at 3:17 PM, John Kalucki wrote:
> > > If you don't receive a limit message, you know that you've received all
> > > possible tweets for the predicate. If you do receive a limit message,
> you
> > > know the precise proportion of tweets received and dropped.
> >
> > > -John Kalucki
> > >http://twitter.com/jkalucki
> > > Twitter Inc.
> >
> > > On Tue, Oct 12, 2010 at 2:36 PM, AA
> wrote:
> >
> > >> Hi everybody!
> > >> Thank you Edward.
> >
> > >> I copy paste part of your answer:
> >
> > >> ["If your filter criteria are sufficiently narrow, you get *all* of
> > >> the public tweets with those keywords sent by users who aren't being
> > >> blocked by Twitter's quality filter." At least that's what the
> > >> documentation has said in the past.]
> >
> > >> -Can anyone confirm this?
> > >> -I think, taking Edward's approach, I've still the same problem : even
> > >> taking a "very narrow" criteria I can never know what's the total, so
> > >> I can'´t know if all the tweets got by streaming are useful or not.
> > >> I think I have to remark that I don't need to know an exact total of
> > >> tweets in a given moment. What I'd like to know is an approximate
> > >> percentage over some approximate total of tweets estimation. I dare to
> > >> think it's part of the "service providing specification".
> >
> > >> I do understand that it can be difficult to exactly define "total of
> > >> tweets" when streaming and having tweets going into Twitter
> > >> permanently but not constantly, but some estimated info would be
> > >> great.
> >
> > >> Thank you all in advance.
> > >> Alejandro.
> >
> > >> On Oct 11, 5:57 pm, "M. Edward (Ed) Borasky" > >> research.net> wrote:
> > >> > Quoting AA :
> >
> > >> > > Hi everybody!
> > >> > > I'm designing an app to do some mining over a corpus of tweets.
> > >> > > I think I'll use streaming api, statuses/filter filtering by
> keywords.
> >
> > >> > > I'd like to know, before starting development, what is the
> percentage
> > >> > > of tweets delivered by this stream over the total tweets
> ('meaning
> > >> > > total tweets' the total of tweets that have the tracking keywords)
> .
> > >> > > This is information is crucial because of statistical confidence:
> a
> > >> > > very little sample may not be significant.
> >
> > >> > > Addittionally, Ive been googling and reading a lot for 3 days and
> I
> > >> > > can't figure out how i can use different 'level accesses'.
> > >> > > I've readhttp://
> > >> dev.twitter.com/pages/streaming_api_methods#statuses-filter
> > >> > > but how can I use this different levels levels of access?
> >
> > >> > > Thanks in advance!
> > >> > > Regards
> > >> > > Alejandro.
> >
> > >> > I actually think the answer to *yout* question is, "If your filter
> > >> > criteria are sufficiently narrow, you get *all* of the public tweets
> > >> > with those keywords sent by users who aren't being blocked by
> > >> > Twitter's quality filter." At least that's what the documentation
> has
> > >> > said in the past.
> >
> > >> > But *my* question is, "How does one determine the total number of
> > >> > tweets, for some definition of total?
> >
> > >> > a. All tweets created, including those that aren't public?
> > >> > b. All public tweets created, including those from "low quality
> users"
> > >> > that don't get indexed by search or sent to the "filter" stream?
> > >> > c. All tweets sent to the inlet of the filter stream and the various
> > >> > elevated access level stream?
> >
> > >> > Remind me again - when does "Snowflake" go live? I haven't looked at
> > >> > Streaming data for a couple months.
> >
> > >> > --
> > >> > M. Edward (Ed) Boraskyhttp://borasky-research.nethttp://
> > >> twitter.com/znmeb
> >
> > >> > "A mathematician is a device for turning coffee into theorems." -
> Paul
> > >> Erdos
> >
> > >> --
> > >> Twitter develo
[twitter-dev] Re: No of statuses extracted by statuses/filter
Thanks a lot!
This is very helpful.
John:
You said:
"If you don't receive a limit message, you know that you've received
all
possible tweets for the predicate"
But:
-The only way to get limited in status/filter is using more keywords
or more users id than is allowed according to access level?
Is there any other way?
-The limit message contains some kind of sum info? (Additionnally,
where can I find the "data spec" for this limit message and for data
returned by status/filter in general?)
Thanks in advance.
Alejandro.
On Oct 12, 7:17 pm, John Kalucki wrote:
> Sorry. Gmail fail / Groups fail.
>
>
>
> On Tue, Oct 12, 2010 at 3:17 PM, John Kalucki wrote:
> > If you don't receive a limit message, you know that you've received all
> > possible tweets for the predicate. If you do receive a limit message, you
> > know the precise proportion of tweets received and dropped.
>
> > -John Kalucki
> >http://twitter.com/jkalucki
> > Twitter Inc.
>
> > On Tue, Oct 12, 2010 at 2:36 PM, AA wrote:
>
> >> Hi everybody!
> >> Thank you Edward.
>
> >> I copy paste part of your answer:
>
> >> ["If your filter criteria are sufficiently narrow, you get *all* of
> >> the public tweets with those keywords sent by users who aren't being
> >> blocked by Twitter's quality filter." At least that's what the
> >> documentation has said in the past.]
>
> >> -Can anyone confirm this?
> >> -I think, taking Edward's approach, I've still the same problem : even
> >> taking a "very narrow" criteria I can never know what's the total, so
> >> I can'´t know if all the tweets got by streaming are useful or not.
> >> I think I have to remark that I don't need to know an exact total of
> >> tweets in a given moment. What I'd like to know is an approximate
> >> percentage over some approximate total of tweets estimation. I dare to
> >> think it's part of the "service providing specification".
>
> >> I do understand that it can be difficult to exactly define "total of
> >> tweets" when streaming and having tweets going into Twitter
> >> permanently but not constantly, but some estimated info would be
> >> great.
>
> >> Thank you all in advance.
> >> Alejandro.
>
> >> On Oct 11, 5:57 pm, "M. Edward (Ed) Borasky" >> research.net> wrote:
> >> > Quoting AA :
>
> >> > > Hi everybody!
> >> > > I'm designing an app to do some mining over a corpus of tweets.
> >> > > I think I'll use streaming api, statuses/filter filtering by keywords.
>
> >> > > I'd like to know, before starting development, what is the percentage
> >> > > of tweets delivered by this stream over the total tweets ('meaning
> >> > > total tweets' the total of tweets that have the tracking keywords) .
> >> > > This is information is crucial because of statistical confidence: a
> >> > > very little sample may not be significant.
>
> >> > > Addittionally, Ive been googling and reading a lot for 3 days and I
> >> > > can't figure out how i can use different 'level accesses'.
> >> > > I've readhttp://
> >> dev.twitter.com/pages/streaming_api_methods#statuses-filter
> >> > > but how can I use this different levels levels of access?
>
> >> > > Thanks in advance!
> >> > > Regards
> >> > > Alejandro.
>
> >> > I actually think the answer to *yout* question is, "If your filter
> >> > criteria are sufficiently narrow, you get *all* of the public tweets
> >> > with those keywords sent by users who aren't being blocked by
> >> > Twitter's quality filter." At least that's what the documentation has
> >> > said in the past.
>
> >> > But *my* question is, "How does one determine the total number of
> >> > tweets, for some definition of total?
>
> >> > a. All tweets created, including those that aren't public?
> >> > b. All public tweets created, including those from "low quality users"
> >> > that don't get indexed by search or sent to the "filter" stream?
> >> > c. All tweets sent to the inlet of the filter stream and the various
> >> > elevated access level stream?
>
> >> > Remind me again - when does "Snowflake" go live? I haven't looked at
> >> > Streaming data for a couple months.
>
> >> > --
> >> > M. Edward (Ed) Boraskyhttp://borasky-research.nethttp://
> >> twitter.com/znmeb
>
> >> > "A mathematician is a device for turning coffee into theorems." - Paul
> >> Erdos
>
> >> --
> >> Twitter developer documentation and resources:http://dev.twitter.com/doc
> >> API updates via Twitter:http://twitter.com/twitterapi
> >> Issues/Enhancements Tracker:
> >>http://code.google.com/p/twitter-api/issues/list
> >> Change your membership to this group:
> >>http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: No of statuses extracted by statuses/filter
Sorry. Gmail fail / Groups fail.
On Tue, Oct 12, 2010 at 3:17 PM, John Kalucki wrote:
> If you don't receive a limit message, you know that you've received all
> possible tweets for the predicate. If you do receive a limit message, you
> know the precise proportion of tweets received and dropped.
>
> -John Kalucki
> http://twitter.com/jkalucki
> Twitter Inc.
>
>
>
> On Tue, Oct 12, 2010 at 2:36 PM, AA wrote:
>
>> Hi everybody!
>> Thank you Edward.
>>
>> I copy paste part of your answer:
>>
>> ["If your filter criteria are sufficiently narrow, you get *all* of
>> the public tweets with those keywords sent by users who aren't being
>> blocked by Twitter's quality filter." At least that's what the
>> documentation has said in the past.]
>>
>> -Can anyone confirm this?
>> -I think, taking Edward's approach, I've still the same problem : even
>> taking a "very narrow" criteria I can never know what's the total, so
>> I can'´t know if all the tweets got by streaming are useful or not.
>> I think I have to remark that I don't need to know an exact total of
>> tweets in a given moment. What I'd like to know is an approximate
>> percentage over some approximate total of tweets estimation. I dare to
>> think it's part of the "service providing specification".
>>
>> I do understand that it can be difficult to exactly define "total of
>> tweets" when streaming and having tweets going into Twitter
>> permanently but not constantly, but some estimated info would be
>> great.
>>
>> Thank you all in advance.
>> Alejandro.
>>
>>
>> On Oct 11, 5:57 pm, "M. Edward (Ed) Borasky" > research.net> wrote:
>> > Quoting AA :
>> >
>> >
>> >
>> >
>> >
>> > > Hi everybody!
>> > > I'm designing an app to do some mining over a corpus of tweets.
>> > > I think I'll use streaming api, statuses/filter filtering by keywords.
>> >
>> > > I'd like to know, before starting development, what is the percentage
>> > > of tweets delivered by this stream over the total tweets ('meaning
>> > > total tweets' the total of tweets that have the tracking keywords) .
>> > > This is information is crucial because of statistical confidence: a
>> > > very little sample may not be significant.
>> >
>> > > Addittionally, Ive been googling and reading a lot for 3 days and I
>> > > can't figure out how i can use different 'level accesses'.
>> > > I've readhttp://
>> dev.twitter.com/pages/streaming_api_methods#statuses-filter
>> > > but how can I use this different levels levels of access?
>> >
>> > > Thanks in advance!
>> > > Regards
>> > > Alejandro.
>> >
>> > I actually think the answer to *yout* question is, "If your filter
>> > criteria are sufficiently narrow, you get *all* of the public tweets
>> > with those keywords sent by users who aren't being blocked by
>> > Twitter's quality filter." At least that's what the documentation has
>> > said in the past.
>> >
>> > But *my* question is, "How does one determine the total number of
>> > tweets, for some definition of total?
>> >
>> > a. All tweets created, including those that aren't public?
>> > b. All public tweets created, including those from "low quality users"
>> > that don't get indexed by search or sent to the "filter" stream?
>> > c. All tweets sent to the inlet of the filter stream and the various
>> > elevated access level stream?
>> >
>> > Remind me again - when does "Snowflake" go live? I haven't looked at
>> > Streaming data for a couple months.
>> >
>> > --
>> > M. Edward (Ed) Boraskyhttp://borasky-research.nethttp://
>> twitter.com/znmeb
>> >
>> > "A mathematician is a device for turning coffee into theorems." - Paul
>> Erdos
>>
>> --
>> Twitter developer documentation and resources: http://dev.twitter.com/doc
>> API updates via Twitter: http://twitter.com/twitterapi
>> Issues/Enhancements Tracker:
>> http://code.google.com/p/twitter-api/issues/list
>> Change your membership to this group:
>> http://groups.google.com/group/twitter-development-talk
>>
>
>
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: No of statuses extracted by statuses/filter
If you don't receive a limit message, you know that you've received all
possible tweets for the predicate. If you do receive a limit message, you
know the precise proportion of tweets received and dropped.
-John Kalucki
http://twitter.com/jkalucki
Twitter Inc.
On Tue, Oct 12, 2010 at 2:36 PM, AA wrote:
> Hi everybody!
> Thank you Edward.
>
> I copy paste part of your answer:
>
> ["If your filter criteria are sufficiently narrow, you get *all* of
> the public tweets with those keywords sent by users who aren't being
> blocked by Twitter's quality filter." At least that's what the
> documentation has said in the past.]
>
> -Can anyone confirm this?
> -I think, taking Edward's approach, I've still the same problem : even
> taking a "very narrow" criteria I can never know what's the total, so
> I can'´t know if all the tweets got by streaming are useful or not.
> I think I have to remark that I don't need to know an exact total of
> tweets in a given moment. What I'd like to know is an approximate
> percentage over some approximate total of tweets estimation. I dare to
> think it's part of the "service providing specification".
>
> I do understand that it can be difficult to exactly define "total of
> tweets" when streaming and having tweets going into Twitter
> permanently but not constantly, but some estimated info would be
> great.
>
> Thank you all in advance.
> Alejandro.
>
>
> On Oct 11, 5:57 pm, "M. Edward (Ed) Borasky" research.net> wrote:
> > Quoting AA :
> >
> >
> >
> >
> >
> > > Hi everybody!
> > > I'm designing an app to do some mining over a corpus of tweets.
> > > I think I'll use streaming api, statuses/filter filtering by keywords.
> >
> > > I'd like to know, before starting development, what is the percentage
> > > of tweets delivered by this stream over the total tweets ('meaning
> > > total tweets' the total of tweets that have the tracking keywords) .
> > > This is information is crucial because of statistical confidence: a
> > > very little sample may not be significant.
> >
> > > Addittionally, Ive been googling and reading a lot for 3 days and I
> > > can't figure out how i can use different 'level accesses'.
> > > I've readhttp://
> dev.twitter.com/pages/streaming_api_methods#statuses-filter
> > > but how can I use this different levels levels of access?
> >
> > > Thanks in advance!
> > > Regards
> > > Alejandro.
> >
> > I actually think the answer to *yout* question is, "If your filter
> > criteria are sufficiently narrow, you get *all* of the public tweets
> > with those keywords sent by users who aren't being blocked by
> > Twitter's quality filter." At least that's what the documentation has
> > said in the past.
> >
> > But *my* question is, "How does one determine the total number of
> > tweets, for some definition of total?
> >
> > a. All tweets created, including those that aren't public?
> > b. All public tweets created, including those from "low quality users"
> > that don't get indexed by search or sent to the "filter" stream?
> > c. All tweets sent to the inlet of the filter stream and the various
> > elevated access level stream?
> >
> > Remind me again - when does "Snowflake" go live? I haven't looked at
> > Streaming data for a couple months.
> >
> > --
> > M. Edward (Ed) Boraskyhttp://borasky-research.nethttp://
> twitter.com/znmeb
> >
> > "A mathematician is a device for turning coffee into theorems." - Paul
> Erdos
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: No of statuses extracted by statuses/filter
Hey Alejandro,
When you receive Tweets through the Streaming API filter method we
will deliver all the Tweets that match the keywords/user_ids you ask
for. If there are more Tweets than your Sample is allowed we will send
a 'rate limited' message indicating how many you missed. This means,
if you don't get a rate limited message, that you got all the Tweets
for that moment in time.
Hope that helps,
@themattharris
Developer Advocate, Twitter
http://twitter.com/themattharris
On Tue, Oct 12, 2010 at 2:36 PM, AA wrote:
> Hi everybody!
> Thank you Edward.
>
> I copy paste part of your answer:
>
> ["If your filter criteria are sufficiently narrow, you get *all* of
> the public tweets with those keywords sent by users who aren't being
> blocked by Twitter's quality filter." At least that's what the
> documentation has said in the past.]
>
> -Can anyone confirm this?
> -I think, taking Edward's approach, I've still the same problem : even
> taking a "very narrow" criteria I can never know what's the total, so
> I can'´t know if all the tweets got by streaming are useful or not.
> I think I have to remark that I don't need to know an exact total of
> tweets in a given moment. What I'd like to know is an approximate
> percentage over some approximate total of tweets estimation. I dare to
> think it's part of the "service providing specification".
>
> I do understand that it can be difficult to exactly define "total of
> tweets" when streaming and having tweets going into Twitter
> permanently but not constantly, but some estimated info would be
> great.
>
> Thank you all in advance.
> Alejandro.
>
>
> On Oct 11, 5:57 pm, "M. Edward (Ed) Borasky" research.net> wrote:
>> Quoting AA :
>>
>>
>>
>>
>>
>> > Hi everybody!
>> > I'm designing an app to do some mining over a corpus of tweets.
>> > I think I'll use streaming api, statuses/filter filtering by keywords.
>>
>> > I'd like to know, before starting development, what is the percentage
>> > of tweets delivered by this stream over the total tweets ('meaning
>> > total tweets' the total of tweets that have the tracking keywords) .
>> > This is information is crucial because of statistical confidence: a
>> > very little sample may not be significant.
>>
>> > Addittionally, Ive been googling and reading a lot for 3 days and I
>> > can't figure out how i can use different 'level accesses'.
>> > I've readhttp://dev.twitter.com/pages/streaming_api_methods#statuses-filter
>> > but how can I use this different levels levels of access?
>>
>> > Thanks in advance!
>> > Regards
>> > Alejandro.
>>
>> I actually think the answer to *yout* question is, "If your filter
>> criteria are sufficiently narrow, you get *all* of the public tweets
>> with those keywords sent by users who aren't being blocked by
>> Twitter's quality filter." At least that's what the documentation has
>> said in the past.
>>
>> But *my* question is, "How does one determine the total number of
>> tweets, for some definition of total?
>>
>> a. All tweets created, including those that aren't public?
>> b. All public tweets created, including those from "low quality users"
>> that don't get indexed by search or sent to the "filter" stream?
>> c. All tweets sent to the inlet of the filter stream and the various
>> elevated access level stream?
>>
>> Remind me again - when does "Snowflake" go live? I haven't looked at
>> Streaming data for a couple months.
>>
>> --
>> M. Edward (Ed) Boraskyhttp://borasky-research.nethttp://twitter.com/znmeb
>>
>> "A mathematician is a device for turning coffee into theorems." - Paul Erdos
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: No of statuses extracted by statuses/filter
Hi everybody!
Thank you Edward.
I copy paste part of your answer:
["If your filter criteria are sufficiently narrow, you get *all* of
the public tweets with those keywords sent by users who aren't being
blocked by Twitter's quality filter." At least that's what the
documentation has said in the past.]
-Can anyone confirm this?
-I think, taking Edward's approach, I've still the same problem : even
taking a "very narrow" criteria I can never know what's the total, so
I can'´t know if all the tweets got by streaming are useful or not.
I think I have to remark that I don't need to know an exact total of
tweets in a given moment. What I'd like to know is an approximate
percentage over some approximate total of tweets estimation. I dare to
think it's part of the "service providing specification".
I do understand that it can be difficult to exactly define "total of
tweets" when streaming and having tweets going into Twitter
permanently but not constantly, but some estimated info would be
great.
Thank you all in advance.
Alejandro.
On Oct 11, 5:57 pm, "M. Edward (Ed) Borasky" wrote:
> Quoting AA :
>
>
>
>
>
> > Hi everybody!
> > I'm designing an app to do some mining over a corpus of tweets.
> > I think I'll use streaming api, statuses/filter filtering by keywords.
>
> > I'd like to know, before starting development, what is the percentage
> > of tweets delivered by this stream over the total tweets ('meaning
> > total tweets' the total of tweets that have the tracking keywords) .
> > This is information is crucial because of statistical confidence: a
> > very little sample may not be significant.
>
> > Addittionally, Ive been googling and reading a lot for 3 days and I
> > can't figure out how i can use different 'level accesses'.
> > I've readhttp://dev.twitter.com/pages/streaming_api_methods#statuses-filter
> > but how can I use this different levels levels of access?
>
> > Thanks in advance!
> > Regards
> > Alejandro.
>
> I actually think the answer to *yout* question is, "If your filter
> criteria are sufficiently narrow, you get *all* of the public tweets
> with those keywords sent by users who aren't being blocked by
> Twitter's quality filter." At least that's what the documentation has
> said in the past.
>
> But *my* question is, "How does one determine the total number of
> tweets, for some definition of total?
>
> a. All tweets created, including those that aren't public?
> b. All public tweets created, including those from "low quality users"
> that don't get indexed by search or sent to the "filter" stream?
> c. All tweets sent to the inlet of the filter stream and the various
> elevated access level stream?
>
> Remind me again - when does "Snowflake" go live? I haven't looked at
> Streaming data for a couple months.
>
> --
> M. Edward (Ed) Boraskyhttp://borasky-research.nethttp://twitter.com/znmeb
>
> "A mathematician is a device for turning coffee into theorems." - Paul Erdos
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
