Re: [twitter-dev] Re: Missing replies to followed user (streaming api)
I'm still thinking about how this might be happening and how we might debug this situation. It isn't trivially reproducible. In the mean time, I've started some reproduction cases that I'm going to let age for a few hours and see what happens there. -John On Tue, Mar 23, 2010 at 9:16 PM, briantroy brian.cosin...@gmail.com wrote: John - Here is a good example: These two tweets came in 2 minutes apart from 2 different 3rd party clients. We got one and not the other: http://bit.ly/cG9emM (Got this one) http://bit.ly/aFvimS (Not this one) No limit messages, re-connects or anything at all unusual about the stream activity during this period. Brian On Mar 23, 4:05 pm, briantroy brian.cosin...@gmail.com wrote: John - We rarely get limit messages - and no, none have coincided with missing replies. Our last limit message was yesterday at: NOTICE: TRACK - NOTICE: Mon, 22 Mar 2010 22:52:28 -0400 Stream is limited The track user is justsignal - the follow user is justsignalshad I'll have to dig through the logs to find the last re-connect time. Just as a note, I didn't get any of your test tweets (as a matter of fact the last I've gotten was at 2:24 pacific. Our last re-connect was before Mon, 22 Mar 2010 15:42:30 -0400 (this is when our latest log file starts). Let me know if I can do anything else to track this down... Brian On Mar 23, 3:31 pm, John Kalucki j...@twitter.com wrote: I couldn't trivially reproduce this. But, I'll say that I suspect that there may indeed be some sort of bug around tracking on screen names and not always getting all tweets. There might be some combination of text reply vs. explicit reply vs. protected vs. private vs. something that causes problems. I've just retested a few combinations, and they seem to work. Did you receive any limit messages on the stream? Could you have been track rate limited at the moment when this message was sent? If send your account and the time, in UTC, that you logged in the stream, I can look to see how many limit messages we've sent to you. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, Mar 23, 2010 at 2:28 PM, briantroy brian.cosin...@gmail.com wrote: Mark - Two separate threads (one user is restricted track, the other is shadow). Track user/thread is running track=briantroy (along with 230 something other terms). The Follow thread/user is follow=4246611 - which is my userid (along with 60 something other user id's). Both threads have logging turned up to list each tweet id they receive. These threads DO NOTHING with the JSON beyond popping it onto a queue. Brian On Mar 23, 2:19 pm, Mark McBride mmcbr...@twitter.com wrote: What track/follow paramaters were you using to get this? follow=briantroytrack=briantroy? ---Mark http://twitter.com/mccv On Tue, Mar 23, 2010 at 1:42 PM, briantroy brian.cosin...@gmail.com wrote: We seem to consistently miss replies to users using the streaming api. Our methodology: If one of our users specifies a twitter user name we both follow the user (using the folllow= predicate) and track on the user's username (using the track= predicate). Our assumption was that this would get structured replies and retweets (via the follow) and unstructured (via the track). This appears to not be the case. We never get some replies. For example: Tweet ID: 10942140971 was never received (in reply to me). Most of these appear to come from 3rd party clients - but this example came from the web. We can confirm (via our logs) that the tweet never comes in on the stream. Anyone else seeing anything like this or have a proven methodology for getting all mentions? Thanks, Brian Roy To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Re: Missing replies to followed user (streaming api)
John - We isolated a bug on our side today that was causing some of the issue. I'm being told (waiting on proof) that there are still some replies that never come in via the stream. I'll update this post when I know more. Thanks, Brian On Mar 24, 6:29 am, John Kalucki j...@twitter.com wrote: I'm still thinking about how this might be happening and how we might debug this situation. It isn't trivially reproducible. In the mean time, I've started some reproduction cases that I'm going to let age for a few hours and see what happens there. -John On Tue, Mar 23, 2010 at 9:16 PM, briantroy brian.cosin...@gmail.com wrote: John - Here is a good example: These two tweets came in 2 minutes apart from 2 different 3rd party clients. We got one and not the other: http://bit.ly/cG9emM(Got this one) http://bit.ly/aFvimS(Not this one) No limit messages, re-connects or anything at all unusual about the stream activity during this period. Brian On Mar 23, 4:05 pm, briantroy brian.cosin...@gmail.com wrote: John - We rarely get limit messages - and no, none have coincided with missing replies. Our last limit message was yesterday at: NOTICE: TRACK - NOTICE: Mon, 22 Mar 2010 22:52:28 -0400 Stream is limited The track user is justsignal - the follow user is justsignalshad I'll have to dig through the logs to find the last re-connect time. Just as a note, I didn't get any of your test tweets (as a matter of fact the last I've gotten was at 2:24 pacific. Our last re-connect was before Mon, 22 Mar 2010 15:42:30 -0400 (this is when our latest log file starts). Let me know if I can do anything else to track this down... Brian On Mar 23, 3:31 pm, John Kalucki j...@twitter.com wrote: I couldn't trivially reproduce this. But, I'll say that I suspect that there may indeed be some sort of bug around tracking on screen names and not always getting all tweets. There might be some combination of text reply vs. explicit reply vs. protected vs. private vs. something that causes problems. I've just retested a few combinations, and they seem to work. Did you receive any limit messages on the stream? Could you have been track rate limited at the moment when this message was sent? If send your account and the time, in UTC, that you logged in the stream, I can look to see how many limit messages we've sent to you. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, Mar 23, 2010 at 2:28 PM, briantroy brian.cosin...@gmail.com wrote: Mark - Two separate threads (one user is restricted track, the other is shadow). Track user/thread is running track=briantroy (along with 230 something other terms). The Follow thread/user is follow=4246611 - which is my userid (along with 60 something other user id's). Both threads have logging turned up to list each tweet id they receive. These threads DO NOTHING with the JSON beyond popping it onto a queue. Brian On Mar 23, 2:19 pm, Mark McBride mmcbr...@twitter.com wrote: What track/follow paramaters were you using to get this? follow=briantroytrack=briantroy? ---Mark http://twitter.com/mccv On Tue, Mar 23, 2010 at 1:42 PM, briantroy brian.cosin...@gmail.com wrote: We seem to consistently miss replies to users using the streaming api. Our methodology: If one of our users specifies a twitter user name we both follow the user (using the folllow= predicate) and track on the user's username (using the track= predicate). Our assumption was that this would get structured replies and retweets (via the follow) and unstructured (via the track). This appears to not be the case. We never get some replies. For example: Tweet ID: 10942140971 was never received (in reply to me). Most of these appear to come from 3rd party clients - but this example came from the web. We can confirm (via our logs) that the tweet never comes in on the stream. Anyone else seeing anything like this or have a proven methodology for getting all mentions? Thanks, Brian Roy To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email
[twitter-dev] Re: Missing replies to followed user (streaming api)
Mark - Two separate threads (one user is restricted track, the other is shadow). Track user/thread is running track=briantroy (along with 230 something other terms). The Follow thread/user is follow=4246611 - which is my userid (along with 60 something other user id's). Both threads have logging turned up to list each tweet id they receive. These threads DO NOTHING with the JSON beyond popping it onto a queue. Brian On Mar 23, 2:19 pm, Mark McBride mmcbr...@twitter.com wrote: What track/follow paramaters were you using to get this? follow=briantroytrack=briantroy? ---Mark http://twitter.com/mccv On Tue, Mar 23, 2010 at 1:42 PM, briantroy brian.cosin...@gmail.com wrote: We seem to consistently miss replies to users using the streaming api. Our methodology: If one of our users specifies a twitter user name we both follow the user (using the folllow= predicate) and track on the user's username (using the track= predicate). Our assumption was that this would get structured replies and retweets (via the follow) and unstructured (via the track). This appears to not be the case. We never get some replies. For example: Tweet ID: 10942140971 was never received (in reply to me). Most of these appear to come from 3rd party clients - but this example came from the web. We can confirm (via our logs) that the tweet never comes in on the stream. Anyone else seeing anything like this or have a proven methodology for getting all mentions? Thanks, Brian Roy To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
Re: [twitter-dev] Re: Missing replies to followed user (streaming api)
I couldn't trivially reproduce this. But, I'll say that I suspect that there may indeed be some sort of bug around tracking on screen names and not always getting all tweets. There might be some combination of text reply vs. explicit reply vs. protected vs. private vs. something that causes problems. I've just retested a few combinations, and they seem to work. Did you receive any limit messages on the stream? Could you have been track rate limited at the moment when this message was sent? If send your account and the time, in UTC, that you logged in the stream, I can look to see how many limit messages we've sent to you. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, Mar 23, 2010 at 2:28 PM, briantroy brian.cosin...@gmail.com wrote: Mark - Two separate threads (one user is restricted track, the other is shadow). Track user/thread is running track=briantroy (along with 230 something other terms). The Follow thread/user is follow=4246611 - which is my userid (along with 60 something other user id's). Both threads have logging turned up to list each tweet id they receive. These threads DO NOTHING with the JSON beyond popping it onto a queue. Brian On Mar 23, 2:19 pm, Mark McBride mmcbr...@twitter.com wrote: What track/follow paramaters were you using to get this? follow=briantroytrack=briantroy? ---Mark http://twitter.com/mccv On Tue, Mar 23, 2010 at 1:42 PM, briantroy brian.cosin...@gmail.com wrote: We seem to consistently miss replies to users using the streaming api. Our methodology: If one of our users specifies a twitter user name we both follow the user (using the folllow= predicate) and track on the user's username (using the track= predicate). Our assumption was that this would get structured replies and retweets (via the follow) and unstructured (via the track). This appears to not be the case. We never get some replies. For example: Tweet ID: 10942140971 was never received (in reply to me). Most of these appear to come from 3rd party clients - but this example came from the web. We can confirm (via our logs) that the tweet never comes in on the stream. Anyone else seeing anything like this or have a proven methodology for getting all mentions? Thanks, Brian Roy To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Re: Missing replies to followed user (streaming api)
John - We rarely get limit messages - and no, none have coincided with missing replies. Our last limit message was yesterday at: NOTICE: TRACK - NOTICE: Mon, 22 Mar 2010 22:52:28 -0400 Stream is limited The track user is justsignal - the follow user is justsignalshad I'll have to dig through the logs to find the last re-connect time. Just as a note, I didn't get any of your test tweets (as a matter of fact the last I've gotten was at 2:24 pacific. Our last re-connect was before Mon, 22 Mar 2010 15:42:30 -0400 (this is when our latest log file starts). Let me know if I can do anything else to track this down... Brian On Mar 23, 3:31 pm, John Kalucki j...@twitter.com wrote: I couldn't trivially reproduce this. But, I'll say that I suspect that there may indeed be some sort of bug around tracking on screen names and not always getting all tweets. There might be some combination of text reply vs. explicit reply vs. protected vs. private vs. something that causes problems. I've just retested a few combinations, and they seem to work. Did you receive any limit messages on the stream? Could you have been track rate limited at the moment when this message was sent? If send your account and the time, in UTC, that you logged in the stream, I can look to see how many limit messages we've sent to you. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, Mar 23, 2010 at 2:28 PM, briantroy brian.cosin...@gmail.com wrote: Mark - Two separate threads (one user is restricted track, the other is shadow). Track user/thread is running track=briantroy (along with 230 something other terms). The Follow thread/user is follow=4246611 - which is my userid (along with 60 something other user id's). Both threads have logging turned up to list each tweet id they receive. These threads DO NOTHING with the JSON beyond popping it onto a queue. Brian On Mar 23, 2:19 pm, Mark McBride mmcbr...@twitter.com wrote: What track/follow paramaters were you using to get this? follow=briantroytrack=briantroy? ---Mark http://twitter.com/mccv On Tue, Mar 23, 2010 at 1:42 PM, briantroy brian.cosin...@gmail.com wrote: We seem to consistently miss replies to users using the streaming api. Our methodology: If one of our users specifies a twitter user name we both follow the user (using the folllow= predicate) and track on the user's username (using the track= predicate). Our assumption was that this would get structured replies and retweets (via the follow) and unstructured (via the track). This appears to not be the case. We never get some replies. For example: Tweet ID: 10942140971 was never received (in reply to me). Most of these appear to come from 3rd party clients - but this example came from the web. We can confirm (via our logs) that the tweet never comes in on the stream. Anyone else seeing anything like this or have a proven methodology for getting all mentions? Thanks, Brian Roy To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Re: Missing replies to followed user (streaming api)
John - Here is a good example: These two tweets came in 2 minutes apart from 2 different 3rd party clients. We got one and not the other: http://bit.ly/cG9emM (Got this one) http://bit.ly/aFvimS (Not this one) No limit messages, re-connects or anything at all unusual about the stream activity during this period. Brian On Mar 23, 4:05 pm, briantroy brian.cosin...@gmail.com wrote: John - We rarely get limit messages - and no, none have coincided with missing replies. Our last limit message was yesterday at: NOTICE: TRACK - NOTICE: Mon, 22 Mar 2010 22:52:28 -0400 Stream is limited The track user is justsignal - the follow user is justsignalshad I'll have to dig through the logs to find the last re-connect time. Just as a note, I didn't get any of your test tweets (as a matter of fact the last I've gotten was at 2:24 pacific. Our last re-connect was before Mon, 22 Mar 2010 15:42:30 -0400 (this is when our latest log file starts). Let me know if I can do anything else to track this down... Brian On Mar 23, 3:31 pm, John Kalucki j...@twitter.com wrote: I couldn't trivially reproduce this. But, I'll say that I suspect that there may indeed be some sort of bug around tracking on screen names and not always getting all tweets. There might be some combination of text reply vs. explicit reply vs. protected vs. private vs. something that causes problems. I've just retested a few combinations, and they seem to work. Did you receive any limit messages on the stream? Could you have been track rate limited at the moment when this message was sent? If send your account and the time, in UTC, that you logged in the stream, I can look to see how many limit messages we've sent to you. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, Mar 23, 2010 at 2:28 PM, briantroy brian.cosin...@gmail.com wrote: Mark - Two separate threads (one user is restricted track, the other is shadow). Track user/thread is running track=briantroy (along with 230 something other terms). The Follow thread/user is follow=4246611 - which is my userid (along with 60 something other user id's). Both threads have logging turned up to list each tweet id they receive. These threads DO NOTHING with the JSON beyond popping it onto a queue. Brian On Mar 23, 2:19 pm, Mark McBride mmcbr...@twitter.com wrote: What track/follow paramaters were you using to get this? follow=briantroytrack=briantroy? ---Mark http://twitter.com/mccv On Tue, Mar 23, 2010 at 1:42 PM, briantroy brian.cosin...@gmail.com wrote: We seem to consistently miss replies to users using the streaming api. Our methodology: If one of our users specifies a twitter user name we both follow the user (using the folllow= predicate) and track on the user's username (using the track= predicate). Our assumption was that this would get structured replies and retweets (via the follow) and unstructured (via the track). This appears to not be the case. We never get some replies. For example: Tweet ID: 10942140971 was never received (in reply to me). Most of these appear to come from 3rd party clients - but this example came from the web. We can confirm (via our logs) that the tweet never comes in on the stream. Anyone else seeing anything like this or have a proven methodology for getting all mentions? Thanks, Brian Roy To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+ unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject. To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
