Re: [twitter-dev] Broken Json Status on Streaming API
In this period of wrong json status, I received either wrong date format of
created_at
2747941206892544 Thu Nov 11 35:42:14 + 2010
2565022072963072 Thw Nov 11 03:35:23 + 2010
256213672896 Tju Nov 11 03:23:54 + 2010
2550619441209344 Thu Nov 11 02:38:0; + 2010
2545567930523648 Vhu Nov 11 02:18:05 + 2010
and so on...
Looks like was only one char problem per status, that mess up everything.
Since Thu Nov 11 15:44:37 + 2010, I get no json parser or corrupt
created_at problem.
Thanks.
On Thu, Nov 11, 2010 at 11:51 PM, Augusto Santos augu...@gemeos.org wrote:
Hi Taylor,
First, thanks for the answer.
I'm using Phirehose lib to PHP, the native json_decode($status,TRUE) from
PHP and after json decode I'm using mysql_real_escape_string for the string
fields. I see now that my log routine use mysql escape either before query.
So this examples are escaped according this mysql procedure.
Here is the amount of tweets with this problem. That's when json_decode
didn't work, so there's no id_str or new_id_str in my $status[] array, then
it's throw an error and log it with the json status. I can send you all
these status if you want it.
DateHour(GMT-2) Count
2010-11-11 13 97
2010-11-11 1 367
2010-11-11 0 521
2010-11-10 23 598
2010-11-10 22 569
2010-11-10 21 577
2010-11-10 20 619
2010-11-10 19 606
2010-11-10 18 603
2010-11-10 17 607
2010-11-10 16 247
2010-11-10 11 9
2010-11-09 22 2
Thanks, Augusto.
On Thu, Nov 11, 2010 at 2:52 PM, Taylor Singletary
taylorsinglet...@twitter.com wrote:
Hi Augusto,
I monitored the sample stream this morning for a few hours for instances
similar to the JSON examples you've provided below and was unable to see the
scenario duplicated. What JSON parser are you using? Is there any other
processing that may have occurred prior to generating your examples below?
Do you know how your library is handling escaped quote values like \ ?
How many of these did you observe?
Thanks,
Taylor
On Thu, Nov 11, 2010 at 5:10 AM, Augusto Santos augu...@gemeos.orgwrote:
I recevied a lot of broken json status from streaming api
Count of brojen json by day
2010-11-09 2
2010-11-10 4435
2010-11-11 888
Examples:
{in_reply_to_status_id_str:2563309119209472,text:@joi4kitten I
have that same
fear.,truncated:false,in_reply_to_user_id_str:16155805,entities:{user_menvions:[{screen_name:joi4kitten,indices:[0,11],name:joi4kitten,id:16155805,id_str:16155805}],urls:[],hashtags:[]},geo:null,in_reply_to_status_id:2563309119209472,place:{country_code:US,country:The
United States of
America,bounding_box:{type:Polygon,coordinates:[[[-76.965351,38.971109],[-76.909147,38.971109],[-76.909147,39.022114],[-76.965353,39.022114]]]},place_type:city,attributes:{},full_name:College
Park, MD,name:College Park,*
id:e4c17912c815124d.url:http:\/\/api.twitter.com
\/1\/geo\/id\/e4c17912c815124d.json*},favorited:false,source:\u003Ca
href=\http:\/\/mobile.twitter.com\ rel=\nofollow\\u003EMobile
Web\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:joi4kitten,coorfinates:null,retweet_count:null,in_reply_to_user_id:16155805,created_at:Thu
Nov 11 03:38:52 +
2010,new_id_str:2565897919139841,new_id:2565897919139841,user:{follow_request_sent:null,lang:en,time_zone:Eastern
Time (US
Canada),screen_name:kellygo,following:null,profile_sidebar_border_color:0A84A5,profile_background_image_url:http:\/\/
a3.twimg.com\/profile_background_images\/8381831\/twitter_background_with_bear_2.jpg,notifications:null,description:I
am not Kelly
Osbourne.,listed_count:12,profile_background_tile:false,profile_background_color:052743,statuses_count:1559,location:College
Park,
Md.,show_all_inline_media:false,profile_use_background_image:true,favourites_count:29,profile_text_color:00,contributors_enabled:false,protected:false,geo_enabled:true,created_at:Sun
Jul 22 02:12:05 + 2007,profile_link_color:0A84A5,name:Kelly
Osborn,friends_count:235,url:http:\/\/kellygo.blogspot.com
,profile_image_url:http:\/\/a1.twimg.com
\/profile_images\/271727429\/kelly_o__posterized_normal.jpg,id:7632582,id_str:7632582,verified:false,utc_offset:-18000,profile_sidebar_fill_color:CFE9E2,followers_count:332},retweeted:false,id:2565897919139841,id_str:2565897919139841}
Here there's a dot where should be a comma, between id and url
{in_reply_to_status_id_str:2546974377119744,text:@ozdj Ah,
suddenly #TweetBurglar makes complete
sense,truncated:false,in_reply_to_user_id_str:15324154,entities:{user_mentions:[{screen_name:ozdj,indices:[0,5],name:Derek
Jenkins,id:15324154,id_str:15324154}],urls:[],hashtags:[{text:TweetBurglar,indices:[19,32]}]},geo:{type:Point,coordinates:[-33.88533896,151.20496326]},in_reply_to_status_id:2546974377119744,place:null,favorited:false,source:\u003Ca
href=\http:\/\/twitter.com\/\ rel=\nofollow\\u003ETwitter for
[twitter-dev] Broken Json Status on Streaming API
I recevied a lot of broken json status from streaming api
Count of brojen json by day
2010-11-09 2
2010-11-10 4435
2010-11-11 888
Examples:
{in_reply_to_status_id_str:2563309119209472,text:@joi4kitten I have
that same
fear.,truncated:false,in_reply_to_user_id_str:16155805,entities:{user_menvions:[{screen_name:joi4kitten,indices:[0,11],name:joi4kitten,id:16155805,id_str:16155805}],urls:[],hashtags:[]},geo:null,in_reply_to_status_id:2563309119209472,place:{country_code:US,country:The
United States of
America,bounding_box:{type:Polygon,coordinates:[[[-76.965351,38.971109],[-76.909147,38.971109],[-76.909147,39.022114],[-76.965353,39.022114]]]},place_type:city,attributes:{},full_name:College
Park, MD,name:College Park,*id:e4c17912c815124d.url:http:\/\/
api.twitter.com\/1\/geo\/id\/e4c17912c815124d.json*},favorited:false,source:\u003Ca
href=\http:\/\/mobile.twitter.com\ rel=\nofollow\\u003EMobile
Web\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:joi4kitten,coorfinates:null,retweet_count:null,in_reply_to_user_id:16155805,created_at:Thu
Nov 11 03:38:52 +
2010,new_id_str:2565897919139841,new_id:2565897919139841,user:{follow_request_sent:null,lang:en,time_zone:Eastern
Time (US
Canada),screen_name:kellygo,following:null,profile_sidebar_border_color:0A84A5,profile_background_image_url:http:\/\/
a3.twimg.com\/profile_background_images\/8381831\/twitter_background_with_bear_2.jpg,notifications:null,description:I
am not Kelly
Osbourne.,listed_count:12,profile_background_tile:false,profile_background_color:052743,statuses_count:1559,location:College
Park,
Md.,show_all_inline_media:false,profile_use_background_image:true,favourites_count:29,profile_text_color:00,contributors_enabled:false,protected:false,geo_enabled:true,created_at:Sun
Jul 22 02:12:05 + 2007,profile_link_color:0A84A5,name:Kelly
Osborn,friends_count:235,url:http:\/\/kellygo.blogspot.com
,profile_image_url:http:\/\/a1.twimg.com
\/profile_images\/271727429\/kelly_o__posterized_normal.jpg,id:7632582,id_str:7632582,verified:false,utc_offset:-18000,profile_sidebar_fill_color:CFE9E2,followers_count:332},retweeted:false,id:2565897919139841,id_str:2565897919139841}
Here there's a dot where should be a comma, between id and url
{in_reply_to_status_id_str:2546974377119744,text:@ozdj Ah, suddenly
#TweetBurglar makes complete
sense,truncated:false,in_reply_to_user_id_str:15324154,entities:{user_mentions:[{screen_name:ozdj,indices:[0,5],name:Derek
Jenkins,id:15324154,id_str:15324154}],urls:[],hashtags:[{text:TweetBurglar,indices:[19,32]}]},geo:{type:Point,coordinates:[-33.88533896,151.20496326]},in_reply_to_status_id:2546974377119744,place:null,favorited:false,source:\u003Ca
href=\http:\/\/twitter.com\/\ rel=\nofollow\\u003ETwitter for
iPhone\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:ozdj,coordinates:{type:Point,coordinates:[151.20496326,-33.88533896]},retweet_count:null,in_reply_to_user_id:15324154,created_at:Thu
Nov 11 03:38:51 +
2010,new_id_str:2565895784243200,new_id:2565895784243200,user:{lang:en,geo_enabled:true,screen_name:coffeetrails,following:null,friends_count:251,profile_sidebar_border_color:362919,verified:false,follow_request_sent:null,notifications:null,time_zone:Sydney,description:Specialising
in organic RA espresso-based coffees, organic teas, fresh sandwiches and
gourmet treats. Shop 5, 2 Lee St Sydney (Henry Deane Plaza, Railway
Sq),profile_background_color:362919,profile_background_image_url:http:\/\/
a3.twimg.com\/profile_background_images\/66956323\/CRW_3458.jpg,location:Central
Station,
Sydney,favourites_count:204,profile_text_color:362919,listed_count:29,profile_background_tile:true,protected:false,statuses_count:1298,crected_at:Wed
Oct 14 05:27:32 + 2009,profile_link_color:7a2a2a,name:Jonno,
Coffee Trails,*show_all_inline_media:fanse*
,profile_use_background_image:true,url:http:\/\/www.facebook.com
\/CoffeeTrails\/,profile_image_url:http:\/\/a2.twimg.com
\/profile_images\/624183782\/CRW_2981_normal.jpg,id:82284263,id_str:82284263,contributors_enabled:false,utc_offset:36000,profile_sidebar_fill_color:c2a887,followers_count:324},retweeted:false,id:2565895784243200,id_str:2565895784243200}
Here there's a 'fanse' in 'show_all_inline_media' where should be a 'false'
Any help?
--
氣
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Broken Json Status on Streaming API
Hi Augusto,
I monitored the sample stream this morning for a few hours for instances
similar to the JSON examples you've provided below and was unable to see the
scenario duplicated. What JSON parser are you using? Is there any other
processing that may have occurred prior to generating your examples below?
Do you know how your library is handling escaped quote values like \ ?
How many of these did you observe?
Thanks,
Taylor
On Thu, Nov 11, 2010 at 5:10 AM, Augusto Santos augu...@gemeos.org wrote:
I recevied a lot of broken json status from streaming api
Count of brojen json by day
2010-11-09 2
2010-11-10 4435
2010-11-11 888
Examples:
{in_reply_to_status_id_str:2563309119209472,text:@joi4kitten I have
that same
fear.,truncated:false,in_reply_to_user_id_str:16155805,entities:{user_menvions:[{screen_name:joi4kitten,indices:[0,11],name:joi4kitten,id:16155805,id_str:16155805}],urls:[],hashtags:[]},geo:null,in_reply_to_status_id:2563309119209472,place:{country_code:US,country:The
United States of
America,bounding_box:{type:Polygon,coordinates:[[[-76.965351,38.971109],[-76.909147,38.971109],[-76.909147,39.022114],[-76.965353,39.022114]]]},place_type:city,attributes:{},full_name:College
Park, MD,name:College Park,*id:e4c17912c815124d.url:http:\/\/
api.twitter.com\/1\/geo\/id\/e4c17912c815124d.json*},favorited:false,source:\u003Ca
href=\http:\/\/mobile.twitter.com\ rel=\nofollow\\u003EMobile
Web\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:joi4kitten,coorfinates:null,retweet_count:null,in_reply_to_user_id:16155805,created_at:Thu
Nov 11 03:38:52 +
2010,new_id_str:2565897919139841,new_id:2565897919139841,user:{follow_request_sent:null,lang:en,time_zone:Eastern
Time (US
Canada),screen_name:kellygo,following:null,profile_sidebar_border_color:0A84A5,profile_background_image_url:http:\/\/
a3.twimg.com\/profile_background_images\/8381831\/twitter_background_with_bear_2.jpg,notifications:null,description:I
am not Kelly
Osbourne.,listed_count:12,profile_background_tile:false,profile_background_color:052743,statuses_count:1559,location:College
Park,
Md.,show_all_inline_media:false,profile_use_background_image:true,favourites_count:29,profile_text_color:00,contributors_enabled:false,protected:false,geo_enabled:true,created_at:Sun
Jul 22 02:12:05 + 2007,profile_link_color:0A84A5,name:Kelly
Osborn,friends_count:235,url:http:\/\/kellygo.blogspot.com
,profile_image_url:http:\/\/a1.twimg.com
\/profile_images\/271727429\/kelly_o__posterized_normal.jpg,id:7632582,id_str:7632582,verified:false,utc_offset:-18000,profile_sidebar_fill_color:CFE9E2,followers_count:332},retweeted:false,id:2565897919139841,id_str:2565897919139841}
Here there's a dot where should be a comma, between id and url
{in_reply_to_status_id_str:2546974377119744,text:@ozdj Ah, suddenly
#TweetBurglar makes complete
sense,truncated:false,in_reply_to_user_id_str:15324154,entities:{user_mentions:[{screen_name:ozdj,indices:[0,5],name:Derek
Jenkins,id:15324154,id_str:15324154}],urls:[],hashtags:[{text:TweetBurglar,indices:[19,32]}]},geo:{type:Point,coordinates:[-33.88533896,151.20496326]},in_reply_to_status_id:2546974377119744,place:null,favorited:false,source:\u003Ca
href=\http:\/\/twitter.com\/\ rel=\nofollow\\u003ETwitter for
iPhone\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:ozdj,coordinates:{type:Point,coordinates:[151.20496326,-33.88533896]},retweet_count:null,in_reply_to_user_id:15324154,created_at:Thu
Nov 11 03:38:51 +
2010,new_id_str:2565895784243200,new_id:2565895784243200,user:{lang:en,geo_enabled:true,screen_name:coffeetrails,following:null,friends_count:251,profile_sidebar_border_color:362919,verified:false,follow_request_sent:null,notifications:null,time_zone:Sydney,description:Specialising
in organic RA espresso-based coffees, organic teas, fresh sandwiches and
gourmet treats. Shop 5, 2 Lee St Sydney (Henry Deane Plaza, Railway
Sq),profile_background_color:362919,profile_background_image_url:http:\/\/
a3.twimg.com\/profile_background_images\/66956323\/CRW_3458.jpg,location:Central
Station,
Sydney,favourites_count:204,profile_text_color:362919,listed_count:29,profile_background_tile:true,protected:false,statuses_count:1298,crected_at:Wed
Oct 14 05:27:32 + 2009,profile_link_color:7a2a2a,name:Jonno,
Coffee Trails,*show_all_inline_media:fanse*
,profile_use_background_image:true,url:http:\/\/www.facebook.com
\/CoffeeTrails\/,profile_image_url:http:\/\/a2.twimg.com
\/profile_images\/624183782\/CRW_2981_normal.jpg,id:82284263,id_str:82284263,contributors_enabled:false,utc_offset:36000,profile_sidebar_fill_color:c2a887,followers_count:324},retweeted:false,id:2565895784243200,id_str:2565895784243200}
Here there's a 'fanse' in 'show_all_inline_media' where should be a 'false'
Any help?
--
氣
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Re: [twitter-dev] Broken Json Status on Streaming API
Hi Taylor,
First, thanks for the answer.
I'm using Phirehose lib to PHP, the native json_decode($status,TRUE) from
PHP and after json decode I'm using mysql_real_escape_string for the string
fields. I see now that my log routine use mysql escape either before query.
So this examples are escaped according this mysql procedure.
Here is the amount of tweets with this problem. That's when json_decode
didn't work, so there's no id_str or new_id_str in my $status[] array, then
it's throw an error and log it with the json status. I can send you all
these status if you want it.
DateHour(GMT-2) Count
2010-11-11 13 97
2010-11-11 1 367
2010-11-11 0 521
2010-11-10 23 598
2010-11-10 22 569
2010-11-10 21 577
2010-11-10 20 619
2010-11-10 19 606
2010-11-10 18 603
2010-11-10 17 607
2010-11-10 16 247
2010-11-10 11 9
2010-11-09 22 2
Thanks, Augusto.
On Thu, Nov 11, 2010 at 2:52 PM, Taylor Singletary
taylorsinglet...@twitter.com wrote:
Hi Augusto,
I monitored the sample stream this morning for a few hours for instances
similar to the JSON examples you've provided below and was unable to see the
scenario duplicated. What JSON parser are you using? Is there any other
processing that may have occurred prior to generating your examples below?
Do you know how your library is handling escaped quote values like \ ?
How many of these did you observe?
Thanks,
Taylor
On Thu, Nov 11, 2010 at 5:10 AM, Augusto Santos augu...@gemeos.orgwrote:
I recevied a lot of broken json status from streaming api
Count of brojen json by day
2010-11-09 2
2010-11-10 4435
2010-11-11 888
Examples:
{in_reply_to_status_id_str:2563309119209472,text:@joi4kitten I have
that same
fear.,truncated:false,in_reply_to_user_id_str:16155805,entities:{user_menvions:[{screen_name:joi4kitten,indices:[0,11],name:joi4kitten,id:16155805,id_str:16155805}],urls:[],hashtags:[]},geo:null,in_reply_to_status_id:2563309119209472,place:{country_code:US,country:The
United States of
America,bounding_box:{type:Polygon,coordinates:[[[-76.965351,38.971109],[-76.909147,38.971109],[-76.909147,39.022114],[-76.965353,39.022114]]]},place_type:city,attributes:{},full_name:College
Park, MD,name:College Park,*id:e4c17912c815124d.url:http:\/\/
api.twitter.com\/1\/geo\/id\/e4c17912c815124d.json*},favorited:false,source:\u003Ca
href=\http:\/\/mobile.twitter.com\ rel=\nofollow\\u003EMobile
Web\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:joi4kitten,coorfinates:null,retweet_count:null,in_reply_to_user_id:16155805,created_at:Thu
Nov 11 03:38:52 +
2010,new_id_str:2565897919139841,new_id:2565897919139841,user:{follow_request_sent:null,lang:en,time_zone:Eastern
Time (US
Canada),screen_name:kellygo,following:null,profile_sidebar_border_color:0A84A5,profile_background_image_url:http:\/\/
a3.twimg.com\/profile_background_images\/8381831\/twitter_background_with_bear_2.jpg,notifications:null,description:I
am not Kelly
Osbourne.,listed_count:12,profile_background_tile:false,profile_background_color:052743,statuses_count:1559,location:College
Park,
Md.,show_all_inline_media:false,profile_use_background_image:true,favourites_count:29,profile_text_color:00,contributors_enabled:false,protected:false,geo_enabled:true,created_at:Sun
Jul 22 02:12:05 + 2007,profile_link_color:0A84A5,name:Kelly
Osborn,friends_count:235,url:http:\/\/kellygo.blogspot.com
,profile_image_url:http:\/\/a1.twimg.com
\/profile_images\/271727429\/kelly_o__posterized_normal.jpg,id:7632582,id_str:7632582,verified:false,utc_offset:-18000,profile_sidebar_fill_color:CFE9E2,followers_count:332},retweeted:false,id:2565897919139841,id_str:2565897919139841}
Here there's a dot where should be a comma, between id and url
{in_reply_to_status_id_str:2546974377119744,text:@ozdj Ah, suddenly
#TweetBurglar makes complete
sense,truncated:false,in_reply_to_user_id_str:15324154,entities:{user_mentions:[{screen_name:ozdj,indices:[0,5],name:Derek
Jenkins,id:15324154,id_str:15324154}],urls:[],hashtags:[{text:TweetBurglar,indices:[19,32]}]},geo:{type:Point,coordinates:[-33.88533896,151.20496326]},in_reply_to_status_id:2546974377119744,place:null,favorited:false,source:\u003Ca
href=\http:\/\/twitter.com\/\ rel=\nofollow\\u003ETwitter for
iPhone\u003C\/a\u003E,contributors:null,in_reply_to_screen_name:ozdj,coordinates:{type:Point,coordinates:[151.20496326,-33.88533896]},retweet_count:null,in_reply_to_user_id:15324154,created_at:Thu
Nov 11 03:38:51 +
2010,new_id_str:2565895784243200,new_id:2565895784243200,user:{lang:en,geo_enabled:true,screen_name:coffeetrails,following:null,friends_count:251,profile_sidebar_border_color:362919,verified:false,follow_request_sent:null,notifications:null,time_zone:Sydney,description:Specialising
in organic RA espresso-based coffees, organic teas, fresh sandwiches and
gourmet treats. Shop 5, 2 Lee St Sydney (Henry Deane Plaza, Railway
Sq),profile_background_color:362919,profile_background_image_url:http:\/\/
