[twitter-dev] Re: listed count?
This issue was first brought up Nov 10 on the issue tracker but of course no response from the Twitter team. http://code.google.com/p/twitter-api/issues/detail?id=1186 On Feb 8, 6:26 pm, waukesha_area waukesha.a...@gmail.com wrote: Is it possible to get a count of how many lists a user belongs to? I am able to get the lists a user belongs to, page through them, and then get a count of them. That seems like a lot of work and bandwidth to find this out.
[twitter-dev] Re: listed count?
Why don't you just simply retrieve the HTML for the user's twitter profile page and look for id=lists_count and just grab the number in this tag. That's what I'm doing now. Of course, Twitter could change the HTML on this page but they probably won't do it often. Quy On Feb 9, 9:20 am, Orian Marx (@orian) or...@orianmarx.com wrote: This issue was first brought up Nov 10 on the issue tracker but of course no response from the Twitter team. http://code.google.com/p/twitter-api/issues/detail?id=1186 On Feb 8, 6:26 pm, waukesha_area waukesha.a...@gmail.com wrote: Is it possible to get a count of how many lists a user belongs to? I am able to get the lists a user belongs to, page through them, and then get a count of them. That seems like a lot of work and bandwidth to find this out.
Re: [twitter-dev] Re: listed count?
Why don't you just simply retrieve the HTML for the user's twitter profile page and look for id=lists_count and just grab the number in this tag. That's what I'm doing now. Of course, Twitter could change the HTML on this page but they probably won't do it often. Screen scraping is an excellent way to get your IP banned. That said, having a total list count would be a very nice feature to add to /show. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- Time wounds all heels. -- Groucho Marx -
[twitter-dev] Re: listed count?
Why would my IP get banned - the API allows developers to retrieve almost every piece of data from user's twitter profiles so developers don't need to scrape. I think if it's a closed site and they want to protect content, then I can understand IP banning but if it's an open system like Twitter, I don't see any reason why my IP would get banned. Maybe someone from the Twitter Platform team can chime in, especially since I only scrape for list counts every 4-6 hours on a small set of users for my application. Quy On Feb 9, 9:54 am, Cameron Kaiser spec...@floodgap.com wrote: Why don't you just simply retrieve the HTML for the user's twitter profile page and look for id=lists_count and just grab the number in this tag. That's what I'm doing now. Of course, Twitter could change the HTML on this page but they probably won't do it often. Screen scraping is an excellent way to get your IP banned. That said, having a total list count would be a very nice feature to add to /show. -- personal:http://www.cameronkaiser.com/-- Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com -- Time wounds all heels. -- Groucho Marx -
Re: [twitter-dev] Re: listed count?
Why would my IP get banned - the API allows developers to retrieve almost every piece of data from user's twitter profiles so developers don't need to scrape. I think if it's a closed site and they want to protect content, then I can understand IP banning but if it's an open system like Twitter, I don't see any reason why my IP would get banned. Maybe someone from the Twitter Platform team can chime in, especially since I only scrape for list counts every 4-6 hours on a small set of users for my application. That's probably why they haven't noticed you (yet). However, screen scraping gets around rate limiting and other controls Twitter places on the API. That's not exactly considered socially agreeable, and it's not fair on other API consumers. It's also against TOS. https://twitter.com/tos -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- It would have been funnier if I didn't have to think. -- Ashley Mills --
Re: [twitter-dev] Re: listed count?
Why can't you just use http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-GET-list-memberships to get the lists the user is listed on...and just do a counter as you go through them? It might require a few extra service calls, but at the moment that seems like the most 'appropriate' way to determine how many lists a given user is currently on. - Kevin http://friendstat.us On Tue, Feb 9, 2010 at 1:10 PM, Cameron Kaiser spec...@floodgap.com wrote: Why would my IP get banned - the API allows developers to retrieve almost every piece of data from user's twitter profiles so developers don't need to scrape. I think if it's a closed site and they want to protect content, then I can understand IP banning but if it's an open system like Twitter, I don't see any reason why my IP would get banned. Maybe someone from the Twitter Platform team can chime in, especially since I only scrape for list counts every 4-6 hours on a small set of users for my application. That's probably why they haven't noticed you (yet). However, screen scraping gets around rate limiting and other controls Twitter places on the API. That's not exactly considered socially agreeable, and it's not fair on other API consumers. It's also against TOS. https://twitter.com/tos -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- It would have been funnier if I didn't have to think. -- Ashley Mills --
Re: [twitter-dev] Re: listed count?
On the TODO list: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/515733c625904ed8/ Another reason to to screen scrape is that building an entire HTML page uses a lot more resources then just returning XML/json. On Tue, Feb 9, 2010 at 10:19, Orian Marx (@orian) or...@orianmarx.comwrote: Agree that scrapping is a bad idea. The question is, why has this particular piece of data (list counts) been available on twitter.com since the lists rollout but exists nowhere in the API. It seems like an oversight, which is why developers are trying to be helpful by logging issues in the issue tracker. There's really no excuse for the fact that no one from Twitter has responded to the logged issue at all, and it was logged almost immediately after the rollout. On Feb 9, 1:10 pm, Cameron Kaiser spec...@floodgap.com wrote: Why would my IP get banned - the API allows developers to retrieve almost every piece of data from user's twitter profiles so developers don't need to scrape. I think if it's a closed site and they want to protect content, then I can understand IP banning but if it's an open system like Twitter, I don't see any reason why my IP would get banned. Maybe someone from the Twitter Platform team can chime in, especially since I only scrape for list counts every 4-6 hours on a small set of users for my application. That's probably why they haven't noticed you (yet). However, screen scraping gets around rate limiting and other controls Twitter places on the API. That's not exactly considered socially agreeable, and it's not fair on other API consumers. It's also against TOS. https://twitter.com/tos -- personal: http://www.cameronkaiser.com/-- Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com -- It would have been funnier if I didn't have to think. -- Ashley Mills -- -- Abraham Williams | Community Advocate | http://abrah.am Project | Out Loud | http://outloud.labs.poseurtech.com This email is: [ ] shareable [x] ask first [ ] private.
