Hi everyone:
I am going to crawl the user relationship information, like, followers
and friends, from Twitter Inc. for my research work.
While, I am confronted with some problems, can anybody give me some
suggestions.
My problem is as follow:
=
Problem A: I try Twitter4j to fetch the relationship in Twitter .
But I always limit by the rate for not owning a white-list key and
secret, and sometimes I also got some errors, like HTTP 414 Error,
denoting request-URI too large.
Thus, is there any other method to escape the rate limitation with
Twitter API?
Problem B:
For this limit, I attempt another method to get the relationship.
I notice that if I input the url http://twitter.com/!#/dannygao/followers
, I could get the followers of dannygao in browser.
If I input the url http://twitter.com/dannygao/followers, I need to
login first, and then, get the followers in browser.
Thus, I use the AsyncHttpClient to crawl this relationship just like
the process of browser.
The code is as follows:
-
AsyncHttpClient client = new AsyncHttpClient();
Response response = client.prepareGet(this.url).execute().get();
if (response.getStatusCode() == 302) {
System.out.print(Redirecting to... );
String redirect = response.getHeader(Location);
this.url = redirect;
System.out.println(this.url);
response = client.prepareGet(this.url).execute().get();
}
return response.getResponseBody(UTF-8);
-
If I input the url http://twitter.com/!#/dannygao/followers, I got
following output.
Connecting url http://twitter.com/!#/dannygao/followers
If I input the url http://twitter.com/dannygao/followers, I get
following output.
Connecting url http://twitter.com/dannygao/followers
Redirecting to...
http://twitter.com/login?redirect_after_login=%2Fdannygao%2Ffollowers
Neither of the scripts I crawled contains the information about user
relationship.
Thus, 1) Can anybody explain what is the difference between these two
urls?
2) If I want to use this method to crawl the twitter use
relationship, can you provide me some suggestions?
===
Many thanks in advance.
Danny
--
Twitter developer documentation and resources: https://dev.twitter.com/doc
API updates via Twitter: https://twitter.com/twitterapi
Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
https://groups.google.com/forum/#!forum/twitter-development-talk
