Hi everyone: I am going to crawl the user relationship information, like, followers and friends, from Twitter Inc. for my research work. While, I am confronted with some problems, can anybody give me some suggestions. My problem is as follow: ========================================= Problem A: I try Twitter4j to fetch the relationship in Twitter . But I always limit by the rate for not owning a white-list key and secret, and sometimes I also got some errors, like HTTP 414 Error, denoting request-URI too large. Thus, is there any other method to escape the rate limitation with Twitter API?
Problem B: For this limit, I attempt another method to get the relationship. I notice that if I input the url http://twitter.com/!#/dannygao/followers , I could get the followers of dannygao in browser. If I input the url http://twitter.com/dannygao/followers, I need to login first, and then, get the followers in browser. Thus, I use the AsyncHttpClient to crawl this relationship just like the process of browser. The code is as follows: ------------------------------------------------- AsyncHttpClient client = new AsyncHttpClient(); Response response = client.prepareGet(this.url).execute().get(); if (response.getStatusCode() == 302) { System.out.print("Redirecting to... "); String redirect = response.getHeader("Location"); this.url = redirect; System.out.println(this.url); response = client.prepareGet(this.url).execute().get(); } return response.getResponseBody("UTF-8"); ------------------------------------------------- If I input the url http://twitter.com/!#/dannygao/followers, I got following output. Connecting url http://twitter.com/!#/dannygao/followers If I input the url http://twitter.com/dannygao/followers, I get following output. Connecting url http://twitter.com/dannygao/followers Redirecting to... http://twitter.com/login?redirect_after_login=%2Fdannygao%2Ffollowers Neither of the scripts I crawled contains the information about user relationship. Thus, 1) Can anybody explain what is the difference between these two urls? 2) If I want to use this method to crawl the twitter use relationship, can you provide me some suggestions? =============================================================================== Many thanks in advance. Danny -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk
