Hey Nischal,
Here's a sample partial configuration for nginx. Note that this proxy
has *no* security. It's enough to get you up and going and its only
security is that only you know the hostname:
upstream twitter_proxy {
server twitter.com;
}
upstream twitter_api_proxy {
server api.twitter.com;
}
server {
listen 80;
server_name securitybyobscuritytwitterproxy.example.com;
location / {
proxy_set_header Host api.twitter.com;
proxy_pass http://twitter_api_proxy;
}
}
Drop the following class as HttpClientImpl in the package
twitter4j.internal.http.alternative. This class could be improved
somewhat (it was written at 2am under launch pressure), but it's
running right now in production:
package twitter4j.internal.http.alternative;
import twitter4j.TwitterException;
import twitter4j.http.Authorization;
import twitter4j.http.OAuthAuthorization;
import twitter4j.internal.http.HttpClientConfiguration;
import twitter4j.internal.http.HttpParameter;
import twitter4j.internal.http.HttpRequest;
import twitter4j.internal.http.HttpResponse;
/**
* Implements an HTTP client that replaces the Twitter API urls with
our proxy.
*/
public class HttpClientImpl extends
twitter4j.internal.http.HttpClientImpl {
private static final long serialVersionUID = 1L;
public HttpClientImpl(HttpClientConfiguration conf) {
super(conf);
}
@Override
public HttpResponse request(HttpRequest req) throws TwitterException
{
HttpRequest httpRequest = new HttpRequest(req.getMethod(),
fixUrl(req.getURL()), req.getParameters(), fixAuth(req
.getAuthorization()), req.getRequestHeaders());
return super.request(httpRequest);
}
private Authorization fixAuth(final Authorization authorization) {
if (authorization instanceof OAuthAuthorization) {
return new Authorization() {
private static final long serialVersionUID = 1L;
@Override
public String
getAuthorizationHeader(HttpRequest req) {
HttpRequest httpRequest = new
HttpRequest(req.getMethod(),
unfixUrl(req.getURL()), req.getParameters(), authorization,
req.getRequestHeaders());
return
authorization.getAuthorizationHeader(httpRequest);
}
@Override
public boolean isEnabled() {
return authorization.isEnabled();
}
};
}
return authorization;
}
@Override
public HttpResponse get(String url) throws TwitterException {
return super.get(fixUrl(url));
}
@Override
public HttpResponse post(String url, HttpParameter[] params) throws
TwitterException {
return super.post(fixUrl(url), params);
}
private String fixUrl(String url) {
return url.replace("http://api.twitter.com";, "http://
securitybyobscuritytwitterproxy.example.com");
}
private String unfixUrl(String url) {
return url.replace("http://
securitybyobscuritytwitterproxy.example.com", "http://
api.twitter.com");
}
}
On Sep 28, 4:00 pm, nischalshetty <[email protected]> wrote:
> @Ikai I have raised an issue with them on this thread here -http://is.gd/fyvVp
>
> One of the engineer had replied saying that some of the IPs might
> still be blocked and he said he'll try to unblock them. But then
> nothing happened. Now I haven't received any replies. Seems like the
> errors have dies down a little, but I'm still not sure if they've
> unblocked all the IPs. I completely agree with you, they should opt
> for something else other than IP range blocking for cloud solutions.
>
> @Matt that seems like a life saver. I have been looking at reverse
> proxies the entire day trying to understand what to do. Can you guide
> me into this nginx thingy? I too use Twitter4j :)
>
> One more thing, please voice your concern on this thread -http://is.gd/fyvVp
> and star the issue as well. I've been telling them to whitelist based
> on user agent but for whatever reason, they do not seem to understand
> at all!
>
> -Nischal
>
> On Sep 29, 1:52 am, Matt Mastracci <[email protected]> wrote:
>
>
>
> > Yeah, we've escalated to the Twitter folks on our end. It's tough on
> > our end because we don't have a list of IP ranges. I think that the
> > Twitter API folks need to start parsing the app-id portion of User-
> > Agent for their rate-limiting and blacklists for the set of external
> > IPs that GAE users - I'll try to communicate that to the resources we
> > have.
>
> > We've worked around it ourselves by putting together an external
> > Twitter API proxy based on nginx. If anyone is interested in the
> > Twitter4J patches required to sign OAuth requests for one host, but
> > deliver them to a proxy, let me know.
>
> > On Sep 28, 8:27 pm, "Ikai Lan (Google)" <[email protected]>
> > wrote:
>
> > > Have you guys raised this with the Twitter API team? We do our best to
> > > resolve issues when App Engine IP addresses become blocked, but it's
> > > important that you voice your concerns with their team as well. As cloud
> > > based solutions become more prominent, IP range based blocking will become
> > > too much of a "nuclear options" for dealing with spam/abuse.
>
> > > --
> > > Ikai Lan
> > > Developer Programs Engineer, Google App Engine
> > > Blogger:http://googleappengine.blogspot.com
> > > Reddit:http://www.reddit.com/r/appengine
> > > Twitter:http://twitter.com/app_engine
>
> > > On Mon, Sep 27, 2010 at 8:50 PM, Matt Mastracci
> > > <[email protected]>wrote:
>
> > > > We're still seeing issues hitting Twitter. I'm seeing a failure rate
> > > > of about 1/10:
>
> > > > Response: com.google.apphosting.api.ApiProxy$ApplicationException:
> > > > ApplicationError: 2: Unknown at
> > > > com.google.apphosting.runtime.ApiProxyImpl
> > > > $AsyncApiFuture.rpcFinished(ApiProxyImpl.java:322) at
> > > > com.google.net.rpc.RpcStub$RpcCallbackDispatcher
> > > > $1.runInContext(RpcStub.java:1025) at com.google.tracing.TraceContext
> > > > $TraceContextRunnable$1.run(TraceContext.java:448) at
> > > > com.google.tracing.TraceContext.runInContext(TraceContext.java:688) at
> > > > com.google.tracing.TraceContext
>
> > > > $AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.jav
> > > > a:
> > > > 326) at com.google.tracing.TraceContext
> > > > $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
> > > > 318) at com.google.tracing.TraceContext
> > > > $TraceContextRunnable.run(TraceContext.java:446) at
> > > > com.google.net.rpc.RpcStub
> > > > $RpcCallbackDispatcher.rpcFinished(RpcStub.java:1046) at
> > > > com.google.net.rpc.RPC.internalFinish(RPC.java:2038) at
> > > > com.google.net.rpc.impl.RpcNetChannel.finishRpc(RpcNetChannel.java:
> > > > 2352) at
> > > > com.google.net.rpc.impl.RpcNetChannel.messageReceived(RpcNetChannel.java:
> > > > 1279) at
> > > > com.google.net.rpc.impl.RpcConnection.parseMessages(RpcConnection.java:
> > > > 319) at
> > > > com.google.net.rpc.impl.RpcConnection.dataReceived(RpcConnection.java:
> > > > 290) at
> > > > com.google.net.async.Connection.handleReadEvent(Connection.java:474)
> > > > at
>
> > > > com.google.net.async.EventDispatcher.processNetworkEvents(EventDispatcher.j
> > > > ava:
> > > > 831) at
> > > > com.google.net.async.EventDispatcher.internalLoop(EventDispatcher.java:
> > > > 207) at com.google.net.async.EventDispatcher.loop(EventDispatcher.java:
> > > > 103) at com.google.net.async.GlobalEventRegistry
> > > > $2.runLoop(GlobalEventRegistry.java:95) at
> > > > com.google.net.async.LoopingEventDispatcher
> > > > $EventDispatcherThread.run(LoopingEventDispatcher.java:384)
>
> > > > On Sep 27, 8:32 pm, "Sean Lynch (Google)" <[email protected]> wrote:
> > > > > Things should be working properly now.
>
> > > > > On Sep 28, 12:36 pm, nischalshetty <[email protected]> wrote:
>
> > > > > > Keeping fingers crossed, hope this gets sorted out soon.
>
> > > > > > -Nischal
>
> > > > > > On Sep 28, 6:46 am, "Ikai Lan (Google)"
> > > > > > <[email protected]<ikai.l%[email protected]>
>
> > > > > > wrote:
>
> > > > > > > Hey everybody,
>
> > > > > > > We're currently being blocked by Twitter. We're reaching out right
> > > > now to
> > > > > > > the Twitter team to resolve the issues.
>
> > > > > > > --
> > > > > > > Ikai Lan
> > > > > > > Developer Programs Engineer, Google App Engine
> > > > > > > Blogger:http://googleappengine.blogspot.com
> > > > > > > Reddit:http://www.reddit.com/r/appengine
> > > > > > > Twitter:http://twitter.com/app_engine
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to [email protected].
> > > > To unsubscribe from this group, send email to
> > > > [email protected]<google-appengine%2Bunsubscrib
> > > > [email protected]>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
