@Matt Awesom! Thanks a lot, I'll try to get one up running on my end. There's apigee.com that can also help in such cases but they do not support the authentication part as far as I could understand. But I do need to set this up, I'm pretty sure in a month or two twitter will be blocking appengine IPs again, it happened a few months back and it'll keep happening every once a while till Twitter doesn't come up with a smarter way to block rogue apps.
I might need your help if things don't go ok when I set this thing up, will hit you up ;) -Nischal On Sep 30, 12:14 am, Matt Mastracci <[email protected]> wrote: > Hey Nischal, > > Here's a sample partial configuration for nginx. Note that this proxy > has *no* security. It's enough to get you up and going and its only > security is that only you know the hostname: > > upstream twitter_proxy { > server twitter.com; > > > } > > upstream twitter_api_proxy { > server api.twitter.com; > > > } > > server { > listen 80; > > > > server_name securitybyobscuritytwitterproxy.example.com; > > > > location / { > proxy_set_header Host api.twitter.com; > > > proxy_pass http://twitter_api_proxy; > > > } > } > > Drop the following class as HttpClientImpl in the package > twitter4j.internal.http.alternative. This class could be improved > somewhat (it was written at 2am under launch pressure), but it's > running right now in production: > > package twitter4j.internal.http.alternative; > > import twitter4j.TwitterException; > import twitter4j.http.Authorization; > import twitter4j.http.OAuthAuthorization; > import twitter4j.internal.http.HttpClientConfiguration; > import twitter4j.internal.http.HttpParameter; > import twitter4j.internal.http.HttpRequest; > import twitter4j.internal.http.HttpResponse; > > /** > * Implements an HTTP client that replaces the Twitter API urls with > our proxy. > */ > public class HttpClientImpl extends > twitter4j.internal.http.HttpClientImpl { > private static final long serialVersionUID = 1L; > > public HttpClientImpl(HttpClientConfiguration conf) { > super(conf); > } > > @Override > public HttpResponse request(HttpRequest req) throws TwitterException > { > HttpRequest httpRequest = new HttpRequest(req.getMethod(), > fixUrl(req.getURL()), req.getParameters(), fixAuth(req > .getAuthorization()), > req.getRequestHeaders()); > return super.request(httpRequest); > } > > private Authorization fixAuth(final Authorization authorization) { > if (authorization instanceof OAuthAuthorization) { > return new Authorization() { > private static final long serialVersionUID = > 1L; > > @Override > public String > getAuthorizationHeader(HttpRequest req) { > HttpRequest httpRequest = new > HttpRequest(req.getMethod(), > unfixUrl(req.getURL()), req.getParameters(), authorization, > > req.getRequestHeaders()); > return > authorization.getAuthorizationHeader(httpRequest); > } > > @Override > public boolean isEnabled() { > return authorization.isEnabled(); > } > }; > } > > return authorization; > } > > @Override > public HttpResponse get(String url) throws TwitterException { > return super.get(fixUrl(url)); > } > > @Override > public HttpResponse post(String url, HttpParameter[] params) throws > TwitterException { > return super.post(fixUrl(url), params); > } > > private String fixUrl(String url) { > return url.replace("http://api.twitter.com";, "http:// > securitybyobscuritytwitterproxy.example.com"); > } > > private String unfixUrl(String url) { > return url.replace("http:// > securitybyobscuritytwitterproxy.example.com", "http:// > api.twitter.com"); > } > > } > > On Sep 28, 4:00 pm, nischalshetty <[email protected]> wrote: > > > > > @Ikai I have raised an issue with them on this thread here > > -http://is.gd/fyvVp > > > One of the engineer had replied saying that some of the IPs might > > still be blocked and he said he'll try to unblock them. But then > > nothing happened. Now I haven't received any replies. Seems like the > > errors have dies down a little, but I'm still not sure if they've > > unblocked all the IPs. I completely agree with you, they should opt > > for something else other than IP range blocking for cloud solutions. > > > @Matt that seems like a life saver. I have been looking at reverse > > proxies the entire day trying to understand what to do. Can you guide > > me into this nginx thingy? I too use Twitter4j :) > > > One more thing, please voice your concern on this thread -http://is.gd/fyvVp > > and star the issue as well. I've been telling them to whitelist based > > on user agent but for whatever reason, they do not seem to understand > > at all! > > > -Nischal > > > On Sep 29, 1:52 am, Matt Mastracci <[email protected]> wrote: > > > > Yeah, we've escalated to the Twitter folks on our end. It's tough on > > > our end because we don't have a list of IP ranges. I think that the > > > Twitter API folks need to start parsing the app-id portion of User- > > > Agent for their rate-limiting and blacklists for the set of external > > > IPs that GAE users - I'll try to communicate that to the resources we > > > have. > > > > We've worked around it ourselves by putting together an external > > > Twitter API proxy based on nginx. If anyone is interested in the > > > Twitter4J patches required to sign OAuth requests for one host, but > > > deliver them to a proxy, let me know. > > > > On Sep 28, 8:27 pm, "Ikai Lan (Google)" <[email protected]> > > > wrote: > > > > > Have you guys raised this with the Twitter API team? We do our best to > > > > resolve issues when App Engine IP addresses become blocked, but it's > > > > important that you voice your concerns with their team as well. As cloud > > > > based solutions become more prominent, IP range based blocking will > > > > become > > > > too much of a "nuclear options" for dealing with spam/abuse. > > > > > -- > > > > Ikai Lan > > > > Developer Programs Engineer, Google App Engine > > > > Blogger:http://googleappengine.blogspot.com > > > > Reddit:http://www.reddit.com/r/appengine > > > > Twitter:http://twitter.com/app_engine > > > > > On Mon, Sep 27, 2010 at 8:50 PM, Matt Mastracci > > > > <[email protected]>wrote: > > > > > > We're still seeing issues hitting Twitter. I'm seeing a failure rate > > > > > of about 1/10: > > > > > > Response: com.google.apphosting.api.ApiProxy$ApplicationException: > > > > > ApplicationError: 2: Unknown at > > > > > com.google.apphosting.runtime.ApiProxyImpl > > > > > $AsyncApiFuture.rpcFinished(ApiProxyImpl.java:322) at > > > > > com.google.net.rpc.RpcStub$RpcCallbackDispatcher > > > > > $1.runInContext(RpcStub.java:1025) at com.google.tracing.TraceContext > > > > > $TraceContextRunnable$1.run(TraceContext.java:448) at > > > > > com.google.tracing.TraceContext.runInContext(TraceContext.java:688) at > > > > > com.google.tracing.TraceContext > > > > > > $AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.jav > > > > > a: > > > > > 326) at com.google.tracing.TraceContext > > > > > $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java: > > > > > 318) at com.google.tracing.TraceContext > > > > > $TraceContextRunnable.run(TraceContext.java:446) at > > > > > com.google.net.rpc.RpcStub > > > > > $RpcCallbackDispatcher.rpcFinished(RpcStub.java:1046) at > > > > > com.google.net.rpc.RPC.internalFinish(RPC.java:2038) at > > > > > com.google.net.rpc.impl.RpcNetChannel.finishRpc(RpcNetChannel.java: > > > > > 2352) at > > > > > com.google.net.rpc.impl.RpcNetChannel.messageReceived(RpcNetChannel.java: > > > > > 1279) at > > > > > com.google.net.rpc.impl.RpcConnection.parseMessages(RpcConnection.java: > > > > > 319) at > > > > > com.google.net.rpc.impl.RpcConnection.dataReceived(RpcConnection.java: > > > > > 290) at > > > > > com.google.net.async.Connection.handleReadEvent(Connection.java:474) > > > > > at > > > > > > com.google.net.async.EventDispatcher.processNetworkEvents(EventDispatcher.j > > > > > ava: > > > > > 831) at > > > > > com.google.net.async.EventDispatcher.internalLoop(EventDispatcher.java: > > > > > 207) at > > > > > com.google.net.async.EventDispatcher.loop(EventDispatcher.java: > > > > > 103) at com.google.net.async.GlobalEventRegistry > > > > > $2.runLoop(GlobalEventRegistry.java:95) at > > > > > com.google.net.async.LoopingEventDispatcher > > > > > $EventDispatcherThread.run(LoopingEventDispatcher.java:384) > > > > > > On Sep 27, 8:32 pm, "Sean Lynch (Google)" <[email protected]> wrote: > > > > > > Things should be working properly now. > > > > > > > On Sep 28, 12:36 pm, nischalshetty <[email protected]> > > > > > > wrote: > > > > > > > > Keeping fingers crossed, hope this gets sorted out soon. > > > > > > > > -Nischal > > > > > > > > On Sep 28, 6:46 am, "Ikai Lan (Google)" > > > > > > > <[email protected]<ikai.l%[email protected]> > > > > > > > > wrote: > > > > > > > > > Hey everybody, > > > > > > > > > We're currently being blocked by Twitter. We're reaching out > > > > > > > > right > > > > > now to > > > > > > > > the Twitter team to resolve the issues. > > > > > > > > > -- > > > > > > > > Ikai Lan > > > > > > > > Developer Programs Engineer, Google App Engine > > > > > > > > Blogger:http://googleappengine.blogspot.com > > > > > > > > Reddit:http://www.reddit.com/r/appengine > > > > > > > > Twitter:http://twitter.com/app_engine > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
