On Mon, Apr 28, 2008 at 02:59:57AM -0700, Kevin Brown wrote: > > > I had a JIRA issue open to migrate to HttpClient instead of using > URLConnection, but I was still leaning towards a wrapper interface of some > sort for various reasons, the main one being that the older HttpClient > interface was difficult to extend. I haven't looked at the 4.0 interface > yet, but I trust your judgment.
I've attached our implementation. (Please note that this was coded
under duress, so it could stand to be cleaned up a bit..))
It appears that oen could extend HttpClient with their concept of
built-in interceptors to implement most of the features we want.
Here's a sample that injects the Accept-Encoding header into outbound
requests:
// Add hooks for gzip/deflate
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException,
IOException {
if (!request.containsHeader("Accept-Encoding")) {
request.addHeader("Accept-Encoding", "gzip,deflate");
}
}
});
> One of the big things I think we should do is drop the tight coupling of the
> auth modes to the http retrieval, and instead have something dedicated to
> dealing with various types of auth. Packing oauth and request signing into
> the fetchers was an interesting experiment, but the end results are barely
> better than the previous revisions where stuff was crammed into the proxy.
> Now that there are multiple places where auth needs to be done, it's even
> uglier to deal with. I think a bunch of us had good ideas here, but they
> didn't really mesh well together and the end result is pretty poor.
Okay. Sounds like it might be possible to factor out the
auth/signing code and then write adapters (like you see above) to
loosely integrate the pieces.
> There's no time like the present to fix this stuff up, I suppose, since
> changes in the last few weeks, and those coming in the next few are dramatic
> enough that anyone doing heavy integration work is already in for a rough
> patch.
Ooof. Anything we can do to make this easier will be welcome.
--
Paul Lindner
hi5 Architect
[EMAIL PROTECTED]
/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * software distributed under the License is distributed on an * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * KIND, either express or implied. See the License for the * specific language governing permissions and limitations under the License. */ package com.hi5.os; import org.apache.shindig.util.InputStreamConsumer; import org.apache.shindig.gadgets.RemoteContentFetcher; import org.apache.shindig.gadgets.RemoteContentRequest; import org.apache.shindig.gadgets.RemoteContent; import org.apache.http.params.HttpParams; import org.apache.http.params.BasicHttpParams; import org.apache.http.params.HttpProtocolParams; import org.apache.http.params.HttpConnectionParams; import org.apache.http.conn.params.HttpConnectionManagerParams; import org.apache.http.conn.scheme.SchemeRegistry; import org.apache.http.conn.scheme.Scheme; import org.apache.http.conn.scheme.PlainSocketFactory; import org.apache.http.conn.ClientConnectionManager; import org.apache.http.conn.ssl.SSLSocketFactory; import org.apache.http.*; import org.apache.http.message.AbstractHttpMessage; import org.apache.http.entity.HttpEntityWrapper; import org.apache.http.entity.StringEntity; import org.apache.http.entity.InputStreamEntity; import org.apache.http.protocol.HttpContext; import org.apache.http.protocol.HTTP; import org.apache.http.client.HttpClient; import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpPost; import org.apache.http.client.params.HttpClientParams; import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager; import org.apache.http.impl.client.DefaultHttpClient; import org.apache.http.impl.client.DefaultHttpRequestRetryHandler; import org.apache.http.conn.routing.HttpRoute; import org.apache.http.conn.params.ConnPerRoute; import java.io.FileNotFoundException; import java.io.InputStream; import java.io.IOException; import java.io.UnsupportedEncodingException; import java.net.HttpURLConnection; import java.net.URLConnection; import java.util.*; import java.util.concurrent.TimeUnit; import java.util.concurrent.ConcurrentHashMap; import java.util.zip.GZIPInputStream; import java.util.zip.Inflater; import java.util.zip.InflaterInputStream; import com.friend.service.AppsService; import com.friend.service.Services; import com.friend.utils.BackgroundTask; import com.friend.data.AppBean; import com.friend.exception.PlatformException; import org.apache.commons.logging.LogFactory; import org.apache.commons.logging.Log; /** * Implementation of a [EMAIL PROTECTED] RemoteObjectFetcher} using standard java.net * classes. */ public class Hi5RemoteContentFetcher implements RemoteContentFetcher { private static final Log log = LogFactory.getLog(Hi5RemoteContentFetcher.class); private static final int CONNECT_TIMEOUT_MS = 15000; private static final int READ_TIMEOUT_MS = 30000; private static ThreadSafeClientConnManager cm; private static final Map<String, Integer> publicAppUrls = new ConcurrentHashMap<String, Integer>(500); // The following background task maintains a list of Public App Urls // Doesn't worry about removing them as of yet.. static { BackgroundTask.getInstance().scheduleWithFixedDelay(new Runnable() { public void run() { AppsService appsService = Services.getService(AppsService.class); Map<Integer, AppBean> publicApps = appsService.getPublicApps(); for (AppBean a : publicApps.values()) { if (!publicAppUrls.containsKey(a.getAppUrl())) publicAppUrls.put(a.getAppUrl(), a.getId()); } } }, 300, 300, TimeUnit.SECONDS); } private final int maxObjSize; AppsService appsService; private static final HttpClient FETCHER = createFetcher(); private static HttpClient createFetcher() { // Create and initialize HTTP parameters HttpParams params = new BasicHttpParams(); HttpConnectionManagerParams.setMaxTotalConnections(params, 1024); HttpConnectionManagerParams.setMaxConnectionsPerRoute(params, new ConnPerRoute() { public int getMaxForRoute(HttpRoute route) { return 128;}}); HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1); HttpProtocolParams.setUserAgent(params, "hi5 fetcher"); HttpProtocolParams.setContentCharset(params, "UTF-8"); HttpConnectionParams.setConnectionTimeout(params, CONNECT_TIMEOUT_MS); HttpConnectionParams.setSoTimeout(params, READ_TIMEOUT_MS); HttpConnectionParams.setStaleCheckingEnabled(params, true); HttpClientParams.setRedirecting(params, true); HttpClientParams.setAuthenticating(params, false); HttpClientParams.setConnectionManagerTimeout(params, CONNECT_TIMEOUT_MS); // Create and initialize scheme registry SchemeRegistry schemeRegistry = new SchemeRegistry(); schemeRegistry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80)); schemeRegistry.register(new Scheme("https", SSLSocketFactory.getSocketFactory(), 443)); ClientConnectionManager cm = new ThreadSafeClientConnManager(params, schemeRegistry); DefaultHttpClient client = new DefaultHttpClient(cm, params); client.setHttpRequestRetryHandler(new DefaultHttpRequestRetryHandler(1,true)); // Add hooks for gzip/deflate client.addRequestInterceptor(new HttpRequestInterceptor() { public void process( final HttpRequest request, final HttpContext context) throws HttpException, IOException { if (!request.containsHeader("Accept-Encoding")) { request.addHeader("Accept-Encoding", "gzip, deflate"); } } }); client.addResponseInterceptor(new HttpResponseInterceptor() { public void process( final HttpResponse response, final HttpContext context) throws HttpException, IOException { HttpEntity entity = response.getEntity(); Header ceheader = entity.getContentEncoding(); if (ceheader != null) { HeaderElement[] codecs = ceheader.getElements(); for (int i = 0; i < codecs.length; i++) { String codecname = codecs[i].getName(); if ("gzip".equalsIgnoreCase(codecname)) { response.setEntity( new GzipDecompressingEntity(response.getEntity())); return; } else if ("deflate".equals(codecname)) { response.setEntity(new DeflateDecompressingEntity(response.getEntity())); return; } } } } }); return client; } static class GzipDecompressingEntity extends HttpEntityWrapper { public GzipDecompressingEntity(final HttpEntity entity) { super(entity); } public InputStream getContent() throws IOException, IllegalStateException { // the wrapped entity's getContent() decides about repeatability InputStream wrappedin = wrappedEntity.getContent(); return new GZIPInputStream(wrappedin); } public long getContentLength() { // length of ungzipped content is not known return -1; } } static class DeflateDecompressingEntity extends HttpEntityWrapper { public DeflateDecompressingEntity(final HttpEntity entity) { super(entity); } public InputStream getContent() throws IOException, IllegalStateException { // the wrapped entity's getContent() decides about repeatability InputStream wrappedin = wrappedEntity.getContent(); return new InflaterInputStream(wrappedin, new Inflater(true)); } public long getContentLength() { // length of ungzipped content is not known return -1; } } /** * Creates a new fetcher capable of retrieving objects [EMAIL PROTECTED] maxObjSize} * bytes or smaller in size. * * @param maxObjSize Maximum size, in bytes, of object to fetch */ public Hi5RemoteContentFetcher(int maxObjSize) { this.maxObjSize = maxObjSize; appsService = Services.getService(AppsService.class); } // set default headers private static void addHeaders(RemoteContentRequest request, AbstractHttpMessage getOrPost) { Map<String, List<String>> reqHeaders = request.getAllHeaders(); for (Map.Entry<String, List<String>> entry : reqHeaders.entrySet()) { List<String> value = entry.getValue(); if (value.size() == 1) { getOrPost.addHeader(entry.getKey(), value.get(0)); } else { StringBuilder headerList = new StringBuilder(); boolean first = false; for (String val : value) { if (!first) { first = true; } else { headerList.append(','); } headerList.append(val); } // Don't set content-length, http-client does not like.. if (!entry.getKey().equalsIgnoreCase("Content-Length")) getOrPost.addHeader(entry.getKey(), headerList.toString()); } } } public RemoteContent fetch(RemoteContentRequest request) { InputStream responseStream = null; HttpPost post = null; HttpGet get = null; HttpResponse response = null; try { // First try using our internal app cache Integer appId = publicAppUrls.get(request.getUri().toString()); if (appId != null) { AppBean a = appsService.getApp(appId); return new RemoteContent(RemoteContent.SC_OK, a.getAppXml().getBytes(), Collections.<String, List<String>>emptyMap()); } // Next pass this off to httpclient if ("POST".equals(request.getMethod())) { post = new HttpPost(request.getUri()); addHeaders(request, post); post.setEntity(new InputStreamEntity(request.getPostBody(), request.getPostBodyLength())); response = FETCHER.execute(post); } else { get = new HttpGet(request.getUri()); addHeaders(request, get); response = FETCHER.execute(get); } if (response == null) throw new IOException("Unknown problem with request"); Map<String, List<String>> headers = new HashMap(); int responseCode = response.getStatusLine().getStatusCode(); if (response.getEntity() != null) responseStream = response.getEntity().getContent(); if (responseStream == null) { throw new IOException("Cannot retrieve " + request.getUri() + " reason " + response.getStatusLine().getReasonPhrase()); } byte[] body = InputStreamConsumer.readToByteArray( responseStream, maxObjSize); response.getEntity().consumeContent(); responseStream.close(); for (Header h : response.getAllHeaders()) { headers.put(h.getName(), Arrays.asList(h.getValue())); } return new RemoteContent(responseCode, body, headers); } catch (Exception e) { // Insure we're clean if (get != null) try {get.abort();} catch (Exception ee) {}; if (post != null) try {post.abort();} catch (Exception ee) {}; // Find timeout exceptions, respond accordingly Class eClass = e.getClass(); if (eClass.equals(org.apache.http.conn.ConnectionPoolTimeoutException.class) || eClass.equals(java.net.SocketTimeoutException.class) || eClass.equals(java.net.SocketTimeoutException.class) || eClass.equals(java.net.SocketException.class) || eClass.equals(org.apache.http.NoHttpResponseException.class) || eClass.equals(InterruptedException.class)) { log.warn("Timeout for " + request.getUri() + " Exception: " + eClass.getName() + " - " + e.getMessage()); return RemoteContent.TIMEOUT; } log.warn("Got Exception fetching " + request.getUri(), e); return RemoteContent.ERROR; } finally { // cleanup any outstanding resources.. if (get != null) try {get.abort();} catch (Exception e) {}; if (post != null) try {post.abort();} catch (Exception e) {}; try { if (responseStream != null) responseStream.close(); } catch (Exception e) {} } } }
pgpkEnCl15fQ4.pgp
Description: PGP signature
