Currently FProxy requires browsers to keep a large number of connections
open if fetching a page containing a lot of subelements which are
fetched separately. Even with a large number of connections such as 50
or 100, there are still pages with more than 50 or 100 images. When
fetching such pages, the browser does not respond to requests for other
pages. (This is true of Mozilla, at least.)
I have decided to make a stab in another direction. FProxy can return a
temporary image when the real image is not yet available, and set
Refresh in the HTTP reply so that the browser will return. Refresh can
even be set to a different URL. When the image is finally available and
the browser returns for the nth time, we give it the image.
It is possible to do the same for other requests, but it seems useless
to return an HTML page saying "fetching page, please wait". That would
only save a single connection too.
The problem is not entirely trivial though. Let me try to walk through
the state diagram. The URL contains information about which state the
request is in currently.
State New: The browser comes in with a fresh request
We need to know whether this is an img request at all. Therefore HTML
pages sent out must have their img src="..." links modified to add an
extra tag. The HTML filter can easily be changed to do that.
Can the request be satisfied from local datastore? If so, send the data
and finish.
Can the request be denied from local datastore? (Yes, this can happen -
as an example, when requesting a non-existent subkey where the parent
key has been fetched already, and therefore all subkeys are known.)
If not, start an asynchronous request, send a temporary image with a
refresh tag, and go to Refresh.
State Refresh
Check if there is a request in flight with equal or greater HTL. If
there is, the image might still arrive. Increase refresh time and go
back to Refresh.
If not, check if the request can be satisfied from local datastore. If
so, send the data and finish.
Similarly, check if the request can be denied from local datastore. If
so, return error and finish.
Otherwise the original request sent must have given up. Return an error
page /without the extra tag saying this is an image request/. This will
just display a broken link, but if the user decides to actually view the
image separately in the browser, Freenet will use normal synchronous
methods and the user can get normal Freenet error messages - and
manually override HTL.
Alternatively, look up the error information that the original request
returned. This is not all that valuable though, since it will still just
display as a broken link.
It is of course also possible for FProxy to automatically increase HTL
and start over. Exponential backoff and a hard limit of <mumble> should
keep the load down.
I have decided to implement a proof of concept. It is not meant as
finished code, just as a prototype. It has some limitations:
It checks the local datastore by issuing a request with HTL=0. I am not
sure whether this is the most efficient way of checking. It does not
react to permanent errors as it should.
It does not check on requests but blindly retries, only checking whether
the data has arrived in the datastore. It does not know when the request
is done, so after 5 retries it gives up.
When it gives up on a request, it keeps the image tag in the error
reply. The image tag should be removed as described above.
It returns a gif for the temporary image no matter what the original
file name was. Since file names are arbitrary this should IMHO be fine,
but it is likely that Internet Explorer will not like having foo.jpeg be
a GIF. It returns correct MIME type though. The gif is just an existing
one I picked, aqua/download.gif. It is quite ugly when stretched.
Matthew J Toseland has been providing valuable feedback on the irc
channel. His recommendation is to keep state for each open request. I
would prefer not to do that, and I think that the above algorithm is
implementable without keeping state. Since state is already kept in
fred, I would prefer to use that instead of having to manage it myself -
including removing state for old connections, and handling the case
where the browser comes back 3 hours later and I removed the state...
It is certainly possible that I am wrong about the state issue, but even
if state has to be kept, I still think this new way of handling images
could be beneficial.
The patch is attached and is also available at
http://amorsen.dk/freenet/Imagehandling-4.diff
A precompiled jar which includes my load estimation changes is available
at http://amorsen.dk/freenet/freenet-imagehandling-4.jar
/Benny
diff -ur freenet-6190/src/freenet/client/http/filter/SaferFilter.java freenet/src/freenet/client/http/filter/SaferFilter.java
--- freenet-6190/src/freenet/client/http/filter/SaferFilter.java 2003-07-29 05:33:15.000000000 +0200
+++ freenet/src/freenet/client/http/filter/SaferFilter.java 2003-09-10 22:39:06.000000000 +0200
@@ -692,10 +692,7 @@
allowedTagsVerifiers.put("base", new TagVerifier("base",
new String[] {"id", "target"},
new String[] {"href"}));
- allowedTagsVerifiers.put("img", new CoreTagVerifier("img",
- new String[] {"alt", "name", "height", "width", "ismap", "align", "border", "hspace", "vspace"},
- new String[] {"src", "longdesc", "usemap"},
- emptyStringArray));
+ allowedTagsVerifiers.put("img", new ImageTagVerifier());
// FIXME: object tag - http://www.w3.org/TR/html4/struct/objects.html#h-13.3
// FIXME: param tag - http://www.w3.org/TR/html4/struct/objects.html#h-13.3.2
// applet tag PROHIBITED - we do not support applets (FIXME?)
@@ -1105,6 +1102,26 @@
}
}
+ static class ImageTagVerifier extends CoreTagVerifier {
+ ImageTagVerifier() {
+ super("img",
+ new String[] {"alt", "name", "height", "width", "ismap", "align", "border", "hspace", "vspace"},
+ new String[] {"src", "longdesc", "usemap"},
+ emptyStringArray);
+ }
+
+ Hashtable sanitizeHash(Hashtable h, ParsedTag p, HTMLParseContext pc) {
+ Hashtable hn = super.sanitizeHash(h, p, pc);
+
+ String src = getHashString(hn, "src");
+ if (src != null) {
+ hn.put("src", src + "?tagtype=img");
+ }
+
+ return hn;
+ }
+ }
+
static class LinkTagVerifier extends CoreTagVerifier {
LinkTagVerifier(String tag, String[] allowedAttrs, String[] uriAttrs, String[] eventAttrs) {
super(tag, allowedAttrs, uriAttrs, eventAttrs);
diff -ur freenet-6190/src/freenet/client/http/FproxyServlet.java freenet/src/freenet/client/http/FproxyServlet.java
--- freenet-6190/src/freenet/client/http/FproxyServlet.java 2003-09-03 22:00:43.000000000 +0200
+++ freenet/src/freenet/client/http/FproxyServlet.java 2003-09-11 21:23:47.000000000 +0200
@@ -374,6 +374,7 @@
String queryHtl = null;
String queryMime = null;
String queryDate = null;
+ String queryTagtype = null;
String queryMaxLogSize = null;
boolean queryRDate = false;
boolean queryVerbose = false;
@@ -477,6 +478,14 @@
queryDateMillis = -1;
}
}
+
+ queryTagtype = req.getParameter("tagtype");
+ if (queryTagtype != null) {
+ queryTagtype = freenet.support.URLDecoder.decode(queryTagtype);
+ if(logDEBUG) logger.log(this, "Read tagtype from query: " + queryTagtype,
+ Logger.DEBUGGING);
+ }
+
} catch (Exception e) {
logger.log(this, "Error while parsing URI", e,
Logger.ERROR);
@@ -498,7 +507,7 @@
String encKey = URLEncoder.encode(queryKey);
queryKey = reconstructRequest(HTMLEncoder.encode(encKey),
queryForce, queryHtl, queryMime,
- queryDate, "");
+ queryDate, "", null);
// ignore the try #
try {
resp.sendRedirect("/" + queryKey);
@@ -575,18 +584,82 @@
FailureListener listener = new FailureListener(logger);
r.addEventListener(listener);
- if (!r.doGet(uri, data, htlUsed)) {
- if (!(queryRDate && listener.dr != null)) {
- if(logDEBUG) logger.log(this, "Request process returned error for "+
- uri+" with htl "+htlUsed, Logger.DEBUG);
- writeErrorMessage(listener.getException(r,uri,req), req,resp,
- null, key, uri, htlUsed, queryDate,
- queryMime, queryForce, tryNum+1);
+ if ((queryTagtype != null) && (queryTagtype.equals("img"))) {
+ long timebefore = System.currentTimeMillis();
+ if (!r.doGet(uri, data, 0)) {
+ if (tryNum == 5) {
+ if(logDEBUG) logger.log(this, "Request process returned error for "+
+ uri+" with htl "+htlUsed, Logger.DEBUG);
+ writeErrorMessage(listener.getException(r,uri,req), req,resp,
+ null, key, uri, htlUsed, queryDate,
+ queryMime, queryForce, tryNum+1);
+ return;
+ }
+
+ long timeafter = System.currentTimeMillis();
+ logger.log(this, "Failed doGet with HTL 0 took: " + (timeafter-timebefore) + "ms", Logger.NORMAL);
+ r.doGet(uri, data, htlUsed, true);
+ try {
+ String image = "aqua/download.gif";
+ resp.setContentType("image/gif");
+
+ InputStream imagenbIS = this.getClass().getResourceAsStream(HtmlTemplate.TEMPLATEDIR
+ + "images/" + image);
+ if (imagenbIS == null) {
+ logger.log(this, "Can't find image " + image, logger.ERROR);
+ resp.sendError(404);
+ return;
+ }
+ resp.addDateHeader("Expires", System.currentTimeMillis()+
+ 60*1000);
+
+ String encKey = "";
+
+ if (key != null) {
+ encKey = URLEncoder.encode(key);
+ logger.log(this, "key is " + key, logger.NORMAL);
+ } else {
+ logger.log(this, "key is null.", logger.NORMAL);
+ }
+
+ String newURL = constructURI(encKey,
+ queryMime, queryDate,
+ queryForce, htlUsed,
+ tryNum+1, queryTagtype);
+
+ logger.log(this, "newURL set to /" + newURL, logger.NORMAL);
+
+ resp.addHeader("Refresh", "45; URL=/" + newURL);
+
+ BufferedInputStream imageIS = new BufferedInputStream(imagenbIS);
+ OutputStream imageOS = resp.getOutputStream();
+ int r1 = imageIS.read();
+ while (r1 != -1) {
+ imageOS.write(r1);
+ r1 = imageIS.read();
+ }
+ } catch (Exception e) {
+ logger.log(this, "Failed to read/write image", e, logger.ERROR);
+ }
+ try {
+ resp.flushBuffer();
+ } catch (IOException e) {}
return;
}
} else {
- if(logDEBUG) logger.log(this, "Request process returned success for "+
- uri+" with htl "+htlUsed, Logger.DEBUGGING);
+ if (!r.doGet(uri, data, htlUsed)) {
+ if (!(queryRDate && listener.dr != null)) {
+ if(logDEBUG) logger.log(this, "Request process returned error for "+
+ uri+" with htl "+htlUsed, Logger.DEBUG);
+ writeErrorMessage(listener.getException(r,uri,req), req,resp,
+ null, key, uri, htlUsed, queryDate,
+ queryMime, queryForce, tryNum+1);
+ return;
+ }
+ } else {
+ if(logDEBUG) logger.log(this, "Request process returned success for "+
+ uri+" with htl "+htlUsed, Logger.DEBUGGING);
+ }
}
if (queryRDate && listener.dr != null) {
@@ -594,7 +667,8 @@
FreenetURI newURI = listener.dr.getTargetForTime(uri, time);
String redirect = reconstructRequest(newURI.toString(false),
queryForce, queryHtl,
- queryMime, null, null);
+ queryMime, null, null,
+ null);
resp.sendRedirect("/" + redirect);
return;
}
@@ -705,7 +779,7 @@
String encKey = URLEncoder.encode(key);
String redirect = "/servlet/SFRequest/" +
reconstructRequest(encKey, "", queryHtl, queryMime,
- queryDate, "");
+ queryDate, "", null);
if(forced) {
char c = (redirect.indexOf('?')!=-1) ? '&' : '?';
redirect += c + "runFilter=false";
@@ -872,7 +946,8 @@
*/
private String reconstructRequest(String queryKey, String queryForce,
String queryHtl, String queryMime,
- String queryDate, String queryTry) {
+ String queryDate, String queryTry,
+ String queryTagtype) {
int count = 0;
if (queryForce != null) {
queryKey += getPrefix(count++) + "force=" + queryForce;
@@ -889,6 +964,10 @@
if (queryTry != null) {
queryKey += getPrefix(count++) + "try=" + queryTry;
}
+ if (queryTagtype != null) {
+ queryKey += getPrefix(count++) + "tagtype=" + queryTagtype;
+ }
+
return queryKey;
}
@@ -1014,7 +1093,7 @@
encKey + extraDate +
"&mime=text/plain"+extraDate+"\">source</A>";
w += ", <a href=\"/"+
- constructURI(encKey, "application/octet-stream", encDate, forceKey, htl, 0)+
+ constructURI(encKey, "application/octet-stream", encDate, forceKey, htl, 0, null)+
"\">force save to disk</a>";
w += ", or <A HREF=\"/\">return</A> to gateway page.</p>";
pw.println(w);
@@ -1117,8 +1196,8 @@
}
pageTmp = refreshPageTmp;
pageTmp.set("REFRESH-TIME", Long.toString(10 << tryNum));
- pageTmp.set("REFRESH-URL", "/" + constructURI(encKey, encMime, encDate, encForce, htlNew, tryNum));
- if(logDEBUG) logger.log(this, "REFRESH-URL:/" + constructURI(encKey, encMime, encDate, encForce, htlNew, tryNum), Logger.DEBUG);
+ pageTmp.set("REFRESH-URL", "/" + constructURI(encKey, encMime, encDate, encForce, htlNew, tryNum, null));
+ if(logDEBUG) logger.log(this, "REFRESH-URL:/" + constructURI(encKey, encMime, encDate, encForce, htlNew, tryNum, null), Logger.DEBUG);
titleBoxTmp.set("TITLE", "Network Error");
if(SimpleAdvanced_ModeUtils.isAdvancedMode(req)) {
pw.println("<p>Couldn't retrieve key: <b>" + encKey + "</b>");
@@ -1217,9 +1296,10 @@
* @param tryNum Number of Retry
*/
private String constructURI(String encKey, String encMime, String encDate,
- String encForce, int htl, int tryNum) {
+ String encForce, int htl, int tryNum, String encTagtype) {
return reconstructRequest(encKey, encForce, Integer.toString(htl),
- encMime, encDate, Integer.toString(tryNum));
+ encMime, encDate, Integer.toString(tryNum),
+ encTagtype);
}
_______________________________________________
Devl mailing list
[EMAIL PROTECTED]
http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
