Mostly a fairly random collection of notes. I'm using WWWOFFLE 2.8d now for offline web browsing. It's a proxy server that saves all the pages you ask for, and in offline mode, returns you an ugly page that tells you it remembered your request. Then, when you go online and give it the "fetch" command, it downloads all the pages you previously asked for, plus all their inline images, stylesheets, Flash files, and so on.
The main differences between WWWOFFLE and a normal caching proxy server: - it caches stuff longer than the HTTP spec says it should, and in offline mode, it serves it up when you ask for it (in lieu of an error page) even if the HTTP spec says it's stale; - it remembers requests it couldn't fulfill in order to fulfill them later. It also has a fairly kickass recursive-get mode. It actually allows me to use e.g. Google Maps offline! Hooray for REST! Some features I wish it had/bugs that annoy me: - being responsive. Actually, I can't tell if the long spinners when Firefox gets pages from it are Firefox's fault or WWWOFFLE's, but I suspect the latter; - noticing DNS server changes, which is apparently impossible to do with the standard resolver library in a portable fashion, other than by restarting the server; - remembering which pages I'd actually asked for, and which ones I was "done with", which clearly requires some better UI; - implicitly recursively getting later pages of multi-page articles (e.g. on Wired and OSNews; clearly this has to have per-site regexes and crap); - storing annotations on the pages so I could search by annotations (it has support for searching your offline cache of part of the web); - a standard blocklist of ad providers, since ads made up a substantial fraction of the hazardous, expensive, battery-draining time that I was online; - better integration into the browser UI --- I'd like to have buttons for "recursively fetch from this page" (which would take me to an options screen, rather than fetch immediately), "block this URL" (likewise, since I'd need to be able to wildcard the URL), "pages that linked to this page", and "already-stored pages linked from this page" --- which should also show up in link colors. - while you can see the list of pending fetches (http://localhost:8080/index/outgoing/?sort=alpha;all) if you see something bad in the list (hundreds of requests for things like http://ad.doubleclick.net/adj/ttm.osnews/ros;tile=2;sz=120x600;ord=08028075), it's too hard to remove it (the remove button is hidden by default and takes you to a different page when you use it) and blacklist it. Blacklisting is an available option, but also hidden by default, and fairly painful to use; also, it doesn't actually remove it from the list of pending fetches. Click "Config", change Path to "Any Path" (two clicks), change Arguments to "Any or none" (two clicks), click "Change URL-SPECIFICATION", page down five times to dont-request, click it, click the "Yes" radio button, click "make change", close window --- 15 UI actions to add each wildcard, and that's once you're looking at the list of pending fetches rather than the page with the offending content, although I suppose the list of pending fetches tells you who the worst offenders are, but it doesn't show where they're linked from or what other similar pages were (although that is available through a few more clicks), so it's hard to distinguish ads from Google Maps images (e.g. kh1.google.com). - a less ugly UI - would it be so hard to use <label> elements in the UI? - better handling of temporary failures, such as timeouts. "Better handling" means "retry". - also it would be nice if I could upload my configuration and list of desired URLs to a server somewhere else (like my colo) which would do the fetches for me while I was offline, and then download a big blob of compressed web pages later. Ideally I could put the configuration and URLs in an encrypted file that I could take to an internet cafe and upload, and download the encrypted blob of compressed web pages in the same way. - when I used Squid for offline web browsing, I added meta http-equiv refreshes (typically refreshing about once every half hour) to all of its error pages, so that any tab displaying an error page would eventually display the web page. - URL blocking that actually worked would be nice too. I added a bunch of domains to the dont-fetch list, but WWWOFFLE kept fetching stuff from them anyway. - hey, it would be nice if I could see why a particular page wasn't fetched after the first time I try to GET it. E.g. if it's an image, the first GET might be as an inline image --- I've seen this happen sometimes. - there's an option to list the "pages" that were requested the last time I was offline. As I said before, it would be nice to have a better display that distinguished pages I actually loaded interactively, by clicking on a link, from inline resources like <iframe>, <script>, <img>, <embed>, and <style> links (which might be difficult without some more browser integration), it would also be nice to be able to see the titles of the pages. As it is, it's difficult to find the dozens of web pages I wanted to read, in among the thousands of Google Maps images. - while wwwoffle does nicely store things on disk in separate files, the content of the HTTP response isn't stored in a separate file --- so you can't use gthumb, gv, file, gzip, and so on, on the files from wwwoffle's cache dirs --- you have to access them through wwwoffle, which means through HTTP. That's a little inconvenient. - At first you might wonder why "sort by file type" doesn't display the content-type it's sorting by. Then you realize that it's actually sorting by the file extension, and is therefore worse than useless: e.g. http://www.folklore.org/StoryView.py?project=Macintosh&story=90_Hours_A_Week_And_Loving_It.txt (which is HTML!) ends up next to http://gnosis.cx/download/gnosis/util/convert/curses_txt2html.py - It's fairly opaque about how it chooses to purge things from cache. There's a "purge" section in the config file, but from my reading of it, it shouldn't delete anything until I haven't read it for four weeks. But in fact it's already deleted a bunch of stuff. - And it would be nice if it remembered more than one version of each page. - strace says it forks a new thread (using clone) for every child request. Perhaps this explains why it is so slow. - And sometimes it tries to send me an error page with "Transfer-Encoding: chunked,chunked" and two layers of chunking, which Firefox doesn't like --- the result is that the error page displays with some three-digit hexadecimal numbers at the top. (I think this should have been fixed in WWWOFFLE 2.8b.) - Every once in a while, especially under heavy load, it serves up the wrong representation --- I got PNGs and GIF89as for a bunch of my HTML pages today. - You can push it on or offline with an HTTP GET. - I caught it doing continuous DNS lookups on www.meetomatic.com, which had an A record, for many minutes --- several requests per second for perhaps half an hour. This wouldn't be so bad except that it did lots of AAAA requests, which failed and thus weren't cached. - Its connect timeouts seem to be painfully short, on the order of tens of seconds. As far as I can tell from reading the changelog (NEWS file), none of the things listed above have been fixed in more recent WWWOFFLEs. Unfortunately I think a good system for this kind of thing implicitly encodes some workflow. At a minimum, each URL is in one of these states: - nobody cares - requested - currently being fetched - waiting to be read - discarded - archived The idea is that from "waiting to be read", a page can go into "discarded" or "archived". I also think you need some kind of categorization system for this --- anyway, I do --- such that some of these states can pertain to a particular category. Normally pages should inherit their categories from the page they were linked from, and you should be able to see and edit the categories for a page in the browser when you have it open.
