Re: [Server-devel] ToDO: Write up Proxy Strategy
On Tue, Jan 13, 2009 at 11:42 AM, Caroline Meeks carol...@solutiongrove.com wrote: rproxy - write down what needs to be done and how much. Caroline and others pointed out today at xocamp that the rproxy notes were never fully documented. There are some emails on this list about it - but nothing complete. I've now put it in the wiki http://wiki.laptop.org/go/XS_Blueprints:rproxy - here's the initial version for discussion... Notes on an rproxy implenentation for the XS. We want to have an optional, easy to switch on rproxy setup on the XS. rproxy can replace or complement the main XS http proxy - if we have a stable, efficient, high quality http proxy on the XS, pairing it with rproxy would be a good idea. As long as we are using Squid, it's not such a good idea :-( =Rationale= rproxy is a proxy that expends HTTP 1.1 proxying and provides much better performance over very bandwidth constrained linkes. It requires an upstream rproxy to match the downstream rproxy. For deployments where the schools are connected to the internet with very limited uplinks, rproxy can be a huge win. Modern internet is made of moderately dynamic content --- the content is mostly static but un-cacheable due to rotating banners or latest news boxes on the side. Even when content changes, the template for the content is often 80% of the HTML downloaded. rproxy provides huge gains with such content. More about what rproxy is * http://rproxy.samba.org/ * http://ozlabs.org/~rusty/rproxy.html =How it would work= For a deployment with limited connectivity into the schools - setup a co-located rproxy server at the ISP - enable rproxy on the XSs deployed, pointing to the ISP Note: the rproxy server at the ISP must only accept requests from valid clients - either from a defined netblock or whitelisted IP addresses. An open rproxy is an open proxy! =Issues with rproxy= * The working version uses an old version of librsync - which may have odd bugs and is generally unsupported, untouched, unknown * There's a half-finished port to the new librsync. Martin Poole started it a while ago, and abandoned it. * Does not stream the content through - blocks until the content is fully fetched before delivery to requester. * Unknown performance and memory footprint when hit by many users concurrently * Needs packaging, init scripts, etc... * Unclear if it behaves reasonably well with ajax streamed requests - it may break or add problematic latencies, etc =TO DO= * Test and ensure it works with modern internet usage patterns. Gmail, youtube, etc... * package, configuration, init scripts... * Test under load, paired with a conventional http proxy - may need performance / memory trimming * Port to modern librsync * Make it work with streamed content (possibly the hardest task!) ... cheers, martin -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] ToDO: Write up Proxy Strategy
On Tue, Jan 13, 2009 at 5:59 PM, Martin Langhoff martin.langh...@gmail.com wrote: http://wiki.laptop.org/go/XS_Blueprints:rproxy - and lest I forget - http://dev.laptop.org/ticket/7815 cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
[Server-devel] interest in new proxy
Hi, My name is Vivek Pai and I'm new to the list. Forgive me if I'm missing anything relating to this query. A long time ago, I visited OLPC to discuss HTTP proxy needs for the schools, and our research group at Princeton had a project that looked like it might be a good fit. In particular, it's a Web proxy that scales to large storage sizes without requiring large memory consumption. It can, for example, use a 1TB USB drive as a store, while requiring only a XO-class laptop to drive it. Is there interest in something like this for the XS server? We're also working on a WAN accelerator that uses this proxy, providing transparent compression of content, even for dynamic Web sites. The original ticket for this was https://dev.laptop.org/ticket/6#comment:8 -Vivek ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] interest in new proxy
Vivek, This sounds very interesting. Could you provide us with more information? Regards, Reuben Vivek Pai wrote: Hi, My name is Vivek Pai and I'm new to the list. Forgive me if I'm missing anything relating to this query. A long time ago, I visited OLPC to discuss HTTP proxy needs for the schools, and our research group at Princeton had a project that looked like it might be a good fit. In particular, it's a Web proxy that scales to large storage sizes without requiring large memory consumption. It can, for example, use a 1TB USB drive as a store, while requiring only a XO-class laptop to drive it. Is there interest in something like this for the XS server? We're also working on a WAN accelerator that uses this proxy, providing transparent compression of content, even for dynamic Web sites. The original ticket for this was https://dev.laptop.org/ticket/6#comment:8 -Vivek ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] interest in new proxy
On Tue, Jan 13, 2009 at 6:39 PM, Vivek Pai vi...@cs.princeton.edu wrote: A long time ago, I visited OLPC to discuss HTTP proxy needs for the schools, and our research group at Princeton had a project that looked like it might be a good fit. Hi Vivek, is it your original hashcache or related to it? There was discussion a while ago of using hashcache - jg was very excited about it, and I was too. We're pretty disappointed with Squid's performance and memory usage. Is there interest in something like this for the XS server? Yes yes yes. It'd be fantastic. What is the licensing status? cheers, martin -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] interest in new proxy
Hi Reuben, The system is called HashCache, and it's a relatively recent development. We'd been corresponding with Jim Gettys and trying to get the license worked out with Eben Moglen. Jim knows the details, but it's basically a proxy built on a new storage engine that's optimized to reduce memory consumption. In its most minimal form, it uses no RAM for the object index, which is what allows us to do the 1TB disk / 256MB RAM configuration. Its performance in this configuration is comparable to Squid, since HashCache has a custom user-level filesystem. It doesn't compete on features with Squid, but it should be possible to integrate the necessary set of features. -Vivek Reuben K. Caron wrote: Vivek, This sounds very interesting. Could you provide us with more information? Regards, Reuben Vivek Pai wrote: Hi, My name is Vivek Pai and I'm new to the list. Forgive me if I'm missing anything relating to this query. A long time ago, I visited OLPC to discuss HTTP proxy needs for the schools, and our research group at Princeton had a project that looked like it might be a good fit. In particular, it's a Web proxy that scales to large storage sizes without requiring large memory consumption. It can, for example, use a 1TB USB drive as a store, while requiring only a XO-class laptop to drive it. Is there interest in something like this for the XS server? We're also working on a WAN accelerator that uses this proxy, providing transparent compression of content, even for dynamic Web sites. The original ticket for this was https://dev.laptop.org/ticket/6#comment:8 -Vivek ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Notes about XS on XO...
On Thu, 2009-01-08 at 19:05 -0500, Martin Langhoff wrote: OK lets backup a second, I'm thinking that the OS could be on a SD disk/NAND with an external USB harddrive holding most of the write intensive directories. thoughts? My plan is to use a high quality SD card, not NAND (jffs2 complexity, slow, small, etc). And the USB hd holding the _large_ directories. Some stuff in /var is somewhat write intensive but very small, so a lazy commit policy will be just the ticket -- I think all of /var on usb might be harder. Just a question, would we be needing a do-it-yourself explanation of the steps involved, or are we just going to distribute an image that you could write to a partition? Both? I have a XO using a 4gig SD card, started life out as xo-1-olpc-stream-8.2.1-devel_ext3.img, with the XS repos added, yum installing xs-pkgs, xs-config now... I've ran into some funny dep issues but brute forced my way round them. I have some notes around, will try to add them to the wiki at some point. I've excluded the kernel from being updated at this point, I'll try a newer kernel at some point later, just waiting for yum to finish and try rebooting this puppy. Woohoo reboot time... rats network is broken... Time to dig/fix, Jerry ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] interest in new proxy
Vivek - Jim Gettys passed on your information to me (he left OLPC last week) and I'd be happy to pick up the conversation with you. Thanks! - Ed Ed McNierney CTO / VP of Engineering One Laptop per Child e...@laptop.org +1 (978) 761-0049 On 1/13/09 9:32 PM, Vivek Pai vi...@cs.princeton.edu wrote: Martin Langhoff wrote: On Tue, Jan 13, 2009 at 6:39 PM, Vivek Pai vi...@cs.princeton.edu wrote: A long time ago, I visited OLPC to discuss HTTP proxy needs for the schools, and our research group at Princeton had a project that looked like it might be a good fit. Hi Vivek, is it your original hashcache or related to it? There was discussion a while ago of using hashcache - jg was very excited about it, and I was too. We're pretty disappointed with Squid's performance and memory usage. The proxy part is HashCache. The WAN accelerator doesn't have a name at the moment, but it's probably a few months away from beta. Is there interest in something like this for the XS server? Yes yes yes. It'd be fantastic. What is the licensing status? I think we'd reached agreement with Eben on something, but we're just waiting at this point. Assuming someone on your end tells him that you're interested in forging ahead, I guess he'll write up the necessary license. It was basically Affero GPL + patent license for the GPL version. -Vivek ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] interest in new proxy
On Tue, Jan 13, 2009 at 9:32 PM, Vivek Pai vi...@cs.princeton.edu wrote: write up the necessary license. It was basically Affero GPL + patent license for the GPL version. That sounds like it'd be workable for us. Fantastic news! m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Notes about XS on XO...
On Tue, Jan 13, 2009 at 9:31 PM, Jerry Vonau jvo...@shaw.ca wrote: Just a question, would we be needing a do-it-yourself explanation of the steps involved, or are we just going to distribute an image that you could write to a partition? Both? I'd say both ;-) I have a XO using a 4gig SD card, started life out as xo-1-olpc-stream-8.2.1-devel_ext3.img, with the XS repos added, yum installing xs-pkgs, xs-config now... I've ran into some funny dep issues but brute forced my way round them. I have some notes around, will try to add them to the wiki at some point. I've excluded the kernel from being updated at this point, I'll try a newer kernel at some point later, just waiting for yum to finish and try rebooting this puppy. Woohoo reboot time... rats network is broken... Cool. Another bit of good news is that I tackled JKatz and he confessed to having (probably) backported the fixes onto the F9 kernel. So maybe a plain XS install to SD may work too... cheers, martin -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
