Re: [Server-devel] Apache proxy CRCsync
Hi Alex, I think you are on the right track, there is a third option which is to add a few extra configuration options to the cache module to make it more aggressive about caching. Basically to cache everything except pages marked 'private' (and possibly even them as long as you can ensure the server is secure). Another aspect of our use of the cache module at the moment is that the disk cache modules has a large portion of code in common with the client module. It would be nice to add a few hooks to this module and reduce our code duplication. That being said, a module that doesn't require modifications to the apache source is an advantage, however I think header hacking would be the only way to achieve this so probably we cant get away with this. Martin's comments about the garbage collection (which just arrived) are an important consideration otherwise the cache could get out of control quite quickly. Toby ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Apache proxy CRCsync
On Mon, Mar 23, 2009 at 10:05 PM, Alex Wulms wrote: > What are your thoughts on this subject? I'm not Toby, but I do have some notes from the chat we had with Rusty back in January. The idea at the time was that - The 'normal' caching proxy would cache things that have good caching headers. Our crcsync smarts are not needed there. That cache has its own garbage collection logic, based on cache headers. - The crcsync proxy wants to cache "the rest" -- the formally uncacheable content, perhaps taking some decision by content type (prefer html, xml, text mimetypes. ignore others?) and its garbage collector has a very different logic (something LRU-ish?) that's a big advantage of this code being in-process with a normal caching proxy, it can ignore the stuff that the caching proxy is handling, and help with the requests that "aren't cacheable" according to their server (most requests these days, unfortunately)... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Is a USB-Ethernet NIC appropriate for the XS?
On Tue, Mar 24, 2009 at 4:49 AM, Bryan Berry wrote: > The ever helpful cjb and Mitch_Bradley directed me to the root of the > problem, the USB-ethernet devices I am using are USB 1.1 which has > horrible throughput. A USB-ethernet device that supports USB 2.0 should > fix the problem. That's great news! I'll also be delighted to hear about what hw you find that works... cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Is a USB-Ethernet NIC appropriate for the XS?
The ever helpful cjb and Mitch_Bradley directed me to the root of the problem, the USB-ethernet devices I am using are USB 1.1 which has horrible throughput. A USB-ethernet device that supports USB 2.0 should fix the problem. Mitch_Bradley: great, do u think I can buy a usb2 ethernet nic for under $50? budget is tight and I need these for 15 XS's AX8817X and AX88772 are the chips Mitch_Bradley: thanks http://www.tigerdirect.com/applications/category/category_slc.asp?CatId=589 Mitch_Bradley: is there any reason that USB-Ethernet is inherently unworkable or is it just a question of getting the right usb-ethernet nic? USB 1.1 sucks rocks for ethernet because the polling for rx packets kills the throughput BryanWB: no, the others are usb2 what Mitch_Bradley said Mitch_Bradley: so usb1.1 will totally crap out w/ 20 users ? USB2.0 has much improved bandwidth and much lower latency for polling usb1.1 will totally crap out with 1 user Mitch_Bradley: thanks a lot guys, you are saving my bacon BryanWB: the number of users won't affect it, other than by sucking proportional to load I didn't even bother supporting USB 1.1 ethernet chips in OFW. It's just not worth it. anyway, yeah. just find an asix dongle. try to buy a name brand device. The no-name ones often just don't work. On Mon, 2009-03-23 at 15:43 +0100, Martin Langhoff wrote: > On Mon, Mar 23, 2009 at 3:10 PM, Bryan Berry wrote: > > We want to use the MSI Wind PC for the XS but it has one big problem. It > > only has 1 on-board NIC and no PCI slots. As you know, the XS requires 2 > > NIC's. We have tried several different USB NIC's and are having serious > > throughput problems. Is this issue one w/ the NIC or the fact that it is > > connected by USB? > > My main worry with USB-connected NICs would be reliability. If the NIC > is reliable, but the throughput a bit below-par, you can still > probably use it as the WAN NIC -- the throughput is likely to be > constrained upstream anyway. > > - When you say throughput problems... what are you getting? How bad > is it? Even if limited, would it be appropriate for the WAN port? > > - The problem may be specific to the driver or NIC hardware -- is > there any discussion on the kernel dev list about it? > > - The problem may be with the USB bus on the MS Wind, hardware or > drivers. Perhaps testing for bus throughput or interrupt handling > helps? > > - Have you tested the reliability of the devices? This may be a larger > problem. > > cheers, > > > martin -- Bryan W. Berry Technology Director OLE Nepal, http://www.olenepal.org ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Is a USB-Ethernet NIC appropriate for the XS?
thks, Martin will investigate this On Mon, 2009-03-23 at 15:43 +0100, Martin Langhoff wrote: > On Mon, Mar 23, 2009 at 3:10 PM, Bryan Berry wrote: > > We want to use the MSI Wind PC for the XS but it has one big problem. It > > only has 1 on-board NIC and no PCI slots. As you know, the XS requires 2 > > NIC's. We have tried several different USB NIC's and are having serious > > throughput problems. Is this issue one w/ the NIC or the fact that it is > > connected by USB? > > My main worry with USB-connected NICs would be reliability. If the NIC > is reliable, but the throughput a bit below-par, you can still > probably use it as the WAN NIC -- the throughput is likely to be > constrained upstream anyway. > > - When you say throughput problems... what are you getting? How bad > is it? Even if limited, would it be appropriate for the WAN port? > > - The problem may be specific to the driver or NIC hardware -- is > there any discussion on the kernel dev list about it? > > - The problem may be with the USB bus on the MS Wind, hardware or > drivers. Perhaps testing for bus throughput or interrupt handling > helps? > > - Have you tested the reliability of the devices? This may be a larger > problem. > > cheers, > > > martin -- Bryan W. Berry Technology Director OLE Nepal, http://www.olenepal.org ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Apache proxy CRCsync
Hi Toby, I did not have much time last week to work on the project but want to continue again this week. I have been thinking about the integration between the standard cache module of apache and the crccache_client cache-handler module. At this moment, the cache module unfortunately does not invoke crccache_client for most dynamic pages; the web/application servers indicate that those pages are not cacheable, either by setting an expiry time in the past or by setting appropriate cache-control headers or a combination of the two. And the cache module respects that, as a good http-citizen. But the whole idea of crccache is that those pages should be stored by crccache_client anyway but get refetched and then delta'd by the crccache_client/crccache_server chain on the next request. So one way or another crccache_client/crccache_server should trick the cache module into caching those dynamic pages. I see two potential ways to make this happen: Option 1) crccache_client (or crccache_server?) modifies the cache related response headers before returning the response back to the cache module. It would modify the headers in such a way that the cache module would decide that the page must be cached but revalidated at the next request. This would require no modifications to the cache module but I do consider it a not-so-clean hack, because we would have to reverse engineer the cache module to understand when to modify the headers and when not to modify the headers. Which is obviously fragile because future enhancements to the cache module could potentially break such logic. Option 2) We introduce some new header(s) that crccache can inject in the response to indicate to the cache module that the pages will be cached by a delta/crcsync aware cache handler. And then we adapt the cache module itself to understand this new header and to cache normally-not-cachable pages if this header is present (and send them for revalidation to the crccache handler upon next request). I see this as a cleaner solution. Though, before immediately starting to implement this solution, I believe that it should be analysed into a little bit more detail. Especially with respect to the future, when crccache will talk to some servers that are crcsync aware and can directly handle the encoding themselves while crccache will at the same time also still talk to many current-gen servers that do not know this http extension. What are your thoughts on this subject? Thanks and kind regards, Alex Op maandag 16 maart 2009, schreef Toby Collett: > Great to hear you got it running, unfortunately I only have about a two > week head start on you with regard to the apache front, so I am sure lots > of things will get neater as we go along. > > 2009/3/16 Alex Wulms > > > Hi Toby, > > > > I managed to get it working on my PC under suse 11.1 with apache 2.2.10. > > > > When I configured a dedicated debug log per virtual server, I noticed > > that the > > crccache_client and crccache_server modules were both invoked in both > > virtual > > servers. Judging from the error log you sent me, that is also the case on > > your server. > > > > I have made following changes to fix the situation: > > > > 1) Move the 'CacheEnable crccache_client' directive (for the 'default' > > virtual > > server) inside the tag. Apparently it is applied globally > > as long as it is outside the tag, regardless of the config > > file in > > which it appears. > > Seems like a sensible change. > > > 2) Introduce a new directive 'CRCCacheServer on'. > > This directive is checked by mod_crccache_server in the > > crccache_server_header_parser_handler. > > It is specified in the tag of the upstream_proxy of the > > virtual > > server. > > Apparently modules get loaded globally and functions like > > the ..._header_parser_handler get invoked for each virtual server, so > > they must check themselves if they should be enabled or disabled in a > > given virtual server. I found this through google, which pointed me to a > > forum where somebody else had faced a similar problem. > > Makes sense > > > I also realize why I only found cached files > > under /var/cache/apache2/mod_crccache_server and not under ..._client. > > It is because the crccache_client.conf and crccache_server.conf file both > > use > > the parameter CacheRoot to store the cache directory. These parameters > > are apparently also global. The fact that they are in two different > > config files > > does not automagically store them in a module specific namespace. So I > > have renamed the parameters to differentiate between the client and the > > server module. > > Actually only the client end should need the CacheRoot at all, the server > side doesnt need caching at all. You could configure a standard apache > cache if you wanted, but it probably wont gain much. > > > I have also noticed that, although the server module reads these > > parameters, > > they actually don't get used by
Re: [Server-devel] mkusbinstall link broken?
On Mon, Mar 23, 2009 at 12:39 AM, Martin Langhoff wrote: > On Mon, Mar 23, 2009 at 11:21 AM, Sameer Verma wrote: >> I've got Squid running and one AA (prototype) and two XOs. The >> "server" pulls in 8 watts at the adapter (110V->12V). Works fine! > > Nice! I see it's rated for 0-45C, what temps are you trying it with? > San Francisco won't cooperate. It stays in the mid 50 F (10 C). At home, its about 68 F (20 C). My wife won't let me use the oven ;-) I ran it all night and it stays mildly warm. I need to get a IR thermometer or some such contraption to measure external body temp. > Will it now be part of your test group's setup? ;-) > > Yes. I tested it with 5 XOs, but would love to test it with our entire OLPC-SF group (typically 25 XOs). We do have 10 XOs collected/bought thus far for Maroantsetra, Madagascar. Those + mine + a few borrowed from here and there should do the trick. BTW, I installed Munin (http://munin.projects.linpro.no/) on it (additional 4.5MB) to monitor long term health. Any +/- on that package? Sameer -- Dr. Sameer Verma, Ph.D. Associate Professor of Information Systems San Francisco State University San Francisco CA 94132 USA http://verma.sfsu.edu/ http://opensource.sfsu.edu/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Is a USB-Ethernet NIC appropriate for the XS?
On Mon, Mar 23, 2009 at 3:10 PM, Bryan Berry wrote: > We want to use the MSI Wind PC for the XS but it has one big problem. It > only has 1 on-board NIC and no PCI slots. As you know, the XS requires 2 > NIC's. We have tried several different USB NIC's and are having serious > throughput problems. Is this issue one w/ the NIC or the fact that it is > connected by USB? My main worry with USB-connected NICs would be reliability. If the NIC is reliable, but the throughput a bit below-par, you can still probably use it as the WAN NIC -- the throughput is likely to be constrained upstream anyway. - When you say throughput problems... what are you getting? How bad is it? Even if limited, would it be appropriate for the WAN port? - The problem may be specific to the driver or NIC hardware -- is there any discussion on the kernel dev list about it? - The problem may be with the USB bus on the MS Wind, hardware or drivers. Perhaps testing for bus throughput or interrupt handling helps? - Have you tested the reliability of the devices? This may be a larger problem. cheers, martin -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
[Server-devel] Apache proxy CRCsync & mozilla gsoc project?
Hi, Tridge just cc'd me on on a GSOC rsync-http mozilla project; given that Martin is coordinating an apache proxy plugin, I thought I'd send a big inclusive mail to make sure we all know about each other! My involvement: a crcsync module in CCAN which can be used as a (simplified) librsync. Cheers! Rusty. ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Apache proxy CRCsync & mozilla gsoc project?
On Mon, Mar 23, 2009 at 7:29 AM, Rusty Russell wrote: > Tridge just cc'd me on on a GSOC rsync-http mozilla project; given that > Martin is coordinating an apache proxy plugin, I thought I'd send a big > inclusive mail to make sure we all know about each other! > > My involvement: a crcsync module in CCAN which can be used as a (simplified) > librsync. Fantastic! I assume the rsync-http now know of the vastly superior karma of crcsync over the 2-hash method of rsync. If the Apache mods and Mozilla speak the same protocol, then machines behind bandwidth-constrained links will be in much better shape. I can see 3G-internet providers pushing this too. Also cc'ing Jim Gettys -- our long-held hope is that the resulting extension to the http protocol is something that can be folded into a future http spec. Pushing buttons to create http-crcs...@lists.laptop.org to serve as a coordination point. cheers, martin -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] help regarding Gsoc ...
2009/3/20 >^< CrOsS FiRe >^< : > hello people ... i needed some help regardin google summer of code. This was > my original idea , here's the abstract (More of a brain dump): Hi! Getting an assignment and posting it, and comparing it to other participants' work is an async task, rather than a synchronous task. For async stuff like this, the main tool on the XS is Moodle, a webbased course mgmt system that is extremely modular. There has been some discussion of having a "scoring" set of test cases on the server side (for programming exercises) as a Moodle module. I would suggest you search the moodle.org website for those discussions, I suspect there may be a contrib module -- if that's the case, you may have a codebase you can extend, polish and improve rather than have to start from scratch. You also mention a whiteboard in the chat facility. That is a complete project on its own, and a fairly complex one too. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] mkusbinstall link broken?
On Mon, Mar 23, 2009 at 11:21 AM, Sameer Verma wrote: > I've got Squid running and one AA (prototype) and two XOs. The > "server" pulls in 8 watts at the adapter (110V->12V). Works fine! Nice! I see it's rated for 0-45C, what temps are you trying it with? Will it now be part of your test group's setup? ;-) m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] mkusbinstall link broken?
On Mon, Mar 23, 2009 at 8:27 AM, Jerry Vonau wrote: >> I tried the mkusbinstall script. It works. Installing from USB is > Yea, at the boot prompt remove the "method= part, you will have to the 'method' part initially works, but fails later, right after formatting the target disk. That's waiting for some explicit anaconda support right? I saw this fail too when testing 0.5.2, and was wondering whether to comment it out until we have a 0.6 with your patched anaconda... cheers, martin -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel