On 11/29/10 11:41 AM, Bostonian wrote: > Based on these tests, here are my guesses: > > i) when stopping tahoe, it pro-actively notifies introducer about its > leave. > > ii) when unplugging the cable, introducer does not know that one node > leaves. As a result, it still tries to connect it when requests.
Yup. The specific issue is the TCP connections between the client and each server. When a server process terminates, the OS kernel closes all of its open sockets, which sends a TCP "FIN" packet to the client, so it can tell that the connection is now closed. It the cable is unplugged instead, the connection is merely unusable (perhaps temporarily). Applications cannot reliably distinguish between this state and simple network congestion or servers being slow. There are two sorts of problems here, with different kinds of solutions for each. The first problem (which isn't the most important one) is how to detect non-explicit connection loss The best you can do here is a heuristic, by sending a keepalive packet every once in a while, and abandoning any connection that has been quiet for too long. Tahoe (or rather the Foolscap library that it uses) pays attention to the "timeout.keepalive" and "timeout.disconnect" settings in tahoe.cfg to control this. The default value causes a keepalive to be sent roughly once every 6 minutes, and relies upon TCP's built-in SO_KEEPALIVE timer to give up on lost connections (which can take anywhere from 10 minutes to several hours, depending on the OS). The second problem (which is the real issue) is how Tahoe's upload/download algorithms deal with very slow and/or "lost" servers (i.e. servers which still look like they're connected, but which never respond to queries). This is improving over time, but some operations can still get stuck waiting for a server to respond, so a "lost" server can cause it to stall for a long time (until TCP gives up on the connection). The recently rewritten immutable file downloader should tolerate lost servers very well. The only case that might trip it up is when a server becomes lost during the middle of a download. We have a plan to improve this, but it requires deciding upon an "impatience" timer, the point at which we give up waiting for one block and switch to using a different server. Mutable file downloads (used for reading directories) should behave the same way. The code is different, but I think it has a similar two-phase behavior (servers that are lost at the beginning of the download are handled well, but servers which become lost during the download can cause a stall). Immutable file uploads are probably more susceptible to stalls with lost servers because the upload starts by asking a bunch of servers if they're willing to hold a share, and there is no "impatience timer" to switch over to different servers when a response is overdue. Likewise mutable files uploads (used when creating a directory) want to hear back from many servers before they will start, so they can be stalled by lost servers too. There are some interesting and deep issues here, particularly for mutable files: you really want the client to try hard to find existing shares when publishing an update, rather than quickly giving up and allowing old shares to remain un-updated, because those old shares can cause rollback and inconsistencies later. Deciding between write availability and consistency is hard. For the most part, we've designed Tahoe around the idea of fairly static grids, where the server nodes stay up for long periods of time, and connections are not coming up and going down quickly. To accomodate more dynamic/transient grid membership, we'll need some new tools. > This seems to an bug with Tahoe. Given the nature of the system, it > will see this situation quite often. Is there any monitoring mechanism > at introducer? Not really. The Introducer is not the one paying attention to whether a server is up or not. It's job is to tell all clients how to contact each server, and once it's delivered that data, its job is done. The clients themselves are responsible for managing connections to the servers. cheers, -Brian _______________________________________________ tahoe-dev mailing list [email protected] http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
