begin quoting Brian Deacon as of Thu, May 10, 2007 at 04:05:44AM -0700: > On Sat, May 05, 2007 at 02:20:35AM -0700, Stewart Stremler wrote: > > So let me see if I have our assumptions correct: > > > > (1) Writes and reads are atomic, but may not be persistent. That is, > > the data in a file will not ever be corrupt > > > > (2) Data in a file might be silently replaced at any time. > > > > (3) Files may be silently removed at any time. > > > > (4) No latency guarantees are made for how long a change to become > > universal. > > > > (5) No ordering guarantees are made for which server updates which > > of its siblings when, or how. > > I think I probably don't completely understand the requirements, but it > sounds like we're all too fascinated with handling locking requirements > on a service that doesn't support it well.
I've got the PDF for the lock-free data structure papers (that Andrew posted links to) up on my home machine. I've kinda stalled on it, but I intend to get back to it. In hindsight, I'm not sure that you're agreeing with assumption #4 in your code. Don't let me impose assumptions if they aren't the ones you're making. > Would the problem get a lot > simpler if you handled the locking locally, so that each node knew that > it was the winner for the particular task it was taking on? Possibly. But that pretty much beings to defeat the purpose of using a distributed system like S3. > I think trying to get S3 to do something it's not good at is maybe not > such a good idea, and being a fairly new service with a lot of traction, > it is likely to better support these scenarios in the future without you > having to build your own mini infrastructure. Infrastructures are a > dangerous black hole because they're so much fun to write. Indeed. And having fun is a GOOD thing. > Or perhaps the gang of machines does their local work, and then just forwards > on the actual S3 manipulation to one server, so that bit of it serializes. > But that's just a variation on handling the locking locally. Yup. > Alright... I just re-read the initial description. Sounds to me like this > would've better been handled with a local message queue. The jobs are dumped > to the queue and your gang of machines are consumers of the queue. > Smarter people than us have solved these problems already. So? It's fun figuring stuff out. Even if you then discover that someone else already invented it, you can see how close you got to their solution. -- Someone else is always smarter. But only you can decide what's pretty. Stewart Stremler -- [email protected] http://www.kernel-panic.org/cgi-bin/mailman/listinfo/kplug-lpsg
