Re: rsync3 design (was Re: Linux 2.2.16 through 2.2.18preX TCP hang bug triggered by rsync)
I'm not a BEEP expert. If you'd like, perhaps we can bring Kris Magnusson into the loop. use it we should certainly learn from it. Agreed. * There seems to be no C implementation yet. Much as I like Java and Python, I think rsync still needs to be done in C, and building this would be a nontrivial part of the new implementation. There's plugin for Apache, but not a stand-alone library. That might be changing. * I hope the new library will use things like sendfile(2) when available, and I'm not sure if that will fit well into a BXXP layer. I've always been a fan of sendfile. I don't know if beep's XML bias (no binary data unless base64?) kills that. * There are already messaging layers that seem more mature, including PSMQDI, COSNotification, Gale, Java Messaging, ... Why not use one of them? Because -- at least for corba iiop-based protocols -- they suck? * Although there's a lot of potential for interesting two-way mirroring projects, I think they're probably not the core requirement. I think I'd rather provide a library that allows people to do rsync-in-BXXP just as they will do rsync-in-HTTP. Makes complete sense. * A concise protocol (within reason) is important for rsync, but the ID explicitly says that it's not a goal for BXXP. I really don't think we should use XML or explicit content types on every request. Fair enough. * I don't think we need multiple streams inside a single connection, although perhaps this is a failure of imagination on my part. Or my confusion. I thought there were like three open TCP streams with bidirectional communication going on. * If we don't need multiple streams, then we don't need asynchronous responses. Allowing for responses to come back out of order will only make the code complex without actually achieving better performance or features. If you can think of a specific way in which async replies would help, by all means say so. Nope. See above. * Although BXXP is making noise, it's not really running any major projects yet that I know of. Unless it seems like a really good fit, I don't see why rsync should be the first. If we design a clean request-response protocol we can always switch later. Sure. But if you're looking at keeping the TCP pipe full and fat, then beware the slippery slope. But since you're at least reading the docs, any duplication will be of *good* efforts. I guess in summary I think BXXP is too complex if we just want to get to a cleaner, extensible, and more reliable version of the current rsync tool. I'll buy that. You know way more than I do. I can totally live with "looks interesting, but nto quite appropriate." It's the *looking* that counts. Thanks. /r$
rsync3 design (was Re: Linux 2.2.16 through 2.2.18preX TCP hang bug triggered by rsync)
On 16 Mar 2001, Rich Salz [EMAIL PROTECTED] wrote: Here's my current idea: it may be na?ve about the TCP bugs, but I hope it will be more reliable and still keep both directions on the network maximally full. You owe it to yourself to look at BEEP and the TCP mapping. It was designed by folks who really know how to optimize ietf-style protocols. Sliding windows, sequence numbers, all that good stuff. I know it's a good design, and I've read the spec already. I'm just not sure that we need a whole other set of sliding windows etc on top of TCP if TCP is already sufficient. I believe they're RFC's now. Also read the draft-mrose-beep-design Internet-Draft. The draft-mrose-beep-design document is a very nice analysis. I'm reading it now. I wonder if adding two-way communications rather than request/response would be a good thing? It might take rsync into an interesting space for push-replication. -- Martin Pool, Human Resource Linuxcare. Inc. +61 2 6262 8990 [EMAIL PROTECTED], http://linuxcare.com.au/ Linuxcare. Putting Open Source to work.
Re: rsync3 design (was Re: Linux 2.2.16 through 2.2.18preX TCP hang bug triggered by rsync)
Rich wrote: You owe it to yourself to look at BEEP and the TCP mapping. It was designed by folks who really know how to optimize ietf-style protocols. Sliding windows, sequence numbers, all that good stuff. It's a good paper, and an interesting design. If we don't actually use it we should certainly learn from it. Their comments about framing and reply codes are good, and I certainly like SASL. (I was thinking of that before when I said PAM.) I'm certainly open to the idea that all these protocols should build on a common framework, but I wonder if the subset of problems they're trying to solve really includes rsync or not. If you don't mind being their advocate for a little longer, I'll list some concerns: * There seems to be no C implementation yet. Much as I like Java and Python, I think rsync still needs to be done in C, and building this would be a nontrivial part of the new implementation. * I hope the new library will use things like sendfile(2) when available, and I'm not sure if that will fit well into a BXXP layer. * There are already messaging layers that seem more mature, including PSMQDI, COSNotification, Gale, Java Messaging, ... Why not use one of them? * Although there's a lot of potential for interesting two-way mirroring projects, I think they're probably not the core requirement. I think I'd rather provide a library that allows people to do rsync-in-BXXP just as they will do rsync-in-HTTP. * A concise protocol (within reason) is important for rsync, but the ID explicitly says that it's not a goal for BXXP. I really don't think we should use XML or explicit content types on every request. * I don't think we need multiple streams inside a single connection, although perhaps this is a failure of imagination on my part. * If we don't need multiple streams, then we don't need asynchronous responses. Allowing for responses to come back out of order will only make the code complex without actually achieving better performance or features. If you can think of a specific way in which async replies would help, by all means say so. * Although BXXP is making noise, it's not really running any major projects yet that I know of. Unless it seems like a really good fit, I don't see why rsync should be the first. If we design a clean request-response protocol we can always switch later. * It might be simpler to go to request/response first, and then switch to BXXP later when we've worked out the kinks. Asynchronous responses or channels might be interesting for rsync-over-UDP. But I think there is already enough to learn in mapping a single specific protocol to UDP, without having another abstraction layer. I guess in summary I think BXXP is too complex if we just want to get to a cleaner, extensible, and more reliable version of the current rsync tool. If we wanted to turn it into something really different, such as allowing a whole group of machines to talk on a channel to stay synchronized, then it could be good. At the moment it doesn't seem justified. I really like modularity, and I'd kind of like to be persuaded not to write our own network layer. But at the moment this doesn't seem like the one to me. -- Martin Pool, Human Resource Linuxcare. Inc. +61 2 6262 8990 [EMAIL PROTECTED], http://linuxcare.com.au/ Linuxcare. Putting Open Source to work.
