If you're interested I have added explanatory comments to the socket-level protocol definition for Captain Proto (formerly pbcap):
http://code.google.com/p/capnproto/source/browse/proto/capnproto.proto I still have no conventions for HTTP transport. On Thu, Dec 10, 2009 at 3:33 PM, Kenton Varda <ken...@google.com> wrote: > On Thu, Dec 10, 2009 at 3:02 PM, Mikhail Opletayev <opleta...@gmail.com>wrote: > >> Will there be a way to directly connect to the service or a client >> must go thru the discovery service each time a connection is open? >> > > Why build this into the protocol when the discovery service is just as > good? Note that the protocol supports sending a request to the returned > service before actually waiting for it to be returned. In other words, a > conversation might look like: > > client: "Open service Foo." > client: "When request 1 is complete, call method Bar() on the result." > server: "Foo opened successfully. Here is a reference." > server: "Bar completed. Here is the response." > > By keeping service naming out of the low-level protocol, we keep it nice > and simple, which makes it easier to write a high-quality implementation. > > I also intend to provide a standard "discovery service" interface, but it > will be a layer that sits on top of the raw protocol implementation, rather > than being built into it. Applications will be able to use the raw protocol > without using the standard discovery service if desired. > > >> > What do you mean? The global "Stream" message exists only to define the >> > protocol. It is not actually used at runtime -- individual messages are >> > read from the stream one at a time. >> >> From pbcap.proto it looked to me as all requests and responses must be >> wrapped inside a Stream message. I guess I'll need to take a look at >> the implementation before I ask more questions, I only had time to >> take a quick peak earlier today. >> > > Maybe what you don't like is the fact that the app-specific request message > is embedded as "bytes" in pbcap.CallRequest? Currently this requires a > reduntant copy of all those bytes. However, I was thinking of adding an > optimization to the protobuf implementation to avoid this -- we could add a > way to construct a new ByteString that is a substring of some other > ByteString in constant time (with no copies). Then, parsing "bytes" fields > -- when the original input is a ByteString -- would not require a copy, > making things efficient. > > >> >> P.S. I hope my questions don't derail this thread too much as the >> original question was about HTTP RPC. >> >> On Dec 10, 4:42 pm, Kenton Varda <ken...@google.com> wrote: >> > On Thu, Dec 10, 2009 at 2:37 PM, Mikhail Opletayev <opleta...@gmail.com >> >wrote: >> > >> > > Interesting. Essentially a discovery service for protobuf RPC. >> > >> > > I am not quite sure what you mean by "pointers to other services". Is >> > > it going to reference them by name or a more complex structure >> > > containing full endpoint information? >> > >> > Currently it references them by an ID number that is tied to the >> particular >> > connection. So, each time a new service object is returned on the >> > connection, a new ID number is assigned to it. >> > >> > > Also, is it going to be an extension to pbcap or something completely >> > > new? >> > >> > Not an extension -- this *is* pbcap. It supports this already. >> > >> > (Note that I'm planning to change the name to "Captain Proto", aka >> > capnproto, to avoid the confusion with pcap.) >> > >> > > The reason I am asking is because some patterns in pbcap (such as >> > > wrapping up everything into a global Stream message) are rather >> > > questionable and not without consequences. >> > >> > What do you mean? The global "Stream" message exists only to define the >> > protocol. It is not actually used at runtime -- individual messages are >> > read from the stream one at a time. >> > >> > >> > >> > > Regards, >> > >> > > On Dec 10, 4:22 pm, Kenton Varda <ken...@google.com> wrote: >> > > > On Thu, Dec 10, 2009 at 8:13 AM, Mikhail Opletayev < >> opleta...@gmail.com >> > > >wrote: >> > >> > > > > It's great news that you working on a standard way to communicate >> > > > > between Protocol Buffers implementations! >> > >> > > > > > You don't need to send the service name at all. The server >> should >> > > > > already >> > > > > > know what kind of service it is exporting. >> > >> > > > > I think its handy to export several services from the same end >> point, >> > > > > especially if you are running RPC over something else than HTTP. >> If >> > > > > you were to run Protocol Buffers RPC over plain sockets you'd >> probably >> > > > > want to publish a bunch of services on the same port. >> > >> > > > This is exactly my point. If you use the service type name to >> identify >> > > the >> > > > service, then you can only export one service of each type. >> Instead, >> > > some >> > > > other name -- having nothing to do with the type name -- should be >> used >> > > to >> > > > identify the service. >> > >> > > > Actually, the implementation I'm working on doesn't even identify >> > > services >> > > > by names. Instead, when you first connect on a port, you connect to >> the >> > > > "default service" for that port. However, the default service can >> send >> > > back >> > > > pointers to other services in RPC responses. So, the default >> service may >> > > > have a method which looks up other services by name, but this is up >> to >> > > the >> > > > application. >> > >> > > > > In Dataflow implementation we use one field (method) but we >> require >> > > > > the method name to be in "serviceName.methodName" format. >> > >> > > > > For the same reason we decided against using HTTP headers to >> transfer >> > > > > the RPC metadata as it binds you to the transport protocol. That's >> why >> > > > > send content length and header length as the first 2 values in the >> > > > > coded stream, then the header message, then the actual data >> message: >> > >> > > > >http://www.dataflow-software.com/docs/pbuf-rpc.html >> > >> > > > > On Dec 10, 3:15 am, Kenton Varda <ken...@google.com> wrote: >> > > > > > 2009/12/10 Romain François <francoisrom...@free.fr> >> > >> > > > > > > On 12/09/2009 09:12 PM, Kenton Varda wrote: >> > >> > > > > > >> Coincidentally, last weekend I started working on an open >> source >> > > > > > >> protobuf-based RPC system. Currently I am defining a >> socket-level >> > > > > > >> protocol, but I also intend to support an HTTP-level protocol >> with >> > > > > > >> optional JSON encoding to allow calls from Javascript. I >> stuck >> > > some >> > > > > > >> totally undocumented code here: >> > >> > > > > > > Thanks. My intention of having it over http is that it can >> > > communicate >> > > > > with >> > > > > > > other languages. I'd be good if we can synchronize our >> protocols. >> > >> > > > > > > I need to make some changes based on what you said on another >> > > thread, >> > > > > and >> > > > > > > then I'll make my java basic server code available. >> > >> > > > > > > http://pbcap.googlecode.com >> > >> > > > > > >> But some coworkers pointed out that the name is confusingly >> > > similar to >> > > > > > >> "pcap", so I'm planning to change it. >> > >> > > > > > >> Currently this is not an official Google project; I'm working >> on >> > > it in >> > > > > > >> my spare time. >> > >> > > > > > >> 2009/12/9 Romain François <francoisrom...@free.fr >> > > > > > >> <mailto:francoisrom...@free.fr>> >> > >> > > > > > >> A request looks like this : >> > >> > > > > > >> ----------------------------------------------------- >> > > > > > >> POST /{service full name}/{method name} HTTP/1.0 >> > >> > > > > > >> I would recommend against putting the service type name in >> the >> > > URL. >> > > > > > >> This makes it impossible to export two services of the same >> type. >> > > > > > >> Instead, you should allow the application to register >> services >> > > under >> > > > > > >> any name it chooses. >> > >> > > > > > > Fair enough. Maybe adding some protobuf specific headers : >> > >> > > > > > > ProtobufService: {service full name} >> > >> > > > > > You don't need to send the service name at all. The server >> should >> > > > > already >> > > > > > know what kind of service it is exporting. >> > >> > > > > > > ProtobufMethod: {method full name} >> > >> > > > > > You do need the method name, though. Inventing new HTTP headers >> > > isn't >> > > > > > usually a good idea as they may confuse proxies and such. >> > >> > > > > > > or encode it as parameters of the url as you said. >> > >> > > > > > > I'd also suggest making the method name be part of the query, >> > > like: >> > >> > > > > > >> POST /someservice?method={method name} >> > >> > > > > > >> This may be a matter of taste, but I feel like a service >> object >> > > should >> > > > > > >> be a single HTTP "resource", rather than have each method be >> a >> > > > > separate >> > > > > > >> resource. >> > >> > > > > > >> Connection: close >> > >> > > > > > >> Why not allow pipelining? >> > >> > > > > > > this was simpler to do a one shot service call, but indeed why >> not. >> > >> > > > > > > Content-Length: {length of the serialized message} >> > >> > > > > > >> {raw bytes of the serialized message} >> > > > > > >> ----------------------------------------------------- >> > >> > > > > > >> And a successful response looks like this: >> > >> > > > > > >> ----------------------------------------------------- >> > > > > > >> HTTP/1.1 200 OK >> > > > > > >> Content-length: {length of the serialized response} >> > >> > > > > > >> {raw bytes of the serialized response} >> > > > > > >> ----------------------------------------------------- >> > >> > > > > > >> Since this is very early in this, I wondered if others >> would >> > > have >> > > > > views >> > > > > > >> on this. >> > >> > > > > > >> http is quite verbose for sending protobuf message around, >> but >> > > it >> > > > > is >> > > > > > >> likely to be implemented for a lot of languages. >> > >> > > > > > >> Regards, >> > >> > > > > > >> Romain >> > >> > > > > > > -- >> > > > > > > Romain Francois >> > > > > > > Professional R Enthusiast >> > > > > > > +33(0) 6 28 91 30 30 >> > > > > > >http://romainfrancois.blog.free.fr >> > > > > > > |-http://tr.im/Gq7i:ohloh >> > > > > > > |-http://tr.im/FtUu:newpackage : highlight >> > > > > > > `-http://tr.im/EAD5:LondonRslides >> > >> > > > > -- >> > >> > > > > You received this message because you are subscribed to the Google >> > > Groups >> > > > > "Protocol Buffers" group. >> > > > > To post to this group, send email to proto...@googlegroups.com. >> > > > > To unsubscribe from this group, send email to >> > > > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com> >> <protobuf%2bunsubscr...@googlegroups.com<protobuf%252bunsubscr...@googlegroups.com> >> > >> > > <protobuf%2bunsubscr...@googlegroups.com<protobuf%252bunsubscr...@googlegroups.com> >> <protobuf%252bunsubscr...@googlegroups.com<protobuf%25252bunsubscr...@googlegroups.com> >> > >> > >> > > > > . >> > > > > For more options, visit this group at >> > > > >http://groups.google.com/group/protobuf?hl=en. >> > >> > > -- >> > > Mikhail Opletayev >> > >http://dataflow-software.com >> > >> > > -- >> > >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Protocol Buffers" group. >> > > To post to this group, send email to proto...@googlegroups.com. >> > > To unsubscribe from this group, send email to >> > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com> >> <protobuf%2bunsubscr...@googlegroups.com<protobuf%252bunsubscr...@googlegroups.com> >> > >> > > . >> > > For more options, visit this group at >> > >http://groups.google.com/group/protobuf?hl=en. >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To post to this group, send email to proto...@googlegroups.com. >> To unsubscribe from this group, send email to >> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/protobuf?hl=en. >> >> >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
