Re: Standard for RPC proto

[fpmc]

Out of the several specifications, a problem I find is they all use
serialized messages as a byte string as part of the message.  That's
inefficient and in the case of C++ involves triple-copying - from
socket buffer to kernel buffer to user buffer to parsed message.  At
the very least that last copy should be avoided (from what I know the
first two can't be avoided at the current moment unless you have
sophisticated hardware).

Also the java version tries to do too much, adding the ability to
query what methods are available.  I think that's best done as a
default RPC service on top of the RPC layer (as in, every rpc server
has a rpc method called "GetKnownMethods()" for example).  Agreeing
with most people here, extra layers like security should be done in
the layer above or below (think RPC over https).

After stripping that all down, you can have a simple protocol such as:
message RpcRequestHeader {
  required uint32 id = 1;
  enum RequestType {
REQUEST  = 0;
CANCEL   = 1;
  }
  optional RequestType type = 2 [default = REQUEST];
  optional string service_name = 3;
  optional string method_name = 4;
}

where every request would send the request header message, followed by
the request message.  Each message is sent as a length/serialized-
message pair, where length is a varint64.  This is what I'm doing so
far as an experimentation in implementing C++ based rpc.

Frank

On Oct 28, 2:37 pm, "Paul P Komkoff Jr" <[EMAIL PROTECTED]> wrote:
> On Tue, Oct 28, 2008 at 9:07 PM, Pavel Shramov <[EMAIL PROTECTED]> wrote:
> > By the way one of the simpliest ways for RPC is to use HTTP transport.
> > It's have some limitations (e.g large overhead for small messages) but
> > also some benefits (e.g many libraries for performing HTTP calls and
> > simple proxying)
>
> Speaking of ugly hacks, one of the intermediate versions of rpc I used
> were, actually, protocol buffers over xmlrpc. messages were serialized
> and then wrapped into xmlrpc.Binary
>
> --
> This message represents the official view of the voices in my head.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Paul P Komkoff Jr]

On Tue, Oct 28, 2008 at 9:07 PM, Pavel Shramov <[EMAIL PROTECTED]> wrote:
> By the way one of the simpliest ways for RPC is to use HTTP transport.
> It's have some limitations (e.g large overhead for small messages) but
> also some benefits (e.g many libraries for performing HTTP calls and
> simple proxying)

Speaking of ugly hacks, one of the intermediate versions of rpc I used
were, actually, protocol buffers over xmlrpc. messages were serialized
and then wrapped into xmlrpc.Binary


-- 
This message represents the official view of the voices in my head.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Pavel Shramov]

On Tue, Oct 28, 2008 at 11:46:19AM -0700, Kenton Varda wrote:
> Ever notice how practically no one uses HTTP auth?  :)
By the way one of the simpliest ways for RPC is to use HTTP transport.
It's have some limitations (e.g large overhead for small messages) but
also some benefits (e.g many libraries for performing HTTP calls and
simple proxying)

All RPC implementations mentioned on wiki use raw link to exchange data.
So I've created simple RPC over HTTP but description [1] is a bit...
incomplete :) If somebody is interested in this approach I'll expand
documentation. Code may be found at [2, 3]

Pavel

P.S. Sorry for self-advertising :)

--
[1] http://grid.pp.ru/wiki/pbufrpc
[2] http://grid.pp.ru/git/?p=psha/pbufrpc/.git
[3] git://grid.pp.ru/psha/pbufrpc

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Kenton Varda]

Ever notice how practically no one uses HTTP auth?  :)

On Tue, Oct 28, 2008 at 1:16 AM, Paul P. Komkoff Jr <[EMAIL PROTECTED]>wrote:

>
> On Oct 28, 2:02 am, "Kenton Varda" <[EMAIL PROTECTED]> wrote:
> > I don't really have a stake in the design of a protobuf-based RPC format.
> >  However, I'd like to point out that the design philosophy we tend to
> prefer
> > at Google is to keep each layer of the system as simple as possible, and
> > implement orthogonal features using separate layers.  Authentication is a
> > great example of something that I would not want to make part of an RPC
> > protocol itself, but rather implement as a layer under it, similar to the
> > way HTTP can operate over SSL.  If you keep the system separate in this
> way,
>
> First, I'm talking about something similar to simple HTTP auth, which
> allows us to authenticate by key/value pair and does not include TLS.
> With support for "struct user_credentials" passed to server method, so
> we can "impersonate" the user.
>
> Also, even before considering paragraph above, if you have the system
> separate that way it will produce incompatible wire formats. My goal
> is to have, at least, lowest common denominator which could be
> implemented in, at least, twisted-python and something-java, in order
> to bootstrap my project now. It would be wonderful if this LCD format
> will have some notion of authentication (or authorization, if
> authentication is performed by separate coexisting entity that
> produces auth cookies).
>
> > it's much easier for people to avoid the overhead of features they don't
> > need, find alternative ways of implementing individual features, and to
> > reuse code in general.
> > Just my opinion.
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Alan Kligman]

Authentication doesn't really belong here. You should either use an
authenticated transport (like HTTPS), or in the layer above (this is
what I'm currently doing).

On Oct 27, 8:54 pm, "Paul P. Komkoff Jr" <[EMAIL PROTECTED]> wrote:
> On 26 окт, 02:53, Alan Kligman <[EMAIL PROTECTED]> wrote:
>
> > I haven't had much to add recently. Protobuf-rpc is based heavily on
> > json-rpc, so there's really nothing new behind it. It works well for
> > my own use and is generic enough to probably work well for most other
> > people.
>
> > Is there a great deal of interest in devising a standard rpc protocol
> > definition?
>
> Yes it is.
> Since everything is trying to design its own RPC format, running into
> the same flaws as everyone else.
> For example, I haven't seen (in protobuf-rpc neither in protorcp) a
> single word about authentification.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Marc Gravell]

Paul - I haven't looked at protobuf-rpc, but protorpc uses a .proto
message for the payload. One up-shot of that is that it should be (in
theory at least) fine to add extension properties to the message -
i.e. you could add a security object as an extended property. A given
server could check for expected extension properties and act
accordingly. The only issue I have with this is that you'd need to put
the options *after* the main body, or push the tags out of order...

No idea if that is a good idea or not...

Marc
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Paul P. Komkoff Jr]

On Oct 28, 2:02 am, "Kenton Varda" <[EMAIL PROTECTED]> wrote:
> I don't really have a stake in the design of a protobuf-based RPC format.
>  However, I'd like to point out that the design philosophy we tend to prefer
> at Google is to keep each layer of the system as simple as possible, and
> implement orthogonal features using separate layers.  Authentication is a
> great example of something that I would not want to make part of an RPC
> protocol itself, but rather implement as a layer under it, similar to the
> way HTTP can operate over SSL.  If you keep the system separate in this way,

First, I'm talking about something similar to simple HTTP auth, which
allows us to authenticate by key/value pair and does not include TLS.
With support for "struct user_credentials" passed to server method, so
we can "impersonate" the user.

Also, even before considering paragraph above, if you have the system
separate that way it will produce incompatible wire formats. My goal
is to have, at least, lowest common denominator which could be
implemented in, at least, twisted-python and something-java, in order
to bootstrap my project now. It would be wonderful if this LCD format
will have some notion of authentication (or authorization, if
authentication is performed by separate coexisting entity that
produces auth cookies).

> it's much easier for people to avoid the overhead of features they don't
> need, find alternative ways of implementing individual features, and to
> reuse code in general.
> Just my opinion.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Marc Gravell]

I'm currently working on the guts of a protorpc layer for protobuf-
net; so yes, any conversations here are very valued. Especially re
test rigs ;-p

I don't have any huge bias for/against either of the cited specs
(protorcp/protobuf-rpc). I just want something that works ;-p

Marc
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Kenton Varda]

I don't really have a stake in the design of a protobuf-based RPC format.
 However, I'd like to point out that the design philosophy we tend to prefer
at Google is to keep each layer of the system as simple as possible, and
implement orthogonal features using separate layers.  Authentication is a
great example of something that I would not want to make part of an RPC
protocol itself, but rather implement as a layer under it, similar to the
way HTTP can operate over SSL.  If you keep the system separate in this way,
it's much easier for people to avoid the overhead of features they don't
need, find alternative ways of implementing individual features, and to
reuse code in general.
Just my opinion.

2008/10/27 Paul P. Komkoff Jr <[EMAIL PROTECTED]>

>
> On 26 окт, 02:53, Alan Kligman <[EMAIL PROTECTED]> wrote:
> > I haven't had much to add recently. Protobuf-rpc is based heavily on
> > json-rpc, so there's really nothing new behind it. It works well for
> > my own use and is generic enough to probably work well for most other
> > people.
> >
> > Is there a great deal of interest in devising a standard rpc protocol
> > definition?
>
> Yes it is.
> Since everything is trying to design its own RPC format, running into
> the same flaws as everyone else.
> For example, I haven't seen (in protobuf-rpc neither in protorcp) a
> single word about authentification.
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Paul P. Komkoff Jr]

On 26 окт, 02:53, Alan Kligman <[EMAIL PROTECTED]> wrote:
> I haven't had much to add recently. Protobuf-rpc is based heavily on
> json-rpc, so there's really nothing new behind it. It works well for
> my own use and is generic enough to probably work well for most other
> people.
>
> Is there a great deal of interest in devising a standard rpc protocol
> definition?

Yes it is.
Since everything is trying to design its own RPC format, running into
the same flaws as everyone else.
For example, I haven't seen (in protobuf-rpc neither in protorcp) a
single word about authentification.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Alan Kligman]

I haven't had much to add recently. Protobuf-rpc is based heavily on
json-rpc, so there's really nothing new behind it. It works well for
my own use and is generic enough to probably work well for most other
people.

Is there a great deal of interest in devising a standard rpc protocol
definition?

On Oct 23, 5:49 pm, Marc Gravell <[EMAIL PROTECTED]> wrote:
> This whole conversation (proto / rpc) seems to have been very quiet
> for a while. I'd be very keen to implement rpc for protobuf-net, so /
> any/ kind of consensus would be good. I suppose I'd default to the
> more Java compatible, but...
>
> Marc
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Marc Gravell]

This whole conversation (proto / rpc) seems to have been very quiet
for a while. I'd be very keen to implement rpc for protobuf-net, so /
any/ kind of consensus would be good. I suppose I'd default to the
more Java compatible, but...

Marc
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Kenton Varda]

Google's RPC system is huge and bloated, and designed explicitly for our
hardware and data center configurations.  So, I don't think publishing our
protocol would be very helpful, and it would take a lot of work just to
document it properly.  I'm not even sure if our protocol makes sense for use
over the general internet (vs. fast, reliable internal networks).  Sorry.

On Wed, Oct 22, 2008 at 9:44 AM, <[EMAIL PROTECTED]> wrote:

>
> To make different RPC implementations based on Protocol Buffers
> compatible with each other, they must use the same method of data
> serialization. However, it's not the case for RPC service fields like
> method name or message id:
>
> Java project Protorcp uses one specificaton (http://
> protorpc.likbilen.com/Protorpcdocprotobuf.html),
> Python protobuf-rpc uses another (http://protobuf-rpc.googlecode.com/
> svn/trunk/protocol/pbrpc.proto
> )
>
> The common point is that both use Protocol Buffers and some proto to
> serialize the service fields.
>
> Since PB are used inside Google for RPC, is it possible to publish the
> Google's variant of RPC serialization? It may become the standard, and
> all implementations would be compatible.
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Standard for RPC proto

[gariev]

To make different RPC implementations based on Protocol Buffers
compatible with each other, they must use the same method of data
serialization. However, it's not the case for RPC service fields like
method name or message id:

Java project Protorcp uses one specificaton (http://
protorpc.likbilen.com/Protorpcdocprotobuf.html),
Python protobuf-rpc uses another (http://protobuf-rpc.googlecode.com/
svn/trunk/protocol/pbrpc.proto)

The common point is that both use Protocol Buffers and some proto to
serialize the service fields.

Since PB are used inside Google for RPC, is it possible to publish the
Google's variant of RPC serialization? It may become the standard, and
all implementations would be compatible.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---