Re: Detecting message type prior parsing a ByteBuffer containing the message
On Jul 23, 12:54 am, Benedikt Hallinger
wrote:
> Hello (and sorry for bothering) and thank you for your clarifying.
>
> > The "type" is simply transmitted as a tag on the wire. The wire format
> > of extensions is just like the wire format of a regular field - the
> > difference is that the containing type (Packet) need not know about
> > the extension fields in the .proto definition. Instead, each payload
> > type that you might include will have a unique extension number. When
> > you include one of the payload messages in your program, it will
> > register its extensions of Packet. Then when a packet message is
> > parsed, if it sees a tag that is not defined in the .proto but is a
> > valid extension, it looks up the extension information and parses it
> > appropriately.
>
> The idea is good and clear, but i have problems figuring out the code.
> The huge problem is, that the PacketManager should be designed protocol
> agnostic, so it
> can deal with all protocols designed in the extension way you described.
>
> This said, if i understood it right, i would write it as following:
> ---File: myProto.proto---
> message Packet {
> extensions n to m;
>
> }
>
> // Payload messages
> // the type field must be the same for all mesages, with differing tag
> number
> message TestMessage {
> extend Packet {
> optional TestMessage type = 10;}
>
> required string msg = 2;
>
> }
>
> -
>
> ---File: PacketHandler.java---
> class PacketHandler {
> ...
>
> // Gets called whenever a new message arrives
> // ByteBuffer is passed to this method containing
> // the raw data from the net.
> public void messageRecieved(ByteBuffer buffer) {
> byte[] bytes = buffer.array(); // fetch bytes from buffer
>
> // parse the header data;
> // this.header is passed in the constructor of PacketHandler;
> // it is an instance of a "Packet" message.
> ExtendableMessage header =
> this.header.newBuilderForType().mergeFrom(bytes).build();
You need to provide the ExtensionRegistry to the builder when invoking
mergeFrom() so that it knows how to parse the extended types.
>
> // lets see what tag number the extension has, this is the type identifier.
> // this is done via the extension registry.
> ExtensionRegistry extReg = ExtensionRegistry.newInstance();
> int type = extReg.findExtensionByName("type").descriptor().getNumber();
This doesn't access anything in the message that was transmitted - it
just sees if there was an extension registered with the name type.
Instead, you can use the reflection methods to determine which fields
are set in the parsed message - maybe something like:
Map field_map = header.getAllFields
();
for (Map.Entry field :
field_map.entrySet()) {
HandleType(field.getKey().number(),// this is the extension
number of the message that was transmitted over the wire
field.getValue());
}
>
> // invoke the right handler for that message, it will take care of the
> rest:
> getHandler(type).handle(header.getExtension(type));
>
> }
>
> ...}
>
> -
>
> However, as i tried that, it gave me errors i cant solve.
> As long as i work with concrete classes and names, all is okay, but that is
> not what i want here. What i want is to avoid that hardcoded "switch"
> statement on types, so it is easier to add new message types without having
> to fiddle around with much code.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Detecting message type prior parsing a ByteBuffer containing the message
Hello (and sorry for bothering) and thank you for your clarifying.
> The "type" is simply transmitted as a tag on the wire. The wire format
> of extensions is just like the wire format of a regular field - the
> difference is that the containing type (Packet) need not know about
> the extension fields in the .proto definition. Instead, each payload
> type that you might include will have a unique extension number. When
> you include one of the payload messages in your program, it will
> register its extensions of Packet. Then when a packet message is
> parsed, if it sees a tag that is not defined in the .proto but is a
> valid extension, it looks up the extension information and parses it
> appropriately.
The idea is good and clear, but i have problems figuring out the code.
The huge problem is, that the PacketManager should be designed protocol
agnostic, so it
can deal with all protocols designed in the extension way you described.
This said, if i understood it right, i would write it as following:
---File: myProto.proto---
message Packet {
extensions n to m;
}
// Payload messages
// the type field must be the same for all mesages, with differing tag
number
message TestMessage {
extend Packet {
optional TestMessage type = 10;
}
required string msg = 2;
}
-
---File: PacketHandler.java---
class PacketHandler {
...
// Gets called whenever a new message arrives
// ByteBuffer is passed to this method containing
// the raw data from the net.
public void messageRecieved(ByteBuffer buffer) {
byte[] bytes = buffer.array(); // fetch bytes from buffer
// parse the header data;
// this.header is passed in the constructor of PacketHandler;
// it is an instance of a "Packet" message.
ExtendableMessage header =
this.header.newBuilderForType().mergeFrom(bytes).build();
// lets see what tag number the extension has, this is the type identifier.
// this is done via the extension registry.
ExtensionRegistry extReg = ExtensionRegistry.newInstance();
int type = extReg.findExtensionByName("type").descriptor().getNumber();
// invoke the right handler for that message, it will take care of the
rest:
getHandler(type).handle(header.getExtension(type));
}
...
}
-
However, as i tried that, it gave me errors i cant solve.
As long as i work with concrete classes and names, all is okay, but that is
not what i want here. What i want is to avoid that hardcoded "switch"
statement on types, so it is easier to add new message types without having
to fiddle around with much code.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Detecting message type prior parsing a ByteBuffer containing the message
On Jul 21, 11:00 pm, Benedikt Hallinger wrote: > Hello, > thank you very much for the ideas. Since i will have a unknowable ammount > of Message types, extensions would be the way to go. > Your were right with the assumption that i use Java. > > The "construct and send that message" stuff is not the problem, the main > problem is, how i should parse the incoming ByteBuffer (or byte[]). > I just cant figure out how i would be able to access the packets type. > What im guesing now is, that since i will only send "Packet" messages (to > stick to the threads examples) in the future, my lib will know that it only > needs to deal with that "wrapper" type. From that i will access the payload > using the extension methods? Right, you would parse a Packet message. > > so the decoder part will work like this: > - get byte array from wire > - parse that data with the Packet-message types builder > (mergeFrom(buffer)) > - now we have a Packet-object with the header fields set and the payload as > extension > - extract the payload object > - push it to the appropriate handler > > But how exactly this would work with the above example? Is the "type" then > transmitted as string? The "type" is simply transmitted as a tag on the wire. The wire format of extensions is just like the wire format of a regular field - the difference is that the containing type (Packet) need not know about the extension fields in the .proto definition. Instead, each payload type that you might include will have a unique extension number. When you include one of the payload messages in your program, it will register its extensions of Packet. Then when a packet message is parsed, if it sees a tag that is not defined in the .proto but is a valid extension, it looks up the extension information and parses it appropriately. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Detecting message type prior parsing a ByteBuffer containing the message
Hello, thank you very much for the ideas. Since i will have a unknowable ammount of Message types, extensions would be the way to go. Your were right with the assumption that i use Java. The "construct and send that message" stuff is not the problem, the main problem is, how i should parse the incoming ByteBuffer (or byte[]). I just cant figure out how i would be able to access the packets type. What im guesing now is, that since i will only send "Packet" messages (to stick to the threads examples) in the future, my lib will know that it only needs to deal with that "wrapper" type. From that i will access the payload using the extension methods? so the decoder part will work like this: - get byte array from wire - parse that data with the Packet-message types builder (mergeFrom(buffer)) - now we have a Packet-object with the header fields set and the payload as extension - extract the payload object - push it to the appropriate handler But how exactly this would work with the above example? Is the "type" then transmitted as string? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Detecting message type prior parsing a ByteBuffer containing the message
By the way you may want to look at the section on "Union Types" in the
docs: http://code.google.com/apis/protocolbuffers/docs/techniques.html#union
On Jul 21, 9:51 am, jasonh wrote:
> How about using extensions to solve this? You could define a generic
> message:
>
> message Packet {
> extensions n to m;
>
> }
>
> // Payload messages
> message TestMessage {
> extend Packet {
> optional TestMessage test = 10;
> }
> required string msg = 2;
>
> }
>
> message TestMessage2 {
> extend Packet {
> optional TestMessage2 test2 = 11;
> }
> required int32 int = 2;
>
> }
>
> Then rather than transmitting bare TestMessage or TestMessage2's, you
> would transmit Packet messages, with only one of the extensions in the
> field. Something like: (assuming Java since you mentioned ByteBuffer)
>
> // Send a TestMessage
> TestMessage.Builder payload_builder = TestMessage.newBuilder();
> // Build payload.
> TestMessage payload = payload_builder.build();
> Packet packet = Packet.newBuilder().setExtension(TestMessage.test,
> payload);
> // Transmit packet
> packet.writeTo(...);
>
> This is effectively equivalent to your proposal - you would assign a
> unique id to each message type, and the proto library will take care
> of transmitting that id on the wire. You would just have to go through
> an extra accessor to get at the payload.
>
> On Jul 21, 6:12 am, Benedikt Hallinger
>
>
>
> wrote:
> > Just to be more precise: the overall goal is, that the messagetype
> > identification stuff should be managed entirely in the proto files. the
> > user should only be concerned to write his handlers and the network
> > protocol definitions.
>
> > Original Message
> > Subject: Detecting message type prior parsing a ByteBuffer containing the
>
> > message
> > Date: Tue, 21 Jul 2009 15:01:38 +0200
> > From: Benedikt Hallinger
> > To:
>
> > Hello,
> > i currently try to build some small framework that eases the usage of
> > protocol buffer for a project of me.
>
> > The system allows for easier protocol handling and is composed of this:
> > - a PacketManager responsible for dispatching recieved messages
> > (bytebuffers) to PacketHandlers
> > - several PacketHandlers that are able to handle one specific message
> > described by a .proto file
>
> > The PacketHandlers know which message type of the available they can handle
> > and tell this information to the PacketManager when they are registered.
> > The overall logic already works, but the main problem is how to detect
> > which kind of message is recieved (and therefore which PacketHandler must
> > be called to handle the message). I tried to solve this by an header
> > message with an id-field with a default value, but this wont work because
> > the id field is not set and therefore cant be extracted from the
> > bytebuffer.
>
> > This problem however can be overcome when at message construction time the
> > type is explicitely set. But i wanted to implement a way the user need not
> > to deal with message ids but can concentrate on the payload data.
> > It would be cool if one can define "hardcoded" fields in the protocol
> > definition file so that those values are always sendet over the wire and
> > are (optionally) immutable.
>
> > ---File: net.proto---
>
> > message Header {
> > optional int32 type = 1;
> > }
> > message TestMessage {
> > optional int32 type = 1 [default=10];
>
> > required string msg = 2;
> > }
> > -
> > ^^ since TestMessage.type is not set in source code, it will not be sent
> > over the wire. I know that with "required" i could force the user to set
> > that field, but it would be good if he didnt need to deal with that:
>
> > ---File: net.proto.advanced---
>
> > message Header {
> > required int32 type = 1;
> > }
> > message TestMessage {
> > optional int32 type = 1 [static=10];
>
> > required string msg = 2;
> > }
> > -
>
> > Most probably im just missing something important as im new to this topic.
> > With best regards!
> > Beni
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Detecting message type prior parsing a ByteBuffer containing the message
How about using extensions to solve this? You could define a generic
message:
message Packet {
extensions n to m;
}
// Payload messages
message TestMessage {
extend Packet {
optional TestMessage test = 10;
}
required string msg = 2;
}
message TestMessage2 {
extend Packet {
optional TestMessage2 test2 = 11;
}
required int32 int = 2;
}
Then rather than transmitting bare TestMessage or TestMessage2's, you
would transmit Packet messages, with only one of the extensions in the
field. Something like: (assuming Java since you mentioned ByteBuffer)
// Send a TestMessage
TestMessage.Builder payload_builder = TestMessage.newBuilder();
// Build payload.
TestMessage payload = payload_builder.build();
Packet packet = Packet.newBuilder().setExtension(TestMessage.test,
payload);
// Transmit packet
packet.writeTo(...);
This is effectively equivalent to your proposal - you would assign a
unique id to each message type, and the proto library will take care
of transmitting that id on the wire. You would just have to go through
an extra accessor to get at the payload.
On Jul 21, 6:12 am, Benedikt Hallinger
wrote:
> Just to be more precise: the overall goal is, that the messagetype
> identification stuff should be managed entirely in the proto files. the
> user should only be concerned to write his handlers and the network
> protocol definitions.
>
>
>
> Original Message
> Subject: Detecting message type prior parsing a ByteBuffer containing the
>
> message
> Date: Tue, 21 Jul 2009 15:01:38 +0200
> From: Benedikt Hallinger
> To:
>
> Hello,
> i currently try to build some small framework that eases the usage of
> protocol buffer for a project of me.
>
> The system allows for easier protocol handling and is composed of this:
> - a PacketManager responsible for dispatching recieved messages
> (bytebuffers) to PacketHandlers
> - several PacketHandlers that are able to handle one specific message
> described by a .proto file
>
> The PacketHandlers know which message type of the available they can handle
> and tell this information to the PacketManager when they are registered.
> The overall logic already works, but the main problem is how to detect
> which kind of message is recieved (and therefore which PacketHandler must
> be called to handle the message). I tried to solve this by an header
> message with an id-field with a default value, but this wont work because
> the id field is not set and therefore cant be extracted from the
> bytebuffer.
>
> This problem however can be overcome when at message construction time the
> type is explicitely set. But i wanted to implement a way the user need not
> to deal with message ids but can concentrate on the payload data.
> It would be cool if one can define "hardcoded" fields in the protocol
> definition file so that those values are always sendet over the wire and
> are (optionally) immutable.
>
> ---File: net.proto---
>
> message Header {
> optional int32 type = 1;
> }
> message TestMessage {
> optional int32 type = 1 [default=10];
>
> required string msg = 2;
> }
> -
> ^^ since TestMessage.type is not set in source code, it will not be sent
> over the wire. I know that with "required" i could force the user to set
> that field, but it would be good if he didnt need to deal with that:
>
> ---File: net.proto.advanced---
>
> message Header {
> required int32 type = 1;
> }
> message TestMessage {
> optional int32 type = 1 [static=10];
>
> required string msg = 2;
> }
> -
>
> Most probably im just missing something important as im new to this topic.
> With best regards!
> Beni
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Detecting message type prior parsing a ByteBuffer containing the message
Just to be more precise: the overall goal is, that the messagetype
identification stuff should be managed entirely in the proto files. the
user should only be concerned to write his handlers and the network
protocol definitions.
Original Message
Subject: Detecting message type prior parsing a ByteBuffer containing the
message
Date: Tue, 21 Jul 2009 15:01:38 +0200
From: Benedikt Hallinger
To:
Hello,
i currently try to build some small framework that eases the usage of
protocol buffer for a project of me.
The system allows for easier protocol handling and is composed of this:
- a PacketManager responsible for dispatching recieved messages
(bytebuffers) to PacketHandlers
- several PacketHandlers that are able to handle one specific message
described by a .proto file
The PacketHandlers know which message type of the available they can handle
and tell this information to the PacketManager when they are registered.
The overall logic already works, but the main problem is how to detect
which kind of message is recieved (and therefore which PacketHandler must
be called to handle the message). I tried to solve this by an header
message with an id-field with a default value, but this wont work because
the id field is not set and therefore cant be extracted from the
bytebuffer.
This problem however can be overcome when at message construction time the
type is explicitely set. But i wanted to implement a way the user need not
to deal with message ids but can concentrate on the payload data.
It would be cool if one can define "hardcoded" fields in the protocol
definition file so that those values are always sendet over the wire and
are (optionally) immutable.
---File: net.proto---
message Header {
optional int32 type = 1;
}
message TestMessage {
optional int32 type = 1 [default=10];
required string msg = 2;
}
-
^^ since TestMessage.type is not set in source code, it will not be sent
over the wire. I know that with "required" i could force the user to set
that field, but it would be good if he didnt need to deal with that:
---File: net.proto.advanced---
message Header {
required int32 type = 1;
}
message TestMessage {
optional int32 type = 1 [static=10];
required string msg = 2;
}
-
Most probably im just missing something important as im new to this topic.
With best regards!
Beni
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
