Re: [protobuf] [De]serialization of messages to java strings
What am I doing wrong? What's the best way to do java string serialization of protobuf messages? The native wire format of protocol buffers is just a sequence of bytes, so it can contain values that are invalid UTF-8 (or any encoding that has invalid byte sequences). Trying to pack that into a String, which holds Unicode character data, isn't going to work well; Strings are welcome to mangle the bytes however they want as long as the same characters are represented. If you want to pass a serialized protocol buffer to something else, you should generally use a ByteString, byte[], or ByteBuffer. If you absolutely have to pass things around as a String, you're going to need to do so in some kind of encoding that supports arbitrary data. For example, you could encode it in Base64. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Java -using GPB -which type of stream should I be using over sockets?
I have created a test Java application that uses ObjectInputStream and ObjectOutputStream over sockets. No problems, it works! I use my GPB class writeTo and parseFrom to send and reconstitute my GPB class using the ObjectInput and Output Streams But: isn't GPB allowing for serialization using bytes? Are Object I/O Streams the best streams to use? Should I be using something else like Data I/O streams? E.g. from JavaDocs, An ObjectOutputStream writes primitive data types and graphs of Java objects to an OutputStream -is there a better, more efficient I/O stream I should be using? Protocol buffer code doesn't use object streams, so when you pass one to its writeTo/parseFrom methods, you're actually passing the object stream as a regular OutputStream or InputStream, ie, your code is already serializing to and from bytes. If you'd rather be explicit about it, you can just switch your stream's type from ObjectOutputStream to OutputStream (similarly with Input) and it should continue to work the same. All that should require is instead of doing something like ObjectOutputStream output = new ObjectOutputStream(socket.getOutputStream()); message.writeTo(output); you do something like OutputStream output = socket.getOutputStream(); message.writeTo(output); For interacting with C++, you should be able to use the standard C++ API for reading in messages, since the data is already on the wire in byte format. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Java -using GPB -which type of stream should I be using over sockets?
If protocol buffers doesn't use object I/O streams, which streams does it use? The argument in Java simply asks for an OutputStream argument, and of course that could mean File, ByteArray, Object, Data, etc I mean that it only depends on the methods in OutputStream. You can provide anything that implements OutputStream (including ObjectOutputStream, if you wish, but it won't encode things using ObjectOutputStream's additional methods for encoding Java objects, it will only write bytes to the stream). And perhaps more importantly, will the choice of Java stream types impact how GPB objects may be reconstituted in C++? At the simplest level I assume this can be done: I can create a GPB class called GPBCar. I have an outputstream (of some type, not sure!) over a socket in Java I want to write to. After creating my GPBCar object, I use the GPB writeTo(output) method on the Java side. My C++ app takes some form of this byte stream and reconstitutes my GPBCar instance by using the equivalent of the parseFrom method from the GPBCar. Yep, that should work fine. My understanding is that I have converted my GPBCar to a series of bytes that represent that GPBCar instance. Now I have a C++ app at the other end of the socket connection. But will the series of bytes be in the correct order (endianness?). Will the outputstream type be particular to Java, or will I encounter problems by using certain specific Java outputstreams? Should I only be using the CodedDataOutputStream provided by GPB? (I note that there is implementations of this stream in both the C++ and Java GPB implementations). The bytes should arrive in the correct order because the wire format specifies the byte order of everything (and the protocol buffer code properly implements that). It could get screwed up if some other system on either end is attempting to modify the byte order, but in general that shouldn't be the case. What kind of OutputStream you use should have no effect, unless it's an OutputStream that transforms the output in some way. You don't have to use CodedOutputStream. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Receiving/Parsing Messages
I just started looking into protobuf for a project of mine.
From the Java Api page I could not really find how to parse a
generated (compiled .proto is present) but unknown message.
So for example: I have messages types MessageA and MessageB. The
client component receives some bytes representing a message of type A
OR B. Do I have to add information of the type of message that's been
send, or is there an easy way of automatically parsing the message
like:
byte[] b; //hols byte representation of message
Message message = foo(b); //parse message
if(message instanceof MessageA)
System.out.println(was type A);
else
System.out.println(was type B);
No, there's no way to do this, because the wire format doesn't include
information about the type of message. It's even possible that the
same set of bytes could be a valid message of both types.
The usual way to handle this is to create a wrapper message that can
hold either, like so:
// Only one field may be filled out
message AorB {
optional MessageA message_a;
optional MessageB message_b;
}
And then you can parse it via:
AorB result = AorB.parseFrom(data);
if (result.hasMessageA()) {
System.out.println(Was type A);
} else if (result.hasMessageB()) {
System.out.println(Was type B);
}
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Java -using GPB -which type of stream should I be using over sockets?
Anyway, for anyone else wanting to use ByteArrayStream in Java for writing a Google , here is some code I've used: ServerSocket serverSocket = new ServerSocket(2004, 10); Socket socket = serverSocket.accept(); OutputStream outputStream = socket.getOutputStream(); outputStream.flush(); MyGPBObjects.MyGPBObject myGPBObject = MyGPBObjectsFactory.instance.createMyGPBObject(); ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream (); myGPBObject.writeTo(byteArrayOutputStream); byteArrayOutputStream.flush(); byteArrayOutputStream.writeTo(outputStream); What you're doing with the ByteArrayOutputStream will do the same thing as: OutputStream outputStream = socket.getOutputStream(); myGPBObject.writeTo(outputStream); except that it takes more time and uses more memory. You don't need an intermediate output stream, you already have the socket's output stream. You can just write directly to that. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Message References
A standard protocol buffer message type ptr_B from which a flat Java
class will be generated does not seem adequat to to so.
Instead some new abstract Class RefMessageT extends Message with a
method T getReferencedObject() would be the solution. ptr_X
messages simply had to be transformed to Java classes extending this
superclass.
My questions so far:
I could write my own code generator for that purpose, right?
Has anybody tried to implement some similar techniques for reference
types on top of protocol buffer?
Is there a completely different approach for this problem?
Am i getting the design and purpose of protobuf wrong when I need/want
such a solution?
When I've seen a similar thing done, it's generally been done like this:
message A {
required int32 ownvalue = 1;
required int32 bref = 2;
}
message B {
required int32 othervalue = 1;
}
message Overall {
repeated B b = 1;
repeated A a = 2;
}
And then you do
Overall overall;
A a;
B b = overall.getB(a.getBref());
Basically, you first send a lookup table, then you send the remaining
items, with references as indexes into the table. To serialize a
graph using that technique, you could send a list of nodes and then a
list of edges. I don't know that it's the best way of doing it, but
it's worked out well where I've seen it.
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Externally compiled protobuf messages
An example:
Foo.jar contains the compiled Java code for the following protobuf:
option java_package = com.foo;
message FooMsg {
optional int32 a = 1;
optional int32 b = 2;
...
}
In my code, I want to do this:
option java_package = com.bar;
message BarMsg {
optional com.foo.FooMsg my_foo = 1;
...
}
However, for the purposes of this discussion, I do not have access to
the proto file for FooMsg, and thus cannot simply do an import
FooMsg.proto.
Just want to know if this is possible as it would be very convenient
for me.
There are two ways to do this, one if you can deal with a dependency
in the other direction, and one if neither can depend on the other.
If you can allow FooMsg to depend on BarMsg, then you can use extensions:
in barmsg.proto:
message BarMsg {
extension 100 to max;
}
in foomsg.proto:
message FooMsg {
optional int32 a = 1;
optional int32 b = 2;
extend BarMsg {
optional FooMsg my_foo = 100;
}
}
Alternatively, if you can't have dependencies in either direction, you
can send opaque messages as bytes, like so:
message BarMsg {
optional bytes my_foo = 1;
}
But you have to do the serializing and deserializing manually:
BarMsg bar = BarMsg.parseFrom(whatever);
FooMsg foo = FooMsg.parseFrom(bar.getMyFoo());
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Builders containing builders
I wish I could pass a builder object to a method and have the method
modify either a value of the builder or a value of a sub-message in
the builder!
I came across this thread, which described exactly the problem I have.
The Car/Engine example in the thread is perfectly illustrative of my
scenario.
http://groups.google.com/group/protobuf/browse_thread/thread/1699791071e92c83/fc77205a755721f0
Has anyone else been in this same situation? What have you done to
ameliorate the problem?
Generally speaking, I use non-PB model objects as my primary in-memory
representation, and then I serialize them on demand to get the PB
representation when I need it. For instance, something like:
public class Engine {
private String make;
private double volumeLiters;
public EnginePb serialize() {
return
EnginePb.newBuilder().setMake(make).setVolumeLiters(volumeLiters).build();
}
}
public class Car {
private Engine engine;
private String make;
private String name;
public CarPb serialize() {
return
CarPb.newBuilder().setMake(make).setName(name).setEngine(engine.serialize()).build();
}
}
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] how to define a one-way rpc message?
the only workaround that i can think of would be creating a Void
message that is declared as the answer and the rpc implementation to
then ignore those.
like this:
message Void
{
}
service myService
{
rpc OneWayWithoutResponse(FirmwareEvent) returns (Void);
}
is there a way how i can avoid this workaround?
Nope, that's the way it's done. Generally speaking, it's recommended
that you create a separate empty response message for each service
method instead of a generic Void message you return from many methods,
so that if you ever do decide to send a response back you don't have
to switch message types, but otherwise that's the usual way it's done.
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How to prevent different protocol buffers from being parseFrom(byte[])
I am confused on how to detect this scenario, and ultimately prevent such things occurring. You can't, at least in the simple way. Protocol buffers on the wire have no type information associated with them, and they're explicitly designed so that they can accept fields they don't expect to be present (for forwards compatibility), so serializing a message of one type and deserializing it as another type will work sometimes. A couple potential solutions come to mind. One is to send all your messages wrapped in another message that includes a type identifier. Another would be to have all your messages include a key field that must be filled in in a specific way (eg, tag 1 in all messages is a int field named tag, and there's a single value for each message type that you require to be filled in). What is most appropriate depends on how you expect a message of one type to be sent to something expecting a different type. (Obviously, the best solution is to set up your well-typed language APIs to keep that from happening.) - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] serializing the message using the compiled file.
I want to serialize the message (id, title and teh text). Can someone please help me out with this. You can call yourmessage.toByteArray() (or toByteString()) to get it as an object, or yourmessage.writeTo(OutputStream) to send it to an OutputStream. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] serialization size from 2.0.x to 2.3.x, also message design best practices
In the case of repeated strings etc (excluding the enum case), I've been toying whether something is possible by associating certain objects / values with unique identifiers on the wire. Potentially this would also allow graph (rather than tree) serialization. This is obviously well into the hazy area of outside what protobuf offers, but possible to represent as valid protobuf fragments / messages, but I'd be interested in people's thoughts... worth investigating? Or silly? We've done that before to save space. In the simplest case, you can just put a repeated string field in your outermost message that's a list of every unique string in the rest of the message, and then anywhere else in the message that you would have put a string, you put an int that says which string should go there. There are downsides to that approach, though. The biggest one is that you can't build your message up piece by piece, you have to build the whole message object in one go, because all of them need to have input into this one global data structure; similarly, you can't process the message on the other side without carrying around the index everywhere you're using any part of the message. It can save a lot of space if you end up repeating the same strings over and over, though. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Self describing messages
I don't want to send the whole enchilada on the wire - just the
descriptor for the message being sent.
As I showed above, I know how to get the descriptor on the wire, but
the other side (getting the descriptor off the wire and parsing the
message using it) is unclear to me.
You'll need to bundle it up into a FileDescriptorProto (on the
receiving side), which is the only unit at which the system can turn
descriptors into actual messages. The code looks something like this:
DescriptorProto proto = whatever;
FileDescriptorProto fileProto =
FileDescriptorProto.newBuilder().addMessageType(proto).build();
FileDescriptor file = FileDescriptor.buildFrom(fileProto, new
FileDescriptor[] {});
Descriptor descriptor = file.getMessageTypes().get(0);
Note that this won't work if you use submessages, because submessage
references in DescriptorProtos include their package, and this way of
doing it doesn't specify a package, so it won't be able to find the
submessage definition (unless you're using the default package, of
course). To fix that, you can either pass the package name along with
the DescriptorProto, or you can munge the DescriptorProto on the
sending side to include the package in its name, resulting in code
like the following.
DescriptorProto proto = whatever;
String name = proto.getName();
int dotIndex = name.lastIndexOf('.');
String protoName = name.substring(dotIndex + 1);
String packageName = (dotIndex == -1) ? : name.substring(0, dotIndex);
DescriptorProto modified =
DescriptorProto.newBuilder(proto).setName(protoName).build();
FileDescriptorProto fileProto =
FileDescriptorProto.newBuilder().addMessageType(modified).setPackage(packageName).build();
FileDescriptor file = FileDescriptor.buildFrom(fileProto, new
FileDescriptor[] {});
Descriptor descriptor = file.getMessageTypes().get(0);
Once you have the descriptor, you just need you use one of the
DynamicMessage.parseFrom() methods to parse the message.
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] can i use protocol buffer to encrypt and send a file
I have a file which i need to encrypt and send so can i use protocol buffer for this buffer? please do reply me. Protocol buffers don't provide any built-in encryption or anything like that, no, all they provide is a serialization format. You could use protocol buffers to send a file you encrypted via some other method, but I don't think that's what you're asking. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Parse PB bytes into a generic message
I need a generic parser in java, which can de-serialize PB bytes buffer into a Message or GeneratedMessage. But, GeneratedMessage provides no such method to merge an arbitary PB bytes and provide an instance of itself. From just a message's bytes you can't turn it into a Message. The serialized format doesn't include type information. You've got a couple choices. First, if you can send the message's Descriptor along with the message, then you can use DynamicMessage.parseFrom(Descriptor, byte[]) to deserialize it. If you can't send the message's Descriptor along, then you can use UnknownFieldSet.parseFrom(byte[]), but that's not going to give you very much at all, just tag numbers, their wiretype, and their value. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Parse PB bytes into a generic message
Since the client and server are remote, i am not sure how i can send Descriptor along with message? All the different kinds of descriptors have associated protocol messages available through their toProto() methods, so you can serialize them, send them over the wire, and deserialize them on the other end. You have to deserialize a FileDescriptorProto, though, so you can either send the appropriate FileDescriptorProto or just send a DescriptorProto and build the file on the receiving end. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] uint32 vs int32: who is the most efficient? (Java)
Then my question is this: why to use int32 type even though uint32 uses less bytes covering the same range? The main reason would be if you ever plan on reading the message from non-Java code. What you've stated is only true in Java because it has no unsigned integral types, so protobufs have to use signed types even when a message declares a field as unsigned. In any language that has unsigned integral types, you would write -1 and the other side would read 4294967295. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Dynamic protocol buffer generation in Java
I'd like to use dynamically generate/use protocol buffer objects in a running Java application. This would be running in a production environment (jre, not jdk) where protoc is not available. Has anyone written a utility in Java to parse a protocol buffer definition and dynamically build Java classes at runtime using something like asm? No need for asm or anything like that, the protobuffer libraries already contain enough to do what you want to do. To create a new message definition, you can use DescriptorProto to create the message, put it in a FileDescriptorProto, and use FileDescriptor.buildFrom() to build it. Now you can get the descriptors out with FileDescriptor.getMessageTypes(). Once you have a message's Descriptor, you can parse messages using DynamicMessage.parseFrom(), and you can create new messages using DynamicMessage.Builder. - Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
