[protobuf] Re: Java: how to use protobuf to send messages quickly over sockets?
I'm a bit embarrassed :) The protobuf version of my code uses about 950MB of memory (the Java Serializable version is only using around 650MB) and I had the java - Xmx flag set too low; in reality protobuf is extremely fast compared to Java Serializable: Java Serializable: 12,000 msgs/sec Protocol Buffers (as described in my first post): 70,000 msgs/sec Protocol Buffers (CodedOutputStream, as described in Evan's post): 73,000 msgs/sec Protocol Buffers (CodedOutputStream, but no flush() after each write): 76,600 msgs/sec Thanks very much for the help, I'm very happy with the performance! Jonathan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
[protobuf] Java: how to use protobuf to send messages quickly over sockets?
I'm looking to get sense if my experience sounds reasonable or if it sounds like I'm doing something very wrong, any insight appreciated! I have a Serializable Java object containing: 4 integers 3 bytes 2 Strings 1 short 1 double I have a vector of about 10M objects I am sending from a server to a client. The ObjectOutputStream for the server is created like this: Socket clientSocket; ... outputStream= new ObjectOutputStream(clientSocket.getOutputStream()); and the ObjectInputStream is created in a similar fashion. I send the Serializable objects like this: outputStream.writeObject(msg); and receive them like this: MyMessage msg = (MyMessage)inputStream.readObject(); On my rather slow server I can receive and process about 12K messages a second. Now for the Protocol Buffers part: I used protoc to create a new Java class using: 8 int32s 2 strings 1 double I create objects like this: MyProtoMessage.MyMessagemsg = MyProtoMessage.MyMessage.newBuilder() ... .build(); and send them like this: byte size = (byte)msg.getSerializedSize(); outputStream.writeByte(size); outputStream.write(msg.toByteArray()); and finally they are read like this: byte size = inputStream.readByte(); byte []bytes = new byte[size]; inputStream.readFully(bytes); MyProtoMessage.MyMessage msg = MyProtoMessage.MyMessage.parseFrom (bytes); With this protobuf approach I can do about 9K a second. I also tried writeDelimited/parseDelimited and it was even slower. Does this sound reasonable? I expected it to be faster than the standard Java serialization approach. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
[protobuf] Re: Java: how to use protobuf to send messages quickly over sockets?
Thanks for the info. I thought the Java serialization metadata would have been large compared to the relevant data, but I guess not! On Oct 29, 5:42 pm, Kenton Varda ken...@google.com wrote: It sounds plausible. There's no fundamental reason why protocol buffers should be faster than Java serialization, at least for simple objects like yours composed of a set of primitive values. Since Java serialization is implemented by the VM, it can probably optimize better than protobufs can. However, the protobuf wire format is considerably simpler, portable to many languages other than Java, and handles extensibility better. Also, I suspect you'll find that Java serialization gets slower with more complex object trees. And finally, Java serialization involves sending a large chunk of metadata over the wire, basically describing each class -- protocol buffers avoids this by assuming that the receiver already knows the type information. But when sending a giant homogeneous array of simple objects, the metadata overhead ends up small. On Thu, Oct 29, 2009 at 1:45 PM, jta23 jonathanhalc...@gmail.com wrote: I'm looking to get sense if my experience sounds reasonable or if it sounds like I'm doing something very wrong, any insight appreciated! I have a Serializable Java object containing: 4 integers 3 bytes 2 Strings 1 short 1 double I have a vector of about 10M objects I am sending from a server to a client. The ObjectOutputStream for the server is created like this: Socket clientSocket; ... outputStream = new ObjectOutputStream(clientSocket.getOutputStream()); and the ObjectInputStream is created in a similar fashion. I send the Serializable objects like this: outputStream.writeObject(msg); and receive them like this: MyMessage msg = (MyMessage)inputStream.readObject(); On my rather slow server I can receive and process about 12K messages a second. Now for the Protocol Buffers part: I used protoc to create a new Java class using: 8 int32s 2 strings 1 double I create objects like this: MyProtoMessage.MyMessage msg = MyProtoMessage.MyMessage.newBuilder() ... .build(); and send them like this: byte size = (byte)msg.getSerializedSize(); outputStream.writeByte(size); outputStream.write(msg.toByteArray()); and finally they are read like this: byte size = inputStream.readByte(); byte []bytes = new byte[size]; inputStream.readFully(bytes); MyProtoMessage.MyMessage msg = MyProtoMessage.MyMessage.parseFrom (bytes); With this protobuf approach I can do about 9K a second. I also tried writeDelimited/parseDelimited and it was even slower. Does this sound reasonable? I expected it to be faster than the standard Java serialization approach. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
