[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
*Attention:* There is an error in the implementation of read_message_delimited! if message + header are less than 4 bytes, the stream will swallow the rest. this is why instead of reading all 4 bytes at once the java implementation will only read the first byte, and then check if more is needed for the header. see readRawVarint32 in https://github.com/google/protobuf/blob/master/java/core/src/main/java/com/google/protobuf/CodedInputStream.java On Monday, October 18, 2010 at 9:11:05 PM UTC+2, prot...@googlecode.com wrote: > > Status: New > Owner: ken...@google.com > Labels: Type-Defect Priority-Medium > > New issue 226 by pkwarren: Python API doesn't support reading/writing > delimited messages > http://code.google.com/p/protobuf/issues/detail?id=226 > > What steps will reproduce the problem? > 1. Use Java Message.writeDelimitedTo(...) to serialize more than one > protobuf in a payload. > 2. Try to read in the serialized messages in Python API. > > What is the expected output? What do you see instead? > Unable to read delimited messages using Python API. Would expect similar > API calls to be available in Python API. > > What version of the product are you using? On what operating system? > Version 2.3.0, Mac OS X > > Please provide any additional information below. > Patches to enable this support in python were contributed in > > http://groups.google.com/group/protobuf/browse_thread/thread/ff78009af538ffcd/691768bdaed14d30?lnk=gst=parseDelimited#691768bdaed14d30 > > > however it doesn't appear they were checked in. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Comment #9 on issue 226 by vladimir...@gmail.com: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 Here's a workaround - assuming you want to do this because you are writing protobuf messages to sockets. The delimited write simply writes the message length as a varint32 before writing the message itself: (apologies for unidiomatic code - bit of a python noob) from google.protobuf.internal.decoder import _DecodeVarint32 from google.protobuf.internal.encoder import _EncodeVarint def write_message_delimited(socket, msg): hdr = [] _EncodeVarint(hdr.append, len(msg)) socket.sendall(.join(hdr)) socket.sendall(msg.SerializeToString()) def read_message_delimited(socket): # int length is at most 4 bytes long hdr_bytes = self.socket.recv(4) (msg_length, hdr_length) = _DecodeVarint32(hdr_bytes, 0) rsp_buffer = io.BytesIO() if hdr_length 4: rsp_buffer.write(hdr_bytes[hdr_length:]) # read the remaining message bytes msg_length = msg_length - (4 - hdr_length) while msg_length 0: rsp_bytes = self.socket.recv(min(8096, msg_length)) rsp_buffer.write(rsp_bytes) msg_length = msg_length - len(rsp_bytes) return MyMessage.ParseFromString(rsp_buffer.getvalue()) -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Comment #10 on issue 226 by vladimir...@gmail.com: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 Here's a workaround - assuming you want to do this because you are writing protobuf messages to sockets. The delimited write simply writes the message length as a varint32 before writing the message itself: (apologies for unidiomatic code - bit of a python noob) from google.protobuf.internal.decoder import _DecodeVarint32 from google.protobuf.internal.encoder import _EncodeVarint def write_message_delimited(socket, msg): hdr = [] _EncodeVarint(hdr.append, len(msg)) socket.sendall(.join(hdr)) socket.sendall(msg.SerializeToString()) def read_message_delimited(socket): # int length is at most 4 bytes long hdr_bytes = socket.recv(4) (msg_length, hdr_length) = _DecodeVarint32(hdr_bytes, 0) rsp_buffer = io.BytesIO() if hdr_length 4: rsp_buffer.write(hdr_bytes[hdr_length:]) # read the remaining message bytes msg_length = msg_length - (4 - hdr_length) while msg_length 0: rsp_bytes = socket.recv(min(8096, msg_length)) rsp_buffer.write(rsp_bytes) msg_length = msg_length - len(rsp_bytes) return MyMessage.ParseFromString(rsp_buffer.getvalue()) -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Comment #8 on issue 226 by vladimir...@gmail.com: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 You have few use cases in your codebase? Lame reason. Disappointing. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Updates: Owner: xiaof...@google.com Comment #6 on issue 226 by xiaof...@google.com: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 I searched around our internal code base and it appears there are very few use cases of the Message.writeDelimitedTo() method in Java. Likely for two reasons: first, most users will just use a repeated message field if they want to send multiple messages over the wire; second, people who want to write multiple messages by themselves will tend to want their own customized format as well (i.e., not just a size prefix but also other meta-data information like the message type name). From my understanding, the writeDelimitedTo() method in Java's MessageLIte interface shouldn't be there in the first place. It just doesn't belong to protobuf core APIs and more of an utility method that very few users would need. For this reason, I don't think we should add its counter-part in Python (or C++). -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Comment #7 on issue 226 by brian.olson: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 The lack of an API for a framed stream of protobuf messages is a feature for lack of which projects have decided to go with Thrift or another system rather than protobuf. -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Comment #5 on issue 226 by nadersal...@gmail.com: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 Any reason why? -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Updates: Status: WontFix Comment #4 on issue 226 by xiaof...@google.com: Python API doesn't support reading/writing delimited messages https://code.google.com/p/protobuf/issues/detail?id=226 (No comment was entered for this change.) -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Comment #2 on issue 226 by brian.olson: Python API doesn't support reading/writing delimited messages http://code.google.com/p/protobuf/issues/detail?id=226 I have a related issue, I've been using C++ CodedOutputStream::WriteVarint32 to write varint size prefixes to files containing multiple protobufs. I'd like Python to officially export google.protobuf.internal.decoder._DecodeVarint or similar. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages
Updates: Labels: -Type-Defect Type-Enhancement Comment #1 on issue 226 by ken...@google.com: Python API doesn't support reading/writing delimited messages http://code.google.com/p/protobuf/issues/detail?id=226 (No comment was entered for this change.) -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
