BTW, I since I can visually pick out the two strings for the first
embedded account message in the hex-dump, and they're followed by "18
00 20 05" (hex), i suspect from the .proto file that the first zero-
value byte is the boolean false value for the 'is_connected' field
(and the "05" is the DISCONNECTED enum).  In case that helps....

Thanks again,
DFlo


On Jun 16, 4:26 pm, DFlo <dflo...@raptr.com> wrote:
> Entirely possible, but given that the expected number of bytes arrive
> on the read-end of the pipe and get accumulated into a buffer, the
> problem for this particular case would seem to occur within
> ParseFromString/MergeFromString.  I'm not aware of zero-valued bytes
> causing problems in any other message.  The entire relevant python
> code near the logged lines is:
>
>     in_stream = QtCore.QDataStream(self.conn)
>     while True:
>
>       bytes_avail = self.conn.bytesAvailable()
>       if bytes_avail > 0:
>         self._readBuf += in_stream.readRawData(bytes_avail)
>       bytes_avail = len(self._readBuf)
>       if bytes_avail == 0:
>         return
>
>       if self._blockSize == 0:
>         self._blockSize = 256*ord(self._readBuf[0]) +
> ord(self._readBuf[1])
>         self._readBuf = self._readBuf[2:]
>         bytes_avail -= 2
>
>       msg_bytes = self._readBuf[0:self._blockSize]
>       self._readBuf = self._readBuf[self._blockSize:]
>       bytes_avail -= self._blockSize
>       log.debug('readHandleServerMsg: got %d-byte block' %
> self._blockSize)
>
>       msg = RaptrClientArbitraryMessage()
>       msg_ok = False;
>       try:
>         msg.ParseFromString(msg_bytes)
>       except Exception, e:
>         log.error('readHandleServerMsg: Bad msg: %s' % e)
>         # ...
>         self._blockSize = 0
>         continue  # the loop over handling the named-pipe and the
> buffered input
>
>       log.debug('readHandleServerMsg: received msg:\n%s'  % str(msg))
>
>       # ...
>
> I'll keep looking....
>
> Thanks,
> DFlo
>
> On Jun 16, 3:29 pm, Kenton Varda <ken...@google.com> wrote:
>
> > I suspect that somewhere in your pipeline, the data is being interpreted as
> > a C-style NUL-terminated string and thus being truncated at the first
> > zero-value byte.
>
> > On Wed, Jun 16, 2010 at 2:20 PM, DFlo <dflo...@raptr.com> wrote:
> > > Keeping this as short as possible, I'm using protobuf to communicate
> > > over a named-pipe on Windows, between a C++ "server" and a Python
> > > "client", and seeing cases where the Python parser clearly does not
> > > get all the data (example follows) or in other cases raises a
> > > "Truncated Message." exception.
>
> > > Is this a known problem?  I didn't see anything obvious posted about
> > > it in this group....  If so, is there a fix or a known workaround?
> > > Otherwise, the system exchanges other messages well into the thousands
> > > without any apparent difficulty; the issue is specifically limited to
> > > this one message class.  Any insight would be greatly appreciated....
>
> > > Here is one example of what's being sent by C++ (including the hex of
> > > the serialized bytestream):
>
> > > 2010/06/16 11:37:17: DEBUG: sendServerMsg: sending (371 bytes):
> > > which_message: QUERY
> > > query {
> > >  sequence_number: 36
> > >  which_query: ACCOUNTS
> > >  accounts {
> > >    username: "raptrguest9351...@raptr.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: false
> > >    status: DISCONNECTED
> > >  }
> > >  accounts {
> > >    username: "raptrguest8707...@raptr.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: false
> > >    status: DISCONNECTED
> > >  }
> > >  accounts {
> > >    username: "raptrguest6531...@raptr.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: false
> > >    status: DISCONNECTED
> > >  }
> > >  accounts {
> > >    username: "d...@raptr.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: true
> > >    status: CONNECTED
> > >  }
> > >  accounts {
> > >    username: "dflo_n...@raptr.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: false
> > >    status: DISCONNECTED
> > >  }
> > >  accounts {
> > >    username: "daveflo...@mac.com"
> > >    protocol: "prpl-aim"
> > >    is_connected: false
> > >    status: CONNECTING
> > >  }
> > >  accounts {
> > >    username: "dwflorek"
> > >    protocol: "prpl-yahoo"
> > >    is_connected: false
> > >    status: CONNECTING
> > >  }
> > >  accounts {
> > >    username: "dflo...@gxlinc.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: false
> > >    status: CONNECTING
> > >  }
> > >  accounts {
> > >    username: "raptr_d...@hotmail.com"
> > >    protocol: "prpl-msn"
> > >    is_connected: false
> > >    status: CONNECTING
> > >  }
> > > }
> > > 2010/06/16 11:37:17: DEBUG: sendServerMsg: bytes: [08 06 32 ff 02 10
> > > 24 38 01 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 39 33 35 31 37 31
> > > 38 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65
> > > 72 18 00 20 05 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 38 37 30 37
> > > 38 33 31 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62
> > > 62 65 72 18 00 20 05 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 36 35
> > > 33 31 32 33 32 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a
> > > 61 62 62 65 72 18 00 20 05 42 21 0a 0e 64 66 6c 6f 40 72 61 70 74 72
> > > 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65 72 18 01 20 03 42 26
> > > 0a 13 64 66 6c 6f 5f 6e 30 30 62 40 72 61 70 74 72 2e 63 6f 6d 12 0b
> > > 70 72 70 6c 2d 6a 61 62 62 65 72 18 00 20 05 42 22 0a 12 64 61 76 65
> > > 66 6c 6f 72 65 6b 40 6d 61 63 2e 63 6f 6d 12 08 70 72 70 6c 2d 61 69
> > > 6d 18 00 20 02 42 1a 0a 08 64 77 66 6c 6f 72 65 6b 12 0a 70 72 70 6c
> > > 2d 79 61 68 6f 6f 18 00 20 02 42 25 0a 12 64 66 6c 6f 72 65 6b 40 67
> > > 78 6c 69 6e 63 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65 72 18
> > > 00 20 02 42 26 0a 16 72 61 70 74 72 5f 64 66 6c 6f 40 68 6f 74 6d 61
> > > 69 6c 2e 63 6f 6d 12 08 70 72 70 6c 2d 6d 73 6e 18 00 20 02]
>
> > > Python gets the correct number of bytes, but fails to parse the entire
> > > message (it misses the enumerated 'status' value, and then doesn't
> > > find the other N account entries either):
>
> > > 2010/06/16 11:37:17: DEBUG: MainThread: readHandleServerMsg: got 371-
> > > byte block
> > > 2010/06/16 11:37:17: DEBUG: MainThread: readHandleServerMsg: received
> > > msg:
> > > which_message: QUERY
> > > query {
> > >  sequence_number: 36
> > >  which_query: ACCOUNTS
> > >  accounts {
> > >    username: "raptrguest9351...@raptr.com"
> > >    protocol: "prpl-jabber"
> > >    is_connected: false
> > >  }
> > > }
>
> > > A stripped down version of my .proto file is:
>
> > > message RaptrClientPing {
> > >    optional int32 timestamp = 1;
> > >    optional int32 sequence_number = 2;
> > >    // ...
> > > }
>
> > > message RaptrClientHeartbeat {
> > >    optional int32 timestamp = 1;
> > >    optional int32 sequence_number = 2;
> > >    // ...
> > > }
>
> > > message RaptrClientError {
> > >    optional int32 timestamp = 1;
> > >    optional int32 sequence_number = 2;
> > >    // ...
> > > }
>
> > > message RaptrClientQuery {
> > >    enum WhichQuery {
> > >        ACCOUNTS = 1;
> > >        // ...
> > >    }
> > >    enum AccountStatus {
> > >        ERROR = 1; CONNECTING = 2; CONNECTED = 3;
> > >        DISCONNECTING = 4; DISCONNECTED = 5;
> > >    }
> > >    message Account {
> > >        optional string username = 1;
> > >        optional string protocol = 2;
> > >        optional bool is_connected = 3;
> > >        optional AccountStatus status = 4;
> > >        optional string error_msg = 5;
> > >    }
> > >    optional int32 timestamp = 1;
> > >    optional int32 sequence_number = 2;
> > >    optional int32 in_response_to = 3;
> > >    optional string username = 4;
> > >    optional string protocol = 5;
> > >    optional string buddy_name = 6;
> > >    optional WhichQuery which_query = 7;
> > >    // response items, depending on what was queried
> > >    repeated Account accounts = 8;
> > >    // ...
> > > }
>
> > > message RaptrClientArbitraryMessage {
> > >    enum WhichMessage {
> > >        NONE = 1;
> > >        // 2 is currently unused
> > >        PING = 3;
> > >        HEARTBEAT = 4;
> > >        ERROR = 5;
> > >        QUERY = 6;
> > >        // ...
> > >    }
>
> > >    // Identifies which field is filled in.
> > >    optional WhichMessage which_message = 1;
>
> > >    // At most one of the following will be filled in.
> > >    // 2 is currently unused
> > >    optional RaptrClientPing ping = 3;
> > >    optional RaptrClientHeartbeat heartbeat = 4;
> > >    optional RaptrClientError error = 5;
> > >    optional RaptrClientQuery query = 6;
> > >    // ...
> > > }
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to