Re: [protobuf] Issue using protocol buffer in Android
Where did the serialized data come from?
What library are you using on Android?
Do you have test code that runs on Android that exhibits the problem?
-- Wink
On Mon, Jun 14, 2010 at 11:34 PM, Rashmi rashmi.malav...@gmail.com wrote:
message info
{
required string username =1;
required string pwd =2;
required string pin =3;
required string imei =4;
required string imsi =5;
required string phoneNumber =6;
required string deviceName =7;
required string userName =8;
required string platform =9;
required string manufacturer =10;
required string model=11;
required string os =12;
required string screenLanguage =13;
required string screenResolution =14;
required string totalExternalMemory =15; // in MB
required string freeExternalMemory =16; // in MB
required string totalPhoneMemory =17; // in MB
required string freePhoneMemory =18; // in MB
}
When i deserialize the deviceinfo proto object, I get an exception
“Protocol Message had invalid wire type”. This is observed in android
build only.
Important Note: Everything works fine when I create and run a java
project.
I have tried with both version of protocol buffer 2.2.0 and 2.3.0.
Any help is appreciated.
Thanks in advance
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: strange behavior for a bool attribute with default value
1. I run it under valgrind. It shows unconditional jump on the line that is calling has_is_valid(), and if I remove that if() block, there is no memcheck or leakcheck problems at all. I have to mention that: a) the messages that I am trying to parse are generated by apps that are using an older version of the proto file, which does not have the optional bool is_valid = 7 [default = true]; in it, but I suppose it shouldn't matter because there is a default value there, and b) other attributes in the message are parsed perfectly fine. 2. I'll see if I could generate the problem with a smaller proto and code and if so, will post it here. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: strange behavior for a bool attribute with default value
what I meant about unconditional jump was in fact Conditional jump or move depends on uninitialised value(s) message. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] python parsing failure, related either to repeated embedded message or enum-type therein
I suspect that somewhere in your pipeline, the data is being interpreted as
a C-style NUL-terminated string and thus being truncated at the first
zero-value byte.
On Wed, Jun 16, 2010 at 2:20 PM, DFlo dflo...@raptr.com wrote:
Keeping this as short as possible, I'm using protobuf to communicate
over a named-pipe on Windows, between a C++ server and a Python
client, and seeing cases where the Python parser clearly does not
get all the data (example follows) or in other cases raises a
Truncated Message. exception.
Is this a known problem? I didn't see anything obvious posted about
it in this group If so, is there a fix or a known workaround?
Otherwise, the system exchanges other messages well into the thousands
without any apparent difficulty; the issue is specifically limited to
this one message class. Any insight would be greatly appreciated
Here is one example of what's being sent by C++ (including the hex of
the serialized bytestream):
2010/06/16 11:37:17: DEBUG: sendServerMsg: sending (371 bytes):
which_message: QUERY
query {
sequence_number: 36
which_query: ACCOUNTS
accounts {
username: raptrguest9351...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: raptrguest8707...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: raptrguest6531...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: d...@raptr.com
protocol: prpl-jabber
is_connected: true
status: CONNECTED
}
accounts {
username: dflo_n...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: daveflo...@mac.com
protocol: prpl-aim
is_connected: false
status: CONNECTING
}
accounts {
username: dwflorek
protocol: prpl-yahoo
is_connected: false
status: CONNECTING
}
accounts {
username: dflo...@gxlinc.com
protocol: prpl-jabber
is_connected: false
status: CONNECTING
}
accounts {
username: raptr_d...@hotmail.com
protocol: prpl-msn
is_connected: false
status: CONNECTING
}
}
2010/06/16 11:37:17: DEBUG: sendServerMsg: bytes: [08 06 32 ff 02 10
24 38 01 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 39 33 35 31 37 31
38 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65
72 18 00 20 05 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 38 37 30 37
38 33 31 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62
62 65 72 18 00 20 05 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 36 35
33 31 32 33 32 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a
61 62 62 65 72 18 00 20 05 42 21 0a 0e 64 66 6c 6f 40 72 61 70 74 72
2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65 72 18 01 20 03 42 26
0a 13 64 66 6c 6f 5f 6e 30 30 62 40 72 61 70 74 72 2e 63 6f 6d 12 0b
70 72 70 6c 2d 6a 61 62 62 65 72 18 00 20 05 42 22 0a 12 64 61 76 65
66 6c 6f 72 65 6b 40 6d 61 63 2e 63 6f 6d 12 08 70 72 70 6c 2d 61 69
6d 18 00 20 02 42 1a 0a 08 64 77 66 6c 6f 72 65 6b 12 0a 70 72 70 6c
2d 79 61 68 6f 6f 18 00 20 02 42 25 0a 12 64 66 6c 6f 72 65 6b 40 67
78 6c 69 6e 63 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65 72 18
00 20 02 42 26 0a 16 72 61 70 74 72 5f 64 66 6c 6f 40 68 6f 74 6d 61
69 6c 2e 63 6f 6d 12 08 70 72 70 6c 2d 6d 73 6e 18 00 20 02]
Python gets the correct number of bytes, but fails to parse the entire
message (it misses the enumerated 'status' value, and then doesn't
find the other N account entries either):
2010/06/16 11:37:17: DEBUG: MainThread: readHandleServerMsg: got 371-
byte block
2010/06/16 11:37:17: DEBUG: MainThread: readHandleServerMsg: received
msg:
which_message: QUERY
query {
sequence_number: 36
which_query: ACCOUNTS
accounts {
username: raptrguest9351...@raptr.com
protocol: prpl-jabber
is_connected: false
}
}
A stripped down version of my .proto file is:
message RaptrClientPing {
optional int32 timestamp = 1;
optional int32 sequence_number = 2;
// ...
}
message RaptrClientHeartbeat {
optional int32 timestamp = 1;
optional int32 sequence_number = 2;
// ...
}
message RaptrClientError {
optional int32 timestamp = 1;
optional int32 sequence_number = 2;
// ...
}
message RaptrClientQuery {
enum WhichQuery {
ACCOUNTS = 1;
// ...
}
enum AccountStatus {
ERROR = 1; CONNECTING = 2; CONNECTED = 3;
DISCONNECTING = 4; DISCONNECTED = 5;
}
message Account {
optional string username = 1;
optional string protocol = 2;
optional bool is_connected = 3;
optional AccountStatus status = 4;
optional string error_msg = 5;
}
optional int32 timestamp = 1;
optional int32 sequence_number = 2;
optional int32 in_response_to = 3;
optional string username = 4;
optional string protocol = 5;
[protobuf] Re: python parsing failure, related either to repeated embedded message or enum-type therein
Entirely possible, but given that the expected number of bytes arrive
on the read-end of the pipe and get accumulated into a buffer, the
problem for this particular case would seem to occur within
ParseFromString/MergeFromString. I'm not aware of zero-valued bytes
causing problems in any other message. The entire relevant python
code near the logged lines is:
in_stream = QtCore.QDataStream(self.conn)
while True:
bytes_avail = self.conn.bytesAvailable()
if bytes_avail 0:
self._readBuf += in_stream.readRawData(bytes_avail)
bytes_avail = len(self._readBuf)
if bytes_avail == 0:
return
if self._blockSize == 0:
self._blockSize = 256*ord(self._readBuf[0]) +
ord(self._readBuf[1])
self._readBuf = self._readBuf[2:]
bytes_avail -= 2
msg_bytes = self._readBuf[0:self._blockSize]
self._readBuf = self._readBuf[self._blockSize:]
bytes_avail -= self._blockSize
log.debug('readHandleServerMsg: got %d-byte block' %
self._blockSize)
msg = RaptrClientArbitraryMessage()
msg_ok = False;
try:
msg.ParseFromString(msg_bytes)
except Exception, e:
log.error('readHandleServerMsg: Bad msg: %s' % e)
# ...
self._blockSize = 0
continue # the loop over handling the named-pipe and the
buffered input
log.debug('readHandleServerMsg: received msg:\n%s' % str(msg))
# ...
I'll keep looking
Thanks,
DFlo
On Jun 16, 3:29 pm, Kenton Varda ken...@google.com wrote:
I suspect that somewhere in your pipeline, the data is being interpreted as
a C-style NUL-terminated string and thus being truncated at the first
zero-value byte.
On Wed, Jun 16, 2010 at 2:20 PM, DFlo dflo...@raptr.com wrote:
Keeping this as short as possible, I'm using protobuf to communicate
over a named-pipe on Windows, between a C++ server and a Python
client, and seeing cases where the Python parser clearly does not
get all the data (example follows) or in other cases raises a
Truncated Message. exception.
Is this a known problem? I didn't see anything obvious posted about
it in this group If so, is there a fix or a known workaround?
Otherwise, the system exchanges other messages well into the thousands
without any apparent difficulty; the issue is specifically limited to
this one message class. Any insight would be greatly appreciated
Here is one example of what's being sent by C++ (including the hex of
the serialized bytestream):
2010/06/16 11:37:17: DEBUG: sendServerMsg: sending (371 bytes):
which_message: QUERY
query {
sequence_number: 36
which_query: ACCOUNTS
accounts {
username: raptrguest9351...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: raptrguest8707...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: raptrguest6531...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: d...@raptr.com
protocol: prpl-jabber
is_connected: true
status: CONNECTED
}
accounts {
username: dflo_n...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: daveflo...@mac.com
protocol: prpl-aim
is_connected: false
status: CONNECTING
}
accounts {
username: dwflorek
protocol: prpl-yahoo
is_connected: false
status: CONNECTING
}
accounts {
username: dflo...@gxlinc.com
protocol: prpl-jabber
is_connected: false
status: CONNECTING
}
accounts {
username: raptr_d...@hotmail.com
protocol: prpl-msn
is_connected: false
status: CONNECTING
}
}
2010/06/16 11:37:17: DEBUG: sendServerMsg: bytes: [08 06 32 ff 02 10
24 38 01 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 39 33 35 31 37 31
38 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65
72 18 00 20 05 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 38 37 30 37
38 33 31 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62
62 65 72 18 00 20 05 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 36 35
33 31 32 33 32 40 72 61 70 74 72 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a
61 62 62 65 72 18 00 20 05 42 21 0a 0e 64 66 6c 6f 40 72 61 70 74 72
2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65 72 18 01 20 03 42 26
0a 13 64 66 6c 6f 5f 6e 30 30 62 40 72 61 70 74 72 2e 63 6f 6d 12 0b
70 72 70 6c 2d 6a 61 62 62 65 72 18 00 20 05 42 22 0a 12 64 61 76 65
66 6c 6f 72 65 6b 40 6d 61 63 2e 63 6f 6d 12 08 70 72 70 6c 2d 61 69
6d 18 00 20 02 42 1a 0a 08 64 77 66 6c 6f 72 65 6b 12 0a 70 72 70 6c
2d 79 61 68 6f 6f 18 00 20 02 42 25 0a 12 64 66 6c 6f 72 65 6b 40 67
78 6c 69 6e 63 2e 63 6f 6d 12 0b 70 72 70 6c 2d 6a 61 62 62 65 72 18
00 20 02 42 26 0a 16 72 61 70 74 72 5f 64 66 6c 6f 40 68 6f 74 6d 61
[protobuf] Re: python parsing failure, related either to repeated embedded message or enum-type therein
Wow, don't I feel like an idiot ... on the sending side, strncpy !=
memcpy.
Even though I allocated a big enough buffer, I didn't copy everything
into it.
I think I've embarrassed myself sufficiently for one day...
hang-dog_expression /
As you were
DFlo
On Jun 16, 4:35 pm, DFlo dflo...@raptr.com wrote:
BTW, I since I can visually pick out the two strings for the first
embedded account message in the hex-dump, and they're followed by 18
00 20 05 (hex), i suspect from the .proto file that the first zero-
value byte is the boolean false value for the 'is_connected' field
(and the 05 is the DISCONNECTED enum). In case that helps
Thanks again,
DFlo
On Jun 16, 4:26 pm, DFlo dflo...@raptr.com wrote:
Entirely possible, but given that the expected number of bytes arrive
on the read-end of the pipe and get accumulated into a buffer, the
problem for this particular case would seem to occur within
ParseFromString/MergeFromString. I'm not aware of zero-valued bytes
causing problems in any other message. The entire relevant python
code near the logged lines is:
in_stream = QtCore.QDataStream(self.conn)
while True:
bytes_avail = self.conn.bytesAvailable()
if bytes_avail 0:
self._readBuf += in_stream.readRawData(bytes_avail)
bytes_avail = len(self._readBuf)
if bytes_avail == 0:
return
if self._blockSize == 0:
self._blockSize = 256*ord(self._readBuf[0]) +
ord(self._readBuf[1])
self._readBuf = self._readBuf[2:]
bytes_avail -= 2
msg_bytes = self._readBuf[0:self._blockSize]
self._readBuf = self._readBuf[self._blockSize:]
bytes_avail -= self._blockSize
log.debug('readHandleServerMsg: got %d-byte block' %
self._blockSize)
msg = RaptrClientArbitraryMessage()
msg_ok = False;
try:
msg.ParseFromString(msg_bytes)
except Exception, e:
log.error('readHandleServerMsg: Bad msg: %s' % e)
# ...
self._blockSize = 0
continue # the loop over handling the named-pipe and the
buffered input
log.debug('readHandleServerMsg: received msg:\n%s' % str(msg))
# ...
I'll keep looking
Thanks,
DFlo
On Jun 16, 3:29 pm, Kenton Varda ken...@google.com wrote:
I suspect that somewhere in your pipeline, the data is being interpreted
as
a C-style NUL-terminated string and thus being truncated at the first
zero-value byte.
On Wed, Jun 16, 2010 at 2:20 PM, DFlo dflo...@raptr.com wrote:
Keeping this as short as possible, I'm using protobuf to communicate
over a named-pipe on Windows, between a C++ server and a Python
client, and seeing cases where the Python parser clearly does not
get all the data (example follows) or in other cases raises a
Truncated Message. exception.
Is this a known problem? I didn't see anything obvious posted about
it in this group If so, is there a fix or a known workaround?
Otherwise, the system exchanges other messages well into the thousands
without any apparent difficulty; the issue is specifically limited to
this one message class. Any insight would be greatly appreciated
Here is one example of what's being sent by C++ (including the hex of
the serialized bytestream):
2010/06/16 11:37:17: DEBUG: sendServerMsg: sending (371 bytes):
which_message: QUERY
query {
sequence_number: 36
which_query: ACCOUNTS
accounts {
username: raptrguest9351...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: raptrguest8707...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: raptrguest6531...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: d...@raptr.com
protocol: prpl-jabber
is_connected: true
status: CONNECTED
}
accounts {
username: dflo_n...@raptr.com
protocol: prpl-jabber
is_connected: false
status: DISCONNECTED
}
accounts {
username: daveflo...@mac.com
protocol: prpl-aim
is_connected: false
status: CONNECTING
}
accounts {
username: dwflorek
protocol: prpl-yahoo
is_connected: false
status: CONNECTING
}
accounts {
username: dflo...@gxlinc.com
protocol: prpl-jabber
is_connected: false
status: CONNECTING
}
accounts {
username: raptr_d...@hotmail.com
protocol: prpl-msn
is_connected: false
status: CONNECTING
}
}
2010/06/16 11:37:17: DEBUG: sendServerMsg: bytes: [08 06 32 ff 02 10
24 38 01 42 2e 0a 1b 72 61 70 74 72 67 75 65 73 74 39 33 35 31 37 31
38 40 72 61 70 74 72 2e
[protobuf] Issue 198 in protobuf: Unnecessarily inefficient calculation of utf-8 encoded lengt
Status: New
Owner: ken...@google.com
Labels: Type-Defect Priority-Medium
New issue 198 by sri...@malhar.net: Unnecessarily inefficient calculation
of utf-8 encoded lengt
http://code.google.com/p/protobuf/issues/detail?id=198
Version: 2.3.0
File: CodedOutputStream.java
public static int computeStringSizeNoTag(final String value) {
try {
final byte[] bytes = value.getBytes(UTF-8);
return computeRawVarint32Size(bytes.length) +
bytes.length;
...
In order to compute the length of the corresponding utf-8 encoding, you
don't have to encode the string and create an array. The following is
enough:
public static int utf8len(String str) {
int len = str.length();
int utf8len = len;
for (int i = 0; i len; i++) {
int c = str.charAt(i) 0x;
if (c 0x80) continue;
int extra = 0;
if (c 0x800)
extra = 1;
else if (c 0x01)
extra = 2;
else
extra = 3;
utf8len += extra;
}
return utf8len;
}
In the most common case of the string being ascii, it amounts to a scan of
the string.
--
You received this message because you are subscribed to the Google Groups Protocol
Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 198 in protobuf: Unnecessarily inefficient calculation of utf-8 encoded lengt
Updates: Status: Fixed Labels: FixedIn-2.3.1 Comment #1 on issue 198 by ken...@google.com: Unnecessarily inefficient calculation of utf-8 encoded lengt http://code.google.com/p/protobuf/issues/detail?id=198 Unfortunately your code is incorrect in the case of code points outside the 16-bit range. Java strings are UTF-16 encoded, meaning they *do* support characters outside the 16-bit range via surrogate pairs. Anyway, a recent change solved this problem by caching the resulting byte array so that it may subsequently be used when serializing, at which point you obviously need the whole array. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
