[protobuf] Max number of repeated fields
We have a protobuf message defined in the following way:
message ModuleEntry
{
optional uint32 id = 1;
optional string name = 2;
optional uint32 type = 3;
repeated ParameterEntry parameter = 4;
repeated ModuleEntry subModules = 5;
optional uint32 usage = 6;
optional string description = 7;
}
The message ModuleEntry can include several other ModuleEntrys
(submodules). One of these submodules can again include several
ParameterEntrys:
message ParameterEntry
{
required uint32 id = 1;
optional ParameterDataType dataType = 2;
optional ParameterType type = 3 [default = MEASUREMENT];
optional string name = 4;
optional int32 integerData = 5;
optional double doubleData = 6;
optional string unit = 7;
enum DynamicSimulation
{
STATIC = 0;
DYNAMIC = 1;
}
enum ParameterType
{
MEASUREMENT = 0;
CONFIGURABLE = 1;
INPUT = 2;
}
}
Our issue is appearing when we are increasing the number of ParameterEntrys
for a submodule over a certain limit (from 18 to 19). The program then
crashes with the following error message:
libprotobuf FATAL ../build/include/google/protobuf/repeated_field.h:659]
CHECK failed: (index) (size()): terminate called after throwing an
instance of 'google::protobuf::FatalException'
what(): CHECK failed: (index) (size()):
I have searched our code to find if we have set any constraints on array
size or similar, but it does not seem so. Is there any limitation to
Protobuf that is causing this behavior, i.e. a max number of repeated
fields? It seems that the number of repeated fields for messages that are
not submodules can be higher though, without crashes occuring. Also,
decreasing the number of the given submodule, or the number of its parent
module (to decrease the total data size) have no effect.
We are using Protobuf version 2.4.1. (I guess this version is kind of
outdated, is it recommendable to upgrade to the latest version?)
Regards
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Max number of repeated fields
On Thu, Jan 30, 2014 at 6:48 AM, Øyvind Bakken oyv...@gmail.com wrote:
We have a protobuf message defined in the following way:
message ModuleEntry
{
optional uint32 id = 1;
optional string name = 2;
optional uint32 type = 3;
repeated ParameterEntry parameter = 4;
repeated ModuleEntry subModules = 5;
optional uint32 usage = 6;
optional string description = 7;
}
The message ModuleEntry can include several other ModuleEntrys
(submodules). One of these submodules can again include several
ParameterEntrys:
message ParameterEntry
{
required uint32 id = 1;
optional ParameterDataType dataType = 2;
optional ParameterType type = 3 [default = MEASUREMENT];
optional string name = 4;
optional int32 integerData = 5;
optional double doubleData = 6;
optional string unit = 7;
enum DynamicSimulation
{
STATIC = 0;
DYNAMIC = 1;
}
enum ParameterType
{
MEASUREMENT = 0;
CONFIGURABLE = 1;
INPUT = 2;
}
}
Our issue is appearing when we are increasing the number of
ParameterEntrys for a submodule over a certain limit (from 18 to 19). The
program then crashes with the following error message:
libprotobuf FATAL ../build/include/google/protobuf/repeated_field.h:659]
CHECK failed: (index) (size()): terminate called after throwing an
instance of 'google::protobuf::FatalException'
what(): CHECK failed: (index) (size()):
You should check the stack trace and see why it's called with an
out-of-band index.
I have searched our code to find if we have set any constraints on array
size or similar, but it does not seem so. Is there any limitation to
Protobuf that is causing this behavior, i.e. a max number of repeated
fields?
There is no such limit in protobuf.
It seems that the number of repeated fields for messages that are not
submodules can be higher though, without crashes occuring. Also, decreasing
the number of the given submodule, or the number of its parent module (to
decrease the total data size) have no effect.
We are using Protobuf version 2.4.1. (I guess this version is kind of
outdated, is it recommendable to upgrade to the latest version?)
Regards
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Max number of repeated fields?
std::bad_alloc comes from your system's memory allocator, not from protocol
buffers. I would guess that the round number is because it creates buckets
for each allocation size, and you aren't allocating anything else that is
the same size as B. So 0x100 is your system's bucket limit.
You could try using tcmalloc (Google's memory allocator) instead.
http://code.google.com/p/google-perftools/
Alternatively, if message B only has a single int32 field, then you could
also try changing message A like:
message A {
repeated int32 b = 1 [packed=true];
}
This would be far more efficient, both in memory and on the wire.
On Thu, Aug 26, 2010 at 9:34 AM, Louis Marascio maras...@gmail.com wrote:
Hi there, I have a simple message defined with a repeated field. The
repeated field type is another message type. For example:
Message A {
repeated B b = 1;
}
Message B {
required uint32 x = 1;
}
I'm building up a single A message in memory with quite a few B's. I
am constantly receveiving a std::bad_alloc exception while trying to
allocate the 16777217 or 0x101 'th msg. This number seems oddly
round.
Is there some builtin limit to the number of repeated values a
repeated field may contain?
Thanks,
Louis
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Max number of repeated fields?
Hi there, I have a simple message defined with a repeated field. The
repeated field type is another message type. For example:
Message A {
repeated B b = 1;
}
Message B {
required uint32 x = 1;
}
I'm building up a single A message in memory with quite a few B's. I
am constantly receveiving a std::bad_alloc exception while trying to
allocate the 16777217 or 0x101 'th msg. This number seems oddly
round.
Is there some builtin limit to the number of repeated values a
repeated field may contain?
Thanks,
Louis
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
