Re: [protobuf] Handeling large amount of data
Ah thanks. That saved a lot of speed (and size)! Data usage is cut in half now (compared to the first implementation). And speed is down to 1/4 of the first implementation! -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Handeling large amount of data
If data size is an issue for you (and compatibility with an existing
protocol isn't), consider adding [packed=true] on the x/y/z arrays. I
would imagine that it will make serialization faster, and should
definitely decrease data size, assuming those arrays are bigger than a
2-3 items ("tag length, item item item item" vs "tag item tag item tag
item").
On Tue, Dec 3, 2013 at 5:23 AM, Mathijs Jonker wrote:
>
>
> Op dinsdag 3 december 2013 10:33:47 UTC+1 schreef Ilia Mirkin:
>>
>> You mentioned you added a reserve function. Are you talking about
>>
>> accelContainer.mutable_data()->Reserve(event.size)?
>
>
> Yes
>
>>
>> Unfortunately that will still construct the object when calling
>> add_data. You may consider getting rid of the sub-object and just
>> having repeated x/y/z. It's not as nice, but should be faster.
>
>
> This method does yield more performance. The data usage is smaller aswell.
> I'll stick to this method and try to optimize further.
>
>> Where does event.data come from? Perhaps it can be a RepeatedPtrField
>> that is populated with the accel data as it comes in? Then you can
>> just use ->Swap() to copy it into place.
>>
>> On Tue, Dec 3, 2013 at 4:24 AM, Mathijs Jonker wrote:
>> > Is there some other way that I can use so the elements are added faster?
>> >
>> > Op maandag 2 december 2013 21:15:41 UTC+1 schreef Mathijs Jonker:
>> >>
>> >> Sorry I meant microseconds, my bad.
>> >> The total time spend however is 0.256 seconds.
>> >> Which I find quite long.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Handeling large amount of data
Op dinsdag 3 december 2013 10:33:47 UTC+1 schreef Ilia Mirkin: > > You mentioned you added a reserve function. Are you talking about > > accelContainer.mutable_data()->Reserve(event.size)? > *Yes * > Unfortunately that will still construct the object when calling > add_data. You may consider getting rid of the sub-object and just > having repeated x/y/z. It's not as nice, but should be faster. > *This method does yield more performance. The data usage is smaller aswell.* *I'll stick to this method and try to optimize further.* > Where does event.data come from? Perhaps it can be a RepeatedPtrField > that is populated with the accel data as it comes in? Then you can > just use ->Swap() to copy it into place. > > On Tue, Dec 3, 2013 at 4:24 AM, Mathijs Jonker > > > wrote: > > Is there some other way that I can use so the elements are added faster? > > > > Op maandag 2 december 2013 21:15:41 UTC+1 schreef Mathijs Jonker: > >> > >> Sorry I meant microseconds, my bad. > >> The total time spend however is 0.256 seconds. > >> Which I find quite long. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Protocol Buffers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to protobuf+u...@googlegroups.com . > > To post to this group, send email to prot...@googlegroups.com. > > > Visit this group at http://groups.google.com/group/protobuf. > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Handeling large amount of data
You mentioned you added a reserve function. Are you talking about accelContainer.mutable_data()->Reserve(event.size)? Unfortunately that will still construct the object when calling add_data. You may consider getting rid of the sub-object and just having repeated x/y/z. It's not as nice, but should be faster. Where does event.data come from? Perhaps it can be a RepeatedPtrField that is populated with the accel data as it comes in? Then you can just use ->Swap() to copy it into place. On Tue, Dec 3, 2013 at 4:24 AM, Mathijs Jonker wrote: > Is there some other way that I can use so the elements are added faster? > > Op maandag 2 december 2013 21:15:41 UTC+1 schreef Mathijs Jonker: >> >> Sorry I meant microseconds, my bad. >> The total time spend however is 0.256 seconds. >> Which I find quite long. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Handeling large amount of data
Is there some other way that I can use so the elements are added faster? Op maandag 2 december 2013 21:15:41 UTC+1 schreef Mathijs Jonker: > > Sorry I meant microseconds, my bad. > The total time spend however is 0.256 seconds. > Which I find quite long. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Handeling large amount of data
Sorry I meant microseconds, my bad. The total time spend however is 0.256 seconds. Which I find quite long. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Handeling large amount of data
On Mon, Dec 2, 2013 at 8:11 AM, Mathijs Jonker wrote:
> Hi there,
>
> I'm using protobuf on a Embedded system.
> While I'm trying to add large amount of data I'm running into the issue
> that the container.add_data() for repeated fields takes long.
>
> I'm trying to add 8000 elements by calling add.
>
> for(size_t i = 0; i < event.size; i++) {
>> // Create a new accel protobuf
>> ::pb_Accel* newAccel = accelContainer.add_data();
>>
>> // Set the data
>>
> newAccel->set_x(event.data[i].X);
>> newAccel->set_y(event.data[i].Y);
>> newAccel->set_z(event.data[i].Z);
>> }
>>
>
> The add_data function takes 32 nanosecs. It does this for each call!
> This gives a total time of 0.256 seconds. I want to optimize this so it
> can go faster.
>
If each call takes 32 nanosecs, shouldn't 8000 calls take 8000 * 32
nanosecs = 256000 nanosecs = 256 microsecs = 0.256 millisecs? It's less
than 1ms and doesn't sound like a big deal.
>
> I've already tried adding a reserve function before I enter the loop but
> that doesn't make any difference.
>
> Any sugestions? Is there perhaps some other way of handeling pointers to
> the protobuf data?
>
> Greets,
> Mathijs
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Handeling large amount of data
Hi there,
I'm using protobuf on a Embedded system.
While I'm trying to add large amount of data I'm running into the issue
that the container.add_data() for repeated fields takes long.
I'm trying to add 8000 elements by calling add.
for(size_t i = 0; i < event.size; i++) {
> // Create a new accel protobuf
> ::pb_Accel* newAccel = accelContainer.add_data();
>
> // Set the data
>
newAccel->set_x(event.data[i].X);
> newAccel->set_y(event.data[i].Y);
> newAccel->set_z(event.data[i].Z);
> }
>
The add_data function takes 32 nanosecs. It does this for each call!
This gives a total time of 0.256 seconds. I want to optimize this so it can
go faster.
I've already tried adding a reserve function before I enter the loop but
that doesn't make any difference.
Any sugestions? Is there perhaps some other way of handeling pointers to
the protobuf data?
Greets,
Mathijs
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
