Java deserialization - any best practices for performances?
When I write out messages using C++ I'm careful to clear messages and
re-use them, is there something equivalent on the java side when
reading those same messages in?
My code looks like:
CodedInputStream stream = CodedInputStream.newInstance(inputStream);
while ( !stream.isAtEnd() )
{
MyMessage.Builder builder = MyMessage.newBuilder();
stream.readMessage(builder, null);
MyMessage myMessage = builder.build();
for ( MessageValue messageValue : myMessage.getValuesList() )
{
..
}
}
I'm passing 150 messages each with 1000 items, so presumably memory is
allocated 150 times for each of the messages...
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
Thanks, yes performance seems really good, though I wouldn't mind seeing the java deserialization faster. From: Kenton Varda [mailto:ken...@google.com] Sent: Tuesday, July 14, 2009 8:06 PM To: Alex Black Cc: protobuf@googlegroups.com Subject: Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms? So, 172 MB/s for composition + serialization. Sounds about right. On Tue, Jul 14, 2009 at 10:46 AM, Alex Black a...@alexblack.ca wrote: Thanks for those tips. I am using tcmalloc, and I'm re-using message for each batch, e.g. I fill it up with say 500 items, send it out, clear it, re-use it. Here are my hopefully accurate timings, each done 100 times, averaged: 1. Baseline (just loops through the data on the server) no protobuf: 191ms 2. Compose messages, serialize them, no I/O or deserialization: 213ms 3. Same as #2 but with IO to a dum java client: 265ms 4. Same as #3 but add java protobuf deserialization: 323ms So from this it looks like: - composing and serializing the messages takes 22ms - sending the data over sockets takes 52ms - deserializing the data in java with protobuf takes 58ms The amount of data being sent is: 3,959,368 bytes in 158,045 messages (composed in batches of 1000). - Alex From: Kenton Varda [mailto:ken...@google.com] Sent: Tuesday, July 14, 2009 3:26 AM To: Alex Black Cc: Protocol Buffers Subject: Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms? OK. If your message composition (or parsing, on the receiving end) takes a lot of time, you might look into how much of that is due to memory allocation. Usually this is a pretty significant fraction. Two good ways to improve that: 1) If your app builds many messages over time and most of them have roughly the same shape (i.e. which fields are set, the size of repeated fields, etc. are usually similar), then you should clear and reuse the same message object rather than allocate a new one each time. This way it will reuse the same memory, avoiding allocation. 2) Use tcmalloc: http://google-perftools.googlecode.com It is often faster than your system's malloc, particularly for multi-threaded C++ apps. All C++ servers at Google use this. On Mon, Jul 13, 2009 at 11:50 PM, Alex Black a...@alexblack.ca wrote: Kenton: I made a mistake with these numbers - pls ignore them - I'll revisit tomorrow. Thx. -Original Message- From: protobuf@googlegroups.com [mailto:proto...@googlegroups.com] On Behalf Of Alex Black Sent: Tuesday, July 14, 2009 2:05 AM To: Protocol Buffers Subject: Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms? ok, I took I/O out of the picture by serializing each message into a pre-allocated buffer, and this time I did a more through measurement. Benchmark 1: Complete scenario - average time 262ms (100 runs) Benchmark 2: Same as # 1 but no IO - average time 250ms (100 runs) Benchmark 3: Same as 2 but with serialization commented out - average time 251ms (100 runs) Benchmark 4: Same as 3 but with message composition commented out too (no protobuf calls) - average time 185 ms (100 runs) So from this I conclude: - My initial #s were wrong - My timings vary too much for each run to really get accurate averages - IO takes about 10ms - Serialization takes ~0ms - Message composition and setting of fields takes ~66ms My message composition is in a loop, the part in the loop looks like
Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
ok, I took I/O out of the picture by serializing each message into a
pre-allocated buffer, and this time I did a more through measurement.
Benchmark 1: Complete scenario
- average time 262ms (100 runs)
Benchmark 2: Same as # 1 but no IO
- average time 250ms (100 runs)
Benchmark 3: Same as 2 but with serialization commented out
- average time 251ms (100 runs)
Benchmark 4: Same as 3 but with message composition commented out too
(no protobuf calls)
- average time 185 ms (100 runs)
So from this I conclude:
- My initial #s were wrong
- My timings vary too much for each run to really get accurate
averages
- IO takes about 10ms
- Serialization takes ~0ms
- Message composition and setting of fields takes ~66ms
My message composition is in a loop, the part in the loop looks like:
uuid_t relatedVertexId;
myProto::IdConfidence* neighborIdConfidence =
pNodeWithNeighbors-
add_neighbors();
// Set the vertex id
neighborIdConfidence-set_id((const void*)
relatedVertexId, 16);
// set the confidence
neighborIdConfidence-set_confidence( confidence );
currentBatchSize++;
if ( currentBatchSize == BatchSize )
{
// Flush out this batch
//stream getNeighborsResponse;
getNeighborsResponse.Clear();
currentBatchSize = 0;
}
On Jul 14, 1:27 am, Kenton Varda ken...@google.com wrote:
Oh, I didn't even know you were including composition in there. My
benchmarks are only for serialization of already-composed messages.
But this still doesn't tell us how much time is spent on network I/O vs.
protobuf serialization. My guess is that once you factor that out, your
performance is pretty close to the benchmarks.
On Mon, Jul 13, 2009 at 10:11 PM, Alex Black a...@alexblack.ca wrote:
If I comment out the actual serialization and sending of the message
(so I am just composing messages, and clearing them each batch) then
the 100ms drops to about 50ms.
On Jul 14, 12:36 am, Alex Black a...@alexblack.ca wrote:
I'm sending a message with about ~150k repeated items in it, total
size is about 3.3mb, and its taking me about 100ms to serialize it and
send it out.
Can I expect to do any better than this? What could I look into to
improve this?
- I have option optimize_for = SPEED; set in my proto file
- I'm compiling with -O3
- I'm sending my message in batches of 1000
- I'm using C++, on ubuntu, x64
- I'm testing all on one machine (e.g. client and server are on one
machine)
My message looks like:
message NodeWithNeighbors
{
required Id nodeId = 1;
repeated IdConfidence neighbors = 2;
}
message GetNeighborsResponse
{
repeated NodeWithNeighbors nodesWithNeighbors = 1;
}
message IdConfidence
{
required bytes id = 1;
required float confidence = 2;
}
Where bytes id is used to send 16byte IDs (uuids).
I'm writing each message (batch) out like this:
CodedOutputStream codedOutputStream(m_ProtoBufStream);
// Write out the size of the message
codedOutputStream.WriteVarint32(message.ByteSize());
// Ask the message to serialize itself to our stream adapter,
which
ultimately calls Write on us
// which we then call Write on our composed stream
message.SerializeWithCachedSizes(codedOutputStream);
In my stream implementation I'm buffering every 16kb, and calling send
on the socket once i have 16kb.
Thanks!
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
Kenton: I made a mistake with these numbers - pls ignore them - I'll revisit
tomorrow.
Thx.
-Original Message-
From: protobuf@googlegroups.com [mailto:proto...@googlegroups.com] On Behalf Of
Alex Black
Sent: Tuesday, July 14, 2009 2:05 AM
To: Protocol Buffers
Subject: Re: Performance: Sending a message with ~150k items, approx 3.3mb, can
I do better than 100ms?
ok, I took I/O out of the picture by serializing each message into a
pre-allocated buffer, and this time I did a more through measurement.
Benchmark 1: Complete scenario
- average time 262ms (100 runs)
Benchmark 2: Same as # 1 but no IO
- average time 250ms (100 runs)
Benchmark 3: Same as 2 but with serialization commented out
- average time 251ms (100 runs)
Benchmark 4: Same as 3 but with message composition commented out too (no
protobuf calls)
- average time 185 ms (100 runs)
So from this I conclude:
- My initial #s were wrong
- My timings vary too much for each run to really get accurate averages
- IO takes about 10ms
- Serialization takes ~0ms
- Message composition and setting of fields takes ~66ms
My message composition is in a loop, the part in the loop looks like:
uuid_t relatedVertexId;
myProto::IdConfidence* neighborIdConfidence =
pNodeWithNeighbors-
add_neighbors();
// Set the vertex id
neighborIdConfidence-set_id((const void*)
relatedVertexId, 16);
// set the confidence
neighborIdConfidence-set_confidence( confidence );
currentBatchSize++;
if ( currentBatchSize == BatchSize )
{
// Flush out this batch
//stream getNeighborsResponse;
getNeighborsResponse.Clear();
currentBatchSize = 0;
}
On Jul 14, 1:27 am, Kenton Varda ken...@google.com wrote:
Oh, I didn't even know you were including composition in there. My
benchmarks are only for serialization of already-composed messages.
But this still doesn't tell us how much time is spent on network I/O vs.
protobuf serialization. My guess is that once you factor that out,
your performance is pretty close to the benchmarks.
On Mon, Jul 13, 2009 at 10:11 PM, Alex Black a...@alexblack.ca wrote:
If I comment out the actual serialization and sending of the message
(so I am just composing messages, and clearing them each batch) then
the 100ms drops to about 50ms.
On Jul 14, 12:36 am, Alex Black a...@alexblack.ca wrote:
I'm sending a message with about ~150k repeated items in it, total
size is about 3.3mb, and its taking me about 100ms to serialize it
and send it out.
Can I expect to do any better than this? What could I look into to
improve this?
- I have option optimize_for = SPEED; set in my proto file
- I'm compiling with -O3
- I'm sending my message in batches of 1000
- I'm using C++, on ubuntu, x64
- I'm testing all on one machine (e.g. client and server are on
one
machine)
My message looks like:
message NodeWithNeighbors
{
required Id nodeId = 1;
repeated IdConfidence neighbors = 2;
}
message GetNeighborsResponse
{
repeated NodeWithNeighbors nodesWithNeighbors = 1;
}
message IdConfidence
{
required bytes id = 1;
required float confidence = 2;
}
Where bytes id is used to send 16byte IDs (uuids).
I'm writing each message (batch) out like this:
CodedOutputStream codedOutputStream(m_ProtoBufStream);
// Write out the size of the message
codedOutputStream.WriteVarint32(message.ByteSize());
// Ask the message to serialize itself to our stream
adapter,
which
ultimately calls Write on us
// which we then call Write on our composed stream
message.SerializeWithCachedSizes(codedOutputStream);
In my stream implementation I'm buffering every 16kb, and calling
send on the socket once i have 16kb.
Thanks!
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
Thanks for those tips. I am using tcmalloc, and I'm re-using message
for each batch, e.g. I fill it up with say 500 items, send it out, clear
it, re-use it.
Here are my hopefully accurate timings, each done 100 times, averaged:
1. Baseline (just loops through the data on the server) no protobuf:
191ms
2. Compose messages, serialize them, no I/O or deserialization: 213ms
3. Same as #2 but with IO to a dum java client: 265ms
4. Same as #3 but add java protobuf deserialization: 323ms
So from this it looks like:
- composing and serializing the messages takes 22ms
- sending the data over sockets takes 52ms
- deserializing the data in java with protobuf takes 58ms
The amount of data being sent is: 3,959,368 bytes in 158,045 messages
(composed in batches of 1000).
- Alex
From: Kenton Varda [mailto:ken...@google.com]
Sent: Tuesday, July 14, 2009 3:26 AM
To: Alex Black
Cc: Protocol Buffers
Subject: Re: Performance: Sending a message with ~150k items, approx
3.3mb, can I do better than 100ms?
OK. If your message composition (or parsing, on the receiving end)
takes a lot of time, you might look into how much of that is due to
memory allocation. Usually this is a pretty significant fraction. Two
good ways to improve that:
1) If your app builds many messages over time and most of them have
roughly the same shape (i.e. which fields are set, the size of
repeated fields, etc. are usually similar), then you should clear and
reuse the same message object rather than allocate a new one each time.
This way it will reuse the same memory, avoiding allocation.
2) Use tcmalloc:
http://google-perftools.googlecode.com
It is often faster than your system's malloc, particularly for
multi-threaded C++ apps. All C++ servers at Google use this.
On Mon, Jul 13, 2009 at 11:50 PM, Alex Black a...@alexblack.ca wrote:
Kenton: I made a mistake with these numbers - pls ignore them -
I'll revisit tomorrow.
Thx.
-Original Message-
From: protobuf@googlegroups.com
[mailto:proto...@googlegroups.com] On Behalf Of Alex Black
Sent: Tuesday, July 14, 2009 2:05 AM
To: Protocol Buffers
Subject: Re: Performance: Sending a message with ~150k items,
approx 3.3mb, can I do better than 100ms?
ok, I took I/O out of the picture by serializing each message
into a pre-allocated buffer, and this time I did a more through
measurement.
Benchmark 1: Complete scenario
- average time 262ms (100 runs)
Benchmark 2: Same as # 1 but no IO
- average time 250ms (100 runs)
Benchmark 3: Same as 2 but with serialization commented out
- average time 251ms (100 runs)
Benchmark 4: Same as 3 but with message composition commented
out too (no protobuf calls)
- average time 185 ms (100 runs)
So from this I conclude:
- My initial #s were wrong
- My timings vary too much for each run to really get accurate
averages
- IO takes about 10ms
- Serialization takes ~0ms
- Message composition and setting of fields takes ~66ms
My message composition is in a loop, the part in the loop looks
like:
uuid_t relatedVertexId;
myProto::IdConfidence*
neighborIdConfidence = pNodeWithNeighbors-
add_neighbors();
// Set the vertex id
neighborIdConfidence-set_id((const
void*) relatedVertexId, 16);
// set the confidence
neighborIdConfidence-set_confidence(
confidence );
currentBatchSize++;
if ( currentBatchSize == BatchSize )
{
// Flush out this batch
//stream getNeighborsResponse;
getNeighborsResponse.Clear();
currentBatchSize = 0;
}
On Jul 14, 1:27 am, Kenton Varda ken...@google.com wrote:
Oh, I didn't even know you were including composition in
there. My
benchmarks are only for serialization of already-composed
messages.
But this still doesn't tell us how much time is spent on
network I/O vs.
protobuf serialization. My guess is that once you factor that
out,
your performance is pretty close to the benchmarks.
On Mon, Jul 13, 2009 at 10:11 PM, Alex Black
a...@alexblack.ca wrote:
If I comment out the actual serialization and sending of the
message
(so I am just composing messages, and clearing them each
batch) then
the 100ms drops to about
Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
If I comment out the actual serialization and sending of the message
(so I am just composing messages, and clearing them each batch) then
the 100ms drops to about 50ms.
On Jul 14, 12:36 am, Alex Black a...@alexblack.ca wrote:
I'm sending a message with about ~150k repeated items in it, total
size is about 3.3mb, and its taking me about 100ms to serialize it and
send it out.
Can I expect to do any better than this? What could I look into to
improve this?
- I have option optimize_for = SPEED; set in my proto file
- I'm compiling with -O3
- I'm sending my message in batches of 1000
- I'm using C++, on ubuntu, x64
- I'm testing all on one machine (e.g. client and server are on one
machine)
My message looks like:
message NodeWithNeighbors
{
required Id nodeId = 1;
repeated IdConfidence neighbors = 2;
}
message GetNeighborsResponse
{
repeated NodeWithNeighbors nodesWithNeighbors = 1;
}
message IdConfidence
{
required bytes id = 1;
required float confidence = 2;
}
Where bytes id is used to send 16byte IDs (uuids).
I'm writing each message (batch) out like this:
CodedOutputStream codedOutputStream(m_ProtoBufStream);
// Write out the size of the message
codedOutputStream.WriteVarint32(message.ByteSize());
// Ask the message to serialize itself to our stream adapter, which
ultimately calls Write on us
// which we then call Write on our composed stream
message.SerializeWithCachedSizes(codedOutputStream);
In my stream implementation I'm buffering every 16kb, and calling send
on the socket once i have 16kb.
Thanks!
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: Howto build protobuf not as library, but directly linking object files?
Hey Bart, I'm not sure I'm seeing the full details of your situation, But on windows I compiled a library libprotobuf.lib, and linked to it statically, meaning it gets built into the exe (or dll) you're building. (I built 2 actually, a release one and a debug one). - Alex -Original Message- From: protobuf@googlegroups.com [mailto:proto...@googlegroups.com] On Behalf Of bart van deenen Sent: Thursday, July 09, 2009 2:44 PM To: Protocol Buffers Subject: Howto build protobuf not as library, but directly linking object files? Hi this question is stupid I know :-) I've succesfully built an application with a shared library, that uses protobuf. No problem whatsoever, on LINUX! Now I have to get the Windows guys involved in it, and they know nothing about shared libraries, linker paths and such. Well neither do I but I don't use Windows, so I have an excuse. I use Qt, which has qmake which generates Makefiles. Can I just compile the files in the protobuf directory directly, and just link with the object files? Which files would I need? All files, are there files that I can skip? . |-- google | `-- protobuf | |-- compiler | | |-- cpp | | |-- java | | `-- python | |-- io | |-- stubs | |-- testdata | `-- testing `-- solaris I figure that by doing it this way, the Windows guys can just open my Qt project file, and compile the whole application (including protobuf) with the same mingw compiler. Any hints? Thanks Bart van Deenen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: GoogleOnceType showing up as a memory leak
Thanks.
From: Kenton Varda [mailto:ken...@google.com]
Sent: Monday, July 06, 2009 1:32 PM
To: Alex Black
Cc: Protocol Buffers
Subject: Re: GoogleOnceType showing up as a memory leak
Yes, this is a known bug resulting from the fact that I developed the
shutdown code on Linux rather than Windows -- this bug only exists on
Windows. It's still not a real leak unless you are dynamically loading
and unloading the protobuf library as a DLL. In any case, this has been
fixed in SVN and the fix will be in the next release.
On Mon, Jul 6, 2009 at 9:05 AM, Alex Black a...@alexblack.ca wrote:
I'm using a memory leak detection tool (ms crtdbg). It runs at
the
end of the program (after main has exited), and is reporting 7
40byte
leaks, each of which corresponds to a new call in
GoogleOnceType,
they're each one of these two:
GoogleOnceType::GoogleOnceType() {
// internal_ may be non-NULL if Init() was already called.
if (internal_ == NULL) internal_ = new
void GoogleOnceType::Init(void (*init_func)()) {
// internal_ may be NULL if we're still in dynamic
initialization
and the
// constructor has not been called yet. As mentioned in
once.h, we
assume
// that the program is still single-threaded at this time, and
therefore it
// should be safe to initialize internal_ like so.
if (internal_ == NULL) internal_ = new GoogleOnceInternal;
EnterCriticalSection(internal_-critical_section);
if (!initialized_) {
init_func();
initialized_ = true;
}
LeaveCriticalSection(internal_-critical_section);
}
I *am* calling google::protobuf::ShutdownProtobufLibrary(); at
the end
of my Main routine (i.e. before the memory leak detection is
done).
Are these real leaks? Is there any way I can avoid seeing them
as
leaks (it'd be nice to have 0 leaks... :)
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
libprotobuf.lib, 29mb?
I'm on windows, using Visual Studio 2008 SP1, compiling a Release x64 build of libprotobuf. The lib file is 29,570KB big, is that expected? I'm using the provided visual studio projects, but I added the x64 configuration to them, perhaps I have some options set incorrectly or something. (The debug lib is 33,575KB) thanks! - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: libprotobuf.lib, 29mb?
Thanks Kenton, thats reassuring. From: Kenton Varda [mailto:ken...@google.com] Sent: Thursday, July 02, 2009 8:56 PM To: Alex Black Cc: Protocol Buffers Subject: Re: libprotobuf.lib, 29mb? On Thu, Jul 2, 2009 at 5:33 PM, Alex Black a...@alexblack.ca wrote: I'm on windows, using Visual Studio 2008 SP1, compiling a Release x64 build of libprotobuf. The lib file is 29,570KB big, is that expected? I was going to say no way, but sure enough, the 32-bit release build is 18,154k, so 64-bit could easily be double that. However, when you actually link a binary with the library, the size is much smaller. protoc.exe, for example, is 1MB, and it includes both libprotobuf.lib and libprotoc.lib, the latter of which is 10MB. I don't know much about MSVC, but perhaps it is including some sort of debug info in this library? Cygwin produces a 14MB libprotobuf.a, but after stripping it is down to 968k. Note that I consider 1MB to be too big for many users, which is why the next version will have a lite mode that sheds all the descriptors and reflection features and should be very small. I'm using the provided visual studio projects, but I added the x64 configuration to them, perhaps I have some options set incorrectly or something. (The debug lib is 33,575KB) thanks! - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: Using CopyingOutputStreamAdaptor and CodedOutputStream
Error:
libprotobuf FATAL
..\src\google\protobuf\io\zero_copy_stream_impl.cc:377] CHECK
failed: buffer_used_ == buffer_size_: BackUp() can only be called after
Next().
Stack trace:
msvcr90d.dll!_NMSG_WRITE(int rterrnum=10) Line 198 C
msvcr90d.dll!abort() Line 68 C
GraphPerf.exe!google::protobuf::internal::LogMessage::Finish() Line
173 C++
GraphPerf.exe!google::protobuf::internal::LogFinisher::operator=(google:
:protobuf::internal::LogMessage other={...}) Line 177 C++
GraphPerf.exe!google::protobuf::io::CopyingOutputStreamAdaptor::BackUp(i
nt count=2202) Line 378 + 0xa0 bytes C++
GraphPerf.exe!google::protobuf::io::CodedOutputStream::~CodedOutputStrea
m() Line 526 C++
GraphPerf.exe!ProtoBufStreamAdaptor::~ProtoBufStreamAdaptor() Line 10
+ 0x59 bytes C++
From: Kenton Varda [mailto:ken...@google.com]
Sent: Monday, June 29, 2009 7:08 PM
To: Alex Black
Cc: Protocol Buffers
Subject: Re: Using CopyingOutputStreamAdaptor and CodedOutputStream
On Sun, Jun 28, 2009 at 7:30 AM, Alex Black a...@alexblack.ca wrote:
Hi, I'm trying to use these to serialize messages to my own
stream.
I have a couple of questions:
1. If I use message.SerializeToCodedStream() do I still need to
write
the size of the message to the stream on my own first?
Yes.
2. My new code looks like this:
MyStream myStream; // this implements CopyingOutputStream
CopyingOutputStreamAdaptor protoBufStream(myStream);
CodedOutputStream codedOutputStream(protoBufStream);
Then I write messages like this:
message.SerializeToCodedStream(codedOutputStream);
When the CodedOutputStream's destructor gets called, my program
aborts
hard.. Looks like there is still 5192 bytes in its buffer, and I
think
I need to call Flush() on it, but there is no flush?
What is the error message? Stack trace?
thx
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Using CopyingOutputStreamAdaptor and CodedOutputStream
Hi, I'm trying to use these to serialize messages to my own stream. I have a couple of questions: 1. If I use message.SerializeToCodedStream() do I still need to write the size of the message to the stream on my own first? Note, you can see the code I was previously using here: http://groups.google.com/group/protobuf/browse_thread/thread/9c9b410549e7eefd/b1268daa2e989c88?lnk=gstq=streaming#b1268daa2e989c88 2. My new code looks like this: MyStream myStream; // this implements CopyingOutputStream CopyingOutputStreamAdaptor protoBufStream(myStream); CodedOutputStream codedOutputStream(protoBufStream); Then I write messages like this: message.SerializeToCodedStream(codedOutputStream); When the CodedOutputStream's destructor gets called, my program aborts hard.. Looks like there is still 5192 bytes in its buffer, and I think I need to call Flush() on it, but there is no flush? thx - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: 'Streaming' messages (say over a socket)
Thanks for pointing out CodedOuptutStream::Varint32Size(), I'll use
that.
My messages are lists of messages, and I am breaking them into batches
of say 1,000 deliberately, so that each is a reasonable size that I can
allocate a buffer for and send over the wire. In one scenario I have
100,000 things to send, so I send them in batches, to avoid allocating
memory for all 100,000 at once.
m_Stream is a light wrapper around a socket, not a c++ stream, I need to
get my head around how to implement a c++ style stream wrapper around a
socket.
- Alex
From: Kenton Varda [mailto:ken...@google.com]
Sent: Wednesday, June 17, 2009 4:42 PM
To: Alex Black
Cc: Christopher Smith; Protocol Buffers
Subject: Re: 'Streaming' messages (say over a socket)
Mostly looks fine.
Note that a varint can be up to 5 bytes. You should probably just use
CodedOutputStream::Varint32Size() to compute the exact size so that you
can allocate a buffer that is exactly large enough.
Also note that if your message is large (say, 10k or more), allocating a
single large buffer may make the memory allocator unhappy. I'm not sure
what type m_Stream is, but you might consider wrapping it in a
ZeroCopyOutputStream and wrapping that in a CodedOutputStream, then
writing to that, so that it doesn't have to buffer the whole message all
at once. Note that CopyingOutputStreamAdaptor makes it pretty easy to
adapt traditional output streams to ZeroCopyOutputStream:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.pr
otobuf.io.zero_copy_stream_impl.html#CopyingOutputStreamAdaptor
On Tue, Jun 16, 2009 at 8:48 PM, Alex Black a...@alexblack.ca wrote:
Thanks, I got something working...how does this look? (ugly I'm
sure...)
On the C++ server side I am looping sending many messages (all
of the same type), and on the Java side I am looping parsing them out.
C++ Server:
int size = message.ByteSize();
EnsureBufferIsAtLeastSize(size + 4);
char* pBuffer = (char*)
google::protobuf::io::CodedOutputStream::WriteVarint32ToArray(size,
(google::protobuf::uint8*) m_pBuffer);
// Serialize the message
bool result = message.SerializeToArray(pBuffer, size);
// Calculate how many bytes the 'size' took
int sizeSize = pBuffer - m_pBuffer;
// Write the message to the stream
m_Stream.Write(m_pBuffer,size + sizeSize);
Java client:
com.google.protobuf.CodedInputStream stream =
com.google.protobuf.CodedInputStream.newInstance(url.openStream());
while ( !stream.isAtEnd() )
{
Foor.Bar.Builder builder = Foo.Bar.newBuilder();
stream.readMessage(builder, null);
Foo.Bar message = builder.build();
}
From: Christopher Smith [mailto:cbsm...@gmail.com]
Sent: Monday, June 15, 2009 2:58 PM
To: Alex Black
Cc: Protocol Buffers
Subject: Re: 'Streaming' messages (say over a socket)
The normal way to do it is to send each Entity as a separate
message. CodedInput/OutputStream is handed for that kind of thing.
--Chris
On Sun, Jun 14, 2009 at 4:14 PM, Alex Black a...@alexblack.ca
wrote:
Is there a way to start sending a message before its
fully composed?
Say we have messages like this:
message Entity
{
required int32 id = 1;
required string name = 2;
}
message Entities
{
repeated Entity entity = 1;
}
If we're sending a message Entities with 1,000 Entity
objects in it,
is there a way to avoid composing the entire message in
memory,
serializing it, and then sending it out?
I'd like to avoid allocating RAM for the entire message,
and just send
it out as I compose it...
thx,
- Alex
clear=all
--
Chris
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com
'Streaming' messages (say over a socket)
Is there a way to start sending a message before its fully composed?
Say we have messages like this:
message Entity
{
required int32 id = 1;
required string name = 2;
}
message Entities
{
repeated Entity entity = 1;
}
If we're sending a message Entities with 1,000 Entity objects in it,
is there a way to avoid composing the entire message in memory,
serializing it, and then sending it out?
I'd like to avoid allocating RAM for the entire message, and just send
it out as I compose it...
thx,
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
