[protobuf] 2.4.0 and lazy UTF-8 conversions in Java

2010-11-30 Thread Blair Zajac
I saw this change coming up in 2.4.0: Added lazy conversion of UTF-8 encoded strings to String objects to improve performance. Is the lazyness thread safe? Without looking at the implementation, then if it isn't thread safe, I would guess this isn't much overhead, but if it is thread safe

Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java

2010-11-30 Thread Evan Jones
On Nov 30, 2010, at 15:58 , Blair Zajac wrote: Added lazy conversion of UTF-8 encoded strings to String objects to improve performance. Is the lazyness thread safe? Without looking at the implementation, then if it isn't thread safe, I would guess this isn't much overhead, but if it is

Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java

2010-11-30 Thread Blair Zajac
On 11/30/10 1:21 PM, Evan Jones wrote: On Nov 30, 2010, at 15:58 , Blair Zajac wrote: Added lazy conversion of UTF-8 encoded strings to String objects to improve performance. Is the lazyness thread safe? Without looking at the implementation, then if it isn't thread safe, I would guess this

Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java

2010-11-30 Thread Kenton Varda
To answer your questions: 1) It is thread-safe. 2) No locking or other synchronization overhead is involved. The string data is stored as an Object pointer which, at any particular time, may point at either a String or a ByteString. When one form or the other is requested, we convert the

[protobuf] Re: Issue 243 in protobuf: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream

2010-11-30 Thread protobuf
Updates: Status: New Comment #4 on issue 243 by ken...@google.com: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream http://code.google.com/p/protobuf/issues/detail?id=243 Woops, for some reason I thought you were talking about C++. Not sure why,

[protobuf] Re: Issue 243 in protobuf: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream

2010-11-30 Thread protobuf
Comment #5 on issue 243 by john.carrino: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream http://code.google.com/p/protobuf/issues/detail?id=243 If the message is over 64MB, the CodedInputStream it creates under the covers will barf during parsing. So I

[protobuf] Re: Issue 243 in protobuf: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream

2010-11-30 Thread protobuf
Updates: Status: Accepted Comment #6 on issue 243 by ken...@google.com: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream http://code.google.com/p/protobuf/issues/detail?id=243 Ah, I see. You are talking about individual messages over 64MB... for

[protobuf] Re: Issue 243 in protobuf: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream

2010-11-30 Thread protobuf
Comment #7 on issue 243 by john.carrino: Improve mergeDelimitedFrom in the java Impl to setSizeLimit on the CodedInputStream http://code.google.com/p/protobuf/issues/detail?id=243 Yeah, this is due to message A having a lot of repeated element B. We could pull B out of A and into the top

[protobuf] Re: Issue 83 in protobuf: protobuf does not compile cleanly in 64-bit mode in Visual Studio 2008

2010-11-30 Thread protobuf
Comment #18 on issue 83 by ken...@google.com: protobuf does not compile cleanly in 64-bit mode in Visual Studio 2008 http://code.google.com/p/protobuf/issues/detail?id=83 Well that's easy enough. Revision 351. Leaving the bug open because I don't think this is the only thing people cared

[protobuf] Handling streaming in Python

2010-11-30 Thread Dan
Hello, Does anyone have or can point me to an example of handling a stream of protobuf messages in python? Specifically what I'm trying to do is parse a stream that looks like the following HeaderSize PB int32 HeaderMessage MessageSize PB int32 Message ... MessageSize PB int32 Message The

[protobuf] Issue 244 in protobuf: Autogenerated Java code producing error: Cannot invoke buildParsed() on the primitive type boolean

2010-11-30 Thread protobuf
Status: New Owner: ken...@google.com Labels: Type-Defect Priority-Medium New issue 244 by kentmchenry: Autogenerated Java code producing error: Cannot invoke buildParsed() on the primitive type boolean http://code.google.com/p/protobuf/issues/detail?id=244 What steps will reproduce the

[protobuf] Re: Issue 244 in protobuf: Autogenerated Java code producing error: Cannot invoke buildParsed() on the primitive type boolean

2010-11-30 Thread protobuf
Updates: Status: WontFix Comment #1 on issue 244 by ken...@google.com: Autogenerated Java code producing error: Cannot invoke buildParsed() on the primitive type boolean http://code.google.com/p/protobuf/issues/detail?id=244 Your protoc version does not match the version of

Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java

2010-11-30 Thread Blair Zajac
On 11/30/10 3:15 PM, Kenton Varda wrote: To answer your questions: 1) It is thread-safe. 2) No locking or other synchronization overhead is involved. The string data is stored as an Object pointer which, at any particular time, may point at either a String or a ByteString. When one form or the

Re: [protobuf] 2.4.0 and lazy UTF-8 conversions in Java

2010-11-30 Thread Kenton Varda
On Tue, Nov 30, 2010 at 1:21 PM, Evan Jones ev...@mit.edu wrote: * I'll guess that it will not be slower if you access all the strings. Currently, the parsing process copies the raw bytes from the input buffer into an individual byte array, then converts that to a String. This is, sadly, the

Re: [protobuf] Handling streaming in Python

2010-11-30 Thread Gregory Szorc
You can load the _DecodeVarint and _VarintEncoder modules from google.protobuf.internal.{decoder,encoder} if you want to risk using an internal interface. See https://github.com/indygreg/zippylog/blob/master/lib/py/zippylog/stream.py for an example (although I can't guarantee the Python code