[protobuf] Re: proposal: options for json format to allow int64

Hi Christian, Are you sure that BigQuery won't accept JSON strings for integers? I'm reading the code for BigQuery's JSON importer and it suggests to me that strings are accepted. Have you had problems with this in the past? On Monday, January 4, 2016 at 5:36:24 PM UTC-8, chri...@lefty.io

[protobuf] Re: GRPC and Ruby, Proto language level 2

Hi Josh, On Friday, January 8, 2016 at 3:17:11 PM UTC-8, Josh Humphries wrote: > > I am actively pursuing moving a proprietary RPC mechanism to GRPC. And the > languages currently needed are Ruby, Go, and Java. > > I didn't realize before trying a few things today that the newly added >

[protobuf] Re: Faster way to convert C++ protobuf into a Python protobuf?

Hi Phil, If you are not doing so already, I would highly recommend using the C++ implementation for Python (instead of the pure-Python one). It will make ParseFromString() in Python much, much faster, which might give you the speed boost you need. There are possible ways of doing more clever

[protobuf] Re: memory leak from ParseFromArray()

Hi Rob, Is there any way to get more information about the objects it says have leaked? I'm surprised that your tool is showing objects being freed on exit that aren't freed by deleting outp and calling google::protobuf::ShutdownProtobufLibrary(). I don't think the protobuf library installs

[protobuf] Re: Any JSON coding question

Hi Tim, I think your analysis is correct. The answer to your question is that it's not possible to transcode Any from JSON <-> protobuf unless you have the schema. That is one of the reasons why the type URL exists as a way to fetch schemas for types you otherwise wouldn't know about. I'm

[protobuf] Re: Minimal Protobuf for deserializing from SerializedArray

Hi Tanweer, You didn't mention which language you're using, but I'm guessing C++ or Java. You can pare down the full library by using the lite runtime -- see details here: https://developers.google.com/protocol-buffers/docs/proto#options Hope this helps, Josh On Tuesday, February 9, 2016 at

[protobuf] Re: What tool does the protobuf project use to generate ref manual from c++ headers?

Hi Tamas, I think the site is mostly hand-written with some light templating. You could probably do a similar thing with any number of open-source site builders like Jekyll. Josh On Tuesday, February 9, 2016 at 11:06:59 AM UTC-8, Tamás Kenéz wrote: > > What tool does the protobuf project use

[protobuf] Re: Questions about Services

Hi Kevin, I think gRPC is probably your best bet. It's true that it isn't in GA yet, but it's far more mature than any other options I'm aware of for protobuf services. You can't "use" generic services per se. Generic services just provide an abstract interface -- someone has to provide an

[protobuf] Re: Unable to get through error: Protocol message contained an invalid tag (zero).

Hi Vishal, The message basically means that the protobuf is corrupted (or isn't a protobuf at all). The message is giving you more detail on the specific kind of data corruption. Protocol buffers don't allow tag number 0 -- you can't define a field like "int32 a = 0". But the binary protobuf

[protobuf] Re: --decode_raw errors with 'Failed to parse input'

It looks like Vinay didn't get a response when posting in 2013. A couple things to keep in mind though: - protobufs are raw binary data, and they can contain NULL bytes. In Vinay's message above, GDB is probably treating the serialized protos as text strings, and so will stop printing at the

[protobuf] Re: Convert java class to proto message. Converting list with abstract or interface types.

Hi Sergey, Protocol Buffers have an extension capability which is similar to what you are describing. The way to describe this in a .proto file would be something like: syntax = "proto2"; message ChatMessage { optional string message = 1; extensions 100 to 200; } // These definitions can

[protobuf] Re: Repeated string java

Hi there, could you post your code snippet and the full exception you are getting? On Friday, January 22, 2016 at 9:54:40 AM UTC-8, John Doe wrote: > > Hi, I'm using protobuf 2.6.1 and I've got a problem using a message with a > simple repeated string: > > message Foo { > . > repeated

[protobuf] Re: Is there a reliable method to dynamically identify and lookup a message from both the Java and Java script API?

HI Keith, Can you post more about what your router needs to do with the wrapped message? What prevents you from just tunneling the serialized message as bytes? What do you gain by knowing its type more specifically and unpacking it? On Friday, June 3, 2016 at 3:52:07 AM UTC-7, Keith Woods

[protobuf] Re: something wrong when parsing protobuf binary file in windows.

If you can write a self-contained reproducible test case, then please file an issue and we'll take a look. Thanks, Josh On Wednesday, February 17, 2016 at 11:09:08 AM UTC-8, Yilong Wei wrote: > > Same here. Trying to cross-compile caffe which uses protobuf, and > caffe::ReadProtoFromBinaryFile

[protobuf] Re: casting from google::protobuf::Message c++

Are you sure that "message" is a MsgFrame class? Since dynamic_cast<>() is failing, I suspect it is not. You should be aware that m1->CopyFrom(m2) can succeed even if m1 and m2 have different C++ classes, as long as both classes have the same descriptor. For example, m1 could be MsgFrame and

[protobuf] Re: Can protobuf work on ppc64le?

I am not sure, and we don't maintain a canonical list of supported platforms for each release. Give it a shot and see if it works. :) On Wednesday, February 17, 2016 at 11:09:08 AM UTC-8, Qian Zhang wrote: > > Hi, > > I'd like to know if the latest stable protobuf release (v2.6.1) can be >

Re: [protobuf] Message Migration Practice?

If the structure of the schema changes, your new schema will be incompatible with your old schema. You might need to convert the data manually to your new schema in this case. On Tuesday, February 16, 2016 at 8:58:58 AM UTC-8, Vincent Dupuis wrote: > > How to migrate when the schema structure

[protobuf] Re: Handle data migration. e.g. moving of field - Java

Yes I think that plan is your best bet. One option is to leave the old field in your source code but comment it out, so you have a record of the field as it was previously defined. You may also want to mark the field numbers and names as "reserved":

[protobuf] Re: Using different versions of Protobuf

Hi there, Many people have run into this issue. To solve it we will be committing to ABI stability for the protobuf 3.0 release. Once 3.0 is released, you will be able to upgrade the core library to a new version without breaking other libraries. Sorry this can't help you for 2.5/2.6, but in

[protobuf] Re: Serializing mixed type arrays with protobuf

If you want an array of mixed-type data, the best option in protobuf is something like this: message Types { oneof types { Foo foo = 1; Bar bar = 2; } } message Container { repeated Types list = 1; } On Thursday, February 11, 2016 at 10:43:10 AM UTC-8, Flo Loferer wrote: > > I

[protobuf] Re: C++ JsonToBinaryStream losing large uint32 values

Thanks a lot for tracking this down. For reference, here it is in our bug tracker: https://github.com/google/protobuf/issues/1243 On Thursday, February 11, 2016 at 3:28:33 AM UTC-8, Ron Ben-Yosef wrote: > > Hi, > > I'm using the proto3 C++ library compiled in MSVC, and noticed some pretty >

[protobuf] Re: report a bug

Hi and thanks for the report. Please file bugs in GitHub issues: https://github.com/google/protobuf/issues On Friday, February 19, 2016 at 3:57:10 PM UTC-8, ykwd.tsing...@gmail.com wrote: > > Hi > > This is a nice project which saves me a lot of times. > > During my use I find a bug in some

[protobuf] Re: Unhelpful compiler warning when a file has no syntax declaration

Hi there -- sorry for the delay on your PR, a couple guys are OOTO right now so we're a bit backed up. I just merged it. In general, we should be monitoring PR's and not letting them sit too long. A conversation first can be good if the change is involved, but for simple changes like this just

Re: [protobuf] Any JSON coding question

On Mon, Mar 14, 2016 at 7:07 PM, Feng Xiao wrote: > On Mon, Mar 14, 2016 at 6:14 PM, Tim Kientzle wrote: >> >> I was considering Any for cases such as logging, data storage, and other >> systems where intermediaries do not need to access or understand

[protobuf] Re: Does proto3 allow recursion?

The easiest way to find out is just to try. :) Both of these examples compile fine with proto3. On Monday, March 21, 2016 at 10:43:23 AM UTC-7, Zachary Deretsky wrote: > > Is recursion handled, for example: > > message Node { > string value = 1; > Node parent = 2; >

Re: [protobuf] Re: Streaming Serialization - Suggestion

Hi Yoav, Chunked encoding is definitely an interesting idea, and I can see the benefits you mentioned. However proto2 and proto3 are more or less frozen from a wire perspective. There are lots of existing clients out there already communicating with proto3, so we're not really at liberty to

[protobuf] Re: ID in protobuf

Hi there, Protocol Buffers don't come with any RPC system built-in. There is a way to generate generic services, but these are just interfaces with no implementation. If you want to do actual RPCs, you will want to integrate with an RPC system like gRPC: http://www.grpc.io/ On Wednesday,

Re: [protobuf] Dynamic/run-time decoding

Hi Mike, Python indeed has descriptor classes that you can use to build proto definitions at runtime. However it does not have a parser for .proto files built in. So if you want to load .proto files at runtime, you'll need to use protoc to build a descriptor (for example, you could shell out

[protobuf] Re: how can i change my protobuf object to byte arrays in javascript

For the moment, you can call msg.serializeBinary(), but make sure you generated the JavaScript with the binary=1 option. In the near future this will be changing. The binary serialization code will get generated into separate files from the data structures themselves. So in the future it will

[protobuf] Re: Public roadmap to RTM?

Hi Reggie, We have not published a roadmap but we are pushing to go to GA (General Availability) as soon as possible. The current plan is for most languages to go GA by the end of this month. There might still be some missing features, but that is the point where we will commit ourselves to a

Re: [protobuf] Pypi version of protoc version 2.4.1 seems to be missing files

Is it possible that something has changed in your environment? I could see this happening if you switched to a different/newer libstdc++. I don't think anyone has touched the packages on PyPI for 2.4.1. I don't think PyPI would let us change them even if we wanted to without changing the

Re: [protobuf] Question about Go protobufs and import_prefix

I don't know the background of the Go import system or go_package option. However this statement concerns me a little: > We have to use go_package to reorganize things that are fine in Java/Ruby, but would be circular package imports in Go. Is this implying that certain .proto files would

[protobuf] Re: Find Descriptor of all messages defined in a proto file

What language are you using? In C++, you can get a FileDescriptor which will contain all types defined in that file. On Friday, July 8, 2016 at 11:50:30 AM UTC-7, Sumit wrote: > > I need to find all messages described in a proto file.Is there some way to > get the descriptor of all messages

[protobuf] Re: What are some protobufs security measures?

This is a complicated question and hard to answer in a general way. When protobuf-Python is compiled to use the C++ protobuf library, it will use C++ for all parsing. So all of the same protections against parsing untrusted input would apply. When you are using the pure-Python protobuf

[protobuf] Re: Protocol buffer classes are basically dumb data holders (like structs in C++);

Can you be more specific about what you think needs fixing? I think the docs are saying that the generated classes don't have any business logic in them. This is true and seems clear to me. On Thursday, March 16, 2017 at 8:29:00 AM UTC-7, Dimitris Servis wrote: > > Hmmm? > > documentation needs

[protobuf] Re: How Do I Build Just the C++ Libraries?

Try: $ cd src && make libprotobuf.la libprotobuf-lite.la I can't guarantee it will compile without fork(), but it should avoid compiling the compiler! On Tuesday, March 14, 2017 at 2:01:43 PM UTC-7, Doug Lewis wrote: > > I'm going to be using protocol buffers on an embedded system using >

[protobuf] Re: Disable arena During Build Process

I don't think there is a way to disable arenas. But this error looks like it is related to thread-local storage. You could try compiling with GOOGLE_PROTOBUF_NO_THREADLOCAL and see if that helps: $ ./configure CPPFLAGS=-DGOOGLE_PROTOBUF_NO_THREADLOCAL On Wednesday, March 15, 2017 at 6:51:42 AM

[protobuf] Re: protoc plugin in Python and unknown extension options

Have you tried just using the FieldDescriptor? I suspect that will work, as it's what some of our own code does. For example see: https://github.com/google/protobuf/blob/master/python/google/protobuf/text_format.py#L819 On Monday, April 24, 2017 at 8:19:23 PM UTC-7, Robert Dyer wrote: > > I'm

[protobuf] Re: Extended field options

If the bitstring has a maximum length that is smaller than 32 or 64 bits, you could declare it as an integer type. Otherwise you might want to declare it as a "bytes" type. On Tuesday, April 18, 2017 at 1:34:01 AM UTC-7, oberon.me...@gmail.com wrote: > > Hello everyone > I am new to protobuf

[protobuf] Re: Linking errors when compiling protobuff 2.5.0 on linux

I don't think we support building with -fvisibility=hidden. That said, it might be a simple fix. You could try tweaking the code in base/port.h according to the suggestions here: https://gcc.gnu.org/wiki/Visibility If you got it to work I think we would welcome a PR on GitHub. On Friday,

Re: [protobuf] new protoc keywords?

I agree with Josh. It is very unlikely that we would introduce new keywords for defining a different kind of service. On Wednesday, April 26, 2017 at 12:45:29 PM UTC-7, Josh Humphries wrote: > > I think the right approach to something like this would be to use the > existing service definition

[protobuf] Re: Linking error

Those functions are marked "inline", so the compiler should not be emitting references to them. Do you have any unusual inlining settings on your compiler/project? On Friday, April 28, 2017 at 3:39:59 AM UTC-7, dbbrealtimesoftw...@gmail.com wrote: > > Hi, > I am starting to use the protobuf

[protobuf] Re: How to convert message to its extend message?

There is no way to do that. A message does not know what message it is contained inside. Pointers only go down to submessages, not up to parent messages. On Friday, April 28, 2017 at 3:49:41 AM UTC-7, Peter Wang wrote: > > Hello, > > 1. message Foo's definition, > message Foo { > extensions

Re: [protobuf] zigzag encode and zigzag decode invoking undefined behavior

If you have a change that makes protobuf code not rely on undefined behavior, and also doesn't sacrifice performance, then we should merge it. Feel free to send a GitHub PR. But make sure the code is C++03 (protobuf doesn't use C++11 yet). On Tuesday, April 25, 2017 at 1:36:31 PM UTC-7,

[protobuf] Re: Does protobuf support : ParseFromArray without typeName ??

In your C struct example, you know the type "T" that has "int v" as a member. If you know that, you can do the same thing with protobuf: // foo.proto message T { optional int32 v = 1; } // main.cc T a; a.set_v(5); a.SerializeToString(); T b; b.ParseFromString(str); On Thursday, January

[protobuf] Re: Arranging protobuf files

I think it is totally up to you, and how you want your builds to work. The only thing I would advise is: make the file path of each .proto file consistent. If you import a file once as "foo/bar/baz.proto", always refer to it as "foo/bar/baz.proto". Don't try to sometimes import it as

[protobuf] Re: Variable-Width Integer Encoding

I believe the encoding you mentioned here is the same as PrefixVarint. See some more info here when WebAssembly was evaluating which varint scheme to use: https://github.com/WebAssembly/design/issues/601 Our existing varint encoding wins on simplicity, and WebAssembly chose it even though

[protobuf] Re: Ptotobuf ->Python

The protobuf library only handles serialization and deserialization. It doesn't know anything about HTTP, compression, etc. You'll need to handle those parts using some other library (like gRPC). On Monday, December 25, 2017 at 9:49:55 AM UTC-8, JAlsi wrote: > > Hello, > > > I have a Web Api

[protobuf] Re: when use python protobuf3 to import Timestamp, get except " KeyError: 'google'"

protobuf3 is a separate package, unrelated to the Google release of protobuf. The regular Google release of protobuf python (--python_out=.) should work with Python 3. On Sunday, December 24, 2017 at 8:18:29 PM UTC-8, Delvin Pan wrote: > > Hi All, > > I have been using protobuf3 for a while

[protobuf] Re: Ruby: Unable to add defs to DescriptorPool: Symtab already has a def named

Can you explain a little bit more why Rails is causing some generated files to be required twice? Unless there is a really good reason for this, it seems better to debug why files are being loaded twice. Protobuf generated files aren't designed to be required multiple times. If we add some

[protobuf] Re: Protobuf c++ creates wrong file struct with protoc (lite and not the full methods files)

ParseFromIStream is not a generated method, so you wouldn't expect to see it in protoc output. ParseFromIStream is a member of the base class Message: https://github.com/google/protobuf/blob/master/src/google/protobuf/message.h#L287 Do your generated classes inherit from Message or MessageLite?

[protobuf] Re: I want to parse binary string into message c++

If you have a proper binary buffer, you should be able to pass it to option.ParseFromString(binaryString). But your HexString2BinaryString() function is not correct. You need to convert each pair of hex digits into a single binary byte. Right now you are converting the hex digits into a

[protobuf] Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

ly to the group. BCCed internal > mailing lists. > > On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman <haber...@google.com> > wrote: > >> Hi all, >> >> I wanted to kick off a conversation about the future of Protobuf and >> Abseil/ABSL in Debian/Ubuntu.

[protobuf] we are considering deprecating/removing the autotools build

Right now you can build protobuf using either autotools (autoconf, automake, etc) or CMake. We are considering deprecating the autotools build in favor of CMake. CMake supports Windows, Linux, and macOS, FreeBSD, OpenBSD, Solaris, and AIX. Other UNIX-like operating systems may work too. Is

Re: [protobuf] Protobuf v3.12.0-rc1 is released

ot good enough due to the > extra space) > > On Tue, May 5, 2020 at 6:50 PM Josh Haberman wrote: > >> On Tue, May 5, 2020 at 6:24 PM Nadav Samet wrote: >> >>> What representations do you mean? In general it is not expected that >>>> oneof names are exposed i

[protobuf] Protobuf v3.12.0-rc2 is released

I have released Protobuf v3.12.0-rc2 to GitHub and the per-language repositories (NuGet, npm, RubyGems, etc). https://github.com/protocolbuffers/protobuf/releases/tag/v3.12.0-rc2 This fixes a few issues that were reported in 3.12.0-rc1 (see the changelog above for more details). Unless there

[protobuf] Protobuf v3.12.0-rc1 is released

I have released Protobuf v3.12.0-rc1 to GitHub and the per-language repositories (NuGet, npm, RubyGems, etc). https://github.com/protocolbuffers/protobuf/releases/tag/v3.12.0-rc1 Please try it out and let us know if you have any issues. The full 3.12.0 release should follow within a week or

Re: [protobuf] Protobuf v3.12.0-rc1 is released

PI or documentation that is consumed by users. But I expect this is a small minority of all code using reflection. Can you give an example of some of the code that would be affected? > -Nadav > > > On Tue, May 5, 2020 at 2:13 PM 'Josh Haberman' via Protocol Buffers < > protobuf

Re: [protobuf] Protobuf v3.12.0-rc1 is released

On Tue, May 5, 2020 at 6:24 PM Nadav Samet wrote: > What representations do you mean? In general it is not expected that oneof >> names are exposed in wire formats. Authors of .proto files are free to >> change them at will without breaking the wire. >> > > An example would be code that produces

Re: [protobuf] Re: Discrepancy in generated code of unsafe_arena_release_foo

that we are focused on that use case. If we loosen the checks in unsafe_arena functions, it could force us to compromise later to keep supporting the heap-allocated case. I don't think we want to loosen these restrictions. > On Tuesday, May 5, 2020 at 2:25:05 PM UTC-

[protobuf] Re: Discrepancy in generated code of unsafe_arena_release_foo

The documentation for unsafe_arena_release_foo() says: > This method should only be used when the parent message is on the arena and the submessage is on the same arena, or an arena with equivalent lifetime. The DCHECK() you highlighted is checking the first part of this requirement.

[protobuf] Re: JsonFormat does not support extensions

Hi Eleanore, Python and Go support this I believe, but the other languages currently do not. There is now a conformance test for this behavior, though it's only "recommended": https://github.com/protocolbuffers/protobuf/blob/63da77bc/conformance/binary_json_conformance_suite.cc#L1868-L1877 So

[protobuf] Protobuf v3.12.0 is released

I have released Protobuf v3.12.0 to GitHub and the per-language repositories (NuGet, npm, RubyGems, etc). https://github.com/protocolbuffers/protobuf/releases/tag/v3.12.0 This is the same as -rc2, except there is a bugfix for serializing TextFormat in Java

Re: [protobuf] Re: Discrepancy in generated code of unsafe_arena_release_foo

protobuf supports arena allocation that's >> not portable (probably assumes certain std::string implementation), thus it >> is not open sourced, what's the plan for string arena allocation support? >> If I don't care about portability, is it possible to get the non-portable >&g

[protobuf] Protocol Buffers Version 25.0 is Released!

This version includes the following non-breaking changes: Compiler - Implement proto2/proto3 with editions (3813b66 ) - Defines Protobuf compiler version strings as macros and separates out

[protobuf] Protocol Buffers Version 24.2 is Released!

This version includes the following non-breaking changes: *C++* - Fixes for 32-bit MSVC. You can view the release on GitHub . If you experience any issues with the release, please file a bug