Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
On Mon, Dec 15, 2014 at 8:33 PM, Sumit Kumar kumar.su...@hotmail.com wrote: Does Arena allocator support custom memory pool override ? Can NUMA aware memory pools be used instead ? No. We can consider adding support for custom memory pools, but I think it might be very hard. The current arena code has its built-in memory allocation mechanism and is highly optimized for that. Supporting custom memory pools will probably require a rewrite of arena code. Any change to the set_allocated ? We added a new generated method unsafe_arena_set_allocated which behaves the same way as the old set_allocated. The new set_allocated implementation will check whether the passed-in object is on the same arena of the containing message and if it's not, a copy will be made. Regards, Sumit Kumar On 13 Dec 2014, at 2:20 am, 'Feng Xiao' via Protocol Buffers protobuf@googlegroups.com wrote: On Fri, Dec 12, 2014 at 5:56 AM, Jeremy Swigart jswig...@gmail.com wrote: Does the arena allocator also get used by messages allocated as children of the root message? Yes. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
Hi Feng, As far I know, protobuf wire format does not have much of an advantage over JSON format on web apps because the payload is usually small enough and encoding/decoding protobuf wire format with Javascript does not necessarily have a better performance than the built-in JSON encoder/decoder. Protobuf has a big advantage over JSON for some web apps. In particular, when you are transferring big amounts of data. In our use case (client-side WebGL vector maps https://www.mapbox.com/blog/mapbox-gl-js/, with lots of numeric data on the wire), Protobuf provides about 2-3 smaller gzipped sizes compared to JSON. In addition, it is much faster to decode — `JSON.parse` parses the data all at once due to the arbitrarily nested nature of JSON, while with Protobuf, you can parse data sequentially, chunk by chunk. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Protobuf Buffers v3.0.0-alpha-1
Does the arena allocator also get used by messages allocated as children of the root message? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
On Fri, Dec 12, 2014 at 5:56 AM, Jeremy Swigart jswig...@gmail.com wrote: Does the arena allocator also get used by messages allocated as children of the root message? Yes. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
On Wed, Dec 10, 2014 at 9:19 PM, chai2010 chaishus...@gmail.com wrote: Feng Xiao, I have some questions: 1. does protobuf3 will include golang compiler? Go protobuf is in its own repository and proto3 will supported there. See: https://github.com/golang/protobuf 2. does protobuf3 have a spec doc (link?) ? We are working on the documentation right now and when it's ready we will publish it on the protobuf developer guide: https://developers.google.com/protocol-buffers/docs/overview Thanks. 2014-12-11 12:51 GMT+08:00 Feng Xiao xiaof...@google.com: Hi all, I just published protobuf v3.0.0-alpha-1 on our github site: https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1 This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2). We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide https://developers.google.com/protocol-buffers/docs/overview. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group. CHANGS === Version 3.0.0-alpha-1 (C++/Java): General * Introduced Protocol Buffers language version 3 (aka proto3). When protobuf was initially opensourced it implemented Protocol Buffers language version 2 (aka proto2), which is why the version number started from v2.0.0. From v3.0.0, a new language version (proto3) is introduced while the old version (proto2) will continue to be supported. The main intent of introducing proto3 is to clean up protobuf before pushing the language as the foundation of Google's new API platform. In proto3, the language is simplified, both for ease of use and to make it available in a wider range of programming languages. At the same time a few features are added to better support common idioms found in APIs. The following are the main new features in language version 3: 1. Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go. 2. Removal of unknown fields. 3. Removal of extensions, which are instead replaced by a new standard type called Any. 4. Fix semantics for unknown enum values. 5. Addition of maps. 6. Addition of a small set of standard types for representation of time, dynamic data, etc. 7. A well-defined encoding in JSON as an alternative to binary proto encoding. This release (v3.0.0-alpha-1) includes partial proto3 support for C++ and Java. Items 6 (well-known types) and 7 (JSON format) in the above feature list are not implemented. A new notion syntax is introduced to specify whether a .proto file uses proto2 or proto3: // foo.proto syntax = proto3; message Bar {...} If omitted, the protocol compiler will generate a warning and proto2 will be used as the default. This warning will be turned into an error in a future release. We recommend that new Protocol Buffers users use proto3. However, we do not generally recommend that existing users migrate from proto2 from proto3 due to API incompatibility, and we will continue to support proto2 for a long time. * Added support for map fields (implemented in C++/Java for both proto2 and proto3). Map fields can be declared using the following syntax: message Foo { mapstring, string values = 1; } Data of a map field will be stored in memory as an unordered map and it can be accessed through generated accessors. C++ * Added arena allocation support (for both proto2 and proto3). Profiling shows memory allocation and deallocation constitutes a significant fraction of CPU-time spent in protobuf code and arena allocation is a technique introduced to reduce this cost. With arena allocation, new objects will be allocated from a large piece of preallocated memory and deallocation of these objects is almost free. Early adoption shows 20% to 50% improvement in some Google binaries. To
Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
Hallo Feng, Am 11.12.2014 um 05:51 schrieb Feng Xiao xiaof...@google.com: Hi all, I just published protobuf v3.0.0-alpha-1 on our github site: https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1 a question on structuring web applications further downstream: you mention node.js and JSON (de)serialisation reading between the lines this would suggest to me a typical protobuf application talking to a web client would talk JSON-serialized protobuf (maybe over a websocket stream) I've used this scheme and while JSON is easy for browser js engines, there are downsides; for instance, (de)serializing doubles from/to JSON usually creates conversion/rounding fuzz - that precludes signing a protobuf object in binary representation because the signature generally wont be valid after JSON conversion. Looser type checking is another drawback. That is why I found an end-to-end binary protobuf transfer and client-side js bindings along the lines of https://github.com/dcodeIO/ProtoBuf.js more robust what's the grand vision here - how are web apps going to talk to protobuf API's server-side? thanks in advance, Michael This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2). We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group. CHANGS === Version 3.0.0-alpha-1 (C++/Java): General * Introduced Protocol Buffers language version 3 (aka proto3). When protobuf was initially opensourced it implemented Protocol Buffers language version 2 (aka proto2), which is why the version number started from v2.0.0. From v3.0.0, a new language version (proto3) is introduced while the old version (proto2) will continue to be supported. The main intent of introducing proto3 is to clean up protobuf before pushing the language as the foundation of Google's new API platform. In proto3, the language is simplified, both for ease of use and to make it available in a wider range of programming languages. At the same time a few features are added to better support common idioms found in APIs. The following are the main new features in language version 3: 1. Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go. 2. Removal of unknown fields. 3. Removal of extensions, which are instead replaced by a new standard type called Any. 4. Fix semantics for unknown enum values. 5. Addition of maps. 6. Addition of a small set of standard types for representation of time, dynamic data, etc. 7. A well-defined encoding in JSON as an alternative to binary proto encoding. This release (v3.0.0-alpha-1) includes partial proto3 support for C++ and Java. Items 6 (well-known types) and 7 (JSON format) in the above feature list are not implemented. A new notion syntax is introduced to specify whether a .proto file uses proto2 or proto3: // foo.proto syntax = proto3; message Bar {...} If omitted, the protocol compiler will generate a warning and proto2 will be used as the default. This warning will be turned into an error in a future release. We recommend that new Protocol Buffers users use proto3. However, we do not generally recommend that existing users migrate from proto2 from proto3 due to API incompatibility, and we will continue to support proto2 for a long time. * Added support for map fields (implemented in C++/Java for both proto2 and proto3). Map fields can be declared using the following syntax: message Foo { mapstring, string values = 1; } Data of a map field will be stored in memory as an unordered map and it can be accessed through generated accessors. C++ * Added arena allocation support (for both proto2 and proto3). Profiling shows memory
Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
On Thu, Dec 11, 2014 at 12:02 PM, Michael Haberler mai...@mah.priv.at wrote: Hallo Feng, Am 11.12.2014 um 05:51 schrieb Feng Xiao xiaof...@google.com: Hi all, I just published protobuf v3.0.0-alpha-1 on our github site: https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1 a question on structuring web applications further downstream: you mention node.js and JSON (de)serialisation reading between the lines this would suggest to me a typical protobuf application talking to a web client would talk JSON-serialized protobuf (maybe over a websocket stream) I've used this scheme and while JSON is easy for browser js engines, there are downsides; for instance, (de)serializing doubles from/to JSON usually creates conversion/rounding fuzz - that precludes signing a protobuf object in binary representation because the signature generally wont be valid after JSON conversion. Looser type checking is another drawback. That is why I found an end-to-end binary protobuf transfer and client-side js bindings along the lines of https://github.com/dcodeIO/ProtoBuf.js more robust what's the grand vision here - how are web apps going to talk to protobuf API's server-side? +liujisi, who is more qualified to answer this than me. I think your reading is correct. We'll publish an protobuf implementation for node.js but not for the Javascript language in general (like the ProtoBuf.js you linked), while JSON format support will be added to all protobuf implementations. Web apps would talk JSON to its server although the server can support both JSON format and protobuf wire format. As far I know, protobuf wire format does not have much of an advantage over JSON format on web apps because the payload is usually small enough and encoding/decoding protobuf wire format with Javascript does not necessarily have a better performance than the built-in JSON encoder/decoder. As most web apps are already using JSON as the data exchange format, supporting JSON format on the sever side so it can talk with JSON clients seems a natural choice here. thanks in advance, Michael This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2). We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group. CHANGS === Version 3.0.0-alpha-1 (C++/Java): General * Introduced Protocol Buffers language version 3 (aka proto3). When protobuf was initially opensourced it implemented Protocol Buffers language version 2 (aka proto2), which is why the version number started from v2.0.0. From v3.0.0, a new language version (proto3) is introduced while the old version (proto2) will continue to be supported. The main intent of introducing proto3 is to clean up protobuf before pushing the language as the foundation of Google's new API platform. In proto3, the language is simplified, both for ease of use and to make it available in a wider range of programming languages. At the same time a few features are added to better support common idioms found in APIs. The following are the main new features in language version 3: 1. Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go. 2. Removal of unknown fields. 3. Removal of extensions, which are instead replaced by a new standard type called Any. 4. Fix semantics for unknown enum values. 5. Addition of maps. 6. Addition of a small set of standard types for representation of time, dynamic data, etc. 7. A well-defined encoding in JSON as an alternative to binary proto encoding. This release (v3.0.0-alpha-1) includes partial proto3 support for C++ and Java. Items 6 (well-known types) and 7 (JSON format) in the above feature list are not implemented. A new notion syntax is introduced to specify whether a .proto file
[protobuf] Protobuf Buffers v3.0.0-alpha-1
Hi all, I just published protobuf v3.0.0-alpha-1 on our github site: https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1 This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2). We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide https://developers.google.com/protocol-buffers/docs/overview. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group. CHANGS === Version 3.0.0-alpha-1 (C++/Java): General * Introduced Protocol Buffers language version 3 (aka proto3). When protobuf was initially opensourced it implemented Protocol Buffers language version 2 (aka proto2), which is why the version number started from v2.0.0. From v3.0.0, a new language version (proto3) is introduced while the old version (proto2) will continue to be supported. The main intent of introducing proto3 is to clean up protobuf before pushing the language as the foundation of Google's new API platform. In proto3, the language is simplified, both for ease of use and to make it available in a wider range of programming languages. At the same time a few features are added to better support common idioms found in APIs. The following are the main new features in language version 3: 1. Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go. 2. Removal of unknown fields. 3. Removal of extensions, which are instead replaced by a new standard type called Any. 4. Fix semantics for unknown enum values. 5. Addition of maps. 6. Addition of a small set of standard types for representation of time, dynamic data, etc. 7. A well-defined encoding in JSON as an alternative to binary proto encoding. This release (v3.0.0-alpha-1) includes partial proto3 support for C++ and Java. Items 6 (well-known types) and 7 (JSON format) in the above feature list are not implemented. A new notion syntax is introduced to specify whether a .proto file uses proto2 or proto3: // foo.proto syntax = proto3; message Bar {...} If omitted, the protocol compiler will generate a warning and proto2 will be used as the default. This warning will be turned into an error in a future release. We recommend that new Protocol Buffers users use proto3. However, we do not generally recommend that existing users migrate from proto2 from proto3 due to API incompatibility, and we will continue to support proto2 for a long time. * Added support for map fields (implemented in C++/Java for both proto2 and proto3). Map fields can be declared using the following syntax: message Foo { mapstring, string values = 1; } Data of a map field will be stored in memory as an unordered map and it can be accessed through generated accessors. C++ * Added arena allocation support (for both proto2 and proto3). Profiling shows memory allocation and deallocation constitutes a significant fraction of CPU-time spent in protobuf code and arena allocation is a technique introduced to reduce this cost. With arena allocation, new objects will be allocated from a large piece of preallocated memory and deallocation of these objects is almost free. Early adoption shows 20% to 50% improvement in some Google binaries. To enable arena support, add the following option to your .proto file: option cc_enable_arenas = true; Protocol compiler will generate additional code to make the generated message classes work with arenas. This does not change the existing API of protobuf messages and does not affect wire format. Your existing code should continue to work after adding this option. In the future we will make this option enabled by default. To actually take advantage of arena allocation, you need to use the arena APIs when creating messages. A quick example of using the arena API: {
Re: [protobuf] Protobuf Buffers v3.0.0-alpha-1
Feng Xiao, I have some questions: 1. does protobuf3 will include golang compiler? 2. does protobuf3 have a spec doc (link?) ? Thanks. 2014-12-11 12:51 GMT+08:00 Feng Xiao xiaof...@google.com: Hi all, I just published protobuf v3.0.0-alpha-1 on our github site: https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1 This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2). We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide https://developers.google.com/protocol-buffers/docs/overview. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group. CHANGS === Version 3.0.0-alpha-1 (C++/Java): General * Introduced Protocol Buffers language version 3 (aka proto3). When protobuf was initially opensourced it implemented Protocol Buffers language version 2 (aka proto2), which is why the version number started from v2.0.0. From v3.0.0, a new language version (proto3) is introduced while the old version (proto2) will continue to be supported. The main intent of introducing proto3 is to clean up protobuf before pushing the language as the foundation of Google's new API platform. In proto3, the language is simplified, both for ease of use and to make it available in a wider range of programming languages. At the same time a few features are added to better support common idioms found in APIs. The following are the main new features in language version 3: 1. Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go. 2. Removal of unknown fields. 3. Removal of extensions, which are instead replaced by a new standard type called Any. 4. Fix semantics for unknown enum values. 5. Addition of maps. 6. Addition of a small set of standard types for representation of time, dynamic data, etc. 7. A well-defined encoding in JSON as an alternative to binary proto encoding. This release (v3.0.0-alpha-1) includes partial proto3 support for C++ and Java. Items 6 (well-known types) and 7 (JSON format) in the above feature list are not implemented. A new notion syntax is introduced to specify whether a .proto file uses proto2 or proto3: // foo.proto syntax = proto3; message Bar {...} If omitted, the protocol compiler will generate a warning and proto2 will be used as the default. This warning will be turned into an error in a future release. We recommend that new Protocol Buffers users use proto3. However, we do not generally recommend that existing users migrate from proto2 from proto3 due to API incompatibility, and we will continue to support proto2 for a long time. * Added support for map fields (implemented in C++/Java for both proto2 and proto3). Map fields can be declared using the following syntax: message Foo { mapstring, string values = 1; } Data of a map field will be stored in memory as an unordered map and it can be accessed through generated accessors. C++ * Added arena allocation support (for both proto2 and proto3). Profiling shows memory allocation and deallocation constitutes a significant fraction of CPU-time spent in protobuf code and arena allocation is a technique introduced to reduce this cost. With arena allocation, new objects will be allocated from a large piece of preallocated memory and deallocation of these objects is almost free. Early adoption shows 20% to 50% improvement in some Google binaries. To enable arena support, add the following option to your .proto file: option cc_enable_arenas = true; Protocol compiler will generate additional code to make the generated message classes work with arenas. This does not change the existing API of protobuf messages and does not affect wire format. Your existing code should continue to work after