[protobuf] Generated source build issues (C++)

[Paul Nienaber (CW)]

Hi,

Joined on another project that's heavily using Protobuf (Apache Arrow) and 
running into strange macro-and-other-symbol visibility issues when I'm 
including generated Protobuf headers in more source files (e.g. in 
application headers).  I'm guessing this may be a more common general 
problem but I didn't find anything similar enough to the issues I'm 
encountering when searching for similar errors.  For example I'm seeing 
(truncated as things get less interesting and somewhat repetitive down the 
log):

/opt/homebrew/bin/ccache /Library/Developer/CommandLineTools/usr/bin/c++ 
-DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING -DARROW_HAVE_NEON 
-DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS -DARROW_WITH_ZLIB 
-DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental 
-DGRPC_USE_CERTIFICATE_VERIFIER -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS 
-DURI_STATIC_BUILD 
-I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src 
-I/Users/pauln/bq/git/arrow/cpp/src 
-I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem 
/opt/homebrew/include -isystem 
/Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem 
/Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem 
/opt/homebrew/opt/openssl@1.1/include -isystem 
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
 
-fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra 
-Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces 
-Wno-unused-parameter -Wno-constant-logical-operand 
-Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed 
-march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot 
/Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC -std=c++17 
-MD -MT 
src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o -MF 
src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d 
-o src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o 
-c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
In file included from 
/Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22: 
In file included from 
/Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24: 
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
 
error: expected expression 
static const uint32_t offsets[]; 
^ 
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
 
error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT' 
struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto { 
^ 
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
 
note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT' 
struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto { 
^ 
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
 
error: no member named 'DescriptorTable' in namespace 
'google::protobuf::internal' 
ARROW_FLIGHT_SQL_EXPORT extern const 
::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable 
descriptor_table_FlightSql_2eproto; 
~~~^ 
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
 
error: expected ';' after top level declarator 
ARROW_FLIGHT_SQL_EXPORT extern const 
::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable 
descriptor_table_FlightSql_2eproto; 
[...]

Thanks in advance for any insight!

Cheers
~Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/37530cbb-e951-44c8-82a4-40716feb97een%40googlegroups.com.

Re: [protobuf] protoc generated c++ files

[Paul Lemay]

Finally, 

The typedef template above was generated properly and I was mislead by the 
template error message that was raised at compiled time. When invoking the 
Request session, the stream parameter passed was wrong. 

mCallData->GetService()->RequestSession(&mCtx, &mStream, 
  mCallData->GetCompletionQueue(), 
  mCallData->GetCompletionQueue(), 
&mProceedFunction);

I mismatched request and response in the ServerAsyncReaderWriter declaration:

  grpc::ServerAsyncReaderWriter mStream

So to the question I was asking, yes, it is possible to have multiple rpc 
streams in a single grpc service.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/1890e403-f199-4784-bfeb-49d4fb6f1a18%40googlegroups.com.

Re: [protobuf] protoc generated c++ files

[Paul Lemay]

Hello Derek,

thanks for your help. 

Yes, the procedure to compile is right. If I keep only 1 async rpc stream 
(i.e., say rpc Session), and I recompile, I get the typedef that I am 
expecting:
typedef WithAsyncMethod_Session AsyncService;

Even if there is multiple async rpc stream there is only 1 typedef produced 
where I would expect many. This lead me to believe that the service must be 
arrange to handle a single async rpc stream or that our proto files are are 
defined properly. Here is the main that includes the other:
syntax = "proto3";

package upfservice.v1;

import "session.proto";
import "hello.proto";

// The Upfu_BfRuntime service definition.
service UPFService {
// Initial handshake initiated by client. 
rpc Hello(stream HelloRequest) returns (stream HelloResponse) {}

// Client shutdown (resource can be cleaned up)
rpc Bye(ByeRequest) returns (ByeResponse) {}

// Ping request
rpc Ping(PingRequest) returns (PingResponse) {}

// Session management bi-directional streaming api.
rpc Session(stream ServiceDataSessionRequest) returns (stream 
ServiceDataSessionResponse) {}

// UR collection server streaming (client call this one then wait for 
continuous UR stream)
rpc UsageReport(UsageReportRequest) returns (stream UsageReports) {}
}



What typedef sould I expect out of there?


On Wednesday, 6 May 2020 21:02:31 UTC-4, Derek Perez wrote:
>
> Here's a link to the docs for the protoc grpc plugin: 
> https://grpc.io/docs/tutorials/basic/cpp/#generating-client-and-server-code
>   
>
> On Wed, May 6, 2020 at 6:00 PM Derek Perez  wrote:
>
>> Are you generating the C++ code with the proper grpc plugin as well? 
>> Services won't generate useful stubs without that.
>>
>> On Wed, May 6, 2020 at 4:52 PM Paul Lemay > > wrote:
>>
>>> Hello,
>>>
>>> I have this proto files where the root file is defined as below:
>>>
>>> syntax = "proto3";
>>>
>>> package upfservice.v1;
>>>
>>> import "session.proto";
>>> import "hello.proto";
>>>
>>> // The Upfu_BfRuntime service definition.
>>> service UPFService {
>>> // Initial handshake initiated by client. 
>>> rpc Hello(stream HelloRequest) returns (stream HelloResponse) {}
>>>
>>> // Client shutdown (resource can be cleaned up)
>>> rpc Bye(ByeRequest) returns (ByeResponse) {}
>>>
>>> // Ping request
>>> rpc Ping(PingRequest) returns (PingResponse) {}
>>>
>>> // Session management bi-directional streaming api.
>>> rpc Session(stream ServiceDataSessionRequest) returns (stream 
>>> ServiceDataSessionResponse) {}
>>>
>>> // UR collection server streaming (client call this one then wait for 
>>> continuous UR stream)
>>> rpc UsageReport(UsageReportRequest) returns (stream UsageReports) {}
>>> }
>>>
>>> The generated file gives me a typedef to work with:
>>>
>>>
>>> typedef 
>>> WithAsyncMethod_Hello>>  
>>> > > > > > AsyncService;
>>>
>>> It does not seems right since I can't use the AsyncService to do 
>>> anything. Is it because there is a limit of 1 asynchronous stream rpc per 
>>> service?
>>>
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prot...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/protobuf/a3941835-afcf-4a60-92fb-844a59837708%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/protobuf/a3941835-afcf-4a60-92fb-844a59837708%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/ae539931-569e-43fb-a32c-ea97e6aeb6cd%40googlegroups.com.

[protobuf] protoc generated c++ files

[Paul Lemay]

Hello,

I have this proto files where the root file is defined as below:

syntax = "proto3";

package upfservice.v1;

import "session.proto";
import "hello.proto";

// The Upfu_BfRuntime service definition.
service UPFService {
// Initial handshake initiated by client. 
rpc Hello(stream HelloRequest) returns (stream HelloResponse) {}

// Client shutdown (resource can be cleaned up)
rpc Bye(ByeRequest) returns (ByeResponse) {}

// Ping request
rpc Ping(PingRequest) returns (PingResponse) {}

// Session management bi-directional streaming api.
rpc Session(stream ServiceDataSessionRequest) returns (stream 
ServiceDataSessionResponse) {}

// UR collection server streaming (client call this one then wait for 
continuous UR stream)
rpc UsageReport(UsageReportRequest) returns (stream UsageReports) {}
}

The generated file gives me a typedef to work with:


typedef 
WithAsyncMethod_Hello > > > > AsyncService;

It does not seems right since I can't use the AsyncService to do anything. 
Is it because there is a limit of 1 asynchronous stream rpc per service?



-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/a3941835-afcf-4a60-92fb-844a59837708%40googlegroups.com.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

['Paul Marks' via Protocol Buffers]

On Tuesday, July 31, 2018 at 6:10:27 AM UTC-7, sanjana gupta wrote:
>
> Hello Marc,
>
> I wanted to let you know that I tried using *fixed32 and fixed64* 
> protobuf types and it has helped me save quite some bytes on the encoded 
> data size. Allow me to show the protobuf message I created which is capable 
> for storing one or multiple v4/v6 IPs :
>
>
All you should need is:
  bytes ip = 1;  // length is 4 for IPv4, 16 for IPv6.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

['Paul Marks' via Protocol Buffers]

Use a bytes field: 4 for IPv4, 16 for IPv6.

Ideally, you should use an IP address library with a "packed bytes as 
std::string" input/output.
The in_addr/in6_addr types are stored as 4 or 16 bytes in RAM.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Making Java JsonFormat and TextFormat extensible

[Paul Gross]

I was wondering if there has been any talk or plan to make TextFormat and 
JsonFormat extensible. For example, we are writing code to log input and 
output protos for gRPC services. In our case, we need to filter certain 
sensitive fields (which we currently mark via custom options in the proto 
file). Ideally, we'd be able to subclass the Printer and use most of the 
logic in JsonFormat or TextFormat. However, the important methods are 
currently private, such as:

https://github.com/google/protobuf/blob/baed06e69443613c531a3580315759c08db6667e/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java#L956-L958
https://github.com/google/protobuf/blob/baed06e69443613c531a3580315759c08db6667e/java/core/src/main/java/com/google/protobuf/TextFormat.java#L385-L388

Instead, we need to re-implement most of the logic in our own formatter 
class. I was also a little surprised to find that the two Printers don't 
share an interface, allowing others to more easily add custom printers.

Is this something that has come up before? Or maybe we're off the beaten 
path?

Thanks,
Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Confused about directory structure from generated Go code

[Paul Johnston]

Consider using the import map 

 
feature of the protoc-gen-go plugin.  Looks something like this:

MAPPING_1="foo/bar.proto=github.com/foo/bar"
MAPPING_1="foo/baz.proto=github.com/foo/baz"
MAPPINGS="MAPPING_1,MAPPING_2"
OPTS="M$MAPPINGS"
--go_out=$OPTS:$OUTDIR


On Thursday, September 1, 2016 at 2:16:25 PM UTC-6, Reggie Burnett wrote:

> We have about 7 proto files that we use for our projects.  Each of the 
> proto files has a different package name and they reference each other 
> using these package names.
>
> I didn't really want to have each generated .go file be in it's own 
> directory so I tried to change the protofiles to generate all the code 
> under the same Golang package.  That didn't seem to work as it was still 
> generating "import" statements to pull in packages that now no longer exist.
>
> So now I have gone back to the original protofiles and generated 
> everything but now we are getting import statements like this:
>
> import  package_name "."
>
> The "." seems to be telling it that the package is in the same directory 
> but since Golang packages all have to be in their own directories that is 
> not working.
>
> Can someone give me some guidance on properly structuring a situation like 
> this?
>
> Thanks
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: what to put in WORKSPACE and/or BUILD files to enable the cc_proto_library rule?

[Paul Johnston]

1. During the loading phase, bring the google/protobuf into your WORKSPACE 
so you can refer to it as @com_github_google_protobuf later.  The bazel 
convention is to have fully qualified names, flattening special characters 
to underscore.

git_repository(
  name = "com_github_google_protobuf",
  remote = "https://github.com/google/protobuf.git";,
  tag = "v3.0.0",
)

2. During the analysis phase, load the function in your BUILD file by 
referring to the skylark bzl filename.  
As the protobuf repo has a BUILD file in the root (which defines it as 
a "package"), you need to refer to the 
bzl relative to the package name.  That's why you need the colon. 

# You can also alias the function to whatever you want with 'foo = 
"cc_proto_library"'
load("@com_github_google_protobuf//:protobuf.bzl", "cc_proto_library")

# These are called "target pattern operators" IIRC
@ = names an external dependency that is loaded in WORKSPACE.
// = relative to the workspace root
: relative to package

3. Call it


cc_proto_library( 
 name = "alice", 
 srcs = ["alice.proto"], 
)

But then, where did the files go?  If it worked, they should be in $(bazel 
info bazel-genfiles)/protoa/alice.pb.h

I've been working on a set of rules for protobuf +/- gRPC 
at https://github.com/pubref/rules_protobuf.  Perhaps it will be of use to 
you.


On Monday, July 18, 2016 at 11:25:45 AM UTC-6, JI Ioannidis wrote:
>
> What is the sequence of magic incantations to create a cc_proto_library 
> rule? I have tried copying protobuf.bzl from the protobuf repo to the root 
> of my workspace, as well as some (at least a year old) .bzl files 
> referenced in this group, to no avail.
>
> My end goal is to have a BUILD file such as 
> $ *cat protoa/BUILD*  
> cc_proto_library( 
>  name = "alice", 
>  srcs = ["alice.proto"], 
> )
>
> then type 
>
> $ *bazel build //protoa:alice*
>
> and have it create the right .pb.h and .pb.cc files
>
> FWIW, I have already installed bazel at sha 42d8290 (compiled, bazel 
> binary copied to ~/bin/) and built and installed protobuf at sha b99577c. 
>
> Thanks!
>
> /ji
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: service without gRPC

[Paul Johnston]

You need to invoke the grpc plugin.  The default protoc tool does not 
generate gRPC stubs on it's own.  Here's what it looks like:

$ protoc --cpp_out=outdir \

  --plugin=protoc-gen-grpc=path/to/the/grpc-cpp-plugin \

  --grpc_out=outdir \

  path/to/your/helloworld.proto

# Generated by protoc

helloworld.pb.h

helloworld.pb.cc

# Generated by grpc plugin

helloworld.grpc.pb.h

helloworld.grpc.pb.cc

I've written a tool for bazel that can do this for you and provide all the 
required build environment, if you're interested in trying it out.

https://github.com/pubref/rules_protobuf

HTH,
Paul

On Friday, August 5, 2016 at 9:03:47 AM UTC-6, Mohamed Koubaa wrote:
>
> Hello,
>
> As per the language guide, defining a service in a .proto file will 
> generate RPC stubs which we can implement (or let gRPC implement). 
> https://developers.google.com/protocol-buffers/docs/proto3#services
>
> We have a zeroMQ service that we are interested in migrating from JSON-RPC 
> to gRPC.  To stage this effort, we are interesting in creating the RPC 
> stubs and using the same zeroMQ to implement them, and then move to gRPC 
> once we deal with an orthogonal service discovery problem that prevents us 
> from dropping zeroMQ immediately.
>
> To my surprise, adding the RPC code to the .proto did not generate any 
> stubs in c++. I am using proto3 beta4 at the moment.  Do I need any 
> additional flags to protoc?
>
> Thanks
> Mohamed Koubaa
> Software Developer
> ANSYS Inc
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Compiling Protobuf on VxWorks C++

[Paul Johnston]

Sorry, that paste was incomplete. Try this one.


*@com_github_google_protobuf//:protobuf*
*@com_github_google_protobuf//:src/google/protobuf/wrappers.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/wire_format.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/type_resolver_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/time_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/message_differencer.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/json_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/utility.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/type_info_test_helper.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/type_info.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/protostream_objectwriter.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/protostream_objectsource.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/proto_writer.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/object_writer.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/json_stream_parser.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/json_objectwriter.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/json_escaping.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/field_mask_utility.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/error_listener.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/default_value_objectwriter.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/datapiece.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/field_mask_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/field_comparator.cc*
*@com_github_google_protobuf//:src/google/protobuf/unknown_field_set.cc*
*@com_github_google_protobuf//:src/google/protobuf/type.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/timestamp.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/text_format.cc*
*@com_github_google_protobuf//:src/google/protobuf/stubs/substitute.cc*
*@com_github_google_protobuf//:src/google/protobuf/stubs/mathlimits.cc*
*@com_github_google_protobuf//:src/google/protobuf/struct.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/source_context.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/service.cc*
*@com_github_google_protobuf//:src/google/protobuf/reflection_ops.cc*
*@com_github_google_protobuf//:src/google/protobuf/message.cc*
*@com_github_google_protobuf//:src/google/protobuf/map_field.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/zero_copy_stream_impl.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/tokenizer.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/strtod.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/printer.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/gzip_stream.cc*
*@com_github_google_protobuf//:src/google/protobuf/generated_message_reflection.cc*
*@com_github_google_protobuf//:src/google/protobuf/field_mask.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/extension_set_heavy.cc*
*@com_github_google_protobuf//:src/google/protobuf/empty.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/dynamic_message.cc*
*@com_github_google_protobuf//:src/google/protobuf/duration.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/descriptor_database.cc*
*@com_github_google_protobuf//:src/google/protobuf/descriptor.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/descriptor.cc*
*@com_github_google_protobuf//:src/google/protobuf/compiler/parser.cc*
*@com_github_google_protobuf//:src/google/protobuf/compiler/importer.cc*
*@com_github_google_protobuf//:src/google/protobuf/api.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/any.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/any.cc*
*@com_github_google_protobuf//:protobuf_lite*
*@com_github_google_protobuf//:src/google/protobuf/wrappers.pb.h*
*@com_github_google_protobuf//:src/google/protobuf/wire_format_lite_inl.h*
*@com_github_google_protobuf//:src/google/protobuf/wire_format_lite.h*
*@com_github_google_protobuf//:src/google/protobuf/wire_format_lite.cc*
*@com_github_google_protobuf//:src/google/protobuf/wire_format.h*
*@com_github_google_protobuf//:src/google/protobuf/util/type_resolver_util.h*
*@com_github_google_protobuf//:src/google/protobuf/util/type_resolver.h*
*@com_github_google_protobuf//:src/google/protobuf/util/time_util.h*
*@com_github_google_protobuf//:src/google/protobuf/util/message_differencer.h*
*@com_github_google_protobuf//:src/google/protobuf/util/json_util.h*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/utility.h*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/type_info_test_helper.h*
*@com_github_google_protobuf//:src/google/protobuf/util/i

[protobuf] Re: Compiling Protobuf on VxWorks C++

[Paul Johnston]

https://github.com/google/protobuf/blob/master/BUILD

I have protobuf available in my workspace as @com_github_google_protobuf

This reports the (file) dependencies required to build the :protobuf rule:

$ bazel query "deps(@com_github_google_protobuf//:protobuf)"

HTH,
Paul


*@com_github_google_protobuf//:protobuf*
*@com_github_google_protobuf//:src/google/protobuf/wrappers.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/wire_format.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/type_resolver_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/time_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/message_differencer.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/json_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/utility.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/type_info_test_helper.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/type_info.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/protostream_objectwriter.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/protostream_objectsource.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/proto_writer.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/object_writer.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/json_stream_parser.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/json_objectwriter.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/json_escaping.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/field_mask_utility.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/error_listener.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/default_value_objectwriter.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/internal/datapiece.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/field_mask_util.cc*
*@com_github_google_protobuf//:src/google/protobuf/util/field_comparator.cc*
*@com_github_google_protobuf//:src/google/protobuf/unknown_field_set.cc*
*@com_github_google_protobuf//:src/google/protobuf/type.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/timestamp.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/text_format.cc*
*@com_github_google_protobuf//:src/google/protobuf/stubs/substitute.cc*
*@com_github_google_protobuf//:src/google/protobuf/stubs/mathlimits.cc*
*@com_github_google_protobuf//:src/google/protobuf/struct.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/source_context.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/service.cc*
*@com_github_google_protobuf//:src/google/protobuf/reflection_ops.cc*
*@com_github_google_protobuf//:src/google/protobuf/message.cc*
*@com_github_google_protobuf//:src/google/protobuf/map_field.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/zero_copy_stream_impl.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/tokenizer.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/strtod.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/printer.cc*
*@com_github_google_protobuf//:src/google/protobuf/io/gzip_stream.cc*
*@com_github_google_protobuf//:src/google/protobuf/generated_message_reflection.cc*
*@com_github_google_protobuf//:src/google/protobuf/field_mask.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/extension_set_heavy.cc*
*@com_github_google_protobuf//:src/google/protobuf/empty.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/dynamic_message.cc*
*@com_github_google_protobuf//:src/google/protobuf/duration.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/descriptor_database.cc*
*@com_github_google_protobuf//:src/google/protobuf/descriptor.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/descriptor.cc*
*@com_github_google_protobuf//:src/google/protobuf/compiler/parser.cc*
*@com_github_google_protobuf//:src/google/protobuf/compiler/importer.cc*
*@com_github_google_protobuf//:src/google/protobuf/api.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/any.pb.cc*
*@com_github_google_protobuf//:src/google/protobuf/any.cc*
*@com_github_google_protobuf//:protobuf_lite*
*@com_github_google_protobuf//:src/google/protobuf/wrappers.pb.h*
*@com_github_google_protobuf//:src/google/protobuf/wire_format_lite_inl.h*
*@com_github_google_protobuf//:src/google/protobuf/wire_format_lite.h*
*@com_github_google_protobuf//:src/google/protobuf/wire_format_lite.cc*
*@com_github_google_protobuf//:src/google/protobuf/wire_format.h*
*@com_github_google_protobuf//:src/google/protobuf/util/type_resolver_util.h*
*@com_github_google_protobuf//:src/google/protobuf/util/type_resolver.h*
*@com_github_google_protobuf//:src/google/protobuf/util/time_util.h*
*@com_github_google_protobuf//:src/google/protobuf/util/message_differencer.h*
*@com_github_google_protobuf//:src/google/pro

Re: [protobuf] Trying to use C++ implementation of python but api_implementation.Type() still returns 'python' instead of 'cpp'

[Paul Johnston]

"--cpp_implementaion" misspelled, should be "--cpp_implementation"

On Monday, August 1, 2016 at 1:28:29 PM UTC-6, Oi Lee wrote:
>
> When I do 
> [localhost python]$ sudo python setup.py build --cpp_implementaion
>
> Installed [...]/protobuf-2.6.1/python/.eggs/google_apputils-0.4.2-py2.7.egg
> Searching for python-gflags>=1.4
> Reading https://pypi.python.org/simple/python-gflags/
> Best match: python-gflags 3.0.5
> Downloading 
> https://pypi.python.org/packages/1f/01/3ca6527f51b7ff26abb635d4e7e2fa8413c7cf191564cc2c1e535f50dec7/python-gflags-3.0.5.tar.gz#md5=0e3a06cb99b9883bf6abacc489f31297
> Processing python-gflags-3.0.5.tar.gz
> Writing /tmp/easy_install-c_jdOd/python-gflags-3.0.5/setup.cfg
> Running python-gflags-3.0.5/setup.py -q bdist_egg --dist-dir 
> /tmp/easy_install-c_jdOd/python-gflags-3.0.5/egg-dist-tmp-M2uDeX
> zip_safe flag not set; analyzing archive contents...
> Moving python_gflags-3.0.5-py2.7.egg to 
> /home/oi.lee/Downloads/protobuf-2.6.1/python/.eggs
>
> Installed [...]/protobuf-2.6.1/python/.eggs/python_gflags-3.0.5-py2.7.egg
> Searching for python-dateutil>=1.4
> Reading https://pypi.python.org/simple/python-dateutil/
> Best match: python-dateutil 2.5.3
> Downloading 
> https://pypi.python.org/packages/b7/9f/ba2b6aaf27e74df59f31b77d1927d5b037cc79a89cda604071f93d289eaf/python-dateutil-2.5.3.zip#md5=52b3f339f41986c25c3a2247e722db17
> Processing python-dateutil-2.5.3.zip
> Writing /tmp/easy_install-F99Iys/python-dateutil-2.5.3/setup.cfg
> Running python-dateutil-2.5.3/setup.py -q bdist_egg --dist-dir 
> /tmp/easy_install-F99Iys/python-dateutil-2.5.3/egg-dist-tmp-YXfHJw
> warning: no previously-included files matching '__pycache__' found 
> anywhere in distribution
> warning: no previously-included files matching '*.py[co]' found anywhere 
> in distribution
> Moving python_dateutil-2.5.3-py2.7.egg to 
> /home/oi.lee/Downloads/protobuf-2.6.1/python/.eggs
>
> Installed [...]/protobuf-2.6.1/python/.eggs/python_dateutil-2.5.3-py2.7.egg
> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>or: setup.py --help [cmd1 cmd2 ...]
>or: setup.py --help-commands
>or: setup.py cmd --help
>
> error: option --cpp_implementaion not recognized
>
>
>
> On Friday, July 29, 2016 at 3:25:10 PM UTC-6, Jie Luo wrote:
>>
>>  I tried and it returns 'cpp' as expected.
>>
>> First, make sure you use the correct commands:
>> python setup.py build --cpp_implementaion
>> python setup.py install --cpp_implementaion
>>
>>
>> Second, Where do you put your .py file to check 
>> api_implementation.Type()? Make sure your file does not under 
>> protobuf/python (DO NOT in the same directory as setup.py)
>>
>> On Fri, Jul 29, 2016 at 12:30 PM, Feng Xiao  wrote:
>>
>>> +Jie
>>>
>>> On Fri, Jul 29, 2016 at 10:10 AM, Oi Lee  wrote:
>>>
 I initially used a pure python implementation of protocol buffers, but 
 I changed it later to use the c++ library implementation. So I installed 
 protobuf using these github instructions 
 
  and 
 then I followed these instructions 
  to 
 use the C++ implementation. But when I use  api_implementation.Type() 
 ,
  it 
 still returns 'python'.  Why is it still using the python implementation?  
 If this function should not be used, then what should I use to check 
 implementation? 
 I'm using protocol buffers 2.6.1 on CentOS. 

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at https://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] bazel rules_protobuf (java)

[Paul Johnston]

I've been struggling to find a clean way to generate protobufs for java 
(with gRPC support) with bazel, so I just released rules_protobuf at 
https://github.com/pubref/rules_protobuf/tree/0.1.1.  

If you are new to http://bazel.io, this might be a good way to try it out. 
 As it will automatically install the protoc and protoc-gen-grpc-java 
binaries into your workspace, it requires no configuration and may be one 
of the easier ways to get started with protobufs and gRPC. Uses protoc 
3.0.0.

Looking forward to native protobuf support in bazel, but until then 
hopefully it fills a need for others with this requirement.  PRs definitely 
welcome for improvements or support for additional language output.


Cheers,
Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Arena allocation with proto v2.x? - release vs syntax

[Paul Baxter]

I had thought arena allocation was available in latest v2.x releases, but 
the latest release 2.6.1 from October 2014 is just prior to adding arena 
allocation.

Is there to be a v2.x release with arena support supporting the additional 
C++ code generation?

Or is it intended to only be available in v3.x releases. Is the idea that 
it is available in v3.x releases when you need to set the proto **syntax** 
v2  i.e. an extended proto2 feature not supported by release stream for v2.x

Could you clarify the intent.

If so, perhaps it should be clarified that for certain features 'proto2 
documentation' refers to proto2 syntax supported in v3 releases.

https://developers.google.com/protocol-buffers/docs/proto

Language guide (proto 2) - maybe reword to add (proto 2 syntax) and make 
clear in the text where a feature is implemented only in release stream 3.x


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] big endian system support dropped (powerpc and mips)?

[Flo Paul]

Hi,
wondering if support for PPC and MIPSEB has been dropped. It's not 
implemented in the platform headers any more (2.5.0) and doesn't cross 
compile on those systems. Has support been dropped for big endian systems? 
It was working in previous versions (2.4.1) and there has been an issue 
488raised but I can't 
find any comments if it's going to be fixed again or why 
it's been dropped. I would also be interested what is missing to make 
theses system work again?

Cheers Flo

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[protobuf] Re: Serialize with Length Prefix

[Paul Shafer]

First time poster and new to protobuf/protobuf-net so forgive me if this 
has already been addressed.  But I've searched quite a bit and cant find 
an"official" answer.

If you want to use the fieldNumber argument in 
your SerializeWithLengthPrefix/TryDeserializeWithLengthPrefix for message 
type identification (or for whatever reason), all works great between two 
C#/.NET apps.  However if you have a c++ app using protobuf talking to a 
C#/.NET app using protobuf-net, the fieldNumber argument will cause grief. 
 It took wireshark, plus diving into protobuf-net code and protobuf code to 
find that the fieldNumber is encoded like (I assume) any other field within 
a message.  Whereas the c++ protobuf has no such notion (that I could find) 
unless you want to use a CodedOutputSTream - which then has both a WriteTag 
and WriteVarint32 (which WriteTag calls).  However WriteTag/WriteVarint32 
does NOT encode the same as the fieldNumber with SerializeWithLengthPrefix. 
 So... what do to?

Again, perhaps this has been addressed already but here's my solution (on 
the c++ side):

#define PROTO_NET_TAG(x) (((x) << 3) | (2 & 7))

...
google::protobuf::io::ZeroCopyOutputStream* raw_output = new 
google::protobuf::io::StringOutputStream(str);
google::protobuf::io::CodedOutputStream * coded = new 
google::protobuf::io::CodedOutputStream(raw_output); 
coded->WriteTag(PROTO_NET_TAG(msgTag));
coded->WriteVarint32(msg->ByteSize());
msg->SerializeToCodedStream(coded);

...



On Monday, August 24, 2009 5:33:26 PM UTC-6, Peter Keen wrote:
>
> I looked at the protobuf-net source a little bit and it looks like if 
> you write the length with WriteLittleEndian32 instead of WriteVarint32 
> you should be good to go, as long as you use PrefixStyle.Fixed32 
> within the C# call to DeserialzeWithLengthPrefix(). 
>
> --Pete 
>
> On Mon, Aug 24, 2009 at 4:17 PM, Jay 
> Thomas> 
> wrote: 
> > 
> > This code ideally has to operate with C# code on the other side of the 
> > socket that uses the DeserializeWithLengthPrefix() method call.  The 
> > message originates in C++ environment on a linux box and terminates in 
> > C# environment on a windows PC.  Why doesn't C++ support the same 
> > methods as C#?  The only document that I've seen that has list of 
> > methods available is the message.h header file, installed into my 
> > Linux usr directory. 
> > 
> > 
> > On Aug 24, 4:05 pm, Peter Keen  wrote: 
> >> You sort of have to roll your own. In my project I'm doing something 
> like this: 
> >> 
> >> coded_output->WriteVarint32(message->ByteSize()); 
> >> message->SerializeToCodedStream(coded_output); 
> >> 
> >> And then on the reading side: 
> >> 
> >> uint32_t size; 
> >> if (! coded_input->ReadVarint32(&size)) { 
> >>   return NULL; // just an example. probably don't want to do this; 
> >> 
> >> } 
> >> 
> >> Message * m = new Message; 
> >> CodedInputStream::Limit limit = coded_input->PushLimit(size); 
> >> message->ParseFromCodedStream(coded_input); 
> >> coded_input->PopLimit(limit); 
> >> 
> >> The biggest thing that helped me along was finding the Limit docs. I 
> >> couldn't figure out how to parse from a ZeroCopyInputStream without 
> >> consuming the entire stream, and then I ran across that and everything 
> >> became much more clear. 
> >> 
> >> --Pete 
> >> 
> >> On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomas 
> wrote: 
> >> 
> >> > Hello 
> >> 
> >> > I am looking for a way to serialize/deserialize with length prefix 
> >> > under c++.  The serialized bytes will sent to a TCP socket.  I am 
> >> > aware that C# has a method SerializeWithLengthPrefix() and 
> >> > DeserializeWithLengthPrefix().  Are there any such analogous methods 
> >> > for C++? Please point out any documentation that I may have missed 
> >> > here. 
> >> 
> >> > Thanks for any assistance. 
> >> 
> >> > Jay 
> > > 
> > 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[protobuf] Code generation error (protobuf-net bug)

[paul . baranov]

Protogen from protobuf-ner r614 generates code that does not compile. The 
.proto file seems to be fine as protogen does not report any errors and 
same .proto file is used to generate c++ code that compiles fine. Here is 
part that causes the problem:

message CosDgp
{
enum Type
{
Offline = 1;
Online = 2;
//... skipped
}

enum RestartReason
{
ResetPin = 1;
PowerCycle = 2;
//...skipped
}

enum ActiveTask
{
HeartBeat = 1;
Task0 = 2;
//... skipped
}

//Cos type
required Type type = 1;

//Panel number
required int32 panel = 2;

//Dgp number
required int32 dgp = 3;

//User number (used for types Isolated, Deisolated, DePolled and Polled)
optional int32 user = 5;

//Restart reason (used for type CpuRestart)
optional RestartReason restart_reason = 6;

//Active task at reset (used for type CpuRestart)
optional ActiveTask active_task = 7;

//Checkpoint (used for type CpuRestart)
optional int32 checkpoint = 8;

//Time and date
required int32 time = 9;
}

And the generated code with the errors:

[global::System.Serializable, 
global::ProtoBuf.ProtoContract(Name=@"CosDgp")]
  public partial class CosDgp : global::ProtoBuf.IExtensible
  {
...
private CosDgp.RestartReason _restart_reason = CosDgp.RestartReason.; 
 <<<<<<<<<<<<<<<<<<<  error on this line
[global::ProtoBuf.ProtoMember(6, IsRequired = false, 
Name=@"restart_reason", DataFormat = 
global::ProtoBuf.DataFormat.TwosComplement)]

[global::System.ComponentModel.DefaultValue(CosDgp.RestartReason.)]  
<<<<<<<<<<<<<<<<<<< 
 error on this line
public CosDgp.RestartReason restart_reason
{
  get { return _restart_reason; }
  set { _restart_reason = value; }
}

private CosDgp.ActiveTask _active_task = 
CosDgp.ActiveTask.;  <<<<<<<<<<<<<<<<<<<  error on this line
[global::ProtoBuf.ProtoMember(7, IsRequired = false, 
Name=@"active_task", DataFormat = 
global::ProtoBuf.DataFormat.TwosComplement)]

[global::System.ComponentModel.DefaultValue(CosDgp.ActiveTask.)]  
<<<<<<<<<<<<<<<<<<< 
 error on this line
public CosDgp.ActiveTask active_task
{
  get { return _active_task; }
  set { _active_task = value; }
}
...

Please avise.

Regards,
Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/protobuf/-/VWfPUfjIeqIJ.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] How to dynamically build a new protobuf from a set of already defined descriptors?

[Paul Di Biase]

 

At my server, we receive Self Described Messages (as defined 
here...
 
which btw wasn't all that easy as there aren't any 'good' examples of this 
in c++ but was definitely manageable). 

At this point I am having no issue creating messages from these 
self-described ones. I can take the FileDescriptorSet, go through each 
FileDescriptorProto,
 
adding each to a DescriptorPool (using 
BuildFile,
 
which also gives me every defined FileDescriptor).

>From here I can create any of the messages which were defined in the 
FileDescriptorSet with a DynamicMessageFactory instanced with the DP and 
calling 
GetPrototype(which
 is very easy to do as our SelfDescribedMessage required the messages 
full_name() and thus we can call the 
FindMessageTypeByNamemethod
 of the DP, giving us the properly encoded Message Prototype).

The question is how can I take each already defined Descriptor or message 
and dynamically BUILD a 'master' message that contains all of the defined 
messages as nested messages. This would primarily be used for saving the 
current state of the messages. Currently we're handling this by just 
instancing a type of each message in the server(to keep a central state 
across different programs). But when we want to 'save off' the current 
state, we're forced to stream them to disk as defined 
here. 
They're streamed one message at a time (with a size prefix). We'd like to 
have ONE message (one to rule them all) instead of the steady stream of 
separate messages. This can be used for other things once it is worked out 
(network based shared state with optimized and easy serialization) 

Since we already have the cross-linked and defined Descriptors, one would 
think there would be an easy way to build 'new' messages from those already 
defined ones. So far the solution has alluded us. We've tried creating our 
own DescriptorProto and adding new fields of the type from our already 
defined Descriptors but got lost (haven't deep dived into this one yet). 
We've also looked at possibly adding them as extensions (unknown at this 
time how to do so). Do we need to create our own 
DescriptorDatabase(also
 unknown at this time how to do so)?

Any insights? I've tried posting a question at 
stackoverflowbut
 haven't had any responses. I apologize if this isn't the correct 
location to ask this question.

Thank You

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/protobuf/-/wap-Qgj0RmgJ.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Improving negative number storage

[Paul TOTH]

Protocol Buffers is great, but there's something I don't understand.

VarInt is great to store positive integers
sint32/64 are stored as "zigzag" VarInt

why int32/64 aren't stored as :
 1) VarInt when positive (type 0)
 2) NegativeVarIn when negative (a new type 6 that store the opposite
value as a positive VarInt)

Then you don't need to specify sint32/64 in the Message and integers
will be stored in the most efficient way each time, regardless of they
sign.

Regards from France
Paul TOTH

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: protocol buffer within a protocol buffer from C++ to Java

[Paul]

thanks kenton.  the string thing worked very well.
Paul

On Oct 26, 9:20 pm, Kenton Varda  wrote:
> The others answered your main question, but there is another problem:
>
> On Mon, Oct 25, 2010 at 6:45 PM, Paul  wrote:
> > char* str_buf;
> > str_buf = new char[meas_rec.ByteSize()];
> > string str = str_buf;
>
> Here you are allocating a char array, and then *copying* it into a
> std::string.  This does nothing useful, and it may actually crash depending
> on the random contents of the allocated array.
>
> These three lines should be replaced with just:
>
>   string str;
>
> You do not have to resize the string ahead of time, as this is done
> automatically by SerializeToString().
>
> > meas_rec.SerializeToString(&str);
> > m.set_meas_rec_str(str);
>
> Incidentally, another, more efficient way to write this would be:
>
> meas_req.SerializeToString(&m.mutable_meas_rec_str());
>
> Also, you probably know this, but just in case:  You can just do:
>
> message Measurement {
>  required string id = 1;
>  optional MeasRec meas_rec = 2;
>
> }
>
> message MeasRec {
>  optional int id = 1;
>
> }
>
> This way you don't have to manually encode the sub-message.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: including common message definitions across .proto files

[Paul]

got it, thanks.

On Oct 26, 11:33 am, Henner Zeller 
wrote:
> On Tue, Oct 26, 2010 at 11:29, Paul  wrote:
> > Hi,
> > Suppose I have two files, file1.proto and file2.proto, and I have a
> > message defined in file1 that I want to use in file2.  How would I
> > include the message from file1?  Is there an "include" statement I
> > would use as in C or C++?
>
> 'import' is the thing you're looking for.
>
> http://code.google.com/apis/protocolbuffers/docs/proto.html#other
>
>
>
> >  file1.proto 
> > message M1 {
> >   optional string id = 1;
> > }
>
> >  file2.proto 
> > message M2 {
> >  optional string id = 1;
> >  optional M1 m1;   // how do I include this?
> > }
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > protobuf+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] including common message definitions across .proto files

[Paul]

Hi,
Suppose I have two files, file1.proto and file2.proto, and I have a
message defined in file1 that I want to use in file2.  How would I
include the message from file1?  Is there an "include" statement I
would use as in C or C++?

 file1.proto 
message M1 {
   optional string id = 1;
}

 file2.proto 
message M2 {
  optional string id = 1;
  optional M1 m1;   // how do I include this?
}

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: protocol buffer within a protocol buffer from C++ to Java

[Paul]

ok thanks.  changing to optional bytes worked.  also, instead of
parseFrom, I ended up using parseDelimitedFrom on the Java side, after
serializing with SerializeToCodedStream on the C++ side.

On Oct 26, 4:59 am, Evan Jones  wrote:
> On Oct 25, 2010, at 21:45 , Paul wrote:
>
> >  optional string meas_rec_str = 2;
>
> Change this to:
>
> optional bytes meas_rec_str = 2;
>
> Evan
>
> --
> Evan Joneshttp://evanjones.ca/

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] protocol buffer within a protocol buffer from C++ to Java

[Paul]

Hi,
I am trying to serialize a protocol buffers message into a string that
is contained in another protocol buffer.  I am doing the serialization
on the C++ side.

// This is the outer protocol buffer
message Measurement {
  required string id = 1;
  optional string meas_rec_str = 2;
}
Measurement m;

// This is the protocol buffer message I want to serialize into a
string and store inside meas_rec.
message MeasRec {
  optional int id = 1;
}

To serialize the MeasRec message, I am doing:
MeasRec meas_rec;
meas_rec.set_id(100);
char* str_buf;
str_buf = new char[meas_rec.ByteSize()];
string str = str_buf;
meas_rec.SerializeToString(&str);
m.set_meas_rec_str(str);

However, when I do this, I encounter the following error on the C++
side:

libprotobuf ERROR google/protobuf/wire_format.cc:1059] Encountered
string containing invalid UTF-8 data while serializing protocol
buffer. Strings must contain only UTF-8; use the 'bytes' type for raw
bytes.

how do I serialize meas_rec into "bytes" instead?

Also, once I have it on the Java side, how do I deserialize?
would I use parseFrom(byte[] data) ?

Thanks,
Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: outer classname for C++

[Paul]

ok thanks.

On Oct 25, 12:49 pm, Daniel Wright  wrote:
> No -- in C++ the message classes are placed directly in a namespace named
> after the package, so there's no "outer" class.
>
> On Mon, Oct 25, 2010 at 11:42 AM, Paul  wrote:
> > Hi,
>
> > In the example of the .proto definition in Java, there is a
>
> > option java_outer_classname = "AddressBookProtos";
>
> > Is there an equivalent statement for C++?
>
> > Thanks,
> > Paul
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] outer classname for C++

[Paul]

Hi,

In the example of the .proto definition in Java, there is a

option java_outer_classname = "AddressBookProtos";

Is there an equivalent statement for C++?

Thanks,
Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Message missing required fields exception when parsing a message that has required fields defaulted

[Paul]

I've gotten this message for reasons other than the field not being
set.  make sure that you have the number of bytes on the delimiters
set correctly on the C++ side.  also, maybe stringProperty should be
string Property (two words).

On Oct 22, 9:12 am, locky  wrote:
> I have a question regarding required string properties and default
> values.
>
> Consider the following message definition.
>
> message StringTestProto {
>   required stringProperty = 1;
>
> }
>
> I have a C++/Java setup where C++ app is producing a message and a
> Java app is consuming it.
>
> If the stringProperty value is set to an empty string I get the
> following exception when trying to parse the
> message once I receive it in the the Java app.
>
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields:
>
> The message is being read as follows:
>
> StringTestProto proto = StringTestProto.parseFrom(data);
>
> From what I can see the issue is as follows:
> As the stringProperty value is the same as the default (an empty
> string) it is not sent over the wire.  When the generated Java code
> tries to build the proto from a byte[ ] (in the buildFrom method), a
> check is done to see if the proto is initialised, this check sees if
> the required fields have been set.  As the stringProperty was never
> sent, the proto does not think it has been set (as it never was) and
> therefore the exception is thrown.
>
> So, how are you supposed to decode a byte[] that contains required
> properties, that may have default values?

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: omitting tag numbers

[Paul]

ok that makes sense.  thanks!

On Oct 22, 4:02 pm, Henner Zeller 
wrote:
> On Fri, Oct 22, 2010 at 15:01, Paul  wrote:
> > Hi,
>
> > This may seem like a basic question, but I find having to label
> > the .proto file with unique tag numbers for each field a little
> > cumbersome, especially if there are a lot of fields.
>
> > message Person {
> >  required string name = 1;
> >  required int32 id = 2;
> >  optional string email = 3;
> > }
>
> > Can I define a .proto file without the tag numbers, like so?
>
> > message Person {
> >  required string name;
> >  required int32 id;
> >  optional string email;
> > }
>
> No.
>
> The reason for this explicit definition is that the protocol buffer is
> 'future compatible': fields written with a particular tag will always
> be written with that tag. Consider you want to re-structure the fields
> in your proto buffer to say (Id, name, email) ... then they would get
> a different 'automatic' tag assigned and you wouldn't be able to read
> files written with older binaries. If the tags are assigned, then
> re-arranging fields in the file does not matter.
>
> -h
>
>
>
> > Thanks,
> > Paul
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > protobuf+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] omitting tag numbers

[Paul]

Hi,

This may seem like a basic question, but I find having to label
the .proto file with unique tag numbers for each field a little
cumbersome, especially if there are a lot of fields.

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

Can I define a .proto file without the tag numbers, like so?

message Person {
  required string name;
  required int32 id;
  optional string email;
}

Thanks,
Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: buffer sizes when sending messages from c++ to java

[Paul]

ok thanks a lot guys!  both ByteCount() and VarintSize32() worked for
me!

Paul

On Oct 20, 3:13 am, Evan Jones  wrote:
> On Oct 20, 2010, at 2:13 , Kenton Varda wrote:
>
> > But you are actually writing a varint32, which can be anywhere  
> > between 1 and 5 bytes depending on the value.
>
> > Use CodedOutputStream::Varint32Size() to compute the number of bytes  
> > needed to encode a particular value.
>
> This has the advantage that you can allocate a buffer of exactly the  
> right size, rather than adding 100 as an estimate. However, you can  
> also find the final size after all the writes with  
> CodedOutputStream::ByteCount()
>
> You should not need to do any byte swapping if you are serializing and  
> deserializing integers using the protobuf API: it handles any required  
> byte swapping for you.
>
> Evan
>
> --
> Evan Joneshttp://evanjones.ca/

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] buffer sizes when sending messages from c++ to java

[Paul]

Hi,

I am sending messages from a C++ client to a Java server, but I am
wondering about the length of the buffer I am using to send it over.
This is the C++ code:

CLIENT SIDE CODE:
*** the protocol buffers message is called snap1 ***

int numBytesForDelim = sizeof(int);
char *snap_buf2;
snap_buf2 = new char[snap1.ByteSize() + 100];

ZeroCopyOutputStream* raw_output = new ArrayOutputStream(snap_buf2,
snap1.ByteSize()+numBytesForDelim);
CodedOutputStream* coded_output = new
CodedOutputStream(raw_output);

coded_output->WriteVarint32(snap1.ByteSize());
snap1.SerializeToCodedStream(coded_output);

delete coded_output;
delete raw_output;

send(socket, snap_buf2, snap1.ByteSize()+numBytesForDelim/2, 0);  //
send over a socket
delete [] snap_buf2;
snap_buf2 = NULL;

When I send this over to the Java server side, the Java code uses
"parseDelimitedFrom" to parse the data.  however, I am not sure why
when I call send, the length of the message has to be snap1.ByteSize()
+numBytesForDelim/2 rather than snap1.ByteSize()+numBytesForDelim.  I
would think that it would be the latter, but when I try it, I get
exceptions on the Java side saying that the protocol message has not
filled out all the fields.  For some reason, it is necessary for me to
cut off the last two bytes when sending over TCP.  Any ideas as to why
this is?

Thanks a lot!

Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: sending a message over TCP from a C++ client to a Java server

[Paul]

ok thanks a lot!  I ended up finding that my TCP send function was not
set up correctly.  Once that was fixed, it worked fine.  Thanks!

On Oct 13, 2:20 pm, Evan Jones  wrote:
> On Oct 13, 2010, at 16:49 , Paul wrote:
>
> > Thanks for the suggestion.  However, I am already prepending the
> > message size on the C++ side in the line:
> > coded_output->WriteVarint64(snap1.ByteSize());
>
> You may want to verify that the exact bytes that come out of  
> msg.SerializeToString (or related) are coming out the other end and  
> getting passed into parseDelimited. It might be helpful if you sent a  
> snippet of code where you are sending and receiving the messages, but  
> I can't think of anything off the top of my head.
>
> Evan
>
> --
> Evan Joneshttp://evanjones.ca/

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: sending a message over TCP from a C++ client to a Java server

[Paul]

Thanks for the suggestion.  However, I am already prepending the
message size on the C++ side in the line:
coded_output->WriteVarint64(snap1.ByteSize());

is there anything else that stands out to you?

thanks for any further help.

On Oct 13, 12:56 pm, Evan Jones  wrote:
> On Oct 13, 2010, at 15:13 , Paul wrote:
>
> > On the client side (in C++), I open a TCP socket connection on the
> > same port with the server's IP address.  I serialize the message using
> > SerializeToCodedStream into an array using ArrayOutputStream.  After
> > serializing it, I send it over the TCP connection using my sendTCP
> > method which uses C++ sockets.
>
> SerializeToCodedStream does *not* prepend the message size. The Java  
> side is expecting that the message will start with the message length,  
> so that is probably why you are getting parse errors. You need to do  
> something like:
>
> codedOutput.WriteVarint32(msg.ByteSize());
> msg.SerializeToCodedStream(codedOutput);
> codedOutput.flush();
>
> ...
>
> Hope this helps,
>
> Evan
>
> (as an aside: the C++ API really should have an equivalent to  
> writeDelimitedTo and parseDelimited on the Java side).
>
> --
> Evan Joneshttp://evanjones.ca/

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] sending a message over TCP from a C++ client to a Java server

[Paul]

Hi,

I am new to protocol buffers, and I am trying to send a protocol
buffers message over TCP from a client computer running C++ to a
server computer running Java.  Both computers are 64 bit.

On the server side (in Java), I open a ServerSocket on a port, and get
a Socket after accepting the client's connection.  From the client
Socket (liveSock), I get an inputstream, which I pass to the handle
function, which calls parseDelimitedFrom on the inputstream.

On the client side (in C++), I open a TCP socket connection on the
same port with the server's IP address.  I serialize the message using
SerializeToCodedStream into an array using ArrayOutputStream.  After
serializing it, I send it over the TCP connection using my sendTCP
method which uses C++ sockets.

When I start running the two sides, the connection gets established.
However, when the client tries to send the data over TCP, the server
handle code crashes at parseDelimitedFrom with an
InvalideProtocolBufferException saying that the Message is missing
required fields.

I'm sure that my C++ TCP sockets work correctly, because I was able to
send text from C++ to Java without any problems.  If it helps, I can
also post the TCP code I have.

I am not sure what I am doing wrong.  I don't know if I am doing
things correctly on the server (Java) side.  Please let me know if you
see any problems!  Thanks a lot!

Paul



SERVER SIDE CODE:

ServerSocket sock = new ServerSocket(7003);

Socket liveSock = sock.accept();

handle(liveSock.getInputStream(), liveSock.getOutputStream());  //
handle defined below



HANDLE METHOD:

*** the protocol buffers message is called Snapshot ***

public static void handle(InputStream in, OutputStream output) {
   try {
   Snapshot snapshot = Snapshot.parseDelimitedFrom(in);
   while(snapshot != null) {
System.out.println("SNAPSHOT: " + snapshot.getId()
snapshot = Snapshot.parseDelimitedFrom(in);
}
   }
   catch (IOException e) {
  System.out.println("exception");
   }
}



CLIENT SIDE CODE:

*** the protocol buffers message is called snap1 ***

int sock = openSocketClientTCP("127.100.100.100", 7003);

char snap_buf2[snap1.ByteSize()+1];

ZeroCopyOutputStream* raw_output = new
ArrayOutputStream(snap_buf2,snap1.ByteSize()+1);
CodedOutputStream* coded_output = new CodedOutputStream(raw_output);

coded_output->WriteVarint64(snap1.ByteSize());
snap1.SerializeToCodedStream(coded_output);

delete coded_output;
delete raw_output;

sendTCP(sock, snap_buf2, snap1.ByteSize() + 1);



-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: serialize to a file using FileOutputStream

[Paul Yang]

thanks!  I tried that and it works now!

On Oct 11, 4:03 pm, Daniel Wright  wrote:
> I think it's being buffered in the FileOutputStream -- you should be sure to
> delete the output streams (in the reverse order that you created them)
> before you close the file.
>
> On Mon, Oct 11, 2010 at 1:07 PM, Paul Yang  wrote:
> > Hi,
> > I am new to protocol buffers, and I am trying to serialize a message
> > to a file.  I need to serialize the file so that in can be opened in
> > Java using parseDelimitedFrom().  However, when I write to the file
> > using SerializeToCodedStream or SerializeWithCachedSizes, nothing
> > happens to the serializedMessage.bin file, and it stays at 0 bytes.
> > Why is nothing being written to the file?  I have attached the code
> > below:
>
> > int fd = open("serializedMessage.bin", O_WRONLY);
>
> > google::protobuf::io::ZeroCopyOutputStream* fileOutput = new
> > google::protobuf::io::FileOutputStream(fd);
>
> > google::protobuf::io::CodedOutputStream* codedOutput = new
> > google::protobuf::io::CodedOutputStream(fileOutput);
>
> > codedOutput->WriteVarint32(message1.ByteSize());
> > message1.SerializeToCodedStream(codedOutput);
> > // also tried snap1.SerializeWithCachedSizes(codedOutput)
>
> > close(fd);
>
> > Thanks!
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] serialize to a file using FileOutputStream

[Paul Yang]

Hi,
I am new to protocol buffers, and I am trying to serialize a message
to a file.  I need to serialize the file so that in can be opened in
Java using parseDelimitedFrom().  However, when I write to the file
using SerializeToCodedStream or SerializeWithCachedSizes, nothing
happens to the serializedMessage.bin file, and it stays at 0 bytes.
Why is nothing being written to the file?  I have attached the code
below:

int fd = open("serializedMessage.bin", O_WRONLY);

google::protobuf::io::ZeroCopyOutputStream* fileOutput = new
google::protobuf::io::FileOutputStream(fd);

google::protobuf::io::CodedOutputStream* codedOutput = new
google::protobuf::io::CodedOutputStream(fileOutput);

codedOutput->WriteVarint32(message1.ByteSize());
message1.SerializeToCodedStream(codedOutput);
// also tried snap1.SerializeWithCachedSizes(codedOutput)

close(fd);

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Any sample messages/data for optimization out there?

[Paul Rudd]

It doesn't have to be. CodedInputStream.newInstance() is very
expensive for small messages.
Eliminating the call yields much better results. For example:

Benchmarking benchmarks.GoogleSpeed$SpeedMessage1 with file ../
google_message1.dat
Deserialize from byte string: 20354111 iterations in 29.817s;
148.4305MB/s
Deserialize from byte array: 20406928 iterations in 29.825s;
148.77573MB/s
Deserialize from memory stream: 9682142 iterations in 29.936s;
70.32546MB/s
Deserialize from memory stream reusing CodedInputStream: 18493403
iterations in 30.029s; 133.90935MB/s

Using this new benchmark:

  final CodedInputStream reuseCodedInputStream =
CodedInputStream.newInstance(inputStream);
  benchmark("Deserialize from memory stream reusing
CodedInputStream", inputData.length, new Action() {
  public void execute() throws IOException {
defaultMessage.newBuilderForType()
  .mergeFrom(reuseCodedInputStream).build();
inputStream.reset();
reuseCodedInputStream.resetSizeCounter();
  }
});

Which actually brings up a feature request -
MessageLite.Builder.mergeDelimitedFrom(InputStream) is too inefficient
for small streamed messages, since it creates a new CodedInputStream
(and 4KB buffer to go with it) every time. The addition of a
mergeDelimitedFrom(CodedInputStream) would be welcomed.



On Mar 8, 2:26 pm, Kenton Varda  wrote:
> Decoding from a "memory stream" is significantly slower than a byte array.
>  The degree of the difference will depend on the data set.  SpeedMessage1 is
> much smaller than SpeedMessage2, therefore differences in one-time costs in
> setting up the parser will be more prominent.  Of course, different
> platforms and Java implementations will also show different results.
>
>
>
> On Mon, Mar 8, 2010 at 8:52 AM, Evan Jones  wrote:
> > On Mar 7, 2010, at 20:36 , Oliver Jowett wrote:
>
> >> Benchmarking benchmarks.GoogleSpeed$SpeedMessage1 with file
> >>> google_message1.dat
> >>> Deserialize from byte string: 17471413 iterations in 30.074s;
> >>> 126.3199MB/s
> >>> Deserialize from byte array: 17389320 iterations in 30.009s;
> >>> 125.99868MB/s
> >>> Deserialize from memory stream: 5050944 iterations in 29.372s;
> >>> 37.391594MB/s
>
> >> I wonder what's happening in the first memory stream case to make it 3x
> >> slower than everything else?
>
> > I just dug up some results from the last time I was playing with
> > ProtoBench, and I got the results being slower, but not 3X. This was on
> > Linux, on a recent Intel Core i7 Xeon (Nehalem architecture), with
> > SpeedMessage1:
>
> > Deserialize from byte string: 25450873 iterations in 30.007s; 184.42299MB/s
> > Deserialize from byte array: 25368774 iterations in 29.745s; 185.44727MB/s
> > Deserialize from memory stream: 15248317 iterations in 29.285s;
> > 113.21699MB/s
>
> > Evan
>
> > --
> > Evan Jones
> >http://evanjones.ca/
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com > om>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

RE: Storing complex graphs of objects?

[Paul Runyan]

Hi Kenton -

 

Thanks for your help.  Obviously defining message types in proto files is 
crucial for being able to exchange messages between different languages.  
However, as a substitute for Java serialisation having to specify message types 
in separate files from the source files seems less than ideal.  

 

What I was thinking of was more along the lines of adding methods to each 
object to be serialised via PB that look like the methods on the classes the PB 
compiler generates --- writeTo() and parseFrom().  These would work analogously 
to the way that writeObject() and readObject() for normal serialisation, but by 
trading away the unrestricted polymorhism that this type of application doesn't 
need it allow it to write ints rather than class names to distinguish types and 
would hopefully also gain from the extra efficiencies of CodedInputStream and 
CodedOutputStream.

 

Best regards,

 

Paul

_
Looking to change your car this year? Find car news, reviews and more 
http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fsecure%2Dau%2Eimrworldwide%2Ecom%2Fcgi%2Dbin%2Fa%2Fci%5F450304%2Fet%5F2%2Fcg%5F801459%2Fpi%5F1004813%2Fai%5F859641&_t=762955845&_r=tig_OCT07&_m=EXT
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

protobuf-2.0.3 on PyPI does not work.

[Paul Carduner]

Hi All,

I am trying to set a dependency on the protobuf package for another
python package I am developing.  Whenever I try to install my python
package, it fails with the error:

Can't find required file: ../src/google/protobuf/descriptor.proto

I've located the problem in protobuf-2.0.3/python/setup.py, line 102
where the setup scripts attempts to compile descriptor.proto... which
is not part of the egg and can therefore not be found.

I think the descriptor_pb2.py file should be included as part of the
egg release and the associated line should be removed from the
setup.py file.

After removing the problem line and rebuilding the egg, I uploaded it
to a private egg repository and that seemed to work.  I would much
prefer using the one on pypi though.

Thanks,
Paul Carduner
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Paul P Komkoff Jr]

On Tue, Oct 28, 2008 at 9:07 PM, Pavel Shramov <[EMAIL PROTECTED]> wrote:
> By the way one of the simpliest ways for RPC is to use HTTP transport.
> It's have some limitations (e.g large overhead for small messages) but
> also some benefits (e.g many libraries for performing HTTP calls and
> simple proxying)

Speaking of ugly hacks, one of the intermediate versions of rpc I used
were, actually, protocol buffers over xmlrpc. messages were serialized
and then wrapped into xmlrpc.Binary


-- 
This message represents the official view of the voices in my head.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Paul P. Komkoff Jr]

On Oct 28, 2:02 am, "Kenton Varda" <[EMAIL PROTECTED]> wrote:
> I don't really have a stake in the design of a protobuf-based RPC format.
>  However, I'd like to point out that the design philosophy we tend to prefer
> at Google is to keep each layer of the system as simple as possible, and
> implement orthogonal features using separate layers.  Authentication is a
> great example of something that I would not want to make part of an RPC
> protocol itself, but rather implement as a layer under it, similar to the
> way HTTP can operate over SSL.  If you keep the system separate in this way,

First, I'm talking about something similar to simple HTTP auth, which
allows us to authenticate by key/value pair and does not include TLS.
With support for "struct user_credentials" passed to server method, so
we can "impersonate" the user.

Also, even before considering paragraph above, if you have the system
separate that way it will produce incompatible wire formats. My goal
is to have, at least, lowest common denominator which could be
implemented in, at least, twisted-python and something-java, in order
to bootstrap my project now. It would be wonderful if this LCD format
will have some notion of authentication (or authorization, if
authentication is performed by separate coexisting entity that
produces auth cookies).

> it's much easier for people to avoid the overhead of features they don't
> need, find alternative ways of implementing individual features, and to
> reuse code in general.
> Just my opinion.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Standard for RPC proto

[Paul P. Komkoff Jr]

On 26 окт, 02:53, Alan Kligman <[EMAIL PROTECTED]> wrote:
> I haven't had much to add recently. Protobuf-rpc is based heavily on
> json-rpc, so there's really nothing new behind it. It works well for
> my own use and is generic enough to probably work well for most other
> people.
>
> Is there a great deal of interest in devising a standard rpc protocol
> definition?

Yes it is.
Since everything is trying to design its own RPC format, running into
the same flaws as everyone else.
For example, I haven't seen (in protobuf-rpc neither in protorcp) a
single word about authentification.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---