[protobuf] Re: Sending protobuf from C++ to Java - Unable to parse in java
Java's parseDelimitedFrom() method expects a length prefix before the
content of the message. In C++, you will need to add this prefix using
CodedOutputStream:
CodedOutputStream coded_output(...);
int length = message.ByteSize();
coded_output.WriteVarint32(length);
message.SerializeToCodedStream(coded_output);
On Thursday, June 11, 2015 at 6:08:33 PM UTC-7, Ashwin Venkataraman wrote:
I am sending a protobuf from a C++ program to a java program via
sockets. I am unable to deserialize the proto in the Java program .I keep
encountering errors like :
1. While parsing a protocol message, the input ended unexpectedly in the
middle of a field. This could mean either than the input has been
truncated or that an embedded message misreported its own length.
2. CodedInputStream encountered a malformed varint
I am using a simple socket in C++ and a ServerSocket in the java program.
I an serializing my proto to a char* buffer before I sent it to the socket.
what am I doing wrong? Kindly help.
I have both my code below:
P.S - the example for proto is taken from the Google example -
AddressBook.proto
*C++ code: ( Client side - sends data )*
#define WIN32_LEAN_AND_MEAN
#include iostream
#include fstream
#include string
#include windows.h
#include winsock2.h
#include ws2tcpip.h
#includeconio.h
#include addressbook.pb.h
#pragma comment (lib, Ws2_32.lib)
#pragma comment (lib, Mswsock.lib)
#pragma comment (lib, AdvApi32.lib)
using namespace std;
// This function fills in a Person message based on user input.
void PromptForAddress(tutorial::Person* person) {
cout Enter person ID number: ;
int id;
cin id;
person-set_id(id);
cin.ignore(256, '\n');
cout Enter name: ;
getline(cin, *person-mutable_name());
cout Enter email address (blank for none): ;
string email;
getline(cin, email);
if (!email.empty()) {
person-set_email(email);
}
while (true) {
cout Enter a phone number (or leave blank to finish): ;
string number;
getline(cin, number);
if (number.empty()) {
break;
}
tutorial::Person::PhoneNumber* phone_number = person-add_phone();
phone_number-set_number(number);
cout Is this a mobile, home, or work phone? ;
string type;
getline(cin, type);
if (type == mobile) {
phone_number-set_type(tutorial::Person::MOBILE);
}
else if (type == home) {
phone_number-set_type(tutorial::Person::HOME);
}
else if (type == work) {
phone_number-set_type(tutorial::Person::WORK);
}
else {
cout Unknown phone type. Using default. endl;
}
}
}
// Main function: Reads the entire address book from a file,
// adds one person based on user input, then writes it back out to the
same
// file.
int main(int argc, char* argv[]) {
// Verify that the version of the library that we linked against is
// compatible with the version of the headers we compiled against.
GOOGLE_PROTOBUF_VERIFY_VERSION;
tutorial::AddressBook address_book;
// Add an address.
PromptForAddress(address_book.add_person());
{
int size = address_book.ByteSize();
char * buffer = new char[size];
address_book.SerializeToArray(buffer, size);
WSADATA wsaData;
SOCKET ConnectSocket = INVALID_SOCKET;
struct addrinfo *result = NULL,
*ptr = NULL,
hints;
int iResult;
// Initialize Winsock
iResult = WSAStartup(MAKEWORD(2, 2), wsaData);
ZeroMemory(hints, sizeof(hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_protocol = IPPROTO_TCP;
// Resolve the server address and port
iResult = getaddrinfo(localhost, 5000, hints, result);
if (iResult != 0) {
printf(getaddrinfo failed with error: %d\n, iResult);
WSACleanup();
return 1;
}
// Attempt to connect to an address until one succeeds
for (ptr = result; ptr != NULL; ptr = ptr-ai_next)
{
// Create a SOCKET for connecting to server
ConnectSocket = socket(ptr-ai_family, ptr-ai_socktype,
ptr-ai_protocol);
// Connect to server.
iResult = connect(ConnectSocket, ptr-ai_addr,
(int)ptr-ai_addrlen);
if (iResult == SOCKET_ERROR) {
closesocket(ConnectSocket);
ConnectSocket = INVALID_SOCKET;
continue;
}
freeaddrinfo(result);
// Send an initial buffer
iResult = send(ConnectSocket, buffer, (int)strlen(buffer), 0);
if (iResult == SOCKET_ERROR) {
printf(send failed with error: %d\n,
[protobuf] Re: Question according assigning tags
On Thursday, June 11, 2015 at 6:08:33 PM UTC-7, caramba230 wrote: Hi, I have a question about assigning tags, the documentation says : tags with values in the range 1 through 15 take one byte to encode, including the identifying number and the field's type (you can find out more about this in Protocol Buffer Encoding https://developers.google.com/protocol-buffers/docs/encoding.html#structure). Tags in the range 16 through 2047 take two bytes. So you should reserve the tags 1 through 15 for very frequently occurring message elements. But if I have a message A with two fields, and his first field is another message who has 15 fields, and the second one has 12 fields. Which fields will be encoded with one byte? The documentation is talking about the number of bytes needed to encode the tag, not the actual content of the field. Whether the field is a message field or a string field or a int32 field, doesn't matter. Thanks -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: How can I read protobuf massages from single file if writer is Java reader is Python?
On Thursday, June 11, 2015 at 6:08:33 PM UTC-7, Misha Gavrilov wrote: Hi all I have a problem with protocol buffers serialised files. I use python for parsing buffer files. But I can't read any message from file, I don't know why. Writer developed on Java and use CodedInputStream and WriteMassangeNoTag for writing buffers file. I wont to parse this file by python. I have all .proto files compiled successful for python. I read PB documentation and find that PB massages must be delimited if they are located in one file. So the questions are next: 1. What is the delimiter CodedInputStream use? 2. Can some one provide code example for python, how to read messages from file one by one. You need to delimite the messages yourself. CodedInputStream/CodedOutputStream does not read/write any delimiters. Basically PB only helps you convert a message object into a chunk of data. It's your responsibility to take care of how to put/read this chunk of data within the file. Code Example (Default code from PB Documentation): JM = proto.myproto_pb2.MyProtoClass() try: f = open(sys.argv[1], rb) # text_format.Merge(f.read(), JM) # print(f.read()) JM.ParseFromString(f.read()) f.close() print (JM) except IOError: print (sys.argv[1] + : Could not open file. Creating a new one.) Return error: google.protobuf.message.DecodeError: Tag had invalid wire type. P.S. If python program write buffer file then python program can read and parse it. So problem happen when java write and python read. Thanks Mike. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Protobuf3: How to use the new timestamp proto type?
On Thursday, June 11, 2015 at 6:08:33 PM UTC-7, Robin wrote:
Hi everyone,
I have installed protobuf3.0.0-alpha-3 on my machine and have the
following proto file:
syntax = proto3;
package my.great.package;
import google/protobuf/timestamp.proto;
option java_outer_classname = TimestampedThingMsg;
message TimestampedThing {
string thing = 1;
Timestamp date = 2;
}
However when I run protoc protobuf/timestamped_thing.proto I get the
following two errors:
protobuf/timestamped_thing.proto: Import google/protobuf/timestamp.proto
was not found or had errors.
protobuf/timestamped_thing.proto:15:5: Timestamp is not defined.
protoc should be able to find these types automatically if they are
installed with protoc. Where did you install the protoc binary? Could you
check whether you can find google/protobuf/timestamp.proto in your include
path? These protos should be installed along with protobuf header files.
How can I import the new Timestamp message type (source file:
https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto)
to use it in my own messages?
The only related example I could find in the docs was for the Any type
where the import looks exactly like mine so not sure how to go about this.
Docs link: https://developers.google.com/protocol-buffers/docs/proto3#any
Thanks!
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
On Wednesday, June 10, 2015 at 9:49:03 PM UTC-7, Jan Kyjovský wrote:
I did as you told me. And tried to compile my proto files with protoc.
Well it eat them up without any problem so I guess that clear that there is
no problem with them but in that case why is it causing such error to me.
As I gave some hint before problem is happening when I am trying to do
following with phone.proto which is being added as second into descriptor
database
compiler::Parser parser;
parser.RecordErrorsTo(errCollector);
bool bV = parser.Parse(tokenizer, result);
std::string strErr = errCollector.GetErrorAndWarnings();
return bV;
Parse returns false and error collector holds following error:
1:1: ERROR: Multiple package definitions.
Do you have nay idea what that could meant?
Could you post the whole content of your code? It would be easier for me to
figure out what's wrong if I can run the code myself.
On Tuesday, 9 June 2015 20:31:38 UTC+2, Feng Xiao wrote:
On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com
wrote:
I have remade code so I now add into descriptor database all protofiles
specified by directory. There is but little problem with error handling.
From nature how I am handling errors it seems that message is possible to
decode but there is one error in error collector which is bothering me:
1:1: ERROR: Multiple package definitions.
*My protos are defined as follows:*
*addressbook_fragmented.proto:*
import phone.proto;
package tutorial;
option java_package = com.example.tutorial;
option java_outer_classname = AddressBookProtos;
message Person {
required string name = 1;
required int32 id = 2;// Unique ID number for this person.
optional string email = 3;
// enum PhoneType {
//MOBILE = 0;
//HOME = 1;
//WORK = 2;
// }
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
optional int32 age = 5;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person person = 1;
}
*phone.proto:*
package tutorial;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
Even though there is error in error collector I can ignore it and decode
message successfully. Yet if I try to remove this package info from
phone.proto itis not detecting any error but FindMessageTypeByName fails to
find data type. I am suspecting since not all references for person data
type I am asking for are resolved it fails.
So I am confused how exactly should be these package keywords be used?
I don't see a problem in your use of package. You could try to run
protoc on these proto files and see if it reports any error. I suspect it's
not the .proto file's problem.
On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:
On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com
wrote:
HI,
I somehow understand what you mean but still I lack in experience with
it. I tried to switch to descriptor database but still I got some
problems.
C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
m_strProtoFile = strFile;
// Build the descriptor pool
m_bConstructed = ParseProtoFile(m_ProtoFileDescr);
if (m_bConstructed)
{
m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
}
}
C_ProtoDecoder::~C_ProtoDecoder()
{
delete m_Pool;
}
bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
strWireFormat, std::string strType)
{
Descriptor *descriptor = (Descriptor
*)m_Pool-FindMessageTypeByName(strType);
Message *message =
m_MessageFactory.GetPrototype(descriptor)-New();
if (descriptor == NULL)
{
char szError[256];
sprintf(szError, Unknown data type %s!, strType.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
bool bFlag = message-ParseFromString(strWireFormat);
if (!bFlag)
{
m_InternalPrinter.AddErrorMessage(Encoding error!);
}
else
{
m_Transformator.MorphToStructs(*message);
}
return true;
}
bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);
if (file_descriptor == -1)
{
char szError[256];
sprintf(szError, Invalid proto file \%s\!,
m_strProtoFile.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
C_ParserErrorCollector errCollector;
io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
result-set_name(m_strProtoFile);
compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
I have used SimpleDescriptorDatabase since it
[protobuf] Re: Protobuf3: How to use the new timestamp proto type?
I installed protobuf with the following homebrew command on OS X 10.10.3:
brew install protobuf --devel
The binary is symlinked to /us/local/bin and the includes are symlinked to
/usr/local/include/google/protobuf/timestamp.proto (which is present as I
confirmed).
Which include path do you mean, the protoc -IPATH? And should it be
sufficient in my .proto file to just use Timestamp or should I use the full
google.protobuf.Timestamp?
On Friday, June 12, 2015 at 8:41:18 AM UTC+2, 肖锋 wrote:
On Thursday, June 11, 2015 at 6:08:33 PM UTC-7, Robin wrote:
Hi everyone,
I have installed protobuf3.0.0-alpha-3 on my machine and have the
following proto file:
syntax = proto3;
package my.great.package;
import google/protobuf/timestamp.proto;
option java_outer_classname = TimestampedThingMsg;
message TimestampedThing {
string thing = 1;
Timestamp date = 2;
}
However when I run protoc protobuf/timestamped_thing.proto I get the
following two errors:
protobuf/timestamped_thing.proto: Import
google/protobuf/timestamp.proto was not found or had errors.
protobuf/timestamped_thing.proto:15:5: Timestamp is not defined.
protoc should be able to find these types automatically if they are
installed with protoc. Where did you install the protoc binary? Could you
check whether you can find google/protobuf/timestamp.proto in your include
path? These protos should be installed along with protobuf header files.
How can I import the new Timestamp message type (source file:
https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto)
to use it in my own messages?
The only related example I could find in the docs was for the Any type
where the import looks exactly like mine so not sure how to go about this.
Docs link: https://developers.google.com/protocol-buffers/docs/proto3#any
Thanks!
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
