Re: Performance Observations using protocol and java serialization
That would be my address, yes. But if this is a large amount of code, I
don't have time to go through it. You really need to narrow it down to
something small which can be easily debugged.
I recommend writing a benchmark which times *only* the parsing and
serialization steps, with protocol buffers and with java serialization,
without any database communication. This would simplify things and would
prove that the problem isn't in the database layer. Once you have that, it
should be a fairly small amount of code, and you should be able to send it
to this list without too much trouble. Please also include your results
with the code so that we don't have to actually run it. Also, have your
benchmark measure the sizes of the encoded messages.
On Thu, Sep 10, 2009 at 7:55 AM, rajesh poorv...@gmail.com wrote:
Hi Kent,
Thanks for the quick reply, It might get really clumsy
if I post the code here..lemme email it to you..can you please provide
me youremail address to which I can send the code to. I have a
'ken...@google.com' on your profile , do you want me to email it to
this id.
On Sep 9, 7:45 pm, Kenton Varda ken...@google.com wrote:
Since you haven't provided any code it's hard to speculate on what may be
going wrong.
On Wed, Sep 9, 2009 at 3:01 PM, rajesh poorv...@gmail.com wrote:
Hi All,
I ran some performance tests to compare the performance for
serializing-persisting-retrieving-desirializing of my POJOs and was
really surprised to see that java-serialization was performing better
than protobuf-serialization. Iam enclosing my proto file based on
which Iam generating my proto buffer messages.
--- protofile---
option java_package = com.equifax.ic.eid.iq.persist.generated;
option java_outer_classname = AllQuestionPackImplProtos;
option optimize_for = SPEED;
message AllQuestionPacksImpl{
optional string transid = 1;
repeated QuestionPack realQuestionPacks = 2;
repeated QuestionPack simulatedQuestionPacks = 3;
}
message QuestionPack{
optional string transkey = 1;
optional bool simulated = 2;
optional GeneratorNameEnum generatorNameType = 3;
optional PersistableQuestion persistableQuestion = 4;
repeated PersistableQuestion options = 5;
optional PersistableHeader persistableHeader = 6;
enum GeneratorNameEnum {
GENERATOR_CONDITIONAL_CANADA_AUTO = 0;
GENERATOR_CONDITIONAL_CANADA_GAS_CARD = 1;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 2;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LOAN = 3;
GENERATOR_CONDITIONAL_CANADA_STUDENT_LOAN = 4;
GENERATOR_FICTIONAL_CANADA_AUTO = 5;
GENERATOR_FICTIONAL_CANADA_GAS_CARD = 6;
GENERATOR_FICTIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 7;
...
...
}
message PersistableHeader {
optional TextParametrization text = 1;
}
message TextParametrization{
optional string asString = 1;
optional string fixedText = 2;
repeated Parameter parameter =3;
optional InteractiveQueryResourceEnum resourceId = 4;
message Parameter{
optional string value = 1;
}
enum InteractiveQueryResourceEnum{
MONTH_1 = 0;
MONTH_2 = 1;
MONTH_3 = 2;
MONTH_4 = 3;
MONTH_5 = 4;
MONTH_6 = 5;
MONTH_7 = 6;
MONTH_8 = 7;
...
..
}
}
message PersistableQuestion{
optional bool real = 1;
repeated PersistableChoice coices = 2;
optional QuestionType questionType = 3;
optional PersistableHeader persistableHeader = 4;
optional TextParametrization text = 5;
message PersistableChoice{
optional bool correct = 1;
optional int32 sortOrder = 2;
optional string simpleText = 3;
optional TextParametrization resourceTextValue = 4;
}
enum QuestionType{
US_MORTGAGE_PROVIDER_REAL = 0;
US_MORTGAGE_PROVIDER_FICTIONAL = 1;
US_MORTGAGE_PAYMENT_REAL = 2;
US_MORTGAGE_PAYMENT_FICTIONAL = 3;
US_AUTO_PROVIDER_REAL = 4;
US_AUTO_PROVIDER_FICTIONAL = 5;
US_AUTO_PAYMENT_REAL = 6;
...
...
}
}
}
protofile---
To reduce the verbosity I have not included all the elements of my
Enums. But as you can see I have large enums. I have included the
option 'option optimize_for = SPEED;' to optimize the proto buffer
for speed, but in vain. The steps I performed are :
1) Compiled the proto fileagainst the protobuf binary to generate the
source.
2) Built my proto message Objects based on the source and the POJO.
3)Serialized the proto message objects
4) Persisted proto message objects to the the Db.
5) Retrieved the blob, desirialized to the proto message object.
6) Built the pojo back from the proto message object.
Steps 2 + 3 +4 are found to be more performance expensive than if I
use java serialization to do the same, same for the retrieval
process.
Is there anything which Iam missing? How do I optimize the proto
buffer to achieve
Re: proto with java c++ python
On Thu, Sep 10, 2009 at 12:51 AM, SuKai sukai090...@gmail.com wrote: In this page http://code.google.com/intl/zh-CN/apis/protocolbuffers/docs/cpptutorial.html, Parsing and Serialization section, * |bool SerializeToString(string* output) const;|: serializes the message and stores the bytes in the given string. Note that the bytes are binary, not text; we only use the |string| class as a convenient container. * |bool ParseFromString(const string data);|: parses a message from the given string. I cannot find these method in V2.2. They are defined on the super-class google::protobuf::MessageLite. http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.message_lite.html#MessageLite Not all methods of protocol message objects are defined in the generated code. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Performance Observations using protocol and java serialization
Kent I did bench mark only those steps(parsing and serialization
withthe size of encoded message), the code is not all that vast just
dint want to paste it here due to readability issues..will email it
to your account..Thanks a lot for your response kent
On Sep 10, 11:08 am, Kenton Varda ken...@google.com wrote:
That would be my address, yes. But if this is a large amount of code, I
don't have time to go through it. You really need to narrow it down to
something small which can be easily debugged.
I recommend writing a benchmark which times *only* the parsing and
serialization steps, with protocol buffers and with java serialization,
without any database communication. This would simplify things and would
prove that the problem isn't in the database layer. Once you have that, it
should be a fairly small amount of code, and you should be able to send it
to this list without too much trouble. Please also include your results
with the code so that we don't have to actually run it. Also, have your
benchmark measure the sizes of the encoded messages.
On Thu, Sep 10, 2009 at 7:55 AM, rajesh poorv...@gmail.com wrote:
Hi Kent,
Thanks for the quick reply, It might get really clumsy
if I post the code here..lemme email it to you..can you please provide
me youremail address to which I can send the code to. I have a
'ken...@google.com' on your profile , do you want me to email it to
this id.
On Sep 9, 7:45 pm, Kenton Varda ken...@google.com wrote:
Since you haven't provided any code it's hard to speculate on what may be
going wrong.
On Wed, Sep 9, 2009 at 3:01 PM, rajesh poorv...@gmail.com wrote:
Hi All,
I ran some performance tests to compare the performance for
serializing-persisting-retrieving-desirializing of my POJOs and was
really surprised to see that java-serialization was performing better
than protobuf-serialization. Iam enclosing my proto file based on
which Iam generating my proto buffer messages.
--- protofile---
option java_package = com.equifax.ic.eid.iq.persist.generated;
option java_outer_classname = AllQuestionPackImplProtos;
option optimize_for = SPEED;
message AllQuestionPacksImpl{
optional string transid = 1;
repeated QuestionPack realQuestionPacks = 2;
repeated QuestionPack simulatedQuestionPacks = 3;
}
message QuestionPack{
optional string transkey = 1;
optional bool simulated = 2;
optional GeneratorNameEnum generatorNameType = 3;
optional PersistableQuestion persistableQuestion = 4;
repeated PersistableQuestion options = 5;
optional PersistableHeader persistableHeader = 6;
enum GeneratorNameEnum {
GENERATOR_CONDITIONAL_CANADA_AUTO = 0;
GENERATOR_CONDITIONAL_CANADA_GAS_CARD = 1;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 2;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LOAN = 3;
GENERATOR_CONDITIONAL_CANADA_STUDENT_LOAN = 4;
GENERATOR_FICTIONAL_CANADA_AUTO = 5;
GENERATOR_FICTIONAL_CANADA_GAS_CARD = 6;
GENERATOR_FICTIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 7;
...
...
}
message PersistableHeader {
optional TextParametrization text = 1;
}
message TextParametrization{
optional string asString = 1;
optional string fixedText = 2;
repeated Parameter parameter =3;
optional InteractiveQueryResourceEnum resourceId = 4;
message Parameter{
optional string value = 1;
}
enum InteractiveQueryResourceEnum{
MONTH_1 = 0;
MONTH_2 = 1;
MONTH_3 = 2;
MONTH_4 = 3;
MONTH_5 = 4;
MONTH_6 = 5;
MONTH_7 = 6;
MONTH_8 = 7;
...
..
}
}
message PersistableQuestion{
optional bool real = 1;
repeated PersistableChoice coices = 2;
optional QuestionType questionType = 3;
optional PersistableHeader persistableHeader = 4;
optional TextParametrization text = 5;
message PersistableChoice{
optional bool correct = 1;
optional int32 sortOrder = 2;
optional string simpleText = 3;
optional TextParametrization resourceTextValue = 4;
}
enum QuestionType{
US_MORTGAGE_PROVIDER_REAL = 0;
US_MORTGAGE_PROVIDER_FICTIONAL = 1;
US_MORTGAGE_PAYMENT_REAL = 2;
US_MORTGAGE_PAYMENT_FICTIONAL = 3;
US_AUTO_PROVIDER_REAL = 4;
US_AUTO_PROVIDER_FICTIONAL = 5;
US_AUTO_PAYMENT_REAL = 6;
...
...
}
}
}
protofile---
To reduce the verbosity I have not included all the elements of my
Enums. But as you can see I have large enums. I have included the
option 'option optimize_for = SPEED;' to optimize the proto buffer
for speed, but in vain. The steps I performed are :
1) Compiled the proto fileagainst the protobuf binary to generate the
source.
2) Built my proto message Objects based on the source and the POJO.
3)Serialized the proto message objects
4)
Re: protoc feature question
Hi, On Thu, Sep 10, 2009 at 4:16 PM, George Georgiev georgi.georg...@citrix.com wrote: Hi, Is there a way to tell to protoc to recompile the files only if the output files are missing or are older than the input file? Isn't this what makefiles are for ? -h --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protoc feature question
Hi, On Thu, Sep 10, 2009 at 4:26 PM, George Georgiev georgi.georg...@citrix.com wrote: Sure, but it is really hard for me to prepare something general since the java output files are located in a different folders based on the package specified in the proto file. I think it will be easier if the protoc provides this feature. mmh, I understand. And if you do it from an ant-buildfile (often used in java projects) then it would be as well hard as ant has not really a notion of file-dependencies (yeah, ant sucks as build-tool). So I guess the best way is to add this feature in your local version of protoc (say, switched on with a commandline flag) and send a patch to this list for review and to be integrated in the mainline. -h Thanks -Original Message- From: henner.zel...@googlemail.com [mailto:henner.zel...@googlemail.com] On Behalf Of Henner Zeller Sent: Thursday, September 10, 2009 4:21 PM To: George Georgiev Cc: Protocol Buffers Subject: Re: protoc feature question Hi, On Thu, Sep 10, 2009 at 4:16 PM, George Georgiev georgi.georg...@citrix.com wrote: Hi, Is there a way to tell to protoc to recompile the files only if the output files are missing or are older than the input file? Isn't this what makefiles are for ? -h --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protoc feature question
On Thu, Sep 10, 2009 at 4:36 PM, Henner Zeller h.zel...@acm.org wrote: Hi, On Thu, Sep 10, 2009 at 4:26 PM, George Georgiev georgi.georg...@citrix.com wrote: Sure, but it is really hard for me to prepare something general since the java output files are located in a different folders based on the package specified in the proto file. I think it will be easier if the protoc provides this feature. mmh, I understand. I don't. Why should be taking on responsibilities of the build system in protoc? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Strange Eclipse compiler error.
Doesn't make any sense to me. Let me know if you find a solution. On Mon, Sep 7, 2009 at 8:16 AM, Brice Figureau brice...@daysofwonder.combrice%2...@daysofwonder.com wrote: Hmm, more information on the issue below: On Mon, 2009-09-07 at 16:52 +0200, Brice Figureau wrote: I'm using eclipse (3.5) on a large project using protobuf v2.1.0, and I'm encountering tons of strange Eclipse java compiler error on every protobuf java generated code. All the errors looks like this one: The method mergeFrom(Message) is ambiguous for the type GeneratedMessage.BuilderBot.PBBot.Builder I created a new java eclipse project and added only the generated java files, and they do compile fine, so the issue is certainly the compilation of the combination of those generated files and my other java files... But I'd appreciate any help... -- Brice Figureau My Blog: http://www.masterzen.fr/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protoc feature question
Hi, On Thu, Sep 10, 2009 at 4:51 PM, Kenton Varda ken...@google.com wrote: On Thu, Sep 10, 2009 at 4:36 PM, Henner Zeller h.zel...@acm.org wrote: Hi, On Thu, Sep 10, 2009 at 4:26 PM, George Georgiev georgi.georg...@citrix.com wrote: Sure, but it is really hard for me to prepare something general since the java output files are located in a different folders based on the package specified in the proto file. I think it will be easier if the protoc provides this feature. mmh, I understand. I don't. Why should be taking on responsibilities of the build system in protoc? I once had to program in java and the build tools available in that world are really crappy. So from that point of view I can understand the pain. Maybe if someone wrote an ant-task for protoc then this could take care of it. -h --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protoc feature question
On Thu, Sep 10, 2009 at 4:59 PM, George Georgiev georgi.georg...@citrix.com wrote: So you suggest if I change the java package in a proto file, I to update the makefile too? That seems reasonable to me. If you changed the package of any of your other Java files it would mean moving source code around which would certainly require updating the makefile. Another idea is to have protoc output the .java files into a temporary directory, and then jar up that whole directory, and consider the .jar to be the output of the operation. That way you have just one output and it's the same regardless of the content of the .proto files. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protoc feature question
Henner Zeller wrote: And if you do it from an ant-buildfile (often used in java projects) then it would be as well hard as ant has not really a notion of file-dependencies (yeah, ant sucks as build-tool). Perhaps you are looking for uptodate or dependset -O --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: protoc feature question
That seems reasonable to me. If you changed the package of any of your other Java files it would mean moving source code around which would certainly require updating the makefile. for me too, but some colleagues find this as extra pain Another idea is to have protoc output the .java files into a temporary directory, and then jar up that whole directory, and consider the .jar to be the output of the operation. That way you have just one output and it's the same regardless of the content of the .proto files. this means I will need to rebuild the whole jar file if any of the proto files is changed It looks like my use case isn't usual, but the way I'm using protobufs involves permanent updates of the proto files. Anyway, I could live with it as is. Thanks, George From: Kenton Varda [mailto:ken...@google.com] Sent: Thursday, September 10, 2009 5:45 PM To: George Georgiev; Protocol Buffers Subject: Re: protoc feature question On Thu, Sep 10, 2009 at 4:59 PM, George Georgiev georgi.georg...@citrix.commailto:georgi.georg...@citrix.com wrote: So you suggest if I change the java package in a proto file, I to update the makefile too? That seems reasonable to me. If you changed the package of any of your other Java files it would mean moving source code around which would certainly require updating the makefile. Another idea is to have protoc output the .java files into a temporary directory, and then jar up that whole directory, and consider the .jar to be the output of the operation. That way you have just one output and it's the same regardless of the content of the .proto files. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
