Re: Performance Observations using protocol and java serialization
That would be my address, yes. But if this is a large amount of code, I
don't have time to go through it. You really need to narrow it down to
something small which can be easily debugged.
I recommend writing a benchmark which times *only* the parsing and
serialization steps, with protocol buffers and with java serialization,
without any database communication. This would simplify things and would
prove that the problem isn't in the database layer. Once you have that, it
should be a fairly small amount of code, and you should be able to send it
to this list without too much trouble. Please also include your results
with the code so that we don't have to actually run it. Also, have your
benchmark measure the sizes of the encoded messages.
On Thu, Sep 10, 2009 at 7:55 AM, rajesh poorv...@gmail.com wrote:
Hi Kent,
Thanks for the quick reply, It might get really clumsy
if I post the code here..lemme email it to you..can you please provide
me youremail address to which I can send the code to. I have a
'ken...@google.com' on your profile , do you want me to email it to
this id.
On Sep 9, 7:45 pm, Kenton Varda ken...@google.com wrote:
Since you haven't provided any code it's hard to speculate on what may be
going wrong.
On Wed, Sep 9, 2009 at 3:01 PM, rajesh poorv...@gmail.com wrote:
Hi All,
I ran some performance tests to compare the performance for
serializing-persisting-retrieving-desirializing of my POJOs and was
really surprised to see that java-serialization was performing better
than protobuf-serialization. Iam enclosing my proto file based on
which Iam generating my proto buffer messages.
--- protofile---
option java_package = com.equifax.ic.eid.iq.persist.generated;
option java_outer_classname = AllQuestionPackImplProtos;
option optimize_for = SPEED;
message AllQuestionPacksImpl{
optional string transid = 1;
repeated QuestionPack realQuestionPacks = 2;
repeated QuestionPack simulatedQuestionPacks = 3;
}
message QuestionPack{
optional string transkey = 1;
optional bool simulated = 2;
optional GeneratorNameEnum generatorNameType = 3;
optional PersistableQuestion persistableQuestion = 4;
repeated PersistableQuestion options = 5;
optional PersistableHeader persistableHeader = 6;
enum GeneratorNameEnum {
GENERATOR_CONDITIONAL_CANADA_AUTO = 0;
GENERATOR_CONDITIONAL_CANADA_GAS_CARD = 1;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 2;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LOAN = 3;
GENERATOR_CONDITIONAL_CANADA_STUDENT_LOAN = 4;
GENERATOR_FICTIONAL_CANADA_AUTO = 5;
GENERATOR_FICTIONAL_CANADA_GAS_CARD = 6;
GENERATOR_FICTIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 7;
...
...
}
message PersistableHeader {
optional TextParametrization text = 1;
}
message TextParametrization{
optional string asString = 1;
optional string fixedText = 2;
repeated Parameter parameter =3;
optional InteractiveQueryResourceEnum resourceId = 4;
message Parameter{
optional string value = 1;
}
enum InteractiveQueryResourceEnum{
MONTH_1 = 0;
MONTH_2 = 1;
MONTH_3 = 2;
MONTH_4 = 3;
MONTH_5 = 4;
MONTH_6 = 5;
MONTH_7 = 6;
MONTH_8 = 7;
...
..
}
}
message PersistableQuestion{
optional bool real = 1;
repeated PersistableChoice coices = 2;
optional QuestionType questionType = 3;
optional PersistableHeader persistableHeader = 4;
optional TextParametrization text = 5;
message PersistableChoice{
optional bool correct = 1;
optional int32 sortOrder = 2;
optional string simpleText = 3;
optional TextParametrization resourceTextValue = 4;
}
enum QuestionType{
US_MORTGAGE_PROVIDER_REAL = 0;
US_MORTGAGE_PROVIDER_FICTIONAL = 1;
US_MORTGAGE_PAYMENT_REAL = 2;
US_MORTGAGE_PAYMENT_FICTIONAL = 3;
US_AUTO_PROVIDER_REAL = 4;
US_AUTO_PROVIDER_FICTIONAL = 5;
US_AUTO_PAYMENT_REAL = 6;
...
...
}
}
}
protofile---
To reduce the verbosity I have not included all the elements of my
Enums. But as you can see I have large enums. I have included the
option 'option optimize_for = SPEED;' to optimize the proto buffer
for speed, but in vain. The steps I performed are :
1) Compiled the proto fileagainst the protobuf binary to generate the
source.
2) Built my proto message Objects based on the source and the POJO.
3)Serialized the proto message objects
4) Persisted proto message objects to the the Db.
5) Retrieved the blob, desirialized to the proto message object.
6) Built the pojo back from the proto message object.
Steps 2 + 3 +4 are found to be more performance expensive than if I
use java serialization to do the same, same for the retrieval
process.
Is there anything which Iam missing? How do I optimize the proto
buffer to achieve
Re: Performance Observations using protocol and java serialization
Kent I did bench mark only those steps(parsing and serialization
withthe size of encoded message), the code is not all that vast just
dint want to paste it here due to readability issues..will email it
to your account..Thanks a lot for your response kent
On Sep 10, 11:08 am, Kenton Varda ken...@google.com wrote:
That would be my address, yes. But if this is a large amount of code, I
don't have time to go through it. You really need to narrow it down to
something small which can be easily debugged.
I recommend writing a benchmark which times *only* the parsing and
serialization steps, with protocol buffers and with java serialization,
without any database communication. This would simplify things and would
prove that the problem isn't in the database layer. Once you have that, it
should be a fairly small amount of code, and you should be able to send it
to this list without too much trouble. Please also include your results
with the code so that we don't have to actually run it. Also, have your
benchmark measure the sizes of the encoded messages.
On Thu, Sep 10, 2009 at 7:55 AM, rajesh poorv...@gmail.com wrote:
Hi Kent,
Thanks for the quick reply, It might get really clumsy
if I post the code here..lemme email it to you..can you please provide
me youremail address to which I can send the code to. I have a
'ken...@google.com' on your profile , do you want me to email it to
this id.
On Sep 9, 7:45 pm, Kenton Varda ken...@google.com wrote:
Since you haven't provided any code it's hard to speculate on what may be
going wrong.
On Wed, Sep 9, 2009 at 3:01 PM, rajesh poorv...@gmail.com wrote:
Hi All,
I ran some performance tests to compare the performance for
serializing-persisting-retrieving-desirializing of my POJOs and was
really surprised to see that java-serialization was performing better
than protobuf-serialization. Iam enclosing my proto file based on
which Iam generating my proto buffer messages.
--- protofile---
option java_package = com.equifax.ic.eid.iq.persist.generated;
option java_outer_classname = AllQuestionPackImplProtos;
option optimize_for = SPEED;
message AllQuestionPacksImpl{
optional string transid = 1;
repeated QuestionPack realQuestionPacks = 2;
repeated QuestionPack simulatedQuestionPacks = 3;
}
message QuestionPack{
optional string transkey = 1;
optional bool simulated = 2;
optional GeneratorNameEnum generatorNameType = 3;
optional PersistableQuestion persistableQuestion = 4;
repeated PersistableQuestion options = 5;
optional PersistableHeader persistableHeader = 6;
enum GeneratorNameEnum {
GENERATOR_CONDITIONAL_CANADA_AUTO = 0;
GENERATOR_CONDITIONAL_CANADA_GAS_CARD = 1;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 2;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LOAN = 3;
GENERATOR_CONDITIONAL_CANADA_STUDENT_LOAN = 4;
GENERATOR_FICTIONAL_CANADA_AUTO = 5;
GENERATOR_FICTIONAL_CANADA_GAS_CARD = 6;
GENERATOR_FICTIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 7;
...
...
}
message PersistableHeader {
optional TextParametrization text = 1;
}
message TextParametrization{
optional string asString = 1;
optional string fixedText = 2;
repeated Parameter parameter =3;
optional InteractiveQueryResourceEnum resourceId = 4;
message Parameter{
optional string value = 1;
}
enum InteractiveQueryResourceEnum{
MONTH_1 = 0;
MONTH_2 = 1;
MONTH_3 = 2;
MONTH_4 = 3;
MONTH_5 = 4;
MONTH_6 = 5;
MONTH_7 = 6;
MONTH_8 = 7;
...
..
}
}
message PersistableQuestion{
optional bool real = 1;
repeated PersistableChoice coices = 2;
optional QuestionType questionType = 3;
optional PersistableHeader persistableHeader = 4;
optional TextParametrization text = 5;
message PersistableChoice{
optional bool correct = 1;
optional int32 sortOrder = 2;
optional string simpleText = 3;
optional TextParametrization resourceTextValue = 4;
}
enum QuestionType{
US_MORTGAGE_PROVIDER_REAL = 0;
US_MORTGAGE_PROVIDER_FICTIONAL = 1;
US_MORTGAGE_PAYMENT_REAL = 2;
US_MORTGAGE_PAYMENT_FICTIONAL = 3;
US_AUTO_PROVIDER_REAL = 4;
US_AUTO_PROVIDER_FICTIONAL = 5;
US_AUTO_PAYMENT_REAL = 6;
...
...
}
}
}
protofile---
To reduce the verbosity I have not included all the elements of my
Enums. But as you can see I have large enums. I have included the
option 'option optimize_for = SPEED;' to optimize the proto buffer
for speed, but in vain. The steps I performed are :
1) Compiled the proto fileagainst the protobuf binary to generate the
source.
2) Built my proto message Objects based on the source and the POJO.
3)Serialized the proto message objects
4)
Re: Performance Observations using protocol and java serialization
Since you haven't provided any code it's hard to speculate on what may be
going wrong.
On Wed, Sep 9, 2009 at 3:01 PM, rajesh poorv...@gmail.com wrote:
Hi All,
I ran some performance tests to compare the performance for
serializing-persisting-retrieving-desirializing of my POJOs and was
really surprised to see that java-serialization was performing better
than protobuf-serialization. Iam enclosing my proto file based on
which Iam generating my proto buffer messages.
--- protofile---
option java_package = com.equifax.ic.eid.iq.persist.generated;
option java_outer_classname = AllQuestionPackImplProtos;
option optimize_for = SPEED;
message AllQuestionPacksImpl{
optional string transid = 1;
repeated QuestionPack realQuestionPacks = 2;
repeated QuestionPack simulatedQuestionPacks = 3;
}
message QuestionPack{
optional string transkey = 1;
optional bool simulated = 2;
optional GeneratorNameEnum generatorNameType = 3;
optional PersistableQuestion persistableQuestion = 4;
repeated PersistableQuestion options = 5;
optional PersistableHeader persistableHeader = 6;
enum GeneratorNameEnum {
GENERATOR_CONDITIONAL_CANADA_AUTO = 0;
GENERATOR_CONDITIONAL_CANADA_GAS_CARD = 1;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 2;
GENERATOR_CONDITIONAL_CANADA_PERSONAL_LOAN = 3;
GENERATOR_CONDITIONAL_CANADA_STUDENT_LOAN = 4;
GENERATOR_FICTIONAL_CANADA_AUTO = 5;
GENERATOR_FICTIONAL_CANADA_GAS_CARD = 6;
GENERATOR_FICTIONAL_CANADA_PERSONAL_LINE_OF_CREDIT = 7;
...
...
}
message PersistableHeader {
optional TextParametrization text = 1;
}
message TextParametrization{
optional string asString = 1;
optional string fixedText = 2;
repeated Parameter parameter =3;
optional InteractiveQueryResourceEnum resourceId = 4;
message Parameter{
optional string value = 1;
}
enum InteractiveQueryResourceEnum{
MONTH_1 = 0;
MONTH_2 = 1;
MONTH_3 = 2;
MONTH_4 = 3;
MONTH_5 = 4;
MONTH_6 = 5;
MONTH_7 = 6;
MONTH_8 = 7;
...
..
}
}
message PersistableQuestion{
optional bool real = 1;
repeated PersistableChoice coices = 2;
optional QuestionType questionType = 3;
optional PersistableHeader persistableHeader = 4;
optional TextParametrization text = 5;
message PersistableChoice{
optional bool correct = 1;
optional int32 sortOrder = 2;
optional string simpleText = 3;
optional TextParametrization resourceTextValue = 4;
}
enum QuestionType{
US_MORTGAGE_PROVIDER_REAL = 0;
US_MORTGAGE_PROVIDER_FICTIONAL = 1;
US_MORTGAGE_PAYMENT_REAL = 2;
US_MORTGAGE_PAYMENT_FICTIONAL = 3;
US_AUTO_PROVIDER_REAL = 4;
US_AUTO_PROVIDER_FICTIONAL = 5;
US_AUTO_PAYMENT_REAL = 6;
...
...
}
}
}
protofile---
To reduce the verbosity I have not included all the elements of my
Enums. But as you can see I have large enums. I have included the
option 'option optimize_for = SPEED;' to optimize the proto buffer
for speed, but in vain. The steps I performed are :
1) Compiled the proto fileagainst the protobuf binary to generate the
source.
2) Built my proto message Objects based on the source and the POJO.
3)Serialized the proto message objects
4) Persisted proto message objects to the the Db.
5) Retrieved the blob, desirialized to the proto message object.
6) Built the pojo back from the proto message object.
Steps 2 + 3 +4 are found to be more performance expensive than if I
use java serialization to do the same, same for the retrieval
process.
Is there anything which Iam missing? How do I optimize the proto
buffer to achieve performance.
Any insights/ help on this is highly appreciated.
Thanks in advance.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
