Thanks Louis, Thanks for reviewing my code also , yes I am going to remove the need of column names per response. The tests I am doing are on the same machine as of now.
Regarding corking mechanism, I see it is planned for 1.1, Do you have any idea when that will be available. ? mean time, If I implemented my scanning RPC server method like flowControlledStreaming in AbstractBenchmark Do you think that will help to get better throughput with large numbers of small messages in a stream ? Thanks Avinash On Wednesday, October 5, 2016 at 4:58:23 AM UTC+5:30, Louis Ryan wrote: > > Have you tested the bandwidth between the two machines using Netperf ? > > You handling of streaming using isReady/onReady is good, we have an > improvement planned to add 'corking' > https://github.com/grpc/grpc-java/issues/994 which can substantially > improve throughput when dealing with large numbers of small messages in a > stream. > > One thing I notice between your two APIs is that in the more structure one > you send the column name for every column value which seems quite > inefficient > > On Tue, Oct 4, 2016 at 1:31 AM, Avinash Dongre <[email protected] > <javascript:>> wrote: > >> Hi Louis, >> >> Thanks, I have made the changes and I see good numbers with async client. >> I have committed the change in my repo. >> >> But I still see very low data transfer rate. What I have observed is that >> when I have big data say ( 1024 bytes ) Then I get rate on the same machine >> around 900-1000 MegaBytes/seconds. >> >> Thanks >> Avinash >> >> Num Of Rows -> 2000000 >> Total Data Size -> 512000000 >> >> Time -> 7.484403756 Seconds >> Data Rate -> 65.20225470316008 MBps >> >> Time -> 5.776407626 Seconds >> Data Rate -> 84.48157256137519 MBps >> >> Time -> 6.010102214 Seconds >> Data Rate -> 81.19662239075524 MBps >> >> Time -> 5.401799312 Seconds >> Data Rate -> 90.34026845757057 MBps >> >> Time -> 5.352594307 Seconds >> Data Rate -> 91.17074300994655 MBps >> >> >> >> >> On Tuesday, October 4, 2016 at 4:48:50 AM UTC+5:30, Louis Ryan wrote: >>> >>> I see you're using a blocking stub to make the streaming calls. I >>> suggest you switch to using an async callback instead here >>> >>> >>> https://github.com/davinash/grpc-bench/blob/master/bench/src/main/java/io/adongre/grpc/formatted/ScanFormattedClient.java#L35 >>> >>> E.g. >>> >>> asyncStub.formatedScan(ScanRequest.newBuilder()..., new >>> StreamObserver() { ... public void onValue( ScanFormattedResponse >>> response) { .... do stuff with response. i.e. replace the while loop in >>> your code } }); >>> .setNumOfColumns(NUMBER_OF_COLUMNS) >>> .setNumOfRows(NUM_OF_ROWS) >>> .setSizeOfEachColumn(SIZE_OF_EACH_COLUMN) >>> .build()); >>> >>> On Sun, Oct 2, 2016 at 3:16 AM, Avinash Dongre <[email protected]> >>> wrote: >>> >>>> Thanks Louis, For helping me here >>>> >>>> Sorry for delayed response >>>> >>>> I am sure I am doing something wrong in my code, but could not figure >>>> out yet. >>>> >>>> I have checked-in my benchmark project here at >>>> https://github.com/davinash/grpc-bench >>>> >>>> Raw Server , client and impl can be found at >>>> https://github.com/davinash/grpc-bench/tree/master/bench/src/main/java/io/adongre/grpc/raw >>>> >>>> Formatted Server, client and Impl can be found at >>>> https://github.com/davinash/grpc-bench/tree/master/bench/src/main/java/io/adongre/grpc/formatted >>>> >>>> With this above code for Raw I get following numbers >>>> >>>> Num Of Rows -> 2000000 >>>> Time -> 68.468212235 Seconds >>>> Total Data Size -> 65536000000 >>>> Data Rate -> 912.8323635132227 MBps >>>> >>>> >>>> >>>> And with formatted one I get following numbers. >>>> >>>> Num Of Rows -> 2000000 >>>> Time -> 358.549532039 Seconds >>>> Total Data Size -> 81952000000 >>>> Data Rate -> 217.97546228982097 MBps >>>> >>>> >>>> Thanks >>>> Avinash >>>> >>>> >>>> >>>> >>>> >>>> On Monday, September 26, 2016 at 10:08:47 PM UTC+5:30, Louis Ryan wrote: >>>>> >>>>> I see that you're using a steaming RPC. Can you send the code from the >>>>> server and client. How you interact with flow control can greatly affect >>>>> performance >>>>> >>>>> -louis (from phone) >>>>> >>>>> On Sep 26, 2016 3:57 AM, "Avinash Dongre" <[email protected]> >>>>> wrote: >>>>> >>>>> Hi All, >>>>> Please help. >>>>> >>>>> Thanks >>>>> Avinash >>>>> >>>>> >>>>> On Saturday, September 24, 2016 at 12:01:02 PM UTC+5:30, Avinash >>>>> Dongre wrote: >>>>>> >>>>>> >>> Now I get around 130-135 MegaBytes/Seconds Speed. >>>>>> >>>>>> This result is on the Same Machine. i.e. gRPC Client and gRPC Servers >>>>>> are running on the same machine. >>>>>> >>>>>> >>>>>> >>>>>> On Saturday, September 24, 2016 at 11:59:53 AM UTC+5:30, Avinash >>>>>> Dongre wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Thanks all for replying/resolving my previous doubts. I am planning >>>>>>> to use gRPC for our project. I have following question related to >>>>>>> performance. >>>>>>> >>>>>>> If I have following proto definition. >>>>>>> >>>>>>> option optimize_for = SPEED; >>>>>>> message ScanRow { >>>>>>> repeated bytes row = 1; >>>>>>> } >>>>>>> message ScanResult { >>>>>>> repeated ScanRow row = 1; >>>>>>> } >>>>>>> message ScanRequest { >>>>>>> int32 numOfColumns = 1; >>>>>>> int32 sizeOfEachColumn = 2; >>>>>>> int64 numOfRows = 3; >>>>>>> int32 batchSize = 4; >>>>>>> } >>>>>>> service ScanService { >>>>>>> rpc Scan (ScanRequest) returns (stream ScanResult) {} >>>>>>> } >>>>>>> >>>>>>> I get around 900-1000 MegaBytes/Seconds on Single Machine and across >>>>>>> two different Physical Machines I am getting 100-110 MegaBytes/Seconds >>>>>>> speed. >>>>>>> >>>>>>> If I change above proto definition to following. >>>>>>> >>>>>>> option optimize_for = SPEED; >>>>>>> message ColumnValues { >>>>>>> bytes columnName = 1; >>>>>>> bytes columnValue = 2; >>>>>>> } >>>>>>> message ScanRow { >>>>>>> int64 rowId = 1; >>>>>>> int64 timeStamp = 2; >>>>>>> repeated ColumnValues columnValue = 3; >>>>>>> } >>>>>>> message ScanResult { >>>>>>> repeated ScanRow row = 1; >>>>>>> } >>>>>>> message ScanRequest { >>>>>>> int32 numOfColumns = 1; >>>>>>> int32 sizeOfEachColumn = 2; >>>>>>> int64 numOfRows = 3; >>>>>>> int32 batchSize = 4; >>>>>>> } >>>>>>> service ScanService { >>>>>>> rpc Scan (ScanRequest) returns (stream ScanResult) {} >>>>>>> } >>>>>>> >>>>>>> Now I get around 130-135 MegaBytes/Seconds Speed. >>>>>>> >>>>>>> >>>>>>> Why it is slow with kind of multi-level proto files. Is there any >>>>>>> Serialization/De-Serialization overhead with this ? >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Avinash >>>>>>> >>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "grpc.io" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/grpc-io/cc74ecdf-3eac-4475-bad0-23fd66ee03b2%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/grpc-io/cc74ecdf-3eac-4475-bad0-23fd66ee03b2%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "grpc.io" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/grpc-io. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/grpc-io/76dcf498-35e2-4e5a-ab50-92e708cdb808%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/grpc-io/76dcf498-35e2-4e5a-ab50-92e708cdb808%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "grpc.io" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/grpc-io. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/99752304-bbb3-4a6b-8e43-225629408e76%40googlegroups.com >> >> <https://groups.google.com/d/msgid/grpc-io/99752304-bbb3-4a6b-8e43-225629408e76%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/b5ce79c1-d47c-4fe8-9035-25e7f43241b0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
