Answered via github: https://github.com/grpc/grpc-go/issues/1228
On Tuesday, May 9, 2017 at 2:18:55 AM UTC-7, Zeymo Wang wrote: > > > I customed grpc-go as gatway proxy which remove IDL and implements > self-TLS1.3 with `cgo` for authenticor to proxy binary-grpc-call forwad to > inner binary-RPC.In production I find some unusual phenomenon when profile > > KVM 4core + 8G + Intel Xeon E3-12xx v2 (Ivy Bridge) 2099Mhz > > connections when profile : 7500 > > 1. it seem much more than I think per conn mem ( 57k/connetion) > > 478: 15663104 [13261: 434536448] @ 0x6eadfa 0x6e39d0 0x6ed51f 0x770e03 > 0x770cde 0x45ce11 > # 0x6eadf9 mwcs/http/h2c/transport.newFramer+0x309 > /src/src/mwcs/http/h2c/transport/http2_util.go:196 > > > 2. gzip malloc much more heap and occupy much more cpu in a periodic time > > 5: 3317760 [1546645: 1026279383040] @ 0x602b51 0x60e903 0x6ef4dc 0x773389 > 0x846da6 0x7d3de3 0x7c8f26 0x77969c 0x77adc0 0x7788e2 0x771c31 0x7717b6 > 0x77667c 0x45ce11 > # 0x602b50 compress/flate.NewWriter+0x30 > /usr/local/go/src/compress/flate/deflate.go:667 > > > dstat > > > 19 1 79 0 0 0| 0 76k|2086k 1021k| 0 0 |9740 9289 > 4 1 95 0 0 1| 0 80k|1974k 981k| 0 0 | 11k 12k > 6 2 92 0 0 0| 0 136k|2220k 1070k| 0 0 | 11k 12k > 3 1 95 0 0 0| 0 104k|2303k 1082k| 0 0 | 11k 11k > 3 1 95 0 0 0| 0 88k|2121k 1015k| 0 0 | 11k 11k > 3 1 96 0 0 0| 0 72k|1723k 855k| 0 0 | 10k 10k > 3 2 95 0 0 1| 0 84k|2291k 956k| 0 0 | 10k 11k > 3 1 96 0 0 0| 0 80k|2107k 983k| 0 0 | 11k 11k > 3 1 95 0 0 0| 0 52k|2273k 1088k| 0 0 | 11k 12k > 3 1 95 0 0 1| 0 80k|2451k 1065k| 0 0 | 11k 12k > 4 1 95 0 0 0| 0 308k|2059k 1016k| 0 0 | 11k 12k > 4 2 94 0 0 0| 0 44k|2256k 1020k| 0 0 | 11k 12k > 5 3 92 0 0 1| 0 88k|1947k 892k| 0 0 | 11k 12k > 4 1 95 0 0 0| 0 128k|2278k 1102k| 0 0 | 12k 13k > 4 1 95 0 0 0| 0 48k|2290k 1002k| 0 0 | 11k 12k > 3 2 95 0 0 1| 0 76k|2674k 1106k| 0 0 | 11k 12k > 3 2 96 0 0 1| 0 80k|1758k 884k| 0 0 | 10k 11k > 3 1 96 0 0 1| 0 112k|1641k 868k| 0 0 | 11k 11k > 20 1 79 0 0 0| 0 48k|1699k 939k| 0 0 |9811 9175 > 3 1 95 0 0 0| 0 84k|2098k 1003k| 0 0 | 11k 12k > 3 1 95 0 0 0| 0 88k|1966k 990k| 0 0 | 11k 11k > > > when i use sync.Pool of *gzip.Writer, the periodic time become more frequently > > > dstat > > > 20 1 78 0 0 1| 0 80k|1950k 774k| 0 0 |9627 9141 > 6 2 91 0 0 1| 0 88k|1688k 955k| 0 0 | 11k 12k > 5 2 93 0 0 1| 0 96k|2167k 1027k| 0 0 | 11k 12k > 6 2 92 0 0 1| 0 52k|2088k 969k| 0 0 | 11k 12k > 6 2 92 0 0 1| 0 88k|1813k 970k| 0 0 | 11k 12k > 5 2 93 0 0 1| 0 124k|2386k 1030k| 0 0 | 11k 12k > 5 2 93 0 0 1| 0 48k|2217k 900k| 0 0 | 10k 11k > 11 3 85 0 0 1| 0 88k|2461k 1042k| 0 0 | 11k 12k > 5 2 92 0 0 1| 0 92k|1924k 956k| 0 0 | 11k 12k > 23 2 75 0 0 1| 0 84k|1886k 839k| 0 0 |9515 8898 > 4 2 93 0 0 1| 0 72k|1687k 830k| 0 0 | 10k 11k > 4 2 94 0 0 1| 0 76k|1235k 638k| 0 0 |9287 9295 > 5 2 93 0 0 1| 0 76k|2205k 966k| 0 0 | 11k 11k > 5 2 93 0 0 1| 0 124k|2194k 968k| 0 0 | 10k 11k > 6 2 92 0 0 1| 0 52k|2383k 1009k| 0 0 | 11k 12k > 5 1 94 0 0 1| 0 112k|1654k 795k| 0 0 |9915 11k > 4 2 95 0 0 0| 0 64k|1431k 636k| 0 0 |8865 8596 > 4 2 94 0 0 1| 0 84k|1683k 847k| 0 0 | 10k 11k > 3 1 95 0 0 1| 0 72k|1212k 654k| 0 0 |9362 9317 > 5 2 93 0 0 1| 0 32k|2011k 929k| 0 0 | 11k 12k > 22 2 75 0 0 1| 0 150k|2234k 999k| 0 0 | 10k 10k > > > 3.runtime.scanobject may be the solution of cpu periodic time high > > > perf top > > > 5.16% mwcs rumtime.scanobject > > 4.36% mwcs compress/flat.(*compress).deflate > > 3.27% mwcs runtime.memclrNoHeapPointers > > 3.22% mwcs runime.heapBitsForObject > > > > Question: > > > 1.connction take upon those more memory is nomarl work? > > 2.why use sync.Pool of *gzip.Writer cause cpu periodic time high frequently? > > 3.generic-grpc-go have no GC problem in high concurrency ? maybe since proto > is more gzip-friendly then json(inner RPC serializer)? > > 4. is any work for gc performance in gprc-go? > > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/74fd7b13-7e05-4d1b-b9a9-a740ce2bf1d4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
