How much data are you pushing over the wire? If you are network-bound adding further TCP connections won’t help. You would need either a faster channel (e.g. 10g) or bonding across multiple 1g NICs.
Anthony > On Jun 9, 2017, at 10:55 PM, Xu, Nan <[email protected]> wrote: > > Just an update, I assign more cpu to my server VM, also give more cpu to > client. it’s not helpful, it’s means bigger box doesn’t help. I guess find > out where is the bound is crucial now because all of our box in prod are big > ones and we want to use them fully. > > From: Michael Stolz [mailto:[email protected]] > Sent: Friday, June 09, 2017 7:32 PM > To: [email protected] > Subject: Re: Geode Performance how to push the cpu to 100%? > > You may need to use tools like top and sat and oostat to find out more > > -- > Mike Stolz > Principal Engineer - Gemfire Product Manager > Mobile: 631-835-4771 > > > On Jun 9, 2017 6:15 PM, "Xu, Nan" <[email protected] <mailto:[email protected]>> > wrote: > Agree that it is more IO bound but the only place I can think of is > TCP/network, but adding more tcp connections don't help. Ideas? > > > Sent with BlackBerry Work (www.blackberry.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackberry.com&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4OtC5nIh8Nm1DFqN6UdqJIhmuqGBE-fxTVeMO_6MSfY&s=vjWLpMVDUpGJoP6pKya7sVCRWNKpfILWtmAqaqJrp_E&e=>) > From: Michael Stolz <[email protected] <mailto:[email protected]>> > Sent: Jun 9, 2017 5:05 PM > To: [email protected] <mailto:[email protected]> > Subject: Re: Geode Performance how to push the cpu to 100%? > > Sounds like you are becoming more and more I/O bound. > > Remember, each put requires a complete round-trip including a layer-7 > acknowledgement back to the caller to put(k,v). > > So the longer each put takes the more I/O wait that thread has to do. > > Have a look at iostat to see how to measure that. > > > -- > Mike Stolz > Principal Engineer, GemFire Product Manager > Mobile: +1-631-835-4771 <tel:(631)%20835-4771> > > On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <[email protected] > <mailto:[email protected]>> wrote: > So I further my testing and start to change message size. > > The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client > still send as quick as possible using 128 tcp connections. I can see with the > message become bigger. My message send rate goes down, which is expected. But > the cpu usage in the server side also goes down, the system become not 100% > cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%. > > I run another program on the server, and able to use the unused cpus . so it > is not the OS. > > My socket buffer size is default, which is about 30k. checked my network > between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s > network bandwidth is not a problem. > > I was hoping just by adding more tcp to client->server will make use all > cpus and get better throughput but it not happen. Any thoughts? > > I don’t want to tune the socket_buffer_size because it surely make one single > tcp more efficient. But if I just open more tcp and I have the bandwidth and > cpu, I should be able to get better performance too. is this assumption right? > > Thanks, > Nan > > From: Xu, Nan > Sent: Friday, June 09, 2017 1:55 PM > To: [email protected] <mailto:[email protected]> > Subject: RE: Geode Performance how to push the cpu to 100%? > > Ya, after I finish my testing, I will do a write at the wiki. > > Nan > > From: Gregory Chase [mailto:[email protected] <mailto:[email protected]>] > Sent: Friday, June 09, 2017 12:59 PM > To: [email protected] <mailto:[email protected]> > Subject: Re: Geode Performance how to push the cpu to 100%? > > Hi Nan, > I think you started a very interesting discussion here that I other users can > benefit from - especially the advice you helped create. > > If your test case is general, I'm wondering if you would be willing to > contribute a short document explaining your throughput test, how you achieved > it, your results, and tips & tricks along the way. > > The article will be useful anywhere, but if you need a place, the Apache > Geode wiki > <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> > would be an excellent place. > > Thanks in advance on behalf of future Geode users. > > -Greg > > On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > Thanks for all the suggestions. As Mike point out, this is because of > the operating system. I start another while(true) {i+=1} program on the > server and cannot make the cpu higher. To this point. I think it’s fair to > say it’s the operating system and in the current setup, I need to turn the > operating system to achieve better performance. > > Thanks, > Nan > From: Vincent Ford [mailto:[email protected] <mailto:[email protected]>] > Sent: Friday, June 09, 2017 12:32 PM > > To: [email protected] <mailto:[email protected]> > Subject: Re: Geode Performance how to push the cpu to 100%? > > Hi, > > As part of your test environment you should use two or more servers, as the > basic design of Geode is around distribution across a cluster of servers. In > a test it is best to use distribution with at least two servers as your > client will be able to distribute it's load across the two different members > and although in a single host test setup you may not see significant initial > improvements but as you scale for production loads across multiple machines > you will get performance and stability benefits from having tested with > distribution from the start. > > The conserve sockets parameter effects the way servers communicate between > themselves by opening additional sockets between the peer/server nodes for > handling updates, so setting this parameter doesn't have as much impact with > a single server test. > > You may need to do virtualization and OS tuning to get the full impact, > increasing the socket buffer sizes should help if you haven't done this > already... > > Vince Ford > GemFire Toolsmith Engineering > Beaverton, OR USA > http://www.pivotal.io > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=> > Open Source Project Geode https://geode.incubator.apache.org/ > <https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=> > > On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <[email protected] > <mailto:[email protected]>> wrote: > Try doing just raw pipe communications between a bunch of processes to see if > you can pin the cpu. > > # cat somebigfile | cat | cat... >/dev/null > > If that can't pin cpu then you know it's the operating system. > > -- > Mike Stolz > Principal Engineer - Gemfire Product Manager > Mobile: 631-835-4771 <tel:(631)%20835-4771> > > On Jun 9, 2017 9:22 AM, "Xu, Nan" <[email protected] <mailto:[email protected]>> > wrote: > Thanks for pointing this out, very useful. > > I change the way I test the performance and get a result I can NOT explain. > > I use 2 separate virtual machine. 1 run client, 1 run server, both siting on > the same physical box. > > The client put small message (about 10 bytes) as quick as possible through > 128 threads. Both client and server have the conserve_socket=false > > I can see there are 128 tcp connections between them and I send about 50,000 > message/s > > Server have 4 core and 3 out of 4 is constantly 100%, but one core is only > 30%. On the server, I only run 1 locator and 1 server and no other program. > Region is PARTITION. I publish about 2000 keys. > > Why there is a core only 30%. My Point is, if I can use the last core more, I > might able to publisher even quicker. > > Thanks, > Nan > > From: Akihiro Kitada [mailto:[email protected] <mailto:[email protected]>] > Sent: Thursday, June 08, 2017 7:13 PM > To: [email protected] <mailto:[email protected]> > Subject: Re: Geode Performance how to push the cpu to 100%? > > Hello Nan, > > Why don't you check Geode statistics > > http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=> > > Maybe, disk I/O or some other causes could be involved... > > > > -- > Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 > <tel:+81%2080-3716-3736> > Support.Pivotal.io > <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> > | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <tel:(877)%20477-2269> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=> > > > 2017-06-09 6:09 GMT+09:00 Xu, Nan <[email protected] <mailto:[email protected]>>: > Hi, > > I am trying to see the limit of geode performance. > > Here is what I did. > > Single machine: OS: centos 7. 8 core. 2.6G. > > Create a single locator and a single server and a single region. > > Only configuration is > Server conserve-sockets= false > Region is PARTITION > Client and server running on the same machine. > > > In my client. I setup a 16 thread pool to get data. But I can only push the > cpu to around 90—93% on the centos. > > Why I cannot push it to 100%? I am suspecting > 1. The tcp connection between the server and client is not fast enough. > Maybe increase the number of tcp connection? I only see one connection > between client and server. > > 2. There are some lock at the server? I realize that I can push the > cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so > maybe there are some other setting I am missing. > > > Thanks, > Nan > > Client side program. > > val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334) > .set("log-level", "info") > .set("conserve-sockets", "false") > .setPoolMinConnections(4) > .setPoolMaxConnections(12) > .setPoolThreadLocalConnections(true) > .create > > val regionFactory: ClientRegionFactory[String, String] = > cache.createClientRegionFactory(ClientRegionShortcut.PROXY) > val region1: Region[String, String] = regionFactory.create(region) > > implicit val ec = > ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16)) > var j = new AtomicLong(0) > for (i <- 1 to 16) yield Future { > while(true){ > val cj = j.addAndGet(1) > region1.get("" + (rnd.nextInt(2000) + 1)) > } > } > > > > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message. > > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message. > > > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message. > > > > -- > Greg Chase > > Product team > http://www.pivotal.io/big-data > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=> > > Pivotal Software > http://www.pivotal.io/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=> > > 650-215-0477 <tel:(650)%20215-0477> > @GregChase > Blog: http://geekmarketing.biz/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=> > > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message. > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available > athttp://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message. > > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available > athttp://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message. > This message, and any attachments, is for the intended recipient(s) only, may > contain information that is privileged, confidential and/or proprietary and > subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer > <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended > recipient, please delete this message.
