You may need to use tools like top and sat and oostat to find out more -- Mike Stolz Principal Engineer - Gemfire Product Manager Mobile: 631-835-4771
On Jun 9, 2017 6:15 PM, "Xu, Nan" <[email protected]> wrote: > Agree that it is more IO bound but the only place I can think of is > TCP/network, but adding more tcp connections don't help. Ideas? > > > Sent with BlackBerry Work (www.blackberry.com) > ------------------------------ > *From: *Michael Stolz <[email protected]> > *Sent: *Jun 9, 2017 5:05 PM > *To: *[email protected] > *Subject: *Re: Geode Performance how to push the cpu to 100%? > > Sounds like you are becoming more and more I/O bound. > > Remember, each put requires a complete round-trip including a layer-7 > acknowledgement back to the caller to put(k,v). > > So the longer each put takes the more I/O wait that thread has to do. > > Have a look at iostat to see how to measure that. > > > -- > Mike Stolz > Principal Engineer, GemFire Product Manager > Mobile: +1-631-835-4771 <(631)%20835-4771> > > On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <[email protected]> wrote: > >> So I further my testing and start to change message size. >> >> >> >> The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. >> Client still send as quick as possible using 128 tcp connections. I can see >> with the message become bigger. My message send rate goes down, which is >> expected. But the cpu usage in the server side also goes down, the system >> become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at >> 2%. >> >> >> >> I run another program on the server, and able to use the unused cpus . so >> it is not the OS. >> >> >> >> My socket buffer size is default, which is about 30k. checked my network >> between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s >> network bandwidth is not a problem. >> >> >> >> I was hoping just by adding more tcp to client->server will make use all >> cpus and get better throughput but it not happen. Any thoughts? >> >> >> >> I don’t want to tune the socket_buffer_size because it surely make one >> single tcp more efficient. But if I just open more tcp and I have the >> bandwidth and cpu, I should be able to get better performance too. is this >> assumption right? >> >> >> >> Thanks, >> >> Nan >> >> >> >> *From:* Xu, Nan >> *Sent:* Friday, June 09, 2017 1:55 PM >> *To:* [email protected] >> *Subject:* RE: Geode Performance how to push the cpu to 100%? >> >> >> >> Ya, after I finish my testing, I will do a write at the wiki. >> >> >> >> Nan >> >> >> >> *From:* Gregory Chase [mailto:[email protected] <[email protected]>] >> *Sent:* Friday, June 09, 2017 12:59 PM >> *To:* [email protected] >> *Subject:* Re: Geode Performance how to push the cpu to 100%? >> >> >> >> Hi Nan, >> I think you started a very interesting discussion here that I other users >> can benefit from - especially the advice you helped create. >> >> >> >> If your test case is general, I'm wondering if you would be willing to >> contribute a short document explaining your throughput test, how you >> achieved it, your results, and tips & tricks along the way. >> >> >> >> The article will be useful anywhere, but if you need a place, the Apache >> Geode wiki >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> >> would be an excellent place. >> >> >> >> Thanks in advance on behalf of future Geode users. >> >> >> >> -Greg >> >> >> >> On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <[email protected]> wrote: >> >> Hi, >> >> >> >> Thanks for all the suggestions. As Mike point out, this is because >> of the operating system. I start another while(true) {i+=1} program on the >> server and cannot make the cpu higher. To this point. I think it’s fair to >> say it’s the operating system and in the current setup, I need to turn the >> operating system to achieve better performance. >> >> >> >> Thanks, >> >> Nan >> >> *From:* Vincent Ford [mailto:[email protected]] >> *Sent:* Friday, June 09, 2017 12:32 PM >> >> >> *To:* [email protected] >> *Subject:* Re: Geode Performance how to push the cpu to 100%? >> >> >> >> Hi, >> >> >> >> As part of your test environment you should use two or more servers, as >> the basic design of Geode is around distribution across a cluster of >> servers. In a test it is best to use distribution with at least two servers >> as your client will be able to distribute it's load across the two >> different members and although in a single host test setup you may not see >> significant initial improvements but as you scale for production loads >> across multiple machines you will get performance and stability benefits >> from having tested with distribution from the start. >> >> >> >> The conserve sockets parameter effects the way servers communicate >> between themselves by opening additional sockets between the peer/server >> nodes for handling updates, so setting this parameter doesn't have as much >> impact with a single server test. >> >> >> >> You may need to do virtualization and OS tuning to get the full impact, >> increasing the socket buffer sizes should help if you haven't done this >> already... >> >> >> *Vince Ford* >> >> GemFire Toolsmith Engineering >> >> Beaverton, OR USA >> >> http://www.pivotal.io >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=> >> >> Open Source Project Geode https://geode.incubator.apache.org/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=> >> >> >> >> On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <[email protected]> wrote: >> >> Try doing just raw pipe communications between a bunch of processes to >> see if you can pin the cpu. >> >> # cat somebigfile | cat | cat... >/dev/null >> >> If that can't pin cpu then you know it's the operating system. >> >> -- >> Mike Stolz >> Principal Engineer - Gemfire Product Manager >> Mobile: 631-835-4771 <(631)%20835-4771> >> >> >> >> On Jun 9, 2017 9:22 AM, "Xu, Nan" <[email protected]> wrote: >> >> Thanks for pointing this out, very useful. >> >> >> >> I change the way I test the performance and get a result I can NOT >> explain. >> >> >> >> I use 2 separate virtual machine. 1 run client, 1 run server, both >> siting on the same physical box. >> >> >> >> The client put small message (about 10 bytes) as quick as possible >> through 128 threads. Both client and server have the conserve_socket=false >> >> >> >> I can see there are 128 tcp connections between them and I send about >> 50,000 message/s >> >> >> >> Server have 4 core and 3 out of 4 is constantly 100%, but one core is >> only 30%. On the server, I only run 1 locator and 1 server and no other >> program. Region is PARTITION. I publish about 2000 keys. >> >> >> >> Why there is a core only 30%. My Point is, if I can use the last core >> more, I might able to publisher even quicker. >> >> >> >> Thanks, >> >> Nan >> >> >> >> *From:* Akihiro Kitada [mailto:[email protected]] >> *Sent:* Thursday, June 08, 2017 7:13 PM >> *To:* [email protected] >> *Subject:* Re: Geode Performance how to push the cpu to 100%? >> >> >> >> Hello Nan, >> >> >> >> Why don't you check Geode statistics >> >> >> >> http://geode.apache.org/docs/guide/11/reference/statistics/s >> tatistics_list.html >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=> >> >> >> >> Maybe, disk I/O or some other causes could be involved... >> >> >> >> >> >> >> -- >> >> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 >> <+81%2080-3716-3736> >> Support.Pivotal.io >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> >> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269> >> [image: support] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> >> [image: twitter] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> >> [image: linkedin] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> >> [image: facebook] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> >> [image: google plus] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> >> [image: youtube] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=> >> >> >> >> >> >> 2017-06-09 6:09 GMT+09:00 Xu, Nan <[email protected]>: >> >> Hi, >> >> >> >> I am trying to see the limit of geode performance. >> >> >> >> Here is what I did. >> >> >> >> Single machine: OS: centos 7. 8 core. 2.6G. >> >> >> >> Create a single locator and a single server and a single region. >> >> >> >> Only configuration is >> >> Server conserve-sockets= false >> >> Region is PARTITION >> >> Client and server running on the same machine. >> >> >> >> >> >> In my client. I setup a 16 thread pool to get data. But I can only push >> the cpu to around 90—93% on the centos. >> >> >> >> Why I cannot push it to 100%? I am suspecting >> >> 1. The tcp connection between the server and client is not fast >> enough. Maybe increase the number of tcp connection? I only see one >> connection between client and server. >> >> 2. There are some lock at the server? I realize that I can push >> the cpu from 50% to 90 by just adding the >> setPoolThreadLocalConnections(true), >> so maybe there are some other setting I am missing. >> >> >> >> Thanks, >> >> Nan >> >> >> >> Client side program. >> >> >> >> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, >> 10334) >> >> .set("log-level", "info") >> >> .set("conserve-sockets", "false") >> >> .setPoolMinConnections(4) >> >> .setPoolMaxConnections(12) >> >> .setPoolThreadLocalConnections(true) >> >> .create >> >> >> >> val regionFactory: ClientRegionFactory[String, String] = >> cache.createClientRegionFactory(ClientRegionShortcut.PROXY) >> >> val region1: Region[String, String] = regionFactory.create(region) >> >> >> >> implicit val ec = ExecutionContext.fromExecutorS >> ervice(Executors.newFixedThreadPool(16)) >> >> var j = new AtomicLong(0) >> >> for (i <- 1 to 16) yield Future { >> >> while(true){ >> >> val cj = j.addAndGet(1) >> >> region1.get("" + (rnd.nextInt(2000) + 1)) >> >> } >> >> } >> >> >> >> >> >> >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> >> >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> >> >> >> >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> >> >> >> >> >> -- >> >> Greg Chase >> >> >> >> Product team >> >> http://www.pivotal.io/big-data >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=> >> >> >> >> Pivotal Software >> >> http://www.pivotal.io/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=> >> >> >> >> 650-215-0477 <(650)%20215-0477> >> >> @GregChase >> >> Blog: http://geekmarketing.biz/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=> >> >> >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> ------------------------------ >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> > > ------------------------------ > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. >
