Hello Nan, >I did some test and found when I have a lot of slow publisher (1 message/second), but 5000 of them. All publisher >connect to a single client region(shared among all publisher), that client region use a lot of connections to geode >server. is that normal?
Yes, it's normal according to the client side configuration and region configuration. Each client connections are pooled for a while according to idle-timeout setting at pool element of client side cache.xml. http://geode.apache.org/docs/guide/12/reference/topics/client-cache.html#cc-pool >Because when I use fewer but faster publisher, I can see a lot less connection being used from client region to >server region. I think from server point of view, it’s almost the same load. Why more connections? I'm not sure how each publisher connect to a single client region. But if each publishers keeps one connection for a while according to idle-timeout setting at least, larger number of publishers could increase more number of connections, rather than faster publishers. By the way, which region type do you have at server side in this context, Replicated region or Partitioned region? If you have Partitioned region and set pr-single-hop-enabled=true at client pool setting (default value), each client pools keeps more number of connections than expected because each client pools keep some connections for all the cache server processes hosting target regions by default. >After a while, server max out 800 connections. Is that safe to increase this number to let’s say 4000? It depends on the number of cache servers in the distributed system in terms of file descriptors usage. Each client-server connections consumes 1 file descriptors. The number of TCP connections between cache servers (between peers, including locators) get increased according to the number of peers because each peers have TCP connections to communicate each others with rest of peers. Moreover, some more TCP connections are consumed for detecting membership failure based on circular network. Anyhow, if you increase max-clients, you have to consider memory usage for TCP connections and maximum number of file descriptors in your system. >Or it’s better to set client cache factory a max connection number? It's one of the workaround. In such case, you have to consider that some of publishers may be timed out according to workloads at client side. By the way, if you have Partitioned region and set pr-single-hop-enabled=true at client pool setting (default value), the number of client-server connection is managed by GemFire automatically and you can't set the maximum number of client pool connections. -- Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 Support.Pivotal.io <http://support.pivotal.io/> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 [image: support] <https://support.pivotal.io/> [image: twitter] <https://twitter.com/pivotal> [image: linkedin] <https://www.linkedin.com/company/3048967> [image: facebook] <https://www.facebook.com/pivotalsoftware> [image: google plus] <https://plus.google.com/+Pivotal> [image: youtube] <https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl> 2017-10-21 5:11 GMT+09:00 Xu, Nan <[email protected]>: > Btw, pool-threadlocal-connection already set to false. > > > > *From:* Xu, Nan > *Sent:* Friday, October 20, 2017 3:09 PM > *To:* [email protected] > *Subject:* [External email from user-return-2141-nxu2=baml. > [email protected]] RE: max out 800 connections > > > > Thanks for the answer. I did some test and found when I have a lot of > slow publisher (1 message/second), but 5000 of them. All publisher connect > to a single client region(shared among all publisher), that client region > use a lot of connections to geode server. is that normal? Because when I > use fewer but faster publisher, I can see a lot less connection being used > from client region to server region. I think from server point of view, > it’s almost the same load. Why more connections? After a while, server max > out 800 connections. Is that safe to increase this number to let’s say > 4000? Or it’s better to set client cache factory a max connection number? > > > > Thanks, > Nan > > > > *From:* Akihiro Kitada [mailto:[email protected] <[email protected]>] > *Sent:* Thursday, October 19, 2017 9:30 PM > *To:* [email protected] > *Subject:* Re: max out 800 connections > > > > Hello Nan, > > > > >[warning 2017/10/19 16:00:47.219 EDT a1 <Handshaker > 0.0.0.0/0.0.0.0:40411 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> > Thread 20> tid=0x11e9] Rejected > > >connection from /171.135.144.25 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__171.135.144.25&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=0GmLAzVpw8ar3wV475-XLR93UTxMrEZzEyIDfcaT1qM&e=> > because current connection count of 800 is greater than or equal to the > > >configured max of 800 > > : > > >Is the first line means max-connection is a server side settting? > > > > It means that the number of client connections for the server > exceeds max-connections setting in the server side setting at cache-server > attribute. > > > > http://geode.apache.org/docs/guide/12/reference/topics/ > cache_xml.html#cache-server > <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_12_reference_topics_cache-5Fxml.html-23cache-2Dserver&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=EZAOFvnX72Vm8BgIawdHZAkICbw9cKfSWK8Je5SVgE0&e=> > > > > >The whole cluster only take 800 client connections? Or it’s a client > side setting, so each client can use up to 800 connections, so if I have 5 > clients, server side will be 4000 connections? > > > > It's a server side configuration for the each servers. Each cache servers > accepts 800 clients connections by default. > > > > >[warning 2017/10/19 16:00:47.220 EDT a1 <Handshaker 0.0.0.0/0.0.0.0:40411 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> > Thread 20> tid=0x11e9] Rejected > > >connection from Server connection from [client host > address=171.135.144.71; client port=45894] because > > >incoming request was rejected by pool possibly due to thread exhaustion > > : > > >Second line means no more thread on the server? I have about 4000 threads > on server. any way to limit geode use certain number of thread? > > > > In this case, "pool" should mean the instance of > java.util.concurrent.ThreadPoolExecutor > used for server connections. The size of this pool equals to > max-connections. So, those both two warning messages basically indicate > that the number of client connections exceeds max-connections setting. > > > > The difference is that the first one is for client connections used for > 'client to server' messages related to a client > > * queue (register interest, create cq, etc.) and the second one is for > other purposes of client connections, according to the source code. > > > > You can find exact parts at > org.apache.geode.internal.cache.tier.sockets.AcceptorImpl > by searching the following words. > > > > * AcceptorImpl_REJECTED_CONNECTION_FROM_0_BECAUSE_ > CURRENT_CONNECTION_COUNT_OF_1_IS_GREATER_THAN_OR_EQUAL_TO_ > THE_CONFIGURED_MAX_OF_2 > > * AcceptorImpl_REJECTED_CONNECTION_FROM_0_BECAUSE_REQUEST_REJECTED_BY_POOL > > > > Thanks. > > > > > > > -- > > Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 > <080-3716-3736> > Support.Pivotal.io > <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=5FY7L2xbyCREkTX91bYnim5siS9b8s0xShxiOX56F4M&e=> > | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 > [image: Image removed by sender. support] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=4egEtrCETGSJB6Yj1FBvcAoaJddujRQ76REMB1CjOiQ&e=> > [image: Image removed by sender. twitter] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=sDNN2JmaGJFGYNQ8SPRpNJzMXWG4cAsRq8jTwBj5Oso&e=> > [image: Image removed by sender. linkedin] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=27JxI5iQLaAHxQAjf4U8ZT5jFTdArFc12nsN2WKC6LA&e=> > [image: Image removed by sender. facebook] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=j6uWCSv_WdfZwG62Zc30Dn4cqwEkGlF6A3RMecWs7hA&e=> > [image: Image removed by sender. google plus] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=scc6bteFZUCqsKOVQXTZ1JSR0ufEQKULdE06eqPoZok&e=> > [image: Image removed by sender. youtube] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=Xk7Bbo5ZLK_RFppNOXZHBkxOi4ZjCbqxiCILhgyEcS4&e=> > > > > > > 2017-10-20 5:54 GMT+09:00 Xu, Nan <[email protected]>: > > Hi, > > > > When load test our geode cluster, see those from our server log > > > > > > [warning 2017/10/19 16:00:47.219 EDT a1 <Handshaker 0.0.0.0/0.0.0.0:40411 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> > Thread 20> tid=0x11e9] Rejected connection from /171.135.144.25 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__171.135.144.25&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=0GmLAzVpw8ar3wV475-XLR93UTxMrEZzEyIDfcaT1qM&e=> > because current connection count of 800 is greater than or equal to the > configured max of 800 > > > > [warning 2017/10/19 16:00:47.220 EDT a1 <Handshaker 0.0.0.0/0.0.0.0:40411 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> > Thread 20> tid=0x11e9] Rejected connection from Server connection from > [client host address=171.135.144.71; client port=45894] because incoming > request was rejected by pool possibly due to thread exhaustion > > > > We are client server setup. > > > > Is the first line means max-connection is a server side settting? The > whole cluster only take 800 client connections? Or it’s a client side > setting, so each client can use up to 800 connections, so if I have 5 > clients, server side will be 4000 connections? > > > > Second line means no more thread on the server? I have about 4000 threads > on server. any way to limit geode use certain number of thread? > > > > Thanks, > > Nan > > > > > > > > ------------------------------ > > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. > > > ------------------------------ > > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. > ------------------------------ > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. >
