Have a look at the client-cache documentation here: http://geode.apache.org/docs/guide/12/reference/topics/client-cache.html
You will see that on the pool, you can configure a number of items related to max connections, min connections, idle timeout, etc. On Mon, Oct 23, 2017 at 9:14 AM, Xu, Nan <[email protected]> wrote: > Thanks, I found if I use cached proxy. For the same 5000 slow clients , I > only use 2 connections, is there a way to setup proxy region have the same > behavior? > > Nan > > > Sent with BlackBerry Work (www.blackberry.com) > ------------------------------ > *From: *Akihiro Kitada <[email protected]> > *Sent: *Oct 22, 2017 8:08 PM > *To: *[email protected] > *Subject: *Re: max out 800 connections > > Hello Nan, > > >I did some test and found when I have a lot of slow publisher (1 > message/second), but 5000 of them. All publisher > >connect to a single client region(shared among all publisher), that > client region use a lot of connections to geode > >server. is that normal? > > Yes, it's normal according to the client side configuration and region > configuration. Each client connections are pooled for a while according > to idle-timeout setting at pool element of client side cache.xml. > > http://geode.apache.org/docs/guide/12/reference/topics/ > client-cache.html#cc-pool > <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_12_reference_topics_client-2Dcache.html-23cc-2Dpool&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=XN7DC4Ewjhhc-0-xFDFA-Xit1ak28QWc8oLouU1kMBg&e=> > > >Because when I use fewer but faster publisher, I can see a lot less > connection being used from client region to > >server region. I think from server point of view, it’s almost the same > load. Why more connections? > > I'm not sure how each publisher connect to a single client region. But if > each publishers keeps one connection for a while according to idle-timeout > setting at least, larger number of publishers could increase more number of > connections, rather than faster publishers. > > By the way, which region type do you have at server side in this context, > Replicated region or Partitioned region? If you have Partitioned region and > set pr-single-hop-enabled=true at client pool setting (default value), each > client pools keeps more number of connections than expected because each > client pools keep some connections for all the cache server processes > hosting target regions by default. > > >After a while, server max out 800 connections. Is that safe to increase > this number to let’s say 4000? > > It depends on the number of cache servers in the distributed system in > terms of file descriptors usage. Each client-server connections consumes 1 > file descriptors. The number of TCP connections between cache servers > (between peers, including locators) get increased according to the number > of peers because each peers have TCP connections to communicate each others > with rest of peers. Moreover, some more TCP connections are consumed for > detecting membership failure based on circular network. > > Anyhow, if you increase max-clients, you have to consider memory usage for > TCP connections and maximum number of file descriptors in your system. > > >Or it’s better to set client cache factory a max connection number? > > It's one of the workaround. In such case, you have to consider that some > of publishers may be timed out according to workloads at client side. > > By the way, if you have Partitioned region and set > pr-single-hop-enabled=true at client pool setting (default value), the > number of client-server connection is managed by GemFire automatically and > you can't set the maximum number of client pool connections. > > > -- > Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 > <+81%2080-3716-3736> > Support.Pivotal.io > <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=21dSS_IyEML_B1Ob4QYO3rnpz2KulMZbM_9u8_ZpVvc&e=> > | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269> > [image: support] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=7ScQVp17l74l65EAJ3Vniof5DXR3roIXIUTBspXxGaI&e=> > [image: twitter] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=9d5rNMiOg5P2HHBg_QGMoID87huKbOLSgujymjoXWpY&e=> > [image: linkedin] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=U3rLzj44E2p1oiZgRe3lgG7FmGN5qXiv4BnKL9UvecI&e=> > [image: facebook] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=4U1E8Gt_KeuGqBGQi787hRjxA1BkT8E5pu9OuqmYEPM&e=> > [image: google plus] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=Unxi9HaUHQfoMmY_MJkQimxjqAC5KwuSh6Tc_sJHNKA&e=> > [image: youtube] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=SEcNGapl3Lztl6TZiXCHsz-eJVHuXOMVkb7SS2Vrip8&s=AL46i2j59jpNd5I--7S9BpWd0x74BYBNd4oFMAXQV5A&e=> > > > > 2017-10-21 5:11 GMT+09:00 Xu, Nan <[email protected]>: > >> Btw, pool-threadlocal-connection already set to false. >> >> >> >> *From:* Xu, Nan >> *Sent:* Friday, October 20, 2017 3:09 PM >> *To:* [email protected] >> *Subject:* [External email from user-return-2141-nxu2=baml.com >> @geode.apache.org] RE: max out 800 connections >> >> >> >> Thanks for the answer. I did some test and found when I have a lot of >> slow publisher (1 message/second), but 5000 of them. All publisher connect >> to a single client region(shared among all publisher), that client region >> use a lot of connections to geode server. is that normal? Because when I >> use fewer but faster publisher, I can see a lot less connection being used >> from client region to server region. I think from server point of view, >> it’s almost the same load. Why more connections? After a while, server max >> out 800 connections. Is that safe to increase this number to let’s say >> 4000? Or it’s better to set client cache factory a max connection number? >> >> >> >> Thanks, >> Nan >> >> >> >> *From:* Akihiro Kitada [mailto:[email protected] <[email protected]>] >> *Sent:* Thursday, October 19, 2017 9:30 PM >> *To:* [email protected] >> *Subject:* Re: max out 800 connections >> >> >> >> Hello Nan, >> >> >> >> >[warning 2017/10/19 16:00:47.219 EDT a1 <Handshaker >> 0.0.0.0/0.0.0.0:40411 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> >> Thread 20> tid=0x11e9] Rejected >> >> >connection from /171.135.144.25 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__171.135.144.25&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=0GmLAzVpw8ar3wV475-XLR93UTxMrEZzEyIDfcaT1qM&e=> >> because current connection count of 800 is greater than or equal to the >> >> >configured max of 800 >> >> : >> >> >Is the first line means max-connection is a server side settting? >> >> >> >> It means that the number of client connections for the server >> exceeds max-connections setting in the server side setting at cache-server >> attribute. >> >> >> >> http://geode.apache.org/docs/guide/12/reference/topics/cache >> _xml.html#cache-server >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_12_reference_topics_cache-5Fxml.html-23cache-2Dserver&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=EZAOFvnX72Vm8BgIawdHZAkICbw9cKfSWK8Je5SVgE0&e=> >> >> >> >> >The whole cluster only take 800 client connections? Or it’s a client >> side setting, so each client can use up to 800 connections, so if I have 5 >> clients, server side will be 4000 connections? >> >> >> >> It's a server side configuration for the each servers. Each cache servers >> accepts 800 clients connections by default. >> >> >> >> >[warning 2017/10/19 16:00:47.220 EDT a1 <Handshaker >> 0.0.0.0/0.0.0.0:40411 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> >> Thread 20> tid=0x11e9] Rejected >> >> >connection from Server connection from [client host >> address=171.135.144.71; client port=45894] because >> >> >incoming request was rejected by pool possibly due to thread exhaustion >> >> : >> >> >Second line means no more thread on the server? I have about 4000 >> threads on server. any way to limit geode use certain number of thread? >> >> >> >> In this case, "pool" should mean the instance of >> java.util.concurrent.ThreadPoolExecutor used for server connections. The >> size of this pool equals to max-connections. So, those both two warning >> messages basically indicate that the number of client connections exceeds >> max-connections setting. >> >> >> >> The difference is that the first one is for client connections used for >> 'client to server' messages related to a client >> >> * queue (register interest, create cq, etc.) and the second one is for >> other purposes of client connections, according to the source code. >> >> >> >> You can find exact parts at >> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl >> by searching the following words. >> >> >> >> * AcceptorImpl_REJECTED_CONNECTION_FROM_0_BECAUSE_CURRENT_ >> CONNECTION_COUNT_OF_1_IS_GREATER_THAN_OR_EQUAL_TO_THE_CONFIGURED_MAX_OF_2 >> >> * AcceptorImpl_REJECTED_CONNECTION_FROM_0_BECAUSE_REQUEST_ >> REJECTED_BY_POOL >> >> >> >> Thanks. >> >> >> >> >> >> >> -- >> >> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 >> <080-3716-3736> >> Support.Pivotal.io >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=5FY7L2xbyCREkTX91bYnim5siS9b8s0xShxiOX56F4M&e=> >> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269> >> [image: Image removed by sender. support] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=4egEtrCETGSJB6Yj1FBvcAoaJddujRQ76REMB1CjOiQ&e=> >> [image: Image removed by sender. twitter] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=sDNN2JmaGJFGYNQ8SPRpNJzMXWG4cAsRq8jTwBj5Oso&e=> >> [image: Image removed by sender. linkedin] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=27JxI5iQLaAHxQAjf4U8ZT5jFTdArFc12nsN2WKC6LA&e=> >> [image: Image removed by sender. facebook] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=j6uWCSv_WdfZwG62Zc30Dn4cqwEkGlF6A3RMecWs7hA&e=> >> [image: Image removed by sender. google plus] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=scc6bteFZUCqsKOVQXTZ1JSR0ufEQKULdE06eqPoZok&e=> >> [image: Image removed by sender. youtube] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=Xk7Bbo5ZLK_RFppNOXZHBkxOi4ZjCbqxiCILhgyEcS4&e=> >> >> >> >> >> >> 2017-10-20 5:54 GMT+09:00 Xu, Nan <[email protected]>: >> >> Hi, >> >> >> >> When load test our geode cluster, see those from our server log >> >> >> >> >> >> [warning 2017/10/19 16:00:47.219 EDT a1 <Handshaker >> 0.0.0.0/0.0.0.0:40411 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> >> Thread 20> tid=0x11e9] Rejected connection from /171.135.144.25 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__171.135.144.25&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=0GmLAzVpw8ar3wV475-XLR93UTxMrEZzEyIDfcaT1qM&e=> >> because current connection count of 800 is greater than or equal to the >> configured max of 800 >> >> >> >> [warning 2017/10/19 16:00:47.220 EDT a1 <Handshaker 0.0.0.0/0.0.0.0:40411 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__0.0.0.0_0.0.0.0-3A40411&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4pec-ejmOPalR_OG4srZgdJe-wgObl1ciBp4drY-Ojk&s=dpA-r8BFw3ZIiLq781hQbvsakBv1is0Sy5zR68rvYxQ&e=> >> Thread 20> tid=0x11e9] Rejected connection from Server connection from >> [client host address=171.135.144.71; client port=45894] because incoming >> request was rejected by pool possibly due to thread exhaustion >> >> >> >> We are client server setup. >> >> >> >> Is the first line means max-connection is a server side settting? The >> whole cluster only take 800 client connections? Or it’s a client side >> setting, so each client can use up to 800 connections, so if I have 5 >> clients, server side will be 4000 connections? >> >> >> >> Second line means no more thread on the server? I have about 4000 threads >> on server. any way to limit geode use certain number of thread? >> >> >> >> Thanks, >> >> Nan >> >> >> >> >> >> >> >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> >> >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> ------------------------------ >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> > > ------------------------------ > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. > -- *Mark Secrist | Director, **Global Education Delivery* [email protected] 970.214.4567 Mobile *pivotal.io <http://www.pivotal.io/>* Follow Us: Twitter <http://www.twitter.com/pivotal> | LinkedIn <http://www.linkedin.com/company/pivotalsoftware> | Facebook <http://www.facebook.com/pivotalsoftware> | YouTube <http://www.youtube.com/gopivotal> | Google+ <https://plus.google.com/105320112436428794490>
