Dan's question is a good one. I was thinking that the issue with the CacheClientUpdater dying was happening a lot and causing the client threads to spin out of control. But if you're not seeing a lot of messages like below in your client log, thats not the case.
[info 2017/02/10 11:07:13.124 GMT <Cache Client Updater Thread on xxx> tid=0x22] Primary subscription endpoint xxx crashed. Scheduling recovery. If your not seeing a lot of message like this, then your client is right on the edge of its resources. The 'java.lang.OutOfMemoryError: unable to create new native thread' means the OS doesn't have the resources to allocate a thread. It could be several things: - the physical memory on the box is exhausted - the user process limit is reached (ulimit -u) What is the physical memory on your client machine? What are your JVM's -Xms and -Xms settings? Is there enough room between the physical memory and the JVM resources for thread stacks? Are you setting -Xss (thread stack size)? You could reduce that to save resources. But before you do anything, you need to figure out what the threads are. Setting remove-unresponsive-clients=true will remove a client whose queue is full. If a client's queue fills up, puts into it are blocked. Which means other client requests are blocked. The main point of setting this property is to prevent cascading issues when a client's queue is full at the expense of causing that client to reconnect. So, there are pros and cons to this property. The maximum-message-count property limits the maximum size of the client's queue before puts block. The default is 230,000 which is mainly to handle bursts of activity. A client's queue should never be this full in normal processing. The Cache Client Updater will try forever to recover its connection to the server. The QueueManagerImpl uses a ScheduledThreadPoolExecutor with a maximum pool size of Integer.MAX_VALUE, so the number of available threads in this area is limited only by the JVM and OS resources. Thanks, Barry Oglesby On Fri, Feb 10, 2017 at 6:27 AM, Thacker, Dharam < [email protected]> wrote: > Thanks Dan & Barry! > > Could not reproduce this issue today but got some hints based on your > guidance with remove-unresponsive-clients and max-message-count. > > Somehow logs didn't give useful information, but I will enable client > stats and verify it once again. > > More could you help me to understand max-message-count's role/meaning if I > enable remove-unresponsive-clients? > > More how many times thread will try to recover connection in case it > crashes? > > Is there any way to increase number of threads than what comes by default > ? (Even is that recommend?) > > Thanks, > Dharam > > Sent with Good Work (www.blackberry.com) > On Feb 9, 2017 11:50 PM, Dan Smith <[email protected]> wrote: > The error says "unable to create a new native thread." Do you know how > many threads you had in this process when it died? The client stats could > show the you the number of threads at least. You might also have an > hs_err_pid file that might list the threads. > > -Dan > > On Thu, Feb 9, 2017 at 9:23 AM, Barry Oglesby <[email protected]> wrote: > >> Dharam, >> >> This stack means that the CacheClientUpdater (which is processing the >> messages on the client) has caught an exception, is crashing and kicking >> off a thread to recover its connection to the server. A high number of >> events alone wouldn't cause this unless the client's queue on the server is >> full and remove-unresponsive-client=true in the server. Is that the >> case? It would be interesting to see the client and server logs as well as >> the server stats to see if we can determine why this is happening. The >> client log may contain the exception that is causing the CacheClientUpdater >> to crash. >> >> Thanks, >> Barry Oglesby >> >> >> On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam < >> [email protected]> wrote: >> >>> Hi Team, >>> >>> >>> >>> I am facing below error while working with continuous query listener. We >>> are receiving high number of events matching to various continuous query >>> listeners which result into below exception. Though task every thread s >>> doing is really very small (Not at all time consuming). >>> >>> >>> >>> Any guideline on below one? Is it like thread after finishing work not >>> going to pool? >>> >>> >>> >>> <gfe:cq-listener-container id=*"cqListenerContainer"* cache= >>> *"event-cache"*> >>> >>> <gfe:listener ref=*"eventListener"* query=*"SELECT * FROM >>> /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and >>> status='PENDING'"*/> >>> >>> </gfe:cq-listener-container> >>> >>> >>> >>> [http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] >>> Servlet.service() for servlet [dispatcherServlet] in context with path [] >>> threw exception [Handler dispatch failed; nested exception is >>> java.lang.OutOfMemoryError: unable to create new native thread] with root >>> cause >>> >>> java.lang.OutOfMemoryError: unable to create new native thread >>> >>> at java.lang.Thread.start0(Native Method) >>> >>> at java.lang.Thread.start(Thread.java:714) >>> >>> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPool >>> Executor.java:950) >>> >>> at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(Threa >>> dPoolExecutor.java:1587) >>> >>> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExec >>> ute(ScheduledThreadPoolExecutor.java:334) >>> >>> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sc >>> heduledThreadPoolExecutor.java:533) >>> >>> at org.apache.geode.cache.client.internal.QueueManagerImpl.sche >>> duleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041) >>> >>> at org.apache.geode.cache.client.internal.QueueManagerImpl.chec >>> kEndpoint(QueueManagerImpl.java:430) >>> >>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda >>> ter.processMessages(CacheClientUpdater.java:1811) >>> >>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda >>> ter.run(CacheClientUpdater.java:463) >>> >>> >>> >>> Thanks & Regards, >>> >>> Dharam >>> >>> This email is confidential and subject to important disclaimers and >>> conditions including on offers for the purchase or sale of securities, >>> accuracy and completeness of information, viruses, confidentiality, legal >>> privilege, and legal entity disclaimers, available at >>> http://www.jpmorgan.com/pages/disclosures/email >>> >> >> > This email is confidential and subject to important disclaimers and > conditions including on offers for the purchase or sale of securities, > accuracy and completeness of information, viruses, confidentiality, legal > privilege, and legal entity disclaimers, available at > http://www.jpmorgan.com/pages/disclosures/email >
