Re: Operation has timed out
2017-02-09 18:45 GMT+09:00 Fady Haikal: > i will try that but this required tomcat restart so i will be > scheduling tonight a restart, anyway from where i can enable the > org.apache.catalina.tribes.MESSAGES i didn't found it in the > logging.properties file > The org.apache.catalina.tribes.MESSAGES is the logger name. You have to add settings to the logging.properties file. The following is a sample. Please refer to. Note for using the org.apache.catalina.tribes.MESSAGES: The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs. So It should get logs in a short period of time. = handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler, 10catalina.org.apache.juli.AsyncFileHandler //omit # FOR DEBUG 10catalina.org.apache.juli.AsyncFileHandler.formatter = org.apache.juli.VerbatimFormatter 10catalina.org.apache.juli.AsyncFileHandler.level = FINEST 10catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 10catalina.org.apache.juli.AsyncFileHandler.prefix = MESSAGES. 10catalina.org.apache.juli.AsyncFileHandler.bufferSize = -1 org.apache.catalina.tribes.MESSAGES.level = FINEST org.apache.catalina.tribes.MESSAGES.handlers = 10catalina.org.apache.juli.AsyncFileHandler > On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino > wrote: > > Hi > > > > Please try mapSendOptions = 8. > > This means that it does not receive ACK of the messages. > > Processing is returned immediately after message transmission is > completed. > > > > As another way, > > Please try the org.apache.catalina.tribes.MESSAGES. (see previous my > > response mail.) > > > > > > > > 2017-02-09 18:23 GMT+09:00 Fady Haikal : > > > >> The is set under the context.xml > >> > >> >>notifyListenersOnReplication="true" > >>mapSendOptions="6"/> > >> > >> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino > >> wrote: > >> > 2017-02-09 18:10 GMT+09:00 Fady Haikal : > >> > > >> >> Keiichi, > >> >> mapSendoption is not configured anyway please check my configuration > >> below > >> >> > >> > > >> > > >> > You have not set . > >> > If you do not use Cluster Manager template, DeltaManager is used by > >> default. > >> > However, in the logs, it seems to be using BackupManager. > >> > Have you set in webapps/{your app} > >> /META-INF/context.xml > >> > or conf/{Engine}/{Host}/{your app}.xml? > >> > > >> > > >> > > >> >> > >> >> > >> >> > >> >> > >> >>>> >> /> > >> >> > >> >> > >> >>>> >> SSLEngine="on" /> > >> >> > >> >>>> >> /> > >> >>>> >> /> > >> >>>> >> /> > >> >> > >> >> > >> >> > >> >> > >> >> >> >> type="org.apache.catalina.UserDatabase" > >> >> description="User database that can be updated and > saved" > >> >> factory="org.apache.catalina.users. > >> >> MemoryUserDatabaseFactory" > >> >> pathname="conf/tomcat-users.xml" /> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> >> >>connectionTimeout="6" maxThreads="500" > >> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false" > >> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" /> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> >> >> jvmRoute="TomcatNode1"> > >> >> > >> >> > >> >> > >> >> >> >>resourceName="UserDatabase"/> > >> >> > >> >> > >> >>>> >> unpackWARs="true" autoDeploy="true" startStopThreads="0"> > >> >> > >> >> > >> >> > >> >> >> >> channelSendOptions="4"> > >> >> > >> >> >> >> className="org.apache.catalina.tribes.membership.McastService" > >> >> address="228.0.0.4" > >> >> port="45560" > >> >> frequency="500" > >> >> dropTime="9000"/> > >> >> >> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver" > >> >> address="auto" > >> >> port="4000" > >> >> autoBind="100" > >> >> selectorTimeout="5000" > >> >> maxThreads="6"/> > >> >> > >> >> >> >> className="org.apache.catalina.tribes.transport. > >> ReplicationTransmitter"> > >> >>>> >> className="org.apache.catalina.tribes.transport.nio. > >> >> PooledParallelSender"/> > >> >> > >> >> >> >> className="org.apache.catalina.tribes.group.interceptors. > >> >>
Re: Operation has timed out
2017-02-09 19:21 GMT+09:00 Fady Haikal: > i already restarted it after changing the parameter to 8 > However, I can not see the log that context has been restarted in the log you showed. You may need to restart Tomcat. > > On Thu, Feb 9, 2017 at 12:16 PM, Keiichi Fujino > wrote: > > It is realy strange. > > Have you failed to restart Context? > > When starting up the AbstractReplicatedMap, the followings should be > logged. > > > > "Initializing AbstractReplicatedMap with context name:xxx" > > "AbstractReplicatedMap[xxx] initialization was completed in xxx ms." > > > > > > 2017-02-09 19:09 GMT+09:00 Fady Haikal : > > > >> i can only see this error related to AbstractReplicatedMap > >> > >> > >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] > >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > >> Unable to send AbstractReplicatedMap.ping message > >> java.lang.NullPointerException > >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap. > >> ping(AbstractReplicatedMap.java:256) > >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat( > >> AbstractReplicatedMap.java:885) > >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat( > >> GroupChannel.java:166) > >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run( > >> GroupChannel.java:697) > >> > >> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino > >> wrote: > >> > These logs seem to be the tomcat memory leak detections logs during > >> context > >> > restart. > >> > It is a different issue from this thread's topic. > >> > > >> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx] > >> > initialization was completed in xxx ms." in your log file? > >> > > >> > 2017-02-09 18:49 GMT+09:00 Fady Haikal : > >> > > >> >> Keiichi, > >> >> once i changed the mapSendOptions and without restarting the below > >> >> shown in the tomcat > >> >> > >> >> > >> >> 09-Feb-2017 12:44:12.299 SEVERE > >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> >> org.apache.catalina.loader.WebappClassLoaderBase. > >> >> checkThreadLocalMapForLeaks > >> >> The web application [imal_services] created a ThreadLocal with key of > >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value > of > >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> >> was stopped. Threads are going to be renewed over time to try and > >> >> avoid a probable memory leak. > >> >> 09-Feb-2017 12:44:12.300 SEVERE > >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> >> org.apache.catalina.loader.WebappClassLoaderBase. > >> >> checkThreadLocalMapForLeaks > >> >> The web application [imal_services] created a ThreadLocal with key of > >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value > of > >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA > >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> >> was stopped. Threads are going to be renewed over time to try and > >> >> avoid a probable memory leak. > >> >> 09-Feb-2017 12:44:12.300 SEVERE > >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> >> org.apache.catalina.loader.WebappClassLoaderBase. > >> >> checkThreadLocalMapForLeaks > >> >> The web application [imal_services] created a ThreadLocal with key of > >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value > of > >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> >> was stopped. Threads are going to be renewed over time to try and > >> >> avoid a probable memory leak. > >> >> 09-Feb-2017 12:44:12.301 SEVERE > >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> >> org.apache.catalina.loader.WebappClassLoaderBase. > >> >> checkThreadLocalMapForLeaks > >> >> The web application [imal_services] created a ThreadLocal with key of > >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value > of > >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT > >> >> Process}]) but failed to remove it when the web application was > >> >> stopped. Threads are going to be renewed over time to try and avoid a > >> >> probable memory leak. > >> >> 09-Feb-2017 12:44:12.301 SEVERE > >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> >> org.apache.catalina.loader.WebappClassLoaderBase. > >> >> checkThreadLocalMapForLeaks > >> >> The web application [imal_services] created a
Re: Operation has timed out
i already restarted it after changing the parameter to 8 On Thu, Feb 9, 2017 at 12:16 PM, Keiichi Fujinowrote: > It is realy strange. > Have you failed to restart Context? > When starting up the AbstractReplicatedMap, the followings should be logged. > > "Initializing AbstractReplicatedMap with context name:xxx" > "AbstractReplicatedMap[xxx] initialization was completed in xxx ms." > > > 2017-02-09 19:09 GMT+09:00 Fady Haikal : > >> i can only see this error related to AbstractReplicatedMap >> >> >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat >> Unable to send AbstractReplicatedMap.ping message >> java.lang.NullPointerException >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap. >> ping(AbstractReplicatedMap.java:256) >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat( >> AbstractReplicatedMap.java:885) >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat( >> GroupChannel.java:166) >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run( >> GroupChannel.java:697) >> >> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino >> wrote: >> > These logs seem to be the tomcat memory leak detections logs during >> context >> > restart. >> > It is a different issue from this thread's topic. >> > >> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx] >> > initialization was completed in xxx ms." in your log file? >> > >> > 2017-02-09 18:49 GMT+09:00 Fady Haikal : >> > >> >> Keiichi, >> >> once i changed the mapSendOptions and without restarting the below >> >> shown in the tomcat >> >> >> >> >> >> 09-Feb-2017 12:44:12.299 SEVERE >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> >> org.apache.catalina.loader.WebappClassLoaderBase. >> >> checkThreadLocalMapForLeaks >> >> The web application [imal_services] created a ThreadLocal with key of >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> >> was stopped. Threads are going to be renewed over time to try and >> >> avoid a probable memory leak. >> >> 09-Feb-2017 12:44:12.300 SEVERE >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> >> org.apache.catalina.loader.WebappClassLoaderBase. >> >> checkThreadLocalMapForLeaks >> >> The web application [imal_services] created a ThreadLocal with key of >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> >> was stopped. Threads are going to be renewed over time to try and >> >> avoid a probable memory leak. >> >> 09-Feb-2017 12:44:12.300 SEVERE >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> >> org.apache.catalina.loader.WebappClassLoaderBase. >> >> checkThreadLocalMapForLeaks >> >> The web application [imal_services] created a ThreadLocal with key of >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> >> was stopped. Threads are going to be renewed over time to try and >> >> avoid a probable memory leak. >> >> 09-Feb-2017 12:44:12.301 SEVERE >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> >> org.apache.catalina.loader.WebappClassLoaderBase. >> >> checkThreadLocalMapForLeaks >> >> The web application [imal_services] created a ThreadLocal with key of >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT >> >> Process}]) but failed to remove it when the web application was >> >> stopped. Threads are going to be renewed over time to try and avoid a >> >> probable memory leak. >> >> 09-Feb-2017 12:44:12.301 SEVERE >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> >> org.apache.catalina.loader.WebappClassLoaderBase. >> >> checkThreadLocalMapForLeaks >> >> The web application [imal_services] created a ThreadLocal with key of >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> >> was stopped. Threads are going to be renewed over time to try and >> >> avoid a
Re: Operation has timed out
It is realy strange. Have you failed to restart Context? When starting up the AbstractReplicatedMap, the followings should be logged. "Initializing AbstractReplicatedMap with context name:xxx" "AbstractReplicatedMap[xxx] initialization was completed in xxx ms." 2017-02-09 19:09 GMT+09:00 Fady Haikal: > i can only see this error related to AbstractReplicatedMap > > > 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > Unable to send AbstractReplicatedMap.ping message > java.lang.NullPointerException > at org.apache.catalina.tribes.tipis.AbstractReplicatedMap. > ping(AbstractReplicatedMap.java:256) > at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat( > AbstractReplicatedMap.java:885) > at org.apache.catalina.tribes.group.GroupChannel.heartbeat( > GroupChannel.java:166) > at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run( > GroupChannel.java:697) > > On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino > wrote: > > These logs seem to be the tomcat memory leak detections logs during > context > > restart. > > It is a different issue from this thread's topic. > > > > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx] > > initialization was completed in xxx ms." in your log file? > > > > 2017-02-09 18:49 GMT+09:00 Fady Haikal : > > > >> Keiichi, > >> once i changed the mapSendOptions and without restarting the below > >> shown in the tomcat > >> > >> > >> 09-Feb-2017 12:44:12.299 SEVERE > >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> org.apache.catalina.loader.WebappClassLoaderBase. > >> checkThreadLocalMapForLeaks > >> The web application [imal_services] created a ThreadLocal with key of > >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> was stopped. Threads are going to be renewed over time to try and > >> avoid a probable memory leak. > >> 09-Feb-2017 12:44:12.300 SEVERE > >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> org.apache.catalina.loader.WebappClassLoaderBase. > >> checkThreadLocalMapForLeaks > >> The web application [imal_services] created a ThreadLocal with key of > >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA > >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> was stopped. Threads are going to be renewed over time to try and > >> avoid a probable memory leak. > >> 09-Feb-2017 12:44:12.300 SEVERE > >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> org.apache.catalina.loader.WebappClassLoaderBase. > >> checkThreadLocalMapForLeaks > >> The web application [imal_services] created a ThreadLocal with key of > >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> was stopped. Threads are going to be renewed over time to try and > >> avoid a probable memory leak. > >> 09-Feb-2017 12:44:12.301 SEVERE > >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> org.apache.catalina.loader.WebappClassLoaderBase. > >> checkThreadLocalMapForLeaks > >> The web application [imal_services] created a ThreadLocal with key of > >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT > >> Process}]) but failed to remove it when the web application was > >> stopped. Threads are going to be renewed over time to try and avoid a > >> probable memory leak. > >> 09-Feb-2017 12:44:12.301 SEVERE > >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> org.apache.catalina.loader.WebappClassLoaderBase. > >> checkThreadLocalMapForLeaks > >> The web application [imal_services] created a ThreadLocal with key of > >> type [com.path.lib.common.util.ThreadAttributes$1] (value > >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application > >> was stopped. Threads are going to be renewed over time to try and > >> avoid a probable memory leak. > >> 09-Feb-2017 12:44:12.302 SEVERE > >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > >> org.apache.catalina.loader.WebappClassLoaderBase. > >> checkThreadLocalMapForLeaks > >> The web application
Re: Operation has timed out
i can only see this error related to AbstractReplicatedMap 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat Unable to send AbstractReplicatedMap.ping message java.lang.NullPointerException at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166) at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujinowrote: > These logs seem to be the tomcat memory leak detections logs during context > restart. > It is a different issue from this thread's topic. > > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx] > initialization was completed in xxx ms." in your log file? > > 2017-02-09 18:49 GMT+09:00 Fady Haikal : > >> Keiichi, >> once i changed the mapSendOptions and without restarting the below >> shown in the tomcat >> >> >> 09-Feb-2017 12:44:12.299 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> org.apache.catalina.loader.WebappClassLoaderBase. >> checkThreadLocalMapForLeaks >> The web application [imal_services] created a ThreadLocal with key of >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> was stopped. Threads are going to be renewed over time to try and >> avoid a probable memory leak. >> 09-Feb-2017 12:44:12.300 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> org.apache.catalina.loader.WebappClassLoaderBase. >> checkThreadLocalMapForLeaks >> The web application [imal_services] created a ThreadLocal with key of >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> was stopped. Threads are going to be renewed over time to try and >> avoid a probable memory leak. >> 09-Feb-2017 12:44:12.300 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> org.apache.catalina.loader.WebappClassLoaderBase. >> checkThreadLocalMapForLeaks >> The web application [imal_services] created a ThreadLocal with key of >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> was stopped. Threads are going to be renewed over time to try and >> avoid a probable memory leak. >> 09-Feb-2017 12:44:12.301 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> org.apache.catalina.loader.WebappClassLoaderBase. >> checkThreadLocalMapForLeaks >> The web application [imal_services] created a ThreadLocal with key of >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT >> Process}]) but failed to remove it when the web application was >> stopped. Threads are going to be renewed over time to try and avoid a >> probable memory leak. >> 09-Feb-2017 12:44:12.301 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> org.apache.catalina.loader.WebappClassLoaderBase. >> checkThreadLocalMapForLeaks >> The web application [imal_services] created a ThreadLocal with key of >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> was stopped. Threads are going to be renewed over time to try and >> avoid a probable memory leak. >> 09-Feb-2017 12:44:12.302 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]] >> org.apache.catalina.loader.WebappClassLoaderBase. >> checkThreadLocalMapForLeaks >> The web application [imal_services] created a ThreadLocal with key of >> type [com.path.lib.common.util.ThreadAttributes$1] (value >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application >> was stopped. Threads are going to be renewed over time to try and >> avoid a probable memory leak. >> 09-Feb-2017 12:44:12.303 SEVERE >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
Re: Operation has timed out
Let's not mix issues here. 1) your Host entry is that means that Tomcat will automatically reload any application when it notices that it has been changed. 2) the parameter you changed is in the application's context, so the application has been changed 3) to reload the application, Tomcat first stops the running application. That's when it noticed all the potential leak issues below. But they have always been there, you just never saw them before. On 09.02.2017 10:49, Fady Haikal wrote: Keiichi, once i changed the mapSendOptions and without restarting the below shown in the tomcat 09-Feb-2017 12:44:12.299 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.300 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.300 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.301 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT Process}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.301 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.302 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.303 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat Unable to send AbstractReplicatedMap.ping message java.lang.NullPointerException at
Re: Operation has timed out
These logs seem to be the tomcat memory leak detections logs during context restart. It is a different issue from this thread's topic. Anyway, Can you see the log likea "AbstractReplicatedMap[xxx] initialization was completed in xxx ms." in your log file? 2017-02-09 18:49 GMT+09:00 Fady Haikal: > Keiichi, > once i changed the mapSendOptions and without restarting the below > shown in the tomcat > > > 09-Feb-2017 12:44:12.299 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > UNKNOWN_PROG_REF}]) but failed to remove it when the web application > was stopped. Threads are going to be renewed over time to try and > avoid a probable memory leak. > 09-Feb-2017 12:44:12.300 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA > UNKNOWN_PROG_REF}]) but failed to remove it when the web application > was stopped. Threads are going to be renewed over time to try and > avoid a probable memory leak. > 09-Feb-2017 12:44:12.300 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > UNKNOWN_PROG_REF}]) but failed to remove it when the web application > was stopped. Threads are going to be renewed over time to try and > avoid a probable memory leak. > 09-Feb-2017 12:44:12.301 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT > Process}]) but failed to remove it when the web application was > stopped. Threads are going to be renewed over time to try and avoid a > probable memory leak. > 09-Feb-2017 12:44:12.301 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > UNKNOWN_PROG_REF}]) but failed to remove it when the web application > was stopped. Threads are going to be renewed over time to try and > avoid a probable memory leak. > 09-Feb-2017 12:44:12.302 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > UNKNOWN_PROG_REF}]) but failed to remove it when the web application > was stopped. Threads are going to be renewed over time to try and > avoid a probable memory leak. > 09-Feb-2017 12:44:12.303 SEVERE > [ContainerBackgroundProcessor[StandardEngine[Catalina]]] > org.apache.catalina.loader.WebappClassLoaderBase. > checkThreadLocalMapForLeaks > The web application [imal_services] created a ThreadLocal with key of > type [com.path.lib.common.util.ThreadAttributes$1] (value > [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of > type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI > UNKNOWN_PROG_REF}]) but failed to remove it when the web application > was stopped. Threads are going to be renewed over time to try and > avoid a probable memory leak. > 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > Unable to send AbstractReplicatedMap.ping message > java.lang.NullPointerException > at org.apache.catalina.tribes.tipis.AbstractReplicatedMap. >
Re: Operation has timed out
Keiichi, once i changed the mapSendOptions and without restarting the below shown in the tomcat 09-Feb-2017 12:44:12.299 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.300 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.300 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.301 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT Process}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.301 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.302 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.303 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [imal_services] created a ThreadLocal with key of type [com.path.lib.common.util.ThreadAttributes$1] (value [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI UNKNOWN_PROG_REF}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1] org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat Unable to send AbstractReplicatedMap.ping message java.lang.NullPointerException at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166) at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) 09-Feb-2017 12:44:14.834 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application
Re: Operation has timed out
i will try that but this required tomcat restart so i will be scheduling tonight a restart, anyway from where i can enable the org.apache.catalina.tribes.MESSAGES i didn't found it in the logging.properties file On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujinowrote: > Hi > > Please try mapSendOptions = 8. > This means that it does not receive ACK of the messages. > Processing is returned immediately after message transmission is completed. > > As another way, > Please try the org.apache.catalina.tribes.MESSAGES. (see previous my > response mail.) > > > > 2017-02-09 18:23 GMT+09:00 Fady Haikal : > >> The is set under the context.xml >> >> >notifyListenersOnReplication="true" >>mapSendOptions="6"/> >> >> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino >> wrote: >> > 2017-02-09 18:10 GMT+09:00 Fady Haikal : >> > >> >> Keiichi, >> >> mapSendoption is not configured anyway please check my configuration >> below >> >> >> > >> > >> > You have not set . >> > If you do not use Cluster Manager template, DeltaManager is used by >> default. >> > However, in the logs, it seems to be using BackupManager. >> > Have you set in webapps/{your app} >> /META-INF/context.xml >> > or conf/{Engine}/{Host}/{your app}.xml? >> > >> > >> > >> >> >> >> >> >> >> >> >> >> > >> /> >> >> >> >> >> >> > >> SSLEngine="on" /> >> >> >> >> > >> /> >> >> > >> /> >> >> > >> /> >> >> >> >> >> >> >> >> >> >> > >> type="org.apache.catalina.UserDatabase" >> >> description="User database that can be updated and saved" >> >> factory="org.apache.catalina.users. >> >> MemoryUserDatabaseFactory" >> >> pathname="conf/tomcat-users.xml" /> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >>connectionTimeout="6" maxThreads="500" >> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false" >> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" /> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> jvmRoute="TomcatNode1"> >> >> >> >> >> >> >> >> > >>resourceName="UserDatabase"/> >> >> >> >> >> >> > >> unpackWARs="true" autoDeploy="true" startStopThreads="0"> >> >> >> >> >> >> >> >> > >> channelSendOptions="4"> >> >> >> >> > >> className="org.apache.catalina.tribes.membership.McastService" >> >> address="228.0.0.4" >> >> port="45560" >> >> frequency="500" >> >> dropTime="9000"/> >> >> > >> className="org.apache.catalina.tribes.transport.nio.NioReceiver" >> >> address="auto" >> >> port="4000" >> >> autoBind="100" >> >> selectorTimeout="5000" >> >> maxThreads="6"/> >> >> >> >> > >> className="org.apache.catalina.tribes.transport. >> ReplicationTransmitter"> >> >> > >> className="org.apache.catalina.tribes.transport.nio. >> >> PooledParallelSender"/> >> >> >> >> > >> className="org.apache.catalina.tribes.group.interceptors. >> >> TcpFailureDetector"/> >> >> > >> className="org.apache.catalina.tribes.group.interceptors. >> >> MessageDispatch15Interceptor"/> >> >> >> >> >> >> > >> >> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|. >> >> *\.html|.*\.css|.*\.txt"/> >> >> > >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> >> >> >> >> > >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/" >> >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/" >> >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/" >> >> watchEnabled="false"/> >> >> >> >> > >> className="org.apache.catalina.ha.session.ClusterSessionListener"/> >> >> >> >> >> >> >> >> >> >> >> >> > >> directory="logs" >> >>prefix="localhost_access_log" suffix=".txt" >> >>pattern="%h %l %u %t %r %s %b" /> >> >> > >> threshold="900" /> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino >> >> wrote: >> >> > Hi >> >> > >> >> > You seems to be using BackupManager. >> >> > What is set for mapSendOptions? >> >> > >> >> > The Cluster module uses the Tomcat JULI logging FW, >> >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES, >> >> you >> >> > can track messages. >> >> > >> >> > Note for
Re: Operation has timed out
Hi Please try mapSendOptions = 8. This means that it does not receive ACK of the messages. Processing is returned immediately after message transmission is completed. As another way, Please try the org.apache.catalina.tribes.MESSAGES. (see previous my response mail.) 2017-02-09 18:23 GMT+09:00 Fady Haikal: > The is set under the context.xml > > notifyListenersOnReplication="true" >mapSendOptions="6"/> > > On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino > wrote: > > 2017-02-09 18:10 GMT+09:00 Fady Haikal : > > > >> Keiichi, > >> mapSendoption is not configured anyway please check my configuration > below > >> > > > > > > You have not set . > > If you do not use Cluster Manager template, DeltaManager is used by > default. > > However, in the logs, it seems to be using BackupManager. > > Have you set in webapps/{your app} > /META-INF/context.xml > > or conf/{Engine}/{Host}/{your app}.xml? > > > > > > > >> > >> > >> > >> > >>>> /> > >> > >> > >>>> SSLEngine="on" /> > >> > >>>> /> > >>>> /> > >>>> /> > >> > >> > >> > >> > >> >> type="org.apache.catalina.UserDatabase" > >> description="User database that can be updated and saved" > >> factory="org.apache.catalina.users. > >> MemoryUserDatabaseFactory" > >> pathname="conf/tomcat-users.xml" /> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >>connectionTimeout="6" maxThreads="500" > >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false" > >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" /> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >> jvmRoute="TomcatNode1"> > >> > >> > >> > >> >>resourceName="UserDatabase"/> > >> > >> > >>>> unpackWARs="true" autoDeploy="true" startStopThreads="0"> > >> > >> > >> > >> >> channelSendOptions="4"> > >> > >> >> className="org.apache.catalina.tribes.membership.McastService" > >> address="228.0.0.4" > >> port="45560" > >> frequency="500" > >> dropTime="9000"/> > >> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver" > >> address="auto" > >> port="4000" > >> autoBind="100" > >> selectorTimeout="5000" > >> maxThreads="6"/> > >> > >> >> className="org.apache.catalina.tribes.transport. > ReplicationTransmitter"> > >>>> className="org.apache.catalina.tribes.transport.nio. > >> PooledParallelSender"/> > >> > >> >> className="org.apache.catalina.tribes.group.interceptors. > >> TcpFailureDetector"/> > >> >> className="org.apache.catalina.tribes.group.interceptors. > >> MessageDispatch15Interceptor"/> > >> > >> > >>>> > >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|. > >> *\.html|.*\.css|.*\.txt"/> > >>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> > >> > >>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/" > >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/" > >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/" > >> watchEnabled="false"/> > >> > >>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/> > >> > >> > >> > >> > >> > >> >> directory="logs" > >>prefix="localhost_access_log" suffix=".txt" > >>pattern="%h %l %u %t %r %s %b" /> > >> >> threshold="900" /> > >> > >> > >> > >> > >> > >> > >> > >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino > >> wrote: > >> > Hi > >> > > >> > You seems to be using BackupManager. > >> > What is set for mapSendOptions? > >> > > >> > The Cluster module uses the Tomcat JULI logging FW, > >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES, > >> you > >> > can track messages. > >> > > >> > Note for using the org.apache.catalina.tribes.MESSAGES: > >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level > >> logs. > >> > So It should get logs in a short period of time. > >> > > >> > > >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal : > >> > > >> >> Guys, we are facing the below errors in Tomcat cluster, please advise > >> >> > >> >> > >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] > >> >>
Re: Operation has timed out
The is set under the context.xml On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujinowrote: > 2017-02-09 18:10 GMT+09:00 Fady Haikal : > >> Keiichi, >> mapSendoption is not configured anyway please check my configuration below >> > > > You have not set . > If you do not use Cluster Manager template, DeltaManager is used by default. > However, in the logs, it seems to be using BackupManager. > Have you set in webapps/{your app} /META-INF/context.xml > or conf/{Engine}/{Host}/{your app}.xml? > > > >> >> >> >> >> > /> >> >> >> > SSLEngine="on" /> >> >> > /> >> > /> >> > /> >> >> >> >> >> > type="org.apache.catalina.UserDatabase" >> description="User database that can be updated and saved" >> factory="org.apache.catalina.users. >> MemoryUserDatabaseFactory" >> pathname="conf/tomcat-users.xml" /> >> >> >> >> >> >> >> >> >> >> >> >connectionTimeout="6" maxThreads="500" >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false" >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" /> >> >> >> >> >> >> >> >> >> >> >> >> >> > jvmRoute="TomcatNode1"> >> >> >> >> >resourceName="UserDatabase"/> >> >> >> > unpackWARs="true" autoDeploy="true" startStopThreads="0"> >> >> >> >> > channelSendOptions="4"> >> >> > className="org.apache.catalina.tribes.membership.McastService" >> address="228.0.0.4" >> port="45560" >> frequency="500" >> dropTime="9000"/> >> > className="org.apache.catalina.tribes.transport.nio.NioReceiver" >> address="auto" >> port="4000" >> autoBind="100" >> selectorTimeout="5000" >> maxThreads="6"/> >> >> > className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> >> > className="org.apache.catalina.tribes.transport.nio. >> PooledParallelSender"/> >> >> > className="org.apache.catalina.tribes.group.interceptors. >> TcpFailureDetector"/> >> > className="org.apache.catalina.tribes.group.interceptors. >> MessageDispatch15Interceptor"/> >> >> >> > >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|. >> *\.html|.*\.css|.*\.txt"/> >> > className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> >> >> > tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/" >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/" >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/" >> watchEnabled="false"/> >> >> > className="org.apache.catalina.ha.session.ClusterSessionListener"/> >> >> >> >> >> >> > directory="logs" >>prefix="localhost_access_log" suffix=".txt" >>pattern="%h %l %u %t %r %s %b" /> >> > threshold="900" /> >> >> >> >> >> >> >> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino >> wrote: >> > Hi >> > >> > You seems to be using BackupManager. >> > What is set for mapSendOptions? >> > >> > The Cluster module uses the Tomcat JULI logging FW, >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES, >> you >> > can track messages. >> > >> > Note for using the org.apache.catalina.tribes.MESSAGES: >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level >> logs. >> > So It should get logs in a short period of time. >> > >> > >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal : >> > >> >> Guys, we are facing the below errors in Tomcat cluster, please advise >> >> >> >> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat >> >> Unable to send AbstractReplicatedMap.ping message >> >> org.apache.catalina.tribes.ChannelException: Operation has timed >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; >> >> at org.apache.catalina.tribes.transport.nio. >> ParallelNioSender.sendMessage( >> >> ParallelNioSender.java:108) >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender. >> >> sendMessage(PooledParallelSender.java:48) >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter. >> >> sendMessage(ReplicationTransmitter.java:54) >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage( >> >> ChannelCoordinator.java:82) >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( >> >>
Re: Operation has timed out
2017-02-09 18:10 GMT+09:00 Fady Haikal: > Keiichi, > mapSendoption is not configured anyway please check my configuration below > You have not set . If you do not use Cluster Manager template, DeltaManager is used by default. However, in the logs, it seems to be using BackupManager. Have you set in webapps/{your app} /META-INF/context.xml or conf/{Engine}/{Host}/{your app}.xml? > > > > >/> > > >SSLEngine="on" /> > >/> >/> >/> > > > > >type="org.apache.catalina.UserDatabase" > description="User database that can be updated and saved" > factory="org.apache.catalina.users. > MemoryUserDatabaseFactory" > pathname="conf/tomcat-users.xml" /> > > > > > > > > > > > connectionTimeout="6" maxThreads="500" > minSpareThreads="25" maxSpareThreads="75" enableLookups="false" > disableUploadTimeout="true" acceptCount="100" redirectPort="8443" /> > > > > > > > > > > > > > jvmRoute="TomcatNode1"> > > > > resourceName="UserDatabase"/> > > >unpackWARs="true" autoDeploy="true" startStopThreads="0"> > > > >channelSendOptions="4"> > > className="org.apache.catalina.tribes.membership.McastService" > address="228.0.0.4" > port="45560" > frequency="500" > dropTime="9000"/> > className="org.apache.catalina.tribes.transport.nio.NioReceiver" > address="auto" > port="4000" > autoBind="100" > selectorTimeout="5000" > maxThreads="6"/> > > className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> >className="org.apache.catalina.tribes.transport.nio. > PooledParallelSender"/> > > className="org.apache.catalina.tribes.group.interceptors. > TcpFailureDetector"/> > className="org.apache.catalina.tribes.group.interceptors. > MessageDispatch15Interceptor"/> > > > > filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|. > *\.html|.*\.css|.*\.txt"/> >className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> > >tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/" > deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/" > watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/" > watchEnabled="false"/> > >className="org.apache.catalina.ha.session.ClusterSessionListener"/> > > > > > > directory="logs" >prefix="localhost_access_log" suffix=".txt" >pattern="%h %l %u %t %r %s %b" /> > threshold="900" /> > > > > > > > > On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino > wrote: > > Hi > > > > You seems to be using BackupManager. > > What is set for mapSendOptions? > > > > The Cluster module uses the Tomcat JULI logging FW, > > so you can enable logging on the org.apache.catalina.tribes.MESSAGES, > you > > can track messages. > > > > Note for using the org.apache.catalina.tribes.MESSAGES: > > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level > logs. > > So It should get logs in a short period of time. > > > > > > 2017-02-06 15:52 GMT+09:00 Fady Haikal : > > > >> Guys, we are facing the below errors in Tomcat cluster, please advise > >> > >> > >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] > >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > >> Unable to send AbstractReplicatedMap.ping message > >> org.apache.catalina.tribes.ChannelException: Operation has timed > >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; > >> at org.apache.catalina.tribes.transport.nio. > ParallelNioSender.sendMessage( > >> ParallelNioSender.java:108) > >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender. > >> sendMessage(PooledParallelSender.java:48) > >> at org.apache.catalina.tribes.transport.ReplicationTransmitter. > >> sendMessage(ReplicationTransmitter.java:54) > >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage( > >> ChannelCoordinator.java:82) > >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( > >> ChannelInterceptorBase.java:76) > >> at org.apache.catalina.tribes.group.interceptors. > >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor. > java:81) > >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( > >> ChannelInterceptorBase.java:76) > >> at
Re: Operation has timed out
Keiichi, mapSendoption is not configured anyway please check my configuration below On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujinowrote: > Hi > > You seems to be using BackupManager. > What is set for mapSendOptions? > > The Cluster module uses the Tomcat JULI logging FW, > so you can enable logging on the org.apache.catalina.tribes.MESSAGES, you > can track messages. > > Note for using the org.apache.catalina.tribes.MESSAGES: > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs. > So It should get logs in a short period of time. > > > 2017-02-06 15:52 GMT+09:00 Fady Haikal : > >> Guys, we are facing the below errors in Tomcat cluster, please advise >> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat >> Unable to send AbstractReplicatedMap.ping message >> org.apache.catalina.tribes.ChannelException: Operation has timed >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; >> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage( >> ParallelNioSender.java:108) >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender. >> sendMessage(PooledParallelSender.java:48) >> at org.apache.catalina.tribes.transport.ReplicationTransmitter. >> sendMessage(ReplicationTransmitter.java:54) >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage( >> ChannelCoordinator.java:82) >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( >> ChannelInterceptorBase.java:76) >> at org.apache.catalina.tribes.group.interceptors. >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( >> ChannelInterceptorBase.java:76) >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector. >> sendMessage(TcpFailureDetector.java:93) >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( >> ChannelInterceptorBase.java:76) >> at org.apache.catalina.tribes.group.GroupChannel.send( >> GroupChannel.java:233) >> at org.apache.catalina.tribes.group.GroupChannel.send( >> GroupChannel.java:186) >> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap. >> ping(AbstractReplicatedMap.java:267) >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat( >> AbstractReplicatedMap.java:885) >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat( >> GroupChannel.java:161) >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run( >> GroupChannel.java:697) >> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to >> process request in NioReceiver >> java.io.IOException: A non-blocking socket operation could not be >> completed immediately >> at sun.nio.ch.SocketDispatcher.close0(Native Method) >> at sun.nio.ch.SocketDispatcher.close(Unknown Source) >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source) >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source) >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source) >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) >> at sun.nio.ch.SelectorImpl.select(Unknown Source) >> at org.apache.catalina.tribes.transport.nio.NioReceiver. >> listen(NioReceiver.java:272) >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run( >> NioReceiver.java:425) >> at java.lang.Thread.run(Unknown Source) >> >> Regards, >> Fady >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > > -- > Keiichi.Fujino - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Operation has timed out
Hi You seems to be using BackupManager. What is set for mapSendOptions? The Cluster module uses the Tomcat JULI logging FW, so you can enable logging on the org.apache.catalina.tribes.MESSAGES, you can track messages. Note for using the org.apache.catalina.tribes.MESSAGES: The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs. So It should get logs in a short period of time. 2017-02-06 15:52 GMT+09:00 Fady Haikal: > Guys, we are facing the below errors in Tomcat cluster, please advise > > > 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > Unable to send AbstractReplicatedMap.ping message > org.apache.catalina.tribes.ChannelException: Operation has timed > out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; > at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage( > ParallelNioSender.java:108) > at org.apache.catalina.tribes.transport.nio.PooledParallelSender. > sendMessage(PooledParallelSender.java:48) > at org.apache.catalina.tribes.transport.ReplicationTransmitter. > sendMessage(ReplicationTransmitter.java:54) > at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage( > ChannelCoordinator.java:82) > at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( > ChannelInterceptorBase.java:76) > at org.apache.catalina.tribes.group.interceptors. > MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) > at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( > ChannelInterceptorBase.java:76) > at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector. > sendMessage(TcpFailureDetector.java:93) > at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage( > ChannelInterceptorBase.java:76) > at org.apache.catalina.tribes.group.GroupChannel.send( > GroupChannel.java:233) > at org.apache.catalina.tribes.group.GroupChannel.send( > GroupChannel.java:186) > at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) > at org.apache.catalina.tribes.tipis.AbstractReplicatedMap. > ping(AbstractReplicatedMap.java:267) > at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat( > AbstractReplicatedMap.java:885) > at org.apache.catalina.tribes.group.GroupChannel.heartbeat( > GroupChannel.java:161) > at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run( > GroupChannel.java:697) > > > 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] > org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to > process request in NioReceiver > java.io.IOException: A non-blocking socket operation could not be > completed immediately > at sun.nio.ch.SocketDispatcher.close0(Native Method) > at sun.nio.ch.SocketDispatcher.close(Unknown Source) > at sun.nio.ch.SocketChannelImpl.kill(Unknown Source) > at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source) > at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source) > at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) > at sun.nio.ch.SelectorImpl.select(Unknown Source) > at org.apache.catalina.tribes.transport.nio.NioReceiver. > listen(NioReceiver.java:272) > at org.apache.catalina.tribes.transport.nio.NioReceiver.run( > NioReceiver.java:425) > at java.lang.Thread.run(Unknown Source) > > Regards, > Fady > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Keiichi.Fujino
Re: Operation has timed out
Ashwin, I tried changing the port but unfortunately i'm still facing the same error On Wed, Feb 8, 2017 at 4:44 PM, ashwin rajputwrote: > Hi Fady, > > can you please try changing default Multicast port to something like : > From port="45564" > to port="45562" or port="45560". > > let me know the outcome will try to fix it and please make sure if you have > firewall enable on system please disable it first and then give a try > otherwise you have to change firewall configuration to allow > multicast traffic. > > Regards, > Ashwin > > On Wed, Feb 8, 2017 at 12:56 PM, Fady Haikal wrote: > >> Hi All, >> As u know we are using 224.0.0.4 multicast IP for tomcat cluster >> (Node1: 10.114.43.102 / Node2: 10.114.43.103) i was trying to ping the >> multicast IP i get reply from 10.114.43.51 IP! >> >> Also i have executed the nslookup for 224.0.0.4 i get the DC IP >> (10.114.43.7) and mcast.net domain: >> >> C:\Users\Administrator>nslookup 224.0.0.4 >> Server: hq-dc02.albaraka.com.sd >> Address: 10.114.43.7 >> >> Name:dvmrp.mcast.net >> Address: 224.0.0.4 >> >> On Wed, Feb 8, 2017 at 8:59 AM, Fady Haikal wrote: >> > Ashwin, >> > I'm using the below configuration, please let me know how i can check >> > if i'm using a unique multicast address and port >> > >> > >> > >> > > > className="org.apache.catalina.tribes.membership.McastService" >> > address="228.0.0.4" >> > port="45564" >> > frequency="500" >> > dropTime="9000"/> >> > > > className="org.apache.catalina.tribes.transport.nio.NioReceiver" >> > address="auto" >> > port="4000" >> > autoBind="100" >> > selectorTimeout="5000" >> > maxThreads="6"/> >> > >> > On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput >> wrote: >> >> I am not sure if anyone has verified below. >> >> >> >> Have you verifyed clustering is using unique multicast address and port. >> >> Cluster multicast address should be unique and not used by any other >> >> cluster. >> >> >> >> Regards, >> >> Ashwin >> >> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" >> wrote: >> >> >> >>> On 07.02.2017 17:20, Fady Haikal wrote: >> >>> >> Christopher, >> For the first time >> >> >>> >> >>> @Christopher : just to make sure you got that bit, buried below : the >> >>> actual replication seems to work fine. The problem is only these >> >>> "unsuccesful ping" messages in the log, which fill the log, and which >> so >> >>> far nobody has managed to find an explanation for. >> >>> >> >>> >> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz >> wrote: >> >> > -BEGIN PGP SIGNED MESSAGE- >> > Hash: SHA256 >> > >> > Fady, >> > >> > On 2/7/17 10:53 AM, Fady Haikal wrote: >> > >> >> ProcessPID Protocol local address local port >> >> Remote Address State Tomcat8.exe 8160 TCP >> >> imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED >> >> >> > >> > Stupid question: was this working in the past, and it stopped >> working? >> > Or are you trying to get this working for the first time? >> > >> > - -chris >> > >> > On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal >> >> wrote: >> >> >> >>> Yes there is a ESTABLISHED connection, the replication of >> >>> sessions is working fine (port 4000 is for tomcat cluster) but we >> >>> also faced this error on the log file >> >>> >> >>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) >> >>> wrote: >> >>> >> On 07.02.2017 16:24, Fady Haikal wrote: >> >> > >> > Hi, telnet IP 4000 is working fine, i installed a tool for >> > network monitoring at the level of IP and Port and i didnt >> > see any disconnection, >> > >> >> >> but did you see a *connection* ? I mean, on the pinging node, >> if you use the Windows "netstat" program, for example as >> netstat -aon -p TCP you should see a list of connections in the >> ESTABLISHED state, of which one of the IP/ports should be your >> target IP:4000 (in the "remote" column). And on the pinged >> node, this port :4000 should be in the "local" column, in >> LISTEN mode (and also probably one in the ESTABLISHED state, if >> they agree.) >> >> Is that the case ? >> >> >> >> and yes i'm sure that no firewall is enabled. >> >> > >> > >> > I saw some strange think on the server that I have tried to >> >
Re: Operation has timed out
Hi Fady, can you please try changing default Multicast port to something like : >From port="45564" to port="45562" or port="45560". let me know the outcome will try to fix it and please make sure if you have firewall enable on system please disable it first and then give a try otherwise you have to change firewall configuration to allow multicast traffic. Regards, Ashwin On Wed, Feb 8, 2017 at 12:56 PM, Fady Haikalwrote: > Hi All, > As u know we are using 224.0.0.4 multicast IP for tomcat cluster > (Node1: 10.114.43.102 / Node2: 10.114.43.103) i was trying to ping the > multicast IP i get reply from 10.114.43.51 IP! > > Also i have executed the nslookup for 224.0.0.4 i get the DC IP > (10.114.43.7) and mcast.net domain: > > C:\Users\Administrator>nslookup 224.0.0.4 > Server: hq-dc02.albaraka.com.sd > Address: 10.114.43.7 > > Name:dvmrp.mcast.net > Address: 224.0.0.4 > > On Wed, Feb 8, 2017 at 8:59 AM, Fady Haikal wrote: > > Ashwin, > > I'm using the below configuration, please let me know how i can check > > if i'm using a unique multicast address and port > > > > > > > > > className="org.apache.catalina.tribes.membership.McastService" > > address="228.0.0.4" > > port="45564" > > frequency="500" > > dropTime="9000"/> > > > className="org.apache.catalina.tribes.transport.nio.NioReceiver" > > address="auto" > > port="4000" > > autoBind="100" > > selectorTimeout="5000" > > maxThreads="6"/> > > > > On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput > wrote: > >> I am not sure if anyone has verified below. > >> > >> Have you verifyed clustering is using unique multicast address and port. > >> Cluster multicast address should be unique and not used by any other > >> cluster. > >> > >> Regards, > >> Ashwin > >> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" > wrote: > >> > >>> On 07.02.2017 17:20, Fady Haikal wrote: > >>> > Christopher, > For the first time > > >>> > >>> @Christopher : just to make sure you got that bit, buried below : the > >>> actual replication seems to work fine. The problem is only these > >>> "unsuccesful ping" messages in the log, which fill the log, and which > so > >>> far nobody has managed to find an explanation for. > >>> > >>> > On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz > wrote: > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > Fady, > > > > On 2/7/17 10:53 AM, Fady Haikal wrote: > > > >> ProcessPID Protocol local address local port > >> Remote Address State Tomcat8.exe 8160 TCP > >> imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED > >> > > > > Stupid question: was this working in the past, and it stopped > working? > > Or are you trying to get this working for the first time? > > > > - -chris > > > > On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal > >> wrote: > >> > >>> Yes there is a ESTABLISHED connection, the replication of > >>> sessions is working fine (port 4000 is for tomcat cluster) but we > >>> also faced this error on the log file > >>> > >>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) > >>> wrote: > >>> > On 07.02.2017 16:24, Fady Haikal wrote: > > > > > Hi, telnet IP 4000 is working fine, i installed a tool for > > network monitoring at the level of IP and Port and i didnt > > see any disconnection, > > > > > but did you see a *connection* ? I mean, on the pinging node, > if you use the Windows "netstat" program, for example as > netstat -aon -p TCP you should see a list of connections in the > ESTABLISHED state, of which one of the IP/ports should be your > target IP:4000 (in the "remote" column). And on the pinged > node, this port :4000 should be in the "local" column, in > LISTEN mode (and also probably one in the ESTABLISHED state, if > they agree.) > > Is that the case ? > > > > and yes i'm sure that no firewall is enabled. > > > > > > > I saw some strange think on the server that I have tried to > > ping the multicast IP (228.0.0.4) and i get reply from > > different IPs in the network, i don't know why and how i get > > those IPs, after checking with the network team they told me > > that those IPs are related to the SAN storage taking into > > consideration that the
Re: Operation has timed out
Hi All, As u know we are using 224.0.0.4 multicast IP for tomcat cluster (Node1: 10.114.43.102 / Node2: 10.114.43.103) i was trying to ping the multicast IP i get reply from 10.114.43.51 IP! Also i have executed the nslookup for 224.0.0.4 i get the DC IP (10.114.43.7) and mcast.net domain: C:\Users\Administrator>nslookup 224.0.0.4 Server: hq-dc02.albaraka.com.sd Address: 10.114.43.7 Name:dvmrp.mcast.net Address: 224.0.0.4 On Wed, Feb 8, 2017 at 8:59 AM, Fady Haikalwrote: > Ashwin, > I'm using the below configuration, please let me know how i can check > if i'm using a unique multicast address and port > > > className="org.apache.catalina.tribes.group.GroupChannel"> > className="org.apache.catalina.tribes.membership.McastService" > address="228.0.0.4" > port="45564" > frequency="500" > dropTime="9000"/> > className="org.apache.catalina.tribes.transport.nio.NioReceiver" > address="auto" > port="4000" > autoBind="100" > selectorTimeout="5000" > maxThreads="6"/> > > On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput wrote: >> I am not sure if anyone has verified below. >> >> Have you verifyed clustering is using unique multicast address and port. >> Cluster multicast address should be unique and not used by any other >> cluster. >> >> Regards, >> Ashwin >> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" wrote: >> >>> On 07.02.2017 17:20, Fady Haikal wrote: >>> Christopher, For the first time >>> >>> @Christopher : just to make sure you got that bit, buried below : the >>> actual replication seems to work fine. The problem is only these >>> "unsuccesful ping" messages in the log, which fill the log, and which so >>> far nobody has managed to find an explanation for. >>> >>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Fady, > > On 2/7/17 10:53 AM, Fady Haikal wrote: > >> ProcessPID Protocol local address local port >> Remote Address State Tomcat8.exe 8160 TCP >> imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED >> > > Stupid question: was this working in the past, and it stopped working? > Or are you trying to get this working for the first time? > > - -chris > > On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal >> wrote: >> >>> Yes there is a ESTABLISHED connection, the replication of >>> sessions is working fine (port 4000 is for tomcat cluster) but we >>> also faced this error on the log file >>> >>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) >>> wrote: >>> On 07.02.2017 16:24, Fady Haikal wrote: > > Hi, telnet IP 4000 is working fine, i installed a tool for > network monitoring at the level of IP and Port and i didnt > see any disconnection, > but did you see a *connection* ? I mean, on the pinging node, if you use the Windows "netstat" program, for example as netstat -aon -p TCP you should see a list of connections in the ESTABLISHED state, of which one of the IP/ports should be your target IP:4000 (in the "remote" column). And on the pinged node, this port :4000 should be in the "local" column, in LISTEN mode (and also probably one in the ESTABLISHED state, if they agree.) Is that the case ? and yes i'm sure that no firewall is enabled. > > > I saw some strange think on the server that I have tried to > ping the multicast IP (228.0.0.4) and i get reply from > different IPs in the network, i don't know why and how i get > those IPs, after checking with the network team they told me > that those IPs are related to the SAN storage taking into > consideration that the Tomcat servers are not connected in > anyway to that SUN storage. > > > On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) > wrote: > >> >> Hi. >> >> This is for the Tomcat/Tribes experts on the list. >> >> I know nothing of Tribes, but the on-line documentation >> seems to say that the communication happens over TCP and >> that the protocol used is not encrypted. Fady previously >> tried a standard "ping" and a "telnet" between the two >> nodes, and that is the base for him mentioning that
Re: Operation has timed out
Ashwin, I'm using the below configuration, please let me know how i can check if i'm using a unique multicast address and port On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajputwrote: > I am not sure if anyone has verified below. > > Have you verifyed clustering is using unique multicast address and port. > Cluster multicast address should be unique and not used by any other > cluster. > > Regards, > Ashwin > On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" wrote: > >> On 07.02.2017 17:20, Fady Haikal wrote: >> >>> Christopher, >>> For the first time >>> >> >> @Christopher : just to make sure you got that bit, buried below : the >> actual replication seems to work fine. The problem is only these >> "unsuccesful ping" messages in the log, which fill the log, and which so >> far nobody has managed to find an explanation for. >> >> >>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz >>> wrote: >>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Fady, On 2/7/17 10:53 AM, Fady Haikal wrote: > ProcessPID Protocol local address local port > Remote Address State Tomcat8.exe 8160 TCP > imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED > Stupid question: was this working in the past, and it stopped working? Or are you trying to get this working for the first time? - -chris On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal > wrote: > >> Yes there is a ESTABLISHED connection, the replication of >> sessions is working fine (port 4000 is for tomcat cluster) but we >> also faced this error on the log file >> >> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) >> wrote: >> >>> On 07.02.2017 16:24, Fady Haikal wrote: >>> Hi, telnet IP 4000 is working fine, i installed a tool for network monitoring at the level of IP and Port and i didnt see any disconnection, >>> >>> >>> but did you see a *connection* ? I mean, on the pinging node, >>> if you use the Windows "netstat" program, for example as >>> netstat -aon -p TCP you should see a list of connections in the >>> ESTABLISHED state, of which one of the IP/ports should be your >>> target IP:4000 (in the "remote" column). And on the pinged >>> node, this port :4000 should be in the "local" column, in >>> LISTEN mode (and also probably one in the ESTABLISHED state, if >>> they agree.) >>> >>> Is that the case ? >>> >>> >>> >>> and yes i'm sure that no firewall is enabled. >>> I saw some strange think on the server that I have tried to ping the multicast IP (228.0.0.4) and i get reply from different IPs in the network, i don't know why and how i get those IPs, after checking with the network team they told me that those IPs are related to the SAN storage taking into consideration that the Tomcat servers are not connected in anyway to that SUN storage. On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) wrote: > > Hi. > > This is for the Tomcat/Tribes experts on the list. > > I know nothing of Tribes, but the on-line documentation > seems to say that the communication happens over TCP and > that the protocol used is not encrypted. Fady previously > tried a standard "ping" and a "telnet" between the two > nodes, and that is the base for him mentioning that "there > is no network disconnection" between the nodes. > Nevertheless, the calling pinging node seems to say that it > times out without getting a response fom the target node. > There is evidently a contradiction there. So this could > still be some kind of network issue. > > Considering that the protocol command for this "ping" > should be known by someone here, would it not be possible > to imagine a little program in some scripting language (or > even java, God forbid), which would open a TCP channel with > the target node IP/port, send such a "ping" message, wait > for a response and report the result ? That would at least > confirm/deny that the problem is with the network. > > The log below does not for example say if the error happens > when opening the TCP communication channel, or after > sending the ping message on it, (Of course, testing the TCP > open could be done with "telnet IP 4000", but I don't know > if Fady tried this). Maybe tribes also already contains > some löw-level debugging options ?
Re: Operation has timed out
I am not sure if anyone has verified below. Have you verifyed clustering is using unique multicast address and port. Cluster multicast address should be unique and not used by any other cluster. Regards, Ashwin On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)"wrote: > On 07.02.2017 17:20, Fady Haikal wrote: > >> Christopher, >> For the first time >> > > @Christopher : just to make sure you got that bit, buried below : the > actual replication seems to work fine. The problem is only these > "unsuccesful ping" messages in the log, which fill the log, and which so > far nobody has managed to find an explanation for. > > >> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz >> wrote: >> >>> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA256 >>> >>> Fady, >>> >>> On 2/7/17 10:53 AM, Fady Haikal wrote: >>> ProcessPID Protocol local address local port Remote Address State Tomcat8.exe 8160 TCP imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED >>> >>> Stupid question: was this working in the past, and it stopped working? >>> Or are you trying to get this working for the first time? >>> >>> - -chris >>> >>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal wrote: > Yes there is a ESTABLISHED connection, the replication of > sessions is working fine (port 4000 is for tomcat cluster) but we > also faced this error on the log file > > On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) > wrote: > >> On 07.02.2017 16:24, Fady Haikal wrote: >> >>> >>> Hi, telnet IP 4000 is working fine, i installed a tool for >>> network monitoring at the level of IP and Port and i didnt >>> see any disconnection, >>> >> >> >> but did you see a *connection* ? I mean, on the pinging node, >> if you use the Windows "netstat" program, for example as >> netstat -aon -p TCP you should see a list of connections in the >> ESTABLISHED state, of which one of the IP/ports should be your >> target IP:4000 (in the "remote" column). And on the pinged >> node, this port :4000 should be in the "local" column, in >> LISTEN mode (and also probably one in the ESTABLISHED state, if >> they agree.) >> >> Is that the case ? >> >> >> >> and yes i'm sure that no firewall is enabled. >> >>> >>> >>> I saw some strange think on the server that I have tried to >>> ping the multicast IP (228.0.0.4) and i get reply from >>> different IPs in the network, i don't know why and how i get >>> those IPs, after checking with the network team they told me >>> that those IPs are related to the SAN storage taking into >>> consideration that the Tomcat servers are not connected in >>> anyway to that SUN storage. >>> >>> >>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) >>> wrote: >>> Hi. This is for the Tomcat/Tribes experts on the list. I know nothing of Tribes, but the on-line documentation seems to say that the communication happens over TCP and that the protocol used is not encrypted. Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is the base for him mentioning that "there is no network disconnection" between the nodes. Nevertheless, the calling pinging node seems to say that it times out without getting a response fom the target node. There is evidently a contradiction there. So this could still be some kind of network issue. Considering that the protocol command for this "ping" should be known by someone here, would it not be possible to imagine a little program in some scripting language (or even java, God forbid), which would open a TCP channel with the target node IP/port, send such a "ping" message, wait for a response and report the result ? That would at least confirm/deny that the problem is with the network. The log below does not for example say if the error happens when opening the TCP communication channel, or after sending the ping message on it, (Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if Fady tried this). Maybe tribes also already contains some löw-level debugging options ? wireshark maybe another option, but it has quite a learning curve. And this is on Windows. By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network policies" there which can have
Re: Operation has timed out
On 07.02.2017 17:20, Fady Haikal wrote: Christopher, For the first time @Christopher : just to make sure you got that bit, buried below : the actual replication seems to work fine. The problem is only these "unsuccesful ping" messages in the log, which fill the log, and which so far nobody has managed to find an explanation for. On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultzwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Fady, On 2/7/17 10:53 AM, Fady Haikal wrote: ProcessPID Protocol local address local port Remote Address State Tomcat8.exe 8160 TCP imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED Stupid question: was this working in the past, and it stopped working? Or are you trying to get this working for the first time? - -chris On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal wrote: Yes there is a ESTABLISHED connection, the replication of sessions is working fine (port 4000 is for tomcat cluster) but we also faced this error on the log file On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) wrote: On 07.02.2017 16:24, Fady Haikal wrote: Hi, telnet IP 4000 is working fine, i installed a tool for network monitoring at the level of IP and Port and i didnt see any disconnection, but did you see a *connection* ? I mean, on the pinging node, if you use the Windows "netstat" program, for example as netstat -aon -p TCP you should see a list of connections in the ESTABLISHED state, of which one of the IP/ports should be your target IP:4000 (in the "remote" column). And on the pinged node, this port :4000 should be in the "local" column, in LISTEN mode (and also probably one in the ESTABLISHED state, if they agree.) Is that the case ? and yes i'm sure that no firewall is enabled. I saw some strange think on the server that I have tried to ping the multicast IP (228.0.0.4) and i get reply from different IPs in the network, i don't know why and how i get those IPs, after checking with the network team they told me that those IPs are related to the SAN storage taking into consideration that the Tomcat servers are not connected in anyway to that SUN storage. On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) wrote: Hi. This is for the Tomcat/Tribes experts on the list. I know nothing of Tribes, but the on-line documentation seems to say that the communication happens over TCP and that the protocol used is not encrypted. Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is the base for him mentioning that "there is no network disconnection" between the nodes. Nevertheless, the calling pinging node seems to say that it times out without getting a response fom the target node. There is evidently a contradiction there. So this could still be some kind of network issue. Considering that the protocol command for this "ping" should be known by someone here, would it not be possible to imagine a little program in some scripting language (or even java, God forbid), which would open a TCP channel with the target node IP/port, send such a "ping" message, wait for a response and report the result ? That would at least confirm/deny that the problem is with the network. The log below does not for example say if the error happens when opening the TCP communication channel, or after sending the ping message on it, (Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if Fady tried this). Maybe tribes also already contains some löw-level debugging options ? wireshark maybe another option, but it has quite a learning curve. And this is on Windows. By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network policies" there which can have wide-ranging side-effects. On 07.02.2017 14:42, Fady Haikal wrote: Hi, issue still not fixed. Tomcat session replication is not able to replicate the key from node to node, please find below the error, taking into consideration that there is no network disconnection between 2 nodes 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn fo Unable to replicate backup key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10 , 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; at
Re: Operation has timed out
Christopher, For the first time On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultzwrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Fady, > > On 2/7/17 10:53 AM, Fady Haikal wrote: >> ProcessPID Protocol local address local port >> Remote Address State Tomcat8.exe 8160 TCP >> imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED > > Stupid question: was this working in the past, and it stopped working? > Or are you trying to get this working for the first time? > > - -chris > >> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal >> wrote: >>> Yes there is a ESTABLISHED connection, the replication of >>> sessions is working fine (port 4000 is for tomcat cluster) but we >>> also faced this error on the log file >>> >>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) >>> wrote: On 07.02.2017 16:24, Fady Haikal wrote: > > Hi, telnet IP 4000 is working fine, i installed a tool for > network monitoring at the level of IP and Port and i didnt > see any disconnection, but did you see a *connection* ? I mean, on the pinging node, if you use the Windows "netstat" program, for example as netstat -aon -p TCP you should see a list of connections in the ESTABLISHED state, of which one of the IP/ports should be your target IP:4000 (in the "remote" column). And on the pinged node, this port :4000 should be in the "local" column, in LISTEN mode (and also probably one in the ESTABLISHED state, if they agree.) Is that the case ? and yes i'm sure that no firewall is enabled. > > > I saw some strange think on the server that I have tried to > ping the multicast IP (228.0.0.4) and i get reply from > different IPs in the network, i don't know why and how i get > those IPs, after checking with the network team they told me > that those IPs are related to the SAN storage taking into > consideration that the Tomcat servers are not connected in > anyway to that SUN storage. > > > On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) > wrote: >> >> Hi. >> >> This is for the Tomcat/Tribes experts on the list. >> >> I know nothing of Tribes, but the on-line documentation >> seems to say that the communication happens over TCP and >> that the protocol used is not encrypted. Fady previously >> tried a standard "ping" and a "telnet" between the two >> nodes, and that is the base for him mentioning that "there >> is no network disconnection" between the nodes. >> Nevertheless, the calling pinging node seems to say that it >> times out without getting a response fom the target node. >> There is evidently a contradiction there. So this could >> still be some kind of network issue. >> >> Considering that the protocol command for this "ping" >> should be known by someone here, would it not be possible >> to imagine a little program in some scripting language (or >> even java, God forbid), which would open a TCP channel with >> the target node IP/port, send such a "ping" message, wait >> for a response and report the result ? That would at least >> confirm/deny that the problem is with the network. >> >> The log below does not for example say if the error happens >> when opening the TCP communication channel, or after >> sending the ping message on it, (Of course, testing the TCP >> open could be done with "telnet IP 4000", but I don't know >> if Fady tried this). Maybe tribes also already contains >> some löw-level debugging options ? wireshark maybe another >> option, but it has quite a learning curve. And this is on >> Windows. >> >> By the way Fady, are you sure that your "Windows Firewall >> with Enhanced Security" is not just dropping TCP packets >> to/from port 40xx (or from "java.exe") ? There are some >> "network policies" there which can have wide-ranging >> side-effects. >> >> >> >> >> On 07.02.2017 14:42, Fady Haikal wrote: >>> >>> >>> Hi, issue still not fixed. Tomcat session replication is >>> not able to replicate the key from node to node, please >>> find below the error, taking into consideration that >>> there is no network disconnection between 2 nodes >>> >>> >>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] >>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn > fo >>> >>> > Unable to replicate backup >>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to >>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10 > , >>> >>> > 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, >>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 >>>
Re: Operation has timed out
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Fady, On 2/7/17 10:53 AM, Fady Haikal wrote: > ProcessPID Protocol local address local port > Remote Address State Tomcat8.exe 8160 TCP > imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED Stupid question: was this working in the past, and it stopped working? Or are you trying to get this working for the first time? - -chris > On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal> wrote: >> Yes there is a ESTABLISHED connection, the replication of >> sessions is working fine (port 4000 is for tomcat cluster) but we >> also faced this error on the log file >> >> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) >> wrote: >>> On 07.02.2017 16:24, Fady Haikal wrote: Hi, telnet IP 4000 is working fine, i installed a tool for network monitoring at the level of IP and Port and i didnt see any disconnection, >>> >>> >>> but did you see a *connection* ? I mean, on the pinging node, >>> if you use the Windows "netstat" program, for example as >>> netstat -aon -p TCP you should see a list of connections in the >>> ESTABLISHED state, of which one of the IP/ports should be your >>> target IP:4000 (in the "remote" column). And on the pinged >>> node, this port :4000 should be in the "local" column, in >>> LISTEN mode (and also probably one in the ESTABLISHED state, if >>> they agree.) >>> >>> Is that the case ? >>> >>> >>> >>> and yes i'm sure that no firewall is enabled. I saw some strange think on the server that I have tried to ping the multicast IP (228.0.0.4) and i get reply from different IPs in the network, i don't know why and how i get those IPs, after checking with the network team they told me that those IPs are related to the SAN storage taking into consideration that the Tomcat servers are not connected in anyway to that SUN storage. On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) wrote: > > Hi. > > This is for the Tomcat/Tribes experts on the list. > > I know nothing of Tribes, but the on-line documentation > seems to say that the communication happens over TCP and > that the protocol used is not encrypted. Fady previously > tried a standard "ping" and a "telnet" between the two > nodes, and that is the base for him mentioning that "there > is no network disconnection" between the nodes. > Nevertheless, the calling pinging node seems to say that it > times out without getting a response fom the target node. > There is evidently a contradiction there. So this could > still be some kind of network issue. > > Considering that the protocol command for this "ping" > should be known by someone here, would it not be possible > to imagine a little program in some scripting language (or > even java, God forbid), which would open a TCP channel with > the target node IP/port, send such a "ping" message, wait > for a response and report the result ? That would at least > confirm/deny that the problem is with the network. > > The log below does not for example say if the error happens > when opening the TCP communication channel, or after > sending the ping message on it, (Of course, testing the TCP > open could be done with "telnet IP 4000", but I don't know > if Fady tried this). Maybe tribes also already contains > some löw-level debugging options ? wireshark maybe another > option, but it has quite a learning curve. And this is on > Windows. > > By the way Fady, are you sure that your "Windows Firewall > with Enhanced Security" is not just dropping TCP packets > to/from port 40xx (or from "java.exe") ? There are some > "network policies" there which can have wide-ranging > side-effects. > > > > > On 07.02.2017 14:42, Fady Haikal wrote: >> >> >> Hi, issue still not fixed. Tomcat session replication is >> not able to replicate the key from node to node, please >> find below the error, taking into consideration that >> there is no network disconnection between 2 nodes >> >> >> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] >> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn fo >> >> Unable to replicate backup >> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to >> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10 , >> >> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, >> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 >> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={}, >> command={}, domain={}, ]. Reason:Operation has timed >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, >> 102}:4000;
Re: Operation has timed out
ProcessPID Protocol local address local port Remote Address State Tomcat8.exe 8160 TCP imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikalwrote: > Yes there is a ESTABLISHED connection, the replication of sessions is > working fine (port 4000 is for tomcat cluster) but we also faced this > error on the log file > > On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) > wrote: >> On 07.02.2017 16:24, Fady Haikal wrote: >>> >>> Hi, >>> telnet IP 4000 is working fine, i installed a tool for network >>> monitoring at the level of IP and Port and i didnt see any >>> disconnection, >> >> >> but did you see a *connection* ? >> I mean, on the pinging node, if you use the Windows "netstat" program, for >> example as >> netstat -aon -p TCP >> you should see a list of connections in the ESTABLISHED state, of which one >> of the IP/ports should be your target IP:4000 (in the "remote" column). >> And on the pinged node, this port :4000 should be in the "local" column, in >> LISTEN mode >> (and also probably one in the ESTABLISHED state, if they agree.) >> >> Is that the case ? >> >> >> >> and yes i'm sure that no firewall is enabled. >>> >>> >>> I saw some strange think on the server that I have tried to ping the >>> multicast IP (228.0.0.4) and i get reply from different IPs in the >>> network, i don't know why and how i get those IPs, after checking with >>> the network team they told me that those IPs are related to the SAN >>> storage taking into consideration that the Tomcat servers are not >>> connected in anyway to that SUN storage. >>> >>> >>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) >>> wrote: Hi. This is for the Tomcat/Tribes experts on the list. I know nothing of Tribes, but the on-line documentation seems to say that the communication happens over TCP and that the protocol used is not encrypted. Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is the base for him mentioning that "there is no network disconnection" between the nodes. Nevertheless, the calling pinging node seems to say that it times out without getting a response fom the target node. There is evidently a contradiction there. So this could still be some kind of network issue. Considering that the protocol command for this "ping" should be known by someone here, would it not be possible to imagine a little program in some scripting language (or even java, God forbid), which would open a TCP channel with the target node IP/port, send such a "ping" message, wait for a response and report the result ? That would at least confirm/deny that the problem is with the network. The log below does not for example say if the error happens when opening the TCP communication channel, or after sending the ping message on it, (Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if Fady tried this). Maybe tribes also already contains some löw-level debugging options ? wireshark maybe another option, but it has quite a learning curve. And this is on Windows. By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network policies" there which can have wide-ranging side-effects. On 07.02.2017 14:42, Fady Haikal wrote: > > > Hi, issue still not fixed. Tomcat session replication is not able to > replicate the key from node to node, please find below the error, > taking into consideration that there is no network disconnection > between 2 nodes > > > 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] > org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo > Unable to replicate backup > key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to > backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, > 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, > securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 > -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. > Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, > 114, 43, 102}:4000; >org.apache.catalina.tribes.ChannelException: Operation has timed > out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; > at > > org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) > at > > org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) > at > >
Re: Operation has timed out
Yes there is a ESTABLISHED connection, the replication of sessions is working fine (port 4000 is for tomcat cluster) but we also faced this error on the log file On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)wrote: > On 07.02.2017 16:24, Fady Haikal wrote: >> >> Hi, >> telnet IP 4000 is working fine, i installed a tool for network >> monitoring at the level of IP and Port and i didnt see any >> disconnection, > > > but did you see a *connection* ? > I mean, on the pinging node, if you use the Windows "netstat" program, for > example as > netstat -aon -p TCP > you should see a list of connections in the ESTABLISHED state, of which one > of the IP/ports should be your target IP:4000 (in the "remote" column). > And on the pinged node, this port :4000 should be in the "local" column, in > LISTEN mode > (and also probably one in the ESTABLISHED state, if they agree.) > > Is that the case ? > > > > and yes i'm sure that no firewall is enabled. >> >> >> I saw some strange think on the server that I have tried to ping the >> multicast IP (228.0.0.4) and i get reply from different IPs in the >> network, i don't know why and how i get those IPs, after checking with >> the network team they told me that those IPs are related to the SAN >> storage taking into consideration that the Tomcat servers are not >> connected in anyway to that SUN storage. >> >> >> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) >> wrote: >>> >>> Hi. >>> >>> This is for the Tomcat/Tribes experts on the list. >>> >>> I know nothing of Tribes, but the on-line documentation seems to say that >>> the communication happens over TCP and that the protocol used is not >>> encrypted. >>> Fady previously tried a standard "ping" and a "telnet" between the two >>> nodes, and that is the base for him mentioning that "there is no network >>> disconnection" between the nodes. >>> Nevertheless, the calling pinging node seems to say that it times out >>> without getting a response fom the target node. There is evidently a >>> contradiction there. >>> So this could still be some kind of network issue. >>> >>> Considering that the protocol command for this "ping" should be known by >>> someone here, would it not be possible to imagine a little program in >>> some >>> scripting language (or even java, God forbid), which would open a TCP >>> channel with the target node IP/port, send such a "ping" message, wait >>> for a >>> response and report the result ? >>> That would at least confirm/deny that the problem is with the network. >>> >>> The log below does not for example say if the error happens when opening >>> the >>> TCP communication channel, or after sending the ping message on it, >>> (Of course, testing the TCP open could be done with "telnet IP 4000", but >>> I >>> don't know if Fady tried this). >>> Maybe tribes also already contains some löw-level debugging options ? >>> wireshark maybe another option, but it has quite a learning curve. >>> And this is on Windows. >>> >>> By the way Fady, are you sure that your "Windows Firewall with Enhanced >>> Security" is not just dropping TCP packets to/from port 40xx (or from >>> "java.exe") ? There are some "network policies" there which can have >>> wide-ranging side-effects. >>> >>> >>> >>> >>> On 07.02.2017 14:42, Fady Haikal wrote: Hi, issue still not fixed. Tomcat session replication is not able to replicate the key from node to node, please find below the error, taking into consideration that there is no network disconnection between 2 nodes 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo Unable to replicate backup key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at
Re: Operation has timed out
On 07.02.2017 16:24, Fady Haikal wrote: Hi, telnet IP 4000 is working fine, i installed a tool for network monitoring at the level of IP and Port and i didnt see any disconnection, but did you see a *connection* ? I mean, on the pinging node, if you use the Windows "netstat" program, for example as netstat -aon -p TCP you should see a list of connections in the ESTABLISHED state, of which one of the IP/ports should be your target IP:4000 (in the "remote" column). And on the pinged node, this port :4000 should be in the "local" column, in LISTEN mode (and also probably one in the ESTABLISHED state, if they agree.) Is that the case ? and yes i'm sure that no firewall is enabled. I saw some strange think on the server that I have tried to ping the multicast IP (228.0.0.4) and i get reply from different IPs in the network, i don't know why and how i get those IPs, after checking with the network team they told me that those IPs are related to the SAN storage taking into consideration that the Tomcat servers are not connected in anyway to that SUN storage. On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)wrote: Hi. This is for the Tomcat/Tribes experts on the list. I know nothing of Tribes, but the on-line documentation seems to say that the communication happens over TCP and that the protocol used is not encrypted. Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is the base for him mentioning that "there is no network disconnection" between the nodes. Nevertheless, the calling pinging node seems to say that it times out without getting a response fom the target node. There is evidently a contradiction there. So this could still be some kind of network issue. Considering that the protocol command for this "ping" should be known by someone here, would it not be possible to imagine a little program in some scripting language (or even java, God forbid), which would open a TCP channel with the target node IP/port, send such a "ping" message, wait for a response and report the result ? That would at least confirm/deny that the problem is with the network. The log below does not for example say if the error happens when opening the TCP communication channel, or after sending the ping message on it, (Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if Fady tried this). Maybe tribes also already contains some löw-level debugging options ? wireshark maybe another option, but it has quite a learning curve. And this is on Windows. By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network policies" there which can have wide-ranging side-effects. On 07.02.2017 14:42, Fady Haikal wrote: Hi, issue still not fixed. Tomcat session replication is not able to replicate the key from node to node, please find below the error, taking into consideration that there is no network disconnection between 2 nodes 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo Unable to replicate backup key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) at org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170) at
Re: Operation has timed out
Hi, telnet IP 4000 is working fine, i installed a tool for network monitoring at the level of IP and Port and i didnt see any disconnection, and yes i'm sure that no firewall is enabled. I saw some strange think on the server that I have tried to ping the multicast IP (228.0.0.4) and i get reply from different IPs in the network, i don't know why and how i get those IPs, after checking with the network team they told me that those IPs are related to the SAN storage taking into consideration that the Tomcat servers are not connected in anyway to that SUN storage. On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)wrote: > Hi. > > This is for the Tomcat/Tribes experts on the list. > > I know nothing of Tribes, but the on-line documentation seems to say that > the communication happens over TCP and that the protocol used is not > encrypted. > Fady previously tried a standard "ping" and a "telnet" between the two > nodes, and that is the base for him mentioning that "there is no network > disconnection" between the nodes. > Nevertheless, the calling pinging node seems to say that it times out > without getting a response fom the target node. There is evidently a > contradiction there. > So this could still be some kind of network issue. > > Considering that the protocol command for this "ping" should be known by > someone here, would it not be possible to imagine a little program in some > scripting language (or even java, God forbid), which would open a TCP > channel with the target node IP/port, send such a "ping" message, wait for a > response and report the result ? > That would at least confirm/deny that the problem is with the network. > > The log below does not for example say if the error happens when opening the > TCP communication channel, or after sending the ping message on it, > (Of course, testing the TCP open could be done with "telnet IP 4000", but I > don't know if Fady tried this). > Maybe tribes also already contains some löw-level debugging options ? > wireshark maybe another option, but it has quite a learning curve. > And this is on Windows. > > By the way Fady, are you sure that your "Windows Firewall with Enhanced > Security" is not just dropping TCP packets to/from port 40xx (or from > "java.exe") ? There are some "network policies" there which can have > wide-ranging side-effects. > > > > > On 07.02.2017 14:42, Fady Haikal wrote: >> >> Hi, issue still not fixed. Tomcat session replication is not able to >> replicate the key from node to node, please find below the error, >> taking into consideration that there is no network disconnection >> between 2 nodes >> >> >> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] >> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo >> Unable to replicate backup >> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to >> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, >> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, >> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 >> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. >> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, >> 114, 43, 102}:4000; >> org.apache.catalina.tribes.ChannelException: Operation has timed >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; >> at >> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) >> at >> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) >> at >> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) >> at >> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) >> at >> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) >> at >> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) >> at >> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) >> at >> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) >> at >> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) >> at >> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) >> at >> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) >> at >> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170) >> at >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040) >> at >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024) >> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647) >> at >>
Re: Operation has timed out
Hi. This is for the Tomcat/Tribes experts on the list. I know nothing of Tribes, but the on-line documentation seems to say that the communication happens over TCP and that the protocol used is not encrypted. Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is the base for him mentioning that "there is no network disconnection" between the nodes. Nevertheless, the calling pinging node seems to say that it times out without getting a response fom the target node. There is evidently a contradiction there. So this could still be some kind of network issue. Considering that the protocol command for this "ping" should be known by someone here, would it not be possible to imagine a little program in some scripting language (or even java, God forbid), which would open a TCP channel with the target node IP/port, send such a "ping" message, wait for a response and report the result ? That would at least confirm/deny that the problem is with the network. The log below does not for example say if the error happens when opening the TCP communication channel, or after sending the ping message on it, (Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if Fady tried this). Maybe tribes also already contains some löw-level debugging options ? wireshark maybe another option, but it has quite a learning curve. And this is on Windows. By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network policies" there which can have wide-ranging side-effects. On 07.02.2017 14:42, Fady Haikal wrote: Hi, issue still not fixed. Tomcat session replication is not able to replicate the key from node to node, please find below the error, taking into consideration that there is no network disconnection between 2 nodes 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo Unable to replicate backup key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) at org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024) at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647) at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374) at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279) at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708) at org.apache.catalina.connector.Request.doGetSession(Request.java:2936) at org.apache.catalina.connector.Request.getSession(Request.java:2260) at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895) at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513) at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137) at
Re: Operation has timed out
Hi, issue still not fixed. Tomcat session replication is not able to replicate the key from node to node, please find below the error, taking into consideration that there is no network disconnection between 2 nodes 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8] org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo Unable to replicate backup key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350, securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ]. Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) at org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024) at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647) at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374) at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279) at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708) at org.apache.catalina.connector.Request.doGetSession(Request.java:2936) at org.apache.catalina.connector.Request.getSession(Request.java:2260) at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895) at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513) at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137) at org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109) at org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60) at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181) at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194) at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at
Re: Operation has timed out
below is the server.xml configuration, as mentioened earlier the issue is related to the cluster configuration, and as per my research i can see that some users are facing the same issue but i didnt found the solution of it On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat)wrote: > On 06.02.2017 17:45, Fady Haikal wrote: >> >> Hi, >> What is the host OS ? Windows Server 2012 >> What is the Tomcat version ? Apache Tomcat/8.0.30 >> >> Is this problem new ? was this working before ? how long ? Since >> cluster implementation >> > > I still don't know tribes, but then my non-educated guess at this point > would be that there is something wrong in your configuration. > Can you copy/paste it here ? (remove sensible things like passwords, public > IP addresses etc..)(but not to the point of making it uncheckable). > > Then maybe some tribes-specialist can take over ? > > >> >> Is there actually something listening on that address/port ? Tomcat >> cluster >> >> the Port 4000 is listening and there is no disconnection between 2 >> nodes ping and telnet are OK >> >> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) >> wrote: >>> >>> On 06.02.2017 17:24, Fady Haikal wrote: Plz can i get some help here? This issue is still occurring and it's filling the log file in the Production server Regards, Fady >>> >>> >>> >>> Hi. >>> If you want quick answers, you should provide more information. >>> What is the host OS ? >>> What is the Tomcat version ? >>> Is this problem new ? was this working before ? how long ? >>> >>> I do not know tribes at all, but according to the logfile below, it seems >>> that something is trying to "ping" the address 10.114.43.103, port 4000, >>> and >>> never getting a response (or at least not within 3000ms). >>> Is there actually something listening on that address/port ? >>> The "netstat" command (available both on Linux and Windows) can tell you. >>> If there is something listening there, can it respond to whatever is >>> ping-ing it ? >>> (routing, firewall, ..) >>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal wrote: > > > Guys, we are facing the below errors in Tomcat cluster, please advise > > > 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > Unable to send AbstractReplicatedMap.ping message >org.apache.catalina.tribes.ChannelException: Operation has timed > out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; > at > > org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) > at > > org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) > at > > org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) > at > > org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) > at > > org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) > at > > org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) > at > > org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) > at > > org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) > at > > org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) > at > > org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) > at > > org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) > at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) > at > > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267) > at > > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) > at > > org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161) > at > > org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) > > > 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] > org.apache.catalina.tribes.transport.nio.NioReceiver.listen
Re: Operation has timed out
On 06.02.2017 17:45, Fady Haikal wrote: Hi, What is the host OS ? Windows Server 2012 What is the Tomcat version ? Apache Tomcat/8.0.30 Is this problem new ? was this working before ? how long ? Since cluster implementation I still don't know tribes, but then my non-educated guess at this point would be that there is something wrong in your configuration. Can you copy/paste it here ? (remove sensible things like passwords, public IP addresses etc..)(but not to the point of making it uncheckable). Then maybe some tribes-specialist can take over ? Is there actually something listening on that address/port ? Tomcat cluster the Port 4000 is listening and there is no disconnection between 2 nodes ping and telnet are OK On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)wrote: On 06.02.2017 17:24, Fady Haikal wrote: Plz can i get some help here? This issue is still occurring and it's filling the log file in the Production server Regards, Fady Hi. If you want quick answers, you should provide more information. What is the host OS ? What is the Tomcat version ? Is this problem new ? was this working before ? how long ? I do not know tribes at all, but according to the logfile below, it seems that something is trying to "ping" the address 10.114.43.103, port 4000, and never getting a response (or at least not within 3000ms). Is there actually something listening on that address/port ? The "netstat" command (available both on Linux and Windows) can tell you. If there is something listening there, can it respond to whatever is ping-ing it ? (routing, firewall, ..) On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal wrote: Guys, we are facing the below errors in Tomcat cluster, please advise 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat Unable to send AbstractReplicatedMap.ping message org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161) at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to process request in NioReceiver java.io.IOException: A non-blocking socket operation could not be completed immediately at sun.nio.ch.SocketDispatcher.close0(Native Method) at sun.nio.ch.SocketDispatcher.close(Unknown Source) at sun.nio.ch.SocketChannelImpl.kill(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source) at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272) at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425) at java.lang.Thread.run(Unknown Source) Regards, Fady - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Operation has timed out
Hi, What is the host OS ? Windows Server 2012 What is the Tomcat version ? Apache Tomcat/8.0.30 Is this problem new ? was this working before ? how long ? Since cluster implementation Is there actually something listening on that address/port ? Tomcat cluster the Port 4000 is listening and there is no disconnection between 2 nodes ping and telnet are OK On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)wrote: > On 06.02.2017 17:24, Fady Haikal wrote: >> >> Plz can i get some help here? >> This issue is still occurring and it's filling the log file in the >> Production server >> >> Regards, >> Fady > > > Hi. > If you want quick answers, you should provide more information. > What is the host OS ? > What is the Tomcat version ? > Is this problem new ? was this working before ? how long ? > > I do not know tribes at all, but according to the logfile below, it seems > that something is trying to "ping" the address 10.114.43.103, port 4000, and > never getting a response (or at least not within 3000ms). > Is there actually something listening on that address/port ? > The "netstat" command (available both on Linux and Windows) can tell you. > If there is something listening there, can it respond to whatever is > ping-ing it ? > (routing, firewall, ..) > >> >> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal wrote: >>> >>> Guys, we are facing the below errors in Tomcat cluster, please advise >>> >>> >>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] >>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat >>> Unable to send AbstractReplicatedMap.ping message >>> org.apache.catalina.tribes.ChannelException: Operation has timed >>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; >>> at >>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) >>> at >>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) >>> at >>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) >>> at >>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) >>> at >>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) >>> at >>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) >>> at >>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) >>> at >>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) >>> at >>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) >>> at >>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) >>> at >>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) >>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) >>> at >>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267) >>> at >>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) >>> at >>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161) >>> at >>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) >>> >>> >>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] >>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to >>> process request in NioReceiver >>> java.io.IOException: A non-blocking socket operation could not be >>> completed immediately >>> at sun.nio.ch.SocketDispatcher.close0(Native Method) >>> at sun.nio.ch.SocketDispatcher.close(Unknown Source) >>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source) >>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source) >>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source) >>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) >>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) >>> at sun.nio.ch.SelectorImpl.select(Unknown Source) >>> at >>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272) >>> at >>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425) >>> at java.lang.Thread.run(Unknown Source) >>> >>> Regards, >>> Fady >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail:
Re: Operation has timed out
On 06.02.2017 17:24, Fady Haikal wrote: Plz can i get some help here? This issue is still occurring and it's filling the log file in the Production server Regards, Fady Hi. If you want quick answers, you should provide more information. What is the host OS ? What is the Tomcat version ? Is this problem new ? was this working before ? how long ? I do not know tribes at all, but according to the logfile below, it seems that something is trying to "ping" the address 10.114.43.103, port 4000, and never getting a response (or at least not within 3000ms). Is there actually something listening on that address/port ? The "netstat" command (available both on Linux and Windows) can tell you. If there is something listening there, can it respond to whatever is ping-ing it ? (routing, firewall, ..) On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikalwrote: Guys, we are facing the below errors in Tomcat cluster, please advise 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat Unable to send AbstractReplicatedMap.ping message org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267) at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161) at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to process request in NioReceiver java.io.IOException: A non-blocking socket operation could not be completed immediately at sun.nio.ch.SocketDispatcher.close0(Native Method) at sun.nio.ch.SocketDispatcher.close(Unknown Source) at sun.nio.ch.SocketChannelImpl.kill(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source) at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source) at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272) at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425) at java.lang.Thread.run(Unknown Source) Regards, Fady - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Operation has timed out
Plz can i get some help here? This issue is still occurring and it's filling the log file in the Production server Regards, Fady On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikalwrote: > Guys, we are facing the below errors in Tomcat cluster, please advise > > > 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1] > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat > Unable to send AbstractReplicatedMap.ping message > org.apache.catalina.tribes.ChannelException: Operation has timed > out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000; > at > org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108) > at > org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48) > at > org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54) > at > org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82) > at > org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) > at > org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81) > at > org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) > at > org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93) > at > org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76) > at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233) > at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186) > at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99) > at > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267) > at > org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885) > at > org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161) > at > org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697) > > > 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver] > org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to > process request in NioReceiver > java.io.IOException: A non-blocking socket operation could not be > completed immediately > at sun.nio.ch.SocketDispatcher.close0(Native Method) > at sun.nio.ch.SocketDispatcher.close(Unknown Source) > at sun.nio.ch.SocketChannelImpl.kill(Unknown Source) > at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source) > at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source) > at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) > at sun.nio.ch.SelectorImpl.select(Unknown Source) > at > org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272) > at > org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425) > at java.lang.Thread.run(Unknown Source) > > Regards, > Fady - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Operation has timed out(3000 ms.).;
I have set it as below. However, still it is showing the same error. Please let me know what to do as our all tomcats are going on high load because of this kind of continuous logging. If there is any change required let us know. ERROR logging: *SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 9}:4000; tcp://{-64, -88, 3, 15}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97) * Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=6/ Manager className=org.apache.catalina.ha.session.BackupManager expireSessionsOnShutdown=false notifyListenersOnReplication=true mapSendOptions=6/ Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=192.168.3.14 port=45564 frequency=500 dropTime=1/ Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=5000 selectorTimeout=5000 timeout=6 maxThreads=6/ Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter timeout=6 Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/ /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ Interceptor className=org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=.*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ On Thu, Mar 22, 2012 at 9:37 PM, Filip Hanik Mailing Lists devli...@hanik.com wrote: take a look at the Sender timeout=... .../ attribute http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-sender.html - Original Message - From: Dhaval Jaiswal dhaval.jais...@via.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, March 21, 2012 10:35:14 PM Subject: Operation has timed out(3000 ms.).; Receiving below errors in catalina log. I have changed the backup manager value of dropTime=3000 to 1 with restart of tomcats. However, still it is not taking effect notifying that Operation has timed out(3000 ms.) Please guide me where should i look to stop this messages. SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 6}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) -- * Dhaval Jaiswal Database System E: dhaval.jais...@via.com T: +91-80-4043 3000 M: +91-8095397843 www.via.com http://www.admin.viaworld.in/callcenter?action1=ADD_EPR_LOG * - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Operation has timed out(3000 ms.).;
take a look at the Sender timeout=... .../ attribute http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-sender.html - Original Message - From: Dhaval Jaiswal dhaval.jais...@via.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, March 21, 2012 10:35:14 PM Subject: Operation has timed out(3000 ms.).; Receiving below errors in catalina log. I have changed the backup manager value of dropTime=3000 to 1 with restart of tomcats. However, still it is not taking effect notifying that Operation has timed out(3000 ms.) Please guide me where should i look to stop this messages. SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 6}:4000; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87) -- * Dhaval Jaiswal Database System E: dhaval.jais...@via.com T: +91-80-4043 3000 M: +91-8095397843 www.via.com http://www.admin.viaworld.in/callcenter?action1=ADD_EPR_LOG * - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org