Re: Operation has timed out

[Keiichi Fujino]

gt;> >>   
> >> >>   
> >> >> 
> >> >>  >> >>resourceName="UserDatabase"/>
> >> >>   
> >> >>
> >> >>>> >> unpackWARs="true" autoDeploy="true" startStopThreads="0">
> >> >>   
> >> >>   
> >> >>
> >> >>   >> >>  channelSendOptions="4">
> >> >>   
> >> >>  >> >> className="org.apache.catalina.tribes.membership.McastService"
> >> >> address="228.0.0.4"
> >> >> port="45560"
> >> >> frequency="500"
> >> >> dropTime="9000"/>
> >> >>  >> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> >> >>   address="auto"
> >> >>   port="4000"
> >> >>   autoBind="100"
> >> >>   selectorTimeout="5000"
> >> >>   maxThreads="6"/>
> >> >>
> >> >>  >> >> className="org.apache.catalina.tribes.transport.
> >> ReplicationTransmitter">
> >> >>>> >> className="org.apache.catalina.tribes.transport.nio.
> >> >> PooledParallelSender"/>
> >> >> 
> >> >>  >> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >> TcpFailureDetector"/>
> >> >>  >> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >> MessageDispatch15Interceptor"/>
> >> >>   
> >> >>
> >> >>>> >>
> >> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> >> >> *\.html|.*\.css|.*\.txt"/>
> >> >>>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
> >> >>
> >> >>>> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> >> >> deployDir="D:/imaljava/
> TomcatNode1/tmp/war-deploy/"
> >> >> watchDir="D:/imaljava/
> TomcatNode1/tmp/war-listen/"
> >> >> watchEnabled="false"/>
> >> >>
> >> >>>> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> >> >> 
> >> >> 
> >> >> 
> >> >>
> >> >> 
> >> >>  >> >> directory="logs"
> >> >>prefix="localhost_access_log" suffix=".txt"
> >> >>pattern="%h %l %u %t "%r" %s %b" />
> >> >>  >> >>  threshold="900" />
> >> >>
> >> >>   
> >> >> 
> >> >>   
> >> >> 
> >> >>
> >> >>
> >> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino 
> >> >> wrote:
> >> >> > Hi
> >> >> >
> >> >> > You seems to be using BackupManager.
> >> >> > What is set for mapSendOptions?
> >> >> >
> >> >> > The Cluster module uses the Tomcat JULI logging FW,
> >> >> > so you can enable logging on the org.apache.catalina.tribes.
> MESSAGES,
> >> >> you
> >> >> > can track messages.
> >> >> >
> >> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace
> level
> >> >> logs.
> >> >> > So It should get logs in a short period of time.
> >> >> >
> >> >> >
> >> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal :
> >> >> >
> >&

Re: Operation has timed out

[Keiichi Fujino]

2017-02-09 19:21 GMT+09:00 Fady Haikal :

> i already restarted it after changing the parameter to 8
>

However, I can not see the log that context has been restarted in the log
you showed.
You may need to restart Tomcat.



>
> On Thu, Feb 9, 2017 at 12:16 PM, Keiichi Fujino 
> wrote:
> > It is realy strange.
> > Have you failed to restart Context?
> > When starting up the AbstractReplicatedMap, the followings should be
> logged.
> >
> > "Initializing AbstractReplicatedMap with context name:xxx"
> > "AbstractReplicatedMap[xxx] initialization was completed in xxx ms."
> >
> >
> > 2017-02-09 19:09 GMT+09:00 Fady Haikal :
> >
> >> i can only see this error related to AbstractReplicatedMap
> >>
> >>
> >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> Unable to send AbstractReplicatedMap.ping message
> >>  java.lang.NullPointerException
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> ping(AbstractReplicatedMap.java:256)
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> AbstractReplicatedMap.java:885)
> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> GroupChannel.java:166)
> >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> >> GroupChannel.java:697)
> >>
> >> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino 
> >> wrote:
> >> > These logs seem to be the tomcat memory leak detections logs during
> >> context
> >> > restart.
> >> > It is a different issue from this thread's topic.
> >> >
> >> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
> >> > initialization was completed in xxx ms." in your log file?
> >> >
> >> > 2017-02-09 18:49 GMT+09:00 Fady Haikal :
> >> >
> >> >> Keiichi,
> >> >> once i changed the mapSendOptions and without restarting the below
> >> >> shown in the tomcat
> >> >>
> >> >>
> >> >> 09-Feb-2017 12:44:12.299 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
> >> >> Process}]) but failed to remove it when the web application was
> >> >> stopped. Threads are going to be renewed over time to try and avoid a
> >> >> probable memory leak.
> >> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.li

Re: Operation has timed out

[Fady Haikal]

i already restarted it after changing the parameter to 8

On Thu, Feb 9, 2017 at 12:16 PM, Keiichi Fujino  wrote:
> It is realy strange.
> Have you failed to restart Context?
> When starting up the AbstractReplicatedMap, the followings should be logged.
>
> "Initializing AbstractReplicatedMap with context name:xxx"
> "AbstractReplicatedMap[xxx] initialization was completed in xxx ms."
>
>
> 2017-02-09 19:09 GMT+09:00 Fady Haikal :
>
>> i can only see this error related to AbstractReplicatedMap
>>
>>
>> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> Unable to send AbstractReplicatedMap.ping message
>>  java.lang.NullPointerException
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> ping(AbstractReplicatedMap.java:256)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> AbstractReplicatedMap.java:885)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> GroupChannel.java:166)
>> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> GroupChannel.java:697)
>>
>> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino 
>> wrote:
>> > These logs seem to be the tomcat memory leak detections logs during
>> context
>> > restart.
>> > It is a different issue from this thread's topic.
>> >
>> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
>> > initialization was completed in xxx ms." in your log file?
>> >
>> > 2017-02-09 18:49 GMT+09:00 Fady Haikal :
>> >
>> >> Keiichi,
>> >> once i changed the mapSendOptions and without restarting the below
>> >> shown in the tomcat
>> >>
>> >>
>> >> 09-Feb-2017 12:44:12.299 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.300 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.300 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.301 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
>> >> Process}]) but failed to remove it when the web application was
>> >> stopped. Threads are going to be renewed over time to try and avoid a
>> >> probable memory leak.
>> >> 09-Feb-2017 12:44:12.301 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.302 SEVERE
>> >> [ContainerBackground

Re: Operation has timed out

[Keiichi Fujino]

It is realy strange.
Have you failed to restart Context?
When starting up the AbstractReplicatedMap, the followings should be logged.

"Initializing AbstractReplicatedMap with context name:xxx"
"AbstractReplicatedMap[xxx] initialization was completed in xxx ms."


2017-02-09 19:09 GMT+09:00 Fady Haikal :

> i can only see this error related to AbstractReplicatedMap
>
>
> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
>  java.lang.NullPointerException
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> ping(AbstractReplicatedMap.java:256)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> GroupChannel.java:166)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> GroupChannel.java:697)
>
> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino 
> wrote:
> > These logs seem to be the tomcat memory leak detections logs during
> context
> > restart.
> > It is a different issue from this thread's topic.
> >
> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
> > initialization was completed in xxx ms." in your log file?
> >
> > 2017-02-09 18:49 GMT+09:00 Fady Haikal :
> >
> >> Keiichi,
> >> once i changed the mapSendOptions and without restarting the below
> >> shown in the tomcat
> >>
> >>
> >> 09-Feb-2017 12:44:12.299 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
> >> Process}]) but failed to remove it when the web application was
> >> stopped. Threads are going to be renewed over time to try and avoid a
> >> probable memory leak.
> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.302 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.

Re: Operation has timed out

[Fady Haikal]

i can only see this error related to AbstractReplicatedMap


09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
 java.lang.NullPointerException
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at 
org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166)
at 
org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)

On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino  wrote:
> These logs seem to be the tomcat memory leak detections logs during context
> restart.
> It is a different issue from this thread's topic.
>
> Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
> initialization was completed in xxx ms." in your log file?
>
> 2017-02-09 18:49 GMT+09:00 Fady Haikal :
>
>> Keiichi,
>> once i changed the mapSendOptions and without restarting the below
>> shown in the tomcat
>>
>>
>> 09-Feb-2017 12:44:12.299 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.300 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.300 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.301 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
>> Process}]) but failed to remove it when the web application was
>> stopped. Threads are going to be renewed over time to try and avoid a
>> probable memory leak.
>> 09-Feb-2017 12:44:12.301 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.302 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.303 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoa

Re: Operation has timed out

[tomcat]

Let's not mix issues here.

1) your Host entry is



that means that Tomcat will automatically reload any application when it notices that it 
has been changed.


2) the parameter you changed is in the application's context, so the application has been 
changed


3) to reload the application, Tomcat first stops the running application.
That's when it noticed all the potential leak issues below.
But they have always been there, you just never saw them before.



On 09.02.2017 10:49, Fady Haikal wrote:

Keiichi,
once i changed the mapSendOptions and without restarting the below
shown in the tomcat


09-Feb-2017 12:44:12.299 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.300 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.300 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.301 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
Process}]) but failed to remove it when the web application was
stopped. Threads are going to be renewed over time to try and avoid a
probable memory leak.
09-Feb-2017 12:44:12.301 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.302 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.303 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
  java.lang.NullPointerException
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(

Re: Operation has timed out

[Keiichi Fujino]

These logs seem to be the tomcat memory leak detections logs during context
restart.
It is a different issue from this thread's topic.

Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
initialization was completed in xxx ms." in your log file?

2017-02-09 18:49 GMT+09:00 Fady Haikal :

> Keiichi,
> once i changed the mapSendOptions and without restarting the below
> shown in the tomcat
>
>
> 09-Feb-2017 12:44:12.299 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.300 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.300 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.301 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
> Process}]) but failed to remove it when the web application was
> stopped. Threads are going to be renewed over time to try and avoid a
> probable memory leak.
> 09-Feb-2017 12:44:12.301 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.302 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.303 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
>  java.lang.NullPointerException
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> ping(AbstractReplicatedMap.java:2

Re: Operation has timed out

[Fady Haikal]

Keiichi,
once i changed the mapSendOptions and without restarting the below
shown in the tomcat


09-Feb-2017 12:44:12.299 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.300 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  A.ALMANA
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.300 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.301 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.SHAIKH  LOGOUT
Process}]) but failed to remove it when the web application was
stopped. Threads are going to be renewed over time to try and avoid a
probable memory leak.
09-Feb-2017 12:44:12.301 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.302 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.303 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS  M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
 java.lang.NullPointerException
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at 
org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166)
at 
org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)

09-Feb-2017 12:44:14.834 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_ser

Re: Operation has timed out

[Fady Haikal]

t;> 
>> >> > >> className="org.apache.catalina.tribes.group.interceptors.
>> >> TcpFailureDetector"/>
>> >> > >> className="org.apache.catalina.tribes.group.interceptors.
>> >> MessageDispatch15Interceptor"/>
>> >>   
>> >>
>> >>   > >>
>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>> >> *\.html|.*\.css|.*\.txt"/>
>> >>   > >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>> >>
>> >>   > >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>> >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>> >> watchEnabled="false"/>
>> >>
>> >>   > >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> >> 
>> >> 
>> >> 
>> >>
>> >> 
>> >> > >> directory="logs"
>> >>prefix="localhost_access_log" suffix=".txt"
>> >>pattern="%h %l %u %t "%r" %s %b" />
>> >> > >>  threshold="900" />
>> >>
>> >>   
>> >> 
>> >>   
>> >> 
>> >>
>> >>
>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino 
>> >> wrote:
>> >> > Hi
>> >> >
>> >> > You seems to be using BackupManager.
>> >> > What is set for mapSendOptions?
>> >> >
>> >> > The Cluster module uses the Tomcat JULI logging FW,
>> >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
>> >> you
>> >> > can track messages.
>> >> >
>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
>> >> logs.
>> >> > So It should get logs in a short period of time.
>> >> >
>> >> >
>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal :
>> >> >
>> >> >> Guys, we are facing the below errors in Tomcat cluster, please advise
>> >> >>
>> >> >>
>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >> >> Unable to send AbstractReplicatedMap.ping message
>> >> >>  org.apache.catalina.tribes.ChannelException: Operation has timed
>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >> >> at org.apache.catalina.tribes.transport.nio.
>> >> ParallelNioSender.sendMessage(
>> >> >> ParallelNioSender.java:108)
>> >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>> >> >> sendMessage(PooledParallelSender.java:48)
>> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>> >> >> sendMessage(ReplicationTransmitter.java:54)
>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>> >> >> ChannelCoordinator.java:82)
>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessage(
>> >> >> ChannelInterceptorBase.java:76)
>> >> >> at org.apache.catalina.tribes.group.interceptors.
>> >> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
>> >> java:81)
>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessage(
>> >> >> ChannelInterceptorBase.java:76)
>> >> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>> >> >> sendMessage(TcpFailureDetector.java:93)
>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessage(
>> >> >> ChannelInterceptorBase.java:76)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> >> GroupChannel.java:233)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> >> GroupChannel.java:186)
>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
>> RpcChannel.java:99)
>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >> >> ping(AbstractReplicatedMap.java:267)
>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> >> >> AbstractReplicatedMap.java:885)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> >> >> GroupChannel.java:161)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
>> HeartbeatThread.run(
>> >> >> GroupChannel.java:697)
>> >> >>
>> >> >>
>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
>> to
>> >> >> process request in NioReceiver
>> >> >>  java.io.IOException: A non-blocking socket operation could not be
>> >> >> completed immediately
>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> >> >> listen(NioReceiver.java:272)
>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> >> >> NioReceiver.java:425)
>> >> >> at java.lang.Thread.run(Unknown Source)
>> >> >>
>> >> >> Regards,
>> >> >> Fady
>> >> >>
>> >> >> 
>> -
>> >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >> >> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > Keiichi.Fujino
>> >>
>> >> -
>> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>
>> >> --
>> >> Keiichi.Fujino
>> >> 
>> >> 
>> >>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>> --
>> Keiichi.Fujino
>> 
>> 
>>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Operation has timed out

[Keiichi Fujino]

loyDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
> >>         watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
> >> watchEnabled="false"/>
> >>
> >>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> >> 
> >> 
> >> 
> >>
> >> 
> >>  >> directory="logs"
> >>prefix="localhost_access_log" suffix=".txt"
> >>pattern="%h %l %u %t "%r" %s %b" />
> >>  >>  threshold="900" />
> >>
> >>   
> >> 
> >>   
> >> 
> >>
> >>
> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino 
> >> wrote:
> >> > Hi
> >> >
> >> > You seems to be using BackupManager.
> >> > What is set for mapSendOptions?
> >> >
> >> > The Cluster module uses the Tomcat JULI logging FW,
> >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
> >> you
> >> > can track messages.
> >> >
> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
> >> logs.
> >> > So It should get logs in a short period of time.
> >> >
> >> >
> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal :
> >> >
> >> >> Guys, we are facing the below errors in Tomcat cluster, please advise
> >> >>
> >> >>
> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> >> Unable to send AbstractReplicatedMap.ping message
> >> >>  org.apache.catalina.tribes.ChannelException: Operation has timed
> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >> >> at org.apache.catalina.tribes.transport.nio.
> >> ParallelNioSender.sendMessage(
> >> >> ParallelNioSender.java:108)
> >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
> >> >> sendMessage(PooledParallelSender.java:48)
> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> >> >> sendMessage(ReplicationTransmitter.java:54)
> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
> >> >> ChannelCoordinator.java:82)
> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessage(
> >> >> ChannelInterceptorBase.java:76)
> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
> >> java:81)
> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessage(
> >> >> ChannelInterceptorBase.java:76)
> >> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> >> >> sendMessage(TcpFailureDetector.java:93)
> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessage(
> >> >> ChannelInterceptorBase.java:76)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> GroupChannel.java:233)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> GroupChannel.java:186)
> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
> RpcChannel.java:99)
> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >> ping(AbstractReplicatedMap.java:267)
> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> >> AbstractReplicatedMap.java:885)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> >> GroupChannel.java:161)
> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> HeartbeatThread.run(
> >> >> GroupChannel.java:697)
> >> >>
> >> >>
> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
> to
> >> >> process request in NioReceiver
> >> >>  java.io.IOException: A non-blocking socket operation could not be
> >> >> completed immediately
> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >> >> listen(NioReceiver.java:272)
> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> >> >> NioReceiver.java:425)
> >> >> at java.lang.Thread.run(Unknown Source)
> >> >>
> >> >> Regards,
> >> >> Fady
> >> >>
> >> >> 
> -
> >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Keiichi.Fujino
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >> --
> >> Keiichi.Fujino
> >> 
> >> 
> >>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
> --
> Keiichi.Fujino
> 
> 
>

Re: Operation has timed out

[Fady Haikal]

The  is set under the context.xml



On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino  wrote:
> 2017-02-09 18:10 GMT+09:00 Fady Haikal :
>
>> Keiichi,
>> mapSendoption is not configured anyway please check my configuration below
>>
>
>
> You have not set .
> If you do not use Cluster Manager template, DeltaManager is used by default.
> However, in the logs, it seems to be using BackupManager.
> Have you set  in webapps/{your app} /META-INF/context.xml
> or conf/{Engine}/{Host}/{your app}.xml?
>
>
>
>> 
>> 
>> 
>> 
>>   > />
>>   
>>   
>>   > SSLEngine="on" />
>>   
>>   > />
>>   > />
>>   > />
>>
>>   
>>   
>> 
>> >   type="org.apache.catalina.UserDatabase"
>>   description="User database that can be updated and saved"
>>   factory="org.apache.catalina.users.
>> MemoryUserDatabaseFactory"
>>   pathname="conf/tomcat-users.xml" />
>>   
>>
>>   
>>   
>>
>> 
>> 
>>
>>
>> 
>> >connectionTimeout="6" maxThreads="500"
>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>> 
>> 
>> 
>> 
>>
>> 
>> 
>>
>>
>> 
>>
>> 
>> > jvmRoute="TomcatNode1">
>>   
>>   
>> 
>> >resourceName="UserDatabase"/>
>>   
>>
>>   > unpackWARs="true" autoDeploy="true" startStopThreads="0">
>>   
>>   
>>
>>  >  channelSendOptions="4">
>>   
>> > className="org.apache.catalina.tribes.membership.McastService"
>> address="228.0.0.4"
>> port="45560"
>> frequency="500"
>> dropTime="9000"/>
>> > className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>   address="auto"
>>   port="4000"
>>   autoBind="100"
>>   selectorTimeout="5000"
>>   maxThreads="6"/>
>>
>> > className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>   > className="org.apache.catalina.tribes.transport.nio.
>> PooledParallelSender"/>
>> 
>> > className="org.apache.catalina.tribes.group.interceptors.
>> TcpFailureDetector"/>
>> > className="org.apache.catalina.tribes.group.interceptors.
>> MessageDispatch15Interceptor"/>
>>   
>>
>>   >
>> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>> *\.html|.*\.css|.*\.txt"/>
>>   > className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>>   > tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>> watchEnabled="false"/>
>>
>>   > className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> 
>> 
>> 
>>
>>     
>> > directory="logs"
>>prefix="localhost_access_log" suffix=".txt"
>>pattern="%h %l %u %t "%r" %s %b" />
>> >  threshold="900" />
>>
>>   
>> 
>>   
>> 
>>
>>
>> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino 
>> wrote:
>> > Hi
>> >
>> > You seems to be using BackupManager.
>> > What is set for mapSendOptions?
>> >
>> > The Cluster module uses the Tomcat JULI logging FW,
>> > so you can enable logging on the org.a

Re: Operation has timed out

[Keiichi Fujino]

2017-02-09 18:10 GMT+09:00 Fady Haikal :

> Keiichi,
> mapSendoption is not configured anyway please check my configuration below
>


You have not set .
If you do not use Cluster Manager template, DeltaManager is used by default.
However, in the logs, it seems to be using BackupManager.
Have you set  in webapps/{your app} /META-INF/context.xml
or conf/{Engine}/{Host}/{your app}.xml?



> 
> 
> 
> 
>/>
>   
>   
>SSLEngine="on" />
>   
>/>
>/>
>/>
>
>   
>   
> 
>type="org.apache.catalina.UserDatabase"
>   description="User database that can be updated and saved"
>   factory="org.apache.catalina.users.
> MemoryUserDatabaseFactory"
>   pathname="conf/tomcat-users.xml" />
>   
>
>   
>   
>
> 
> 
>
>
> 
> connectionTimeout="6" maxThreads="500"
> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
> 
> 
> 
> 
>
> 
> 
>
>
> 
>
> 
>  jvmRoute="TomcatNode1">
>   
>   
> 
> resourceName="UserDatabase"/>
>   
>
>unpackWARs="true" autoDeploy="true" startStopThreads="0">
>   
>   
>
>channelSendOptions="4">
>   
>  className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45560"
> frequency="500"
> dropTime="9000"/>
>  className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>   address="auto"
>   port="4000"
>   autoBind="100"
>   selectorTimeout="5000"
>   maxThreads="6"/>
>
>  className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>className="org.apache.catalina.tribes.transport.nio.
> PooledParallelSender"/>
> 
>  className="org.apache.catalina.tribes.group.interceptors.
> TcpFailureDetector"/>
>  className="org.apache.catalina.tribes.group.interceptors.
> MessageDispatch15Interceptor"/>
>   
>
>   
> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> *\.html|.*\.css|.*\.txt"/>
>className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
>tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
> watchEnabled="false"/>
>
>className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> 
> 
> 
>
> 
>  directory="logs"
>prefix="localhost_access_log" suffix=".txt"
>pattern="%h %l %u %t "%r" %s %b" />
>   threshold="900" />
>
>   
> 
>   
> 
>
>
> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino 
> wrote:
> > Hi
> >
> > You seems to be using BackupManager.
> > What is set for mapSendOptions?
> >
> > The Cluster module uses the Tomcat JULI logging FW,
> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
> you
> > can track messages.
> >
> > Note for using the org.apache.catalina.tribes.MESSAGES:
> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
> logs.
> > So It should get logs in a short period of time.
> >
> >
> > 2017-02-06 15:52 GMT+09:00 Fady Haikal :
> >
> >> Guys, we are facing the below errors in Tomcat cluster, please advise
> >>
> >>
> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> Unable to send AbstractReplicatedMap.ping message
> >>  org

Re: Operation has timed out

[Fady Haikal]

Keiichi,
mapSendoption is not configured anyway please check my configuration below





  
  
  
  
  
  
  
  

  
  


  

  
  




















  
  


  

  
  
  

 
  




  



  

  
  

  

  








  

  



On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino  wrote:
> Hi
>
> You seems to be using BackupManager.
> What is set for mapSendOptions?
>
> The Cluster module uses the Tomcat JULI logging FW,
> so you can enable logging on the org.apache.catalina.tribes.MESSAGES, you
> can track messages.
>
> Note for using the org.apache.catalina.tribes.MESSAGES:
> The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs.
> So It should get logs in a short period of time.
>
>
> 2017-02-06 15:52 GMT+09:00 Fady Haikal :
>
>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>
>>
>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> Unable to send AbstractReplicatedMap.ping message
>>  org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(
>> ParallelNioSender.java:108)
>> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>> sendMessage(PooledParallelSender.java:48)
>> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>> sendMessage(ReplicationTransmitter.java:54)
>> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>> ChannelCoordinator.java:82)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.interceptors.
>> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>> sendMessage(TcpFailureDetector.java:93)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.GroupChannel.send(
>> GroupChannel.java:233)
>> at org.apache.catalina.tribes.group.GroupChannel.send(
>> GroupChannel.java:186)
>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> ping(AbstractReplicatedMap.java:267)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> AbstractReplicatedMap.java:885)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> GroupChannel.java:161)
>> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> GroupChannel.java:697)
>>
>>
>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>> process request in NioReceiver
>>  java.io.IOException: A non-blocking socket operation could not be
>> completed immediately
>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> listen(NioReceiver.java:272)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> NioReceiver.java:425)
>> at java.lang.Thread.run(Unknown Source)
>>
>> Regards,
>> Fady
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>
>
> --
> Keiichi.Fujino

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Operation has timed out

[Keiichi Fujino]

Hi

You seems to be using BackupManager.
What is set for mapSendOptions?

The Cluster module uses the Tomcat JULI logging FW,
so you can enable logging on the org.apache.catalina.tribes.MESSAGES, you
can track messages.

Note for using the org.apache.catalina.tribes.MESSAGES:
The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs.
So It should get logs in a short period of time.


2017-02-06 15:52 GMT+09:00 Fady Haikal :

> Guys, we are facing the below errors in Tomcat cluster, please advise
>
>
> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
>  org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(
> ParallelNioSender.java:108)
> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
> sendMessage(PooledParallelSender.java:48)
> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> sendMessage(ReplicationTransmitter.java:54)
> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
> ChannelCoordinator.java:82)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.
> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> sendMessage(TcpFailureDetector.java:93)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.GroupChannel.send(
> GroupChannel.java:233)
> at org.apache.catalina.tribes.group.GroupChannel.send(
> GroupChannel.java:186)
> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> ping(AbstractReplicatedMap.java:267)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> GroupChannel.java:161)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> GroupChannel.java:697)
>
>
> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
> process request in NioReceiver
>  java.io.IOException: A non-blocking socket operation could not be
> completed immediately
> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> listen(NioReceiver.java:272)
> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> NioReceiver.java:425)
> at java.lang.Thread.run(Unknown Source)
>
> Regards,
> Fady
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


-- 
Keiichi.Fujino

Re: Operation has timed out

[Fady Haikal]

>> also faced this error on the log file
>> >>>>>>>
>> >>>>>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>> >>>>>>>  wrote:
>> >>>>>>>
>> >>>>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>> >>>>>>>>> network monitoring at the level of IP and Port and i didnt
>> >>>>>>>>> see any disconnection,
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> but did you see a *connection* ? I mean, on the pinging node,
>> >>>>>>>> if you use the Windows "netstat" program, for example as
>> >>>>>>>> netstat -aon -p TCP you should see a list of connections in the
>> >>>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
>> >>>>>>>> target IP:4000 (in the "remote" column). And on the pinged
>> >>>>>>>> node, this port :4000 should be in the "local" column, in
>> >>>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>> >>>>>>>> they agree.)
>> >>>>>>>>
>> >>>>>>>> Is that the case ?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> and yes i'm sure that no firewall is enabled.
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> I saw some strange think on the server that I have tried to
>> >>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>> >>>>>>>>> different IPs in the network, i don't know why and how i get
>> >>>>>>>>> those IPs, after checking with the network team they told me
>> >>>>>>>>> that those IPs are related to the SAN storage taking into
>> >>>>>>>>> consideration that the Tomcat servers are not connected in
>> >>>>>>>>> anyway to that SUN storage.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>> >>>>>>>>>  wrote:
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Hi.
>> >>>>>>>>>>
>> >>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
>> >>>>>>>>>>
>> >>>>>>>>>> I know nothing of Tribes, but the on-line documentation
>> >>>>>>>>>> seems to say that the communication happens over TCP and
>> >>>>>>>>>> that the protocol used is not encrypted. Fady previously
>> >>>>>>>>>> tried a standard "ping" and a "telnet" between the two
>> >>>>>>>>>> nodes, and that is the base for him mentioning that "there
>> >>>>>>>>>> is no network disconnection" between the nodes.
>> >>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>> >>>>>>>>>> times out without getting a response fom the target node.
>> >>>>>>>>>> There is evidently a contradiction there. So this could
>> >>>>>>>>>> still be some kind of network issue.
>> >>>>>>>>>>
>> >>>>>>>>>> Considering that the protocol command for this "ping"
>> >>>>>>>>>> should be known by someone here, would it not be possible
>> >>>>>>>>>> to imagine a little program in some scripting language (or
>> >>>>>>>>>> even java, God forbid), which would open a TCP channel with
>&g

Re: Operation has timed out

[ashwin rajput]

 the level of IP and Port and i didnt
> >>>>>>>>> see any disconnection,
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> but did you see a *connection* ? I mean, on the pinging node,
> >>>>>>>> if you use the Windows "netstat" program, for example as
> >>>>>>>> netstat -aon -p TCP you should see a list of connections in the
> >>>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
> >>>>>>>> target IP:4000 (in the "remote" column). And on the pinged
> >>>>>>>> node, this port :4000 should be in the "local" column, in
> >>>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
> >>>>>>>> they agree.)
> >>>>>>>>
> >>>>>>>> Is that the case ?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> and yes i'm sure that no firewall is enabled.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I saw some strange think on the server that I have tried to
> >>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
> >>>>>>>>> different IPs in the network, i don't know why and how i get
> >>>>>>>>> those IPs, after checking with the network team they told me
> >>>>>>>>> that those IPs are related to the SAN storage taking into
> >>>>>>>>> consideration that the Tomcat servers are not connected in
> >>>>>>>>> anyway to that SUN storage.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
> >>>>>>>>>  wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi.
> >>>>>>>>>>
> >>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
> >>>>>>>>>>
> >>>>>>>>>> I know nothing of Tribes, but the on-line documentation
> >>>>>>>>>> seems to say that the communication happens over TCP and
> >>>>>>>>>> that the protocol used is not encrypted. Fady previously
> >>>>>>>>>> tried a standard "ping" and a "telnet" between the two
> >>>>>>>>>> nodes, and that is the base for him mentioning that "there
> >>>>>>>>>> is no network disconnection" between the nodes.
> >>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
> >>>>>>>>>> times out without getting a response fom the target node.
> >>>>>>>>>> There is evidently a contradiction there. So this could
> >>>>>>>>>> still be some kind of network issue.
> >>>>>>>>>>
> >>>>>>>>>> Considering that the protocol command for this "ping"
> >>>>>>>>>> should be known by someone here, would it not be possible
> >>>>>>>>>> to imagine a little program in some scripting language (or
> >>>>>>>>>> even java, God forbid), which would open a TCP channel with
> >>>>>>>>>> the target node IP/port, send such a "ping" message, wait
> >>>>>>>>>> for a response and report the result ? That would at least
> >>>>>>>>>> confirm/deny that the problem is with the network.
> >>>>>>>>>>
> >>>>>>>>>> The log below does not for example say if the error happens
> >>>>>>>>>> when opening the TCP communication channel, or after
> >>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
> >>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
> >>>>>>>>>> if Fady t

Re: Operation has timed out

[Fady Haikal]

ED state, if
>>>>>>>> they agree.)
>>>>>>>>
>>>>>>>> Is that the case ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> and yes i'm sure that no firewall is enabled.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I saw some strange think on the server that I have tried to
>>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>>>>>>> different IPs in the network, i don't know why and how i get
>>>>>>>>> those IPs, after checking with the network team they told me
>>>>>>>>> that those IPs are related to the SAN storage taking into
>>>>>>>>> consideration that the Tomcat servers are not connected in
>>>>>>>>> anyway to that SUN storage.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>>
>>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>>>>>>
>>>>>>>>>> I know nothing of Tribes, but the on-line documentation
>>>>>>>>>> seems to say that the communication happens over TCP and
>>>>>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>>>>>> is no network disconnection" between the nodes.
>>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>>>>>> times out without getting a response fom the target node.
>>>>>>>>>> There is evidently a contradiction there. So this could
>>>>>>>>>> still be some kind of network issue.
>>>>>>>>>>
>>>>>>>>>> Considering that the protocol command for this "ping"
>>>>>>>>>> should be known by someone here, would it not be possible
>>>>>>>>>> to imagine a little program in some scripting language (or
>>>>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>>>>> for a response and report the result ? That would at least
>>>>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>>>>
>>>>>>>>>> The log below does not for example say if the error happens
>>>>>>>>>> when opening the TCP communication channel, or after
>>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>>>>> Windows.
>>>>>>>>>>
>>>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>>>>> "network policies" there which can have wide-ranging
>>>>>>>>>> side-effects.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>

Re: Operation has timed out

[Fady Haikal]

w nothing of Tribes, but the on-line documentation
>>>>>>>>> seems to say that the communication happens over TCP and
>>>>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>>>>> is no network disconnection" between the nodes.
>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>>>>> times out without getting a response fom the target node.
>>>>>>>>> There is evidently a contradiction there. So this could
>>>>>>>>> still be some kind of network issue.
>>>>>>>>>
>>>>>>>>> Considering that the protocol command for this "ping"
>>>>>>>>> should be known by someone here, would it not be possible
>>>>>>>>> to imagine a little program in some scripting language (or
>>>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>>>> for a response and report the result ? That would at least
>>>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>>>
>>>>>>>>> The log below does not for example say if the error happens
>>>>>>>>> when opening the TCP communication channel, or after
>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>>>> Windows.
>>>>>>>>>
>>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>>>> "network policies" there which can have wide-ranging
>>>>>>>>> side-effects.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>>>>> not able to replicate the key from node to node, please
>>>>>>>>>> find below the error, taking into consideration that
>>>>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>>
>>>>>>>>> fo
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> Unable to replicate backup
>>>>
>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
>>>>>>>>>>
>>>>>>>>> ,
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>
>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>>>>> 102}:4000; org.apache.catalina.tribes.C

Re: Operation has timed out

[ashwin rajput]

gt;>> times out without getting a response fom the target node.
>>>>>>>> There is evidently a contradiction there. So this could
>>>>>>>> still be some kind of network issue.
>>>>>>>>
>>>>>>>> Considering that the protocol command for this "ping"
>>>>>>>> should be known by someone here, would it not be possible
>>>>>>>> to imagine a little program in some scripting language (or
>>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>>> for a response and report the result ? That would at least
>>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>>
>>>>>>>> The log below does not for example say if the error happens
>>>>>>>> when opening the TCP communication channel, or after
>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>>> Windows.
>>>>>>>>
>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>>> "network policies" there which can have wide-ranging
>>>>>>>> side-effects.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>>>> not able to replicate the key from node to node, please
>>>>>>>>> find below the error, taking into consideration that
>>>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>
>>>>>>>> fo
>>>
>>>>
>>>>>>>>>
>>>>>>>>> Unable to replicate backup
>>>
>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
>>>>>>>>>
>>>>>>>> ,
>>>
>>>>
>>>>>>>>>
>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>
>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
>>>>>>>>>
>>>>>>>> ssage(ParallelNioSender.java:108)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
>>>>>>>>>
>>>>>>>> dMessage(PooledParallelSender.java:48)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>

Re: Operation has timed out

[tomcat]

On 07.02.2017 17:20, Fady Haikal wrote:

Christopher,
For the first time


@Christopher : just to make sure you got that bit, buried below : the actual replication 
seems to work fine. The problem is only these "unsuccesful ping" messages in the log, 
which fill the log, and which so far nobody has managed to find an explanation for.




On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
 wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Fady,

On 2/7/17 10:53 AM, Fady Haikal wrote:

ProcessPID   Protocol   local address  local port
Remote Address  State Tomcat8.exe 8160 TCP
imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED


Stupid question: was this working in the past, and it stopped working?
Or are you trying to get this working for the first time?

- -chris


On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal 
wrote:

Yes there is a ESTABLISHED connection, the replication of
sessions is working fine (port 4000 is for tomcat cluster) but we
also faced this error on the log file

On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
 wrote:

On 07.02.2017 16:24, Fady Haikal wrote:


Hi, telnet IP 4000 is working fine, i installed a tool for
network monitoring at the level of IP and Port and i didnt
see any disconnection,



but did you see a *connection* ? I mean, on the pinging node,
if you use the Windows "netstat" program, for example as
netstat -aon -p TCP you should see a list of connections in the
ESTABLISHED state, of which one of the IP/ports should be your
target IP:4000 (in the "remote" column). And on the pinged
node, this port :4000 should be in the "local" column, in
LISTEN mode (and also probably one in the ESTABLISHED state, if
they agree.)

Is that the case ?



and yes i'm sure that no firewall is enabled.



I saw some strange think on the server that I have tried to
ping the multicast IP (228.0.0.4) and i get reply from
different IPs in the network, i don't know why and how i get
those IPs, after checking with the network team they told me
that those IPs are related to the SAN storage taking into
consideration that the Tomcat servers are not connected in
anyway to that SUN storage.


On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
 wrote:


Hi.

This is for the Tomcat/Tribes experts on the list.

I know nothing of Tribes, but the on-line documentation
seems to say that the communication happens over TCP and
that the protocol used is not encrypted. Fady previously
tried a standard "ping" and a "telnet" between the two
nodes, and that is the base for him mentioning that "there
is no network disconnection" between the nodes.
Nevertheless, the calling pinging node seems to say that it
times out without getting a response fom the target node.
There is evidently a contradiction there. So this could
still be some kind of network issue.

Considering that the protocol command for this "ping"
should be known by someone here, would it not be possible
to imagine a little program in some scripting language (or
even java, God forbid), which would open a TCP channel with
the target node IP/port, send such a "ping" message, wait
for a response and report the result ? That would at least
confirm/deny that the problem is with the network.

The log below does not for example say if the error happens
when opening the TCP communication channel, or after
sending the ping message on it, (Of course, testing the TCP
open could be done with "telnet IP 4000", but I don't know
if Fady tried this). Maybe tribes also already contains
some löw-level debugging options ? wireshark maybe another
option, but it has quite a learning curve. And this is on
Windows.

By the way Fady, are you sure that your "Windows Firewall
with Enhanced Security" is not just dropping TCP packets
to/from port 40xx (or from "java.exe") ? There are some
"network policies" there which can have wide-ranging
side-effects.




On 07.02.2017 14:42, Fady Haikal wrote:



Hi, issue still not fixed. Tomcat session replication is
not able to replicate the key from node to node, please
find below the error, taking into consideration that
there is no network disconnection between 2 nodes


07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn

fo




Unable to replicate backup

key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10

,




114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,

securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
command={}, domain={}, ]. Reason:Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
102}:4000; org.apache.catalina.tribes.ChannelException:
Operation has timed out(3000 ms.).; Faulty
membe

Re: Operation has timed out

[Fady Haikal]

s quite a learning curve. And this is on
>>>>>> Windows.
>>>>>>
>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>> "network policies" there which can have wide-ranging
>>>>>> side-effects.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>> not able to replicate the key from node to node, please
>>>>>>> find below the error, taking into consideration that
>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>
>>>>>>>
>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
> fo
>>>>>>>
>>>>>>>
> Unable to replicate backup
>>>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
> ,
>>>>>>>
>>>>>>>
> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
> ssage(ParallelNioSender.java:108)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
> dMessage(PooledParallelSender.java:48)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
> essage(ReplicationTransmitter.java:54)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
> hannelCoordinator.java:82)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
> ge(ChannelInterceptorBase.java:76)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
> ge(ChannelInterceptorBase.java:76)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> sendMessage(TcpFailureDetector.java:93)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
> ge(ChannelInterceptorBase.java:76)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
> ava:233)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
> ava:186)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
> fo(LazyReplicatedMap.java:170)
>>>>>>>
>>>>>>>
> a

Re: Operation has timed out

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Fady,

On 2/7/17 10:53 AM, Fady Haikal wrote:
> ProcessPID   Protocol   local address  local port
> Remote Address  State Tomcat8.exe 8160 TCP
> imal14-app24000 imal14-app1.albaraka.com.sdESTABLISHED

Stupid question: was this working in the past, and it stopped working?
Or are you trying to get this working for the first time?

- -chris

> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal 
> wrote:
>> Yes there is a ESTABLISHED connection, the replication of
>> sessions is working fine (port 4000 is for tomcat cluster) but we
>> also faced this error on the log file
>> 
>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>>  wrote:
>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>> 
>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>> network monitoring at the level of IP and Port and i didnt
>>>> see any disconnection,
>>> 
>>> 
>>> but did you see a *connection* ? I mean, on the pinging node,
>>> if you use the Windows "netstat" program, for example as 
>>> netstat -aon -p TCP you should see a list of connections in the
>>> ESTABLISHED state, of which one of the IP/ports should be your
>>> target IP:4000 (in the "remote" column). And on the pinged
>>> node, this port :4000 should be in the "local" column, in 
>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>> they agree.)
>>> 
>>> Is that the case ?
>>> 
>>> 
>>> 
>>> and yes i'm sure that no firewall is enabled.
>>>> 
>>>> 
>>>> I saw some strange think on the server that I have tried to
>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>> different IPs in the network, i don't know why and how i get
>>>> those IPs, after checking with the network team they told me
>>>> that those IPs are related to the SAN storage taking into
>>>> consideration that the Tomcat servers are not connected in
>>>> anyway to that SUN storage.
>>>> 
>>>> 
>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>>>>  wrote:
>>>>> 
>>>>> Hi.
>>>>> 
>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>> 
>>>>> I know nothing of Tribes, but the on-line documentation
>>>>> seems to say that the communication happens over TCP and
>>>>> that the protocol used is not encrypted. Fady previously
>>>>> tried a standard "ping" and a "telnet" between the two 
>>>>> nodes, and that is the base for him mentioning that "there
>>>>> is no network disconnection" between the nodes. 
>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>> times out without getting a response fom the target node.
>>>>> There is evidently a contradiction there. So this could
>>>>> still be some kind of network issue.
>>>>> 
>>>>> Considering that the protocol command for this "ping"
>>>>> should be known by someone here, would it not be possible
>>>>> to imagine a little program in some scripting language (or
>>>>> even java, God forbid), which would open a TCP channel with
>>>>> the target node IP/port, send such a "ping" message, wait 
>>>>> for a response and report the result ? That would at least
>>>>> confirm/deny that the problem is with the network.
>>>>> 
>>>>> The log below does not for example say if the error happens
>>>>> when opening the TCP communication channel, or after
>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>> if Fady tried this). Maybe tribes also already contains
>>>>> some löw-level debugging options ? wireshark maybe another
>>>>> option, but it has quite a learning curve. And this is on
>>>>> Windows.
>>>>> 
>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>> "network policies" there which can have wide-ranging
&

Re: Operation has timed out

[Fady Haikal]

ProcessPID   Protocol   local address  local port Remote
Address  State
Tomcat8.exe 8160 TCP imal14-app24000
imal14-app1.albaraka.com.sdESTABLISHED

On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal  wrote:
> Yes there is a ESTABLISHED connection, the replication of sessions is
> working fine (port 4000 is for tomcat cluster) but we also faced this
> error on the log file
>
> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)  
> wrote:
>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>
>>> Hi,
>>> telnet IP 4000 is working fine, i installed a tool for network
>>> monitoring at the level of IP and Port and i didnt see any
>>> disconnection,
>>
>>
>> but did you see a *connection* ?
>> I mean, on the pinging node, if you use the Windows "netstat" program, for
>> example as
>> netstat -aon -p TCP
>> you should see a list of connections in the ESTABLISHED state, of which one
>> of the IP/ports should be your target IP:4000 (in the "remote" column).
>> And on the pinged node, this port :4000 should be in the "local" column, in
>> LISTEN mode
>> (and also probably one in the ESTABLISHED state, if they agree.)
>>
>> Is that the case ?
>>
>>
>>
>> and yes i'm sure that no firewall is enabled.
>>>
>>>
>>> I saw some strange think on the server that I have tried to ping the
>>> multicast IP (228.0.0.4) and i get reply from different IPs in the
>>> network, i don't know why and how i get those IPs, after checking with
>>> the network team they told me that those IPs are related to the SAN
>>> storage taking into consideration that the Tomcat servers are not
>>> connected in anyway to that SUN storage.
>>>
>>>
>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) 
>>> wrote:
>>>>
>>>> Hi.
>>>>
>>>> This is for the Tomcat/Tribes experts on the list.
>>>>
>>>> I know nothing of Tribes, but the on-line documentation seems to say that
>>>> the communication happens over TCP and that the protocol used is not
>>>> encrypted.
>>>> Fady previously tried a standard "ping" and a "telnet" between the two
>>>> nodes, and that is the base for him mentioning that "there is no network
>>>> disconnection" between the nodes.
>>>> Nevertheless, the calling pinging node seems to say that it times out
>>>> without getting a response fom the target node.  There is evidently a
>>>> contradiction there.
>>>> So this could still be some kind of network issue.
>>>>
>>>> Considering that the protocol command for this "ping" should be known by
>>>> someone here, would it not be possible to imagine a little program in
>>>> some
>>>> scripting language (or even java, God forbid), which would open a TCP
>>>> channel with the target node IP/port, send such a "ping" message, wait
>>>> for a
>>>> response and report the result ?
>>>> That would at least confirm/deny that the problem is with the network.
>>>>
>>>> The log below does not for example say if the error happens when opening
>>>> the
>>>> TCP communication channel, or after sending the ping message on it,
>>>> (Of course, testing the TCP open could be done with "telnet IP 4000", but
>>>> I
>>>> don't know if Fady tried this).
>>>> Maybe tribes also already contains some löw-level debugging options ?
>>>> wireshark maybe another option, but it has quite a learning curve.
>>>> And this is on Windows.
>>>>
>>>> By the way Fady, are you sure that your "Windows Firewall with Enhanced
>>>> Security" is not just dropping TCP packets to/from port 40xx (or from
>>>> "java.exe") ? There are some "network policies" there which can have
>>>> wide-ranging side-effects.
>>>>
>>>>
>>>>
>>>>
>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>
>>>>>
>>>>> Hi, issue still not fixed. Tomcat session replication is not able to
>>>>> replicate the key from node to node, please find below the error,
>>>>> taking into consideration that there is no network disconnection
>>>>> 

Re: Operation has timed out

[Fady Haikal]

Yes there is a ESTABLISHED connection, the replication of sessions is
working fine (port 4000 is for tomcat cluster) but we also faced this
error on the log file

On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)  wrote:
> On 07.02.2017 16:24, Fady Haikal wrote:
>>
>> Hi,
>> telnet IP 4000 is working fine, i installed a tool for network
>> monitoring at the level of IP and Port and i didnt see any
>> disconnection,
>
>
> but did you see a *connection* ?
> I mean, on the pinging node, if you use the Windows "netstat" program, for
> example as
> netstat -aon -p TCP
> you should see a list of connections in the ESTABLISHED state, of which one
> of the IP/ports should be your target IP:4000 (in the "remote" column).
> And on the pinged node, this port :4000 should be in the "local" column, in
> LISTEN mode
> (and also probably one in the ESTABLISHED state, if they agree.)
>
> Is that the case ?
>
>
>
> and yes i'm sure that no firewall is enabled.
>>
>>
>> I saw some strange think on the server that I have tried to ping the
>> multicast IP (228.0.0.4) and i get reply from different IPs in the
>> network, i don't know why and how i get those IPs, after checking with
>> the network team they told me that those IPs are related to the SAN
>> storage taking into consideration that the Tomcat servers are not
>> connected in anyway to that SUN storage.
>>
>>
>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) 
>> wrote:
>>>
>>> Hi.
>>>
>>> This is for the Tomcat/Tribes experts on the list.
>>>
>>> I know nothing of Tribes, but the on-line documentation seems to say that
>>> the communication happens over TCP and that the protocol used is not
>>> encrypted.
>>> Fady previously tried a standard "ping" and a "telnet" between the two
>>> nodes, and that is the base for him mentioning that "there is no network
>>> disconnection" between the nodes.
>>> Nevertheless, the calling pinging node seems to say that it times out
>>> without getting a response fom the target node.  There is evidently a
>>> contradiction there.
>>> So this could still be some kind of network issue.
>>>
>>> Considering that the protocol command for this "ping" should be known by
>>> someone here, would it not be possible to imagine a little program in
>>> some
>>> scripting language (or even java, God forbid), which would open a TCP
>>> channel with the target node IP/port, send such a "ping" message, wait
>>> for a
>>> response and report the result ?
>>> That would at least confirm/deny that the problem is with the network.
>>>
>>> The log below does not for example say if the error happens when opening
>>> the
>>> TCP communication channel, or after sending the ping message on it,
>>> (Of course, testing the TCP open could be done with "telnet IP 4000", but
>>> I
>>> don't know if Fady tried this).
>>> Maybe tribes also already contains some löw-level debugging options ?
>>> wireshark maybe another option, but it has quite a learning curve.
>>> And this is on Windows.
>>>
>>> By the way Fady, are you sure that your "Windows Firewall with Enhanced
>>> Security" is not just dropping TCP packets to/from port 40xx (or from
>>> "java.exe") ? There are some "network policies" there which can have
>>> wide-ranging side-effects.
>>>
>>>
>>>
>>>
>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>
>>>>
>>>> Hi, issue still not fixed. Tomcat session replication is not able to
>>>> replicate the key from node to node, please find below the error,
>>>> taking into consideration that there is no network disconnection
>>>> between 2 nodes
>>>>
>>>>
>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>>>> Unable to replicate backup
>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>>>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>>>> Reason:Operation has timed out(300

Re: Operation has timed out

[tomcat]

On 07.02.2017 16:24, Fady Haikal wrote:

Hi,
telnet IP 4000 is working fine, i installed a tool for network
monitoring at the level of IP and Port and i didnt see any
disconnection,


but did you see a *connection* ?
I mean, on the pinging node, if you use the Windows "netstat" program, for 
example as
netstat -aon -p TCP
you should see a list of connections in the ESTABLISHED state, of which one of the 
IP/ports should be your target IP:4000 (in the "remote" column).

And on the pinged node, this port :4000 should be in the "local" column, in 
LISTEN mode
(and also probably one in the ESTABLISHED state, if they agree.)

Is that the case ?


and yes i'm sure that no firewall is enabled.


I saw some strange think on the server that I have tried to ping the
multicast IP (228.0.0.4) and i get reply from different IPs in the
network, i don't know why and how i get those IPs, after checking with
the network team they told me that those IPs are related to the SAN
storage taking into consideration that the Tomcat servers are not
connected in anyway to that SUN storage.


On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)  wrote:

Hi.

This is for the Tomcat/Tribes experts on the list.

I know nothing of Tribes, but the on-line documentation seems to say that
the communication happens over TCP and that the protocol used is not
encrypted.
Fady previously tried a standard "ping" and a "telnet" between the two
nodes, and that is the base for him mentioning that "there is no network
disconnection" between the nodes.
Nevertheless, the calling pinging node seems to say that it times out
without getting a response fom the target node.  There is evidently a
contradiction there.
So this could still be some kind of network issue.

Considering that the protocol command for this "ping" should be known by
someone here, would it not be possible to imagine a little program in some
scripting language (or even java, God forbid), which would open a TCP
channel with the target node IP/port, send such a "ping" message, wait for a
response and report the result ?
That would at least confirm/deny that the problem is with the network.

The log below does not for example say if the error happens when opening the
TCP communication channel, or after sending the ping message on it,
(Of course, testing the TCP open could be done with "telnet IP 4000", but I
don't know if Fady tried this).
Maybe tribes also already contains some löw-level debugging options ?
wireshark maybe another option, but it has quite a learning curve.
And this is on Windows.

By the way Fady, are you sure that your "Windows Firewall with Enhanced
Security" is not just dropping TCP packets to/from port 40xx (or from
"java.exe") ? There are some "network policies" there which can have
wide-ranging side-effects.




On 07.02.2017 14:42, Fady Haikal wrote:


Hi, issue still not fixed. Tomcat session replication is not able to
replicate the key from node to node, please find below the error,
taking into consideration that there is no network disconnection
between 2 nodes


07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
Unable to replicate backup
key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
-85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
114, 43, 102}:4000;
   org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at
org.apache.catalina.tribes.group.GroupCh

Re: Operation has timed out

[Fady Haikal]

Hi,
telnet IP 4000 is working fine, i installed a tool for network
monitoring at the level of IP and Port and i didnt see any
disconnection, and yes i'm sure that no firewall is enabled.

I saw some strange think on the server that I have tried to ping the
multicast IP (228.0.0.4) and i get reply from different IPs in the
network, i don't know why and how i get those IPs, after checking with
the network team they told me that those IPs are related to the SAN
storage taking into consideration that the Tomcat servers are not
connected in anyway to that SUN storage.


On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)  wrote:
> Hi.
>
> This is for the Tomcat/Tribes experts on the list.
>
> I know nothing of Tribes, but the on-line documentation seems to say that
> the communication happens over TCP and that the protocol used is not
> encrypted.
> Fady previously tried a standard "ping" and a "telnet" between the two
> nodes, and that is the base for him mentioning that "there is no network
> disconnection" between the nodes.
> Nevertheless, the calling pinging node seems to say that it times out
> without getting a response fom the target node.  There is evidently a
> contradiction there.
> So this could still be some kind of network issue.
>
> Considering that the protocol command for this "ping" should be known by
> someone here, would it not be possible to imagine a little program in some
> scripting language (or even java, God forbid), which would open a TCP
> channel with the target node IP/port, send such a "ping" message, wait for a
> response and report the result ?
> That would at least confirm/deny that the problem is with the network.
>
> The log below does not for example say if the error happens when opening the
> TCP communication channel, or after sending the ping message on it,
> (Of course, testing the TCP open could be done with "telnet IP 4000", but I
> don't know if Fady tried this).
> Maybe tribes also already contains some löw-level debugging options ?
> wireshark maybe another option, but it has quite a learning curve.
> And this is on Windows.
>
> By the way Fady, are you sure that your "Windows Firewall with Enhanced
> Security" is not just dropping TCP packets to/from port 40xx (or from
> "java.exe") ? There are some "network policies" there which can have
> wide-ranging side-effects.
>
>
>
>
> On 07.02.2017 14:42, Fady Haikal wrote:
>>
>> Hi, issue still not fixed. Tomcat session replication is not able to
>> replicate the key from node to node, please find below the error,
>> taking into consideration that there is no network disconnection
>> between 2 nodes
>>
>>
>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>> Unable to replicate backup
>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
>> 114, 43, 102}:4000;
>>   org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
&

Re: Operation has timed out

[tomcat]

Hi.

This is for the Tomcat/Tribes experts on the list.

I know nothing of Tribes, but the on-line documentation seems to say that the 
communication happens over TCP and that the protocol used is not encrypted.
Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is 
the base for him mentioning that "there is no network disconnection" between the nodes.
Nevertheless, the calling pinging node seems to say that it times out without getting a 
response fom the target node.  There is evidently a contradiction there.

So this could still be some kind of network issue.

Considering that the protocol command for this "ping" should be known by someone here, 
would it not be possible to imagine a little program in some scripting language (or even 
java, God forbid), which would open a TCP channel with the target node IP/port, send such 
a "ping" message, wait for a response and report the result ?

That would at least confirm/deny that the problem is with the network.

The log below does not for example say if the error happens when opening the TCP 
communication channel, or after sending the ping message on it,
(Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if 
Fady tried this).

Maybe tribes also already contains some löw-level debugging options ?
wireshark maybe another option, but it has quite a learning curve.
And this is on Windows.

By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not 
just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network 
policies" there which can have wide-ranging side-effects.




On 07.02.2017 14:42, Fady Haikal wrote:

Hi, issue still not fixed. Tomcat session replication is not able to
replicate the key from node to node, please find below the error,
taking into consideration that there is no network disconnection
between 2 nodes


07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
Unable to replicate backup
key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
-85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
114, 43, 102}:4000;
  org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
at 
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at 
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at 
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at 
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at 
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
at org.apache.catalina.connector.Request.getSession(Request.java:2260)
at 
org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
at 
javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
at 
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
at 
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
at 
org.apache.jasper.runtime.PageContextImpl.initializ

Re: Operation has timed out

[Fady Haikal]

Hi, issue still not fixed. Tomcat session replication is not able to
replicate the key from node to node, please find below the error,
taking into consideration that there is no network disconnection
between 2 nodes


07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
Unable to replicate backup
key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
-85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
114, 43, 102}:4000;
 org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
at 
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at 
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at 
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at 
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at 
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
at org.apache.catalina.connector.Request.getSession(Request.java:2260)
at 
org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
at 
javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
at 
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
at 
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
at 
org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
at 
org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
at 
org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
at 
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
at 
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
at 
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
at 
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at

AW: Operation has timed out

[Kreuser, Peter]

remove sensible things like passwords, public
> > IP addresses etc..)(but not to the point of making it uncheckable).
> >
> > Then maybe some tribes-specialist can take over ?
> >
> >
> >>
> >> Is there actually something listening on that address/port ? Tomcat
> >> cluster
> >>
> >> the Port 4000 is listening and there is no disconnection between 2
> >> nodes ping and telnet are OK
> >>
> >> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) 
> >> wrote:
> >>>
> >>> On 06.02.2017 17:24, Fady Haikal wrote:
> >>>>
> >>>>
> >>>> Plz can i get some help here?
> >>>> This issue is still occurring and it's filling the log file in the
> >>>> Production server
> >>>>
> >>>> Regards,
> >>>> Fady
> >>>
> >>>
> >>>
> >>> Hi.
> >>> If you want quick answers, you should provide more information.
> >>> What is the host OS ?
> >>> What is the Tomcat version ?
> >>> Is this problem new ? was this working before ? how long ?
> >>>
> >>> I do not know tribes at all, but according to the logfile below, it seems
> >>> that something is trying to "ping" the address 10.114.43.103, port 4000,
> >>> and
> >>> never getting a response (or at least not within 3000ms).
> >>> Is there actually something listening on that address/port ?
> >>> The "netstat" command (available both on Linux and Windows) can tell you.
> >>> If there is something listening there, can it respond to whatever is
> >>> ping-ing it ?
> >>> (routing, firewall, ..)
> >>>
> >>>>
> >>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal 
> >>>> wrote:
> >>>>>
> >>>>>
> >>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
> >>>>>
> >>>>>
> >>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >>>>> Unable to send AbstractReplicatedMap.ping message
> >>>>>org.apache.catalina.tribes.ChannelException: Operation has timed
> >>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
> >>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
> >>>>> at
> >>>>>
>

Re: Operation has timed out

[Fady Haikal]

below is the server.xml configuration, as mentioened earlier the issue
is related to the cluster configuration, and as per my research i can
see that some users are facing the same issue but i didnt  found the
solution of it






  
  
  
  
  
  
  
  

  
  


  

  
  




















  
  


  

  
  
  

 
  




  



  

  
  

  

  








  

  


On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat)  wrote:
> On 06.02.2017 17:45, Fady Haikal wrote:
>>
>> Hi,
>> What is the host OS ? Windows Server 2012
>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>
>> Is this problem new ? was this working before ? how long ? Since
>> cluster implementation
>>
>
> I still don't know tribes, but then my non-educated guess at this point
> would be that there is something wrong in your configuration.
> Can you copy/paste it here ? (remove sensible things like passwords, public
> IP addresses etc..)(but not to the point of making it uncheckable).
>
> Then maybe some tribes-specialist can take over ?
>
>
>>
>> Is there actually something listening on that address/port ? Tomcat
>> cluster
>>
>> the Port 4000 is listening and there is no disconnection between 2
>> nodes ping and telnet are OK
>>
>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) 
>> wrote:
>>>
>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>
>>>>
>>>> Plz can i get some help here?
>>>> This issue is still occurring and it's filling the log file in the
>>>> Production server
>>>>
>>>> Regards,
>>>> Fady
>>>
>>>
>>>
>>> Hi.
>>> If you want quick answers, you should provide more information.
>>> What is the host OS ?
>>> What is the Tomcat version ?
>>> Is this problem new ? was this working before ? how long ?
>>>
>>> I do not know tribes at all, but according to the logfile below, it seems
>>> that something is trying to "ping" the address 10.114.43.103, port 4000,
>>> and
>>> never getting a response (or at least not within 3000ms).
>>> Is there actually something listening on that address/port ?
>>> The "netstat" command (available both on Linux and Windows) can tell you.
>>> If there is something listening there, can it respond to whatever is
>>> ping-ing it ?
>>> (routing, firewall, ..)
>>>
>>>>
>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal 
>>>> wrote:
>>>>>
>>>>>
>>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>>>
>>>>>
>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>org.apache.catalina.tribes.ChannelException: Operation has timed
>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>&

Re: Operation has timed out

[tomcat]

On 06.02.2017 17:45, Fady Haikal wrote:

Hi,
What is the host OS ? Windows Server 2012
What is the Tomcat version ? Apache Tomcat/8.0.30

Is this problem new ? was this working before ? how long ? Since
cluster implementation



I still don't know tribes, but then my non-educated guess at this point would be that 
there is something wrong in your configuration.
Can you copy/paste it here ? (remove sensible things like passwords, public IP addresses 
etc..)(but not to the point of making it uncheckable).


Then maybe some tribes-specialist can take over ?



Is there actually something listening on that address/port ? Tomcat cluster

the Port 4000 is listening and there is no disconnection between 2
nodes ping and telnet are OK

On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)  wrote:

On 06.02.2017 17:24, Fady Haikal wrote:


Plz can i get some help here?
This issue is still occurring and it's filling the log file in the
Production server

Regards,
Fady



Hi.
If you want quick answers, you should provide more information.
What is the host OS ?
What is the Tomcat version ?
Is this problem new ? was this working before ? how long ?

I do not know tribes at all, but according to the logfile below, it seems
that something is trying to "ping" the address 10.114.43.103, port 4000, and
never getting a response (or at least not within 3000ms).
Is there actually something listening on that address/port ?
The "netstat" command (available both on Linux and Windows) can tell you.
If there is something listening there, can it respond to whatever is
ping-ing it ?
(routing, firewall, ..)



On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal  wrote:


Guys, we are facing the below errors in Tomcat cluster, please advise


06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
   org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at
org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
at
org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)


06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
process request in NioReceiver
   java.io.IOException: A non-blocking socket operation could not be
completed immediately
at sun.nio.ch.SocketDispatcher.close0(Native Method)
at sun.nio.ch.SocketDispatcher.close(Unknown Source)
at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
at
org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
at java.lang.Thread.run(Unknown Source)

Regards,
Fady



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-

Re: Operation has timed out

[Fady Haikal]

Hi,
What is the host OS ? Windows Server 2012
What is the Tomcat version ? Apache Tomcat/8.0.30

Is this problem new ? was this working before ? how long ? Since
cluster implementation


Is there actually something listening on that address/port ? Tomcat cluster

the Port 4000 is listening and there is no disconnection between 2
nodes ping and telnet are OK

On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)  wrote:
> On 06.02.2017 17:24, Fady Haikal wrote:
>>
>> Plz can i get some help here?
>> This issue is still occurring and it's filling the log file in the
>> Production server
>>
>> Regards,
>> Fady
>
>
> Hi.
> If you want quick answers, you should provide more information.
> What is the host OS ?
> What is the Tomcat version ?
> Is this problem new ? was this working before ? how long ?
>
> I do not know tribes at all, but according to the logfile below, it seems
> that something is trying to "ping" the address 10.114.43.103, port 4000, and
> never getting a response (or at least not within 3000ms).
> Is there actually something listening on that address/port ?
> The "netstat" command (available both on Linux and Windows) can tell you.
> If there is something listening there, can it respond to whatever is
> ping-ing it ?
> (routing, firewall, ..)
>
>>
>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal  wrote:
>>>
>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>
>>>
>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>> Unable to send AbstractReplicatedMap.ping message
>>>   org.apache.catalina.tribes.ChannelException: Operation has timed
>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>
>>>
>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>>> process request in NioReceiver
>>>   java.io.IOException: A non-blocking socket operation could not be
>>> completed immediately
>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioR

Re: Operation has timed out

[tomcat]

On 06.02.2017 17:24, Fady Haikal wrote:

Plz can i get some help here?
This issue is still occurring and it's filling the log file in the
Production server

Regards,
Fady


Hi.
If you want quick answers, you should provide more information.
What is the host OS ?
What is the Tomcat version ?
Is this problem new ? was this working before ? how long ?

I do not know tribes at all, but according to the logfile below, it seems that something 
is trying to "ping" the address 10.114.43.103, port 4000, and never getting a response (or 
at least not within 3000ms).

Is there actually something listening on that address/port ?
The "netstat" command (available both on Linux and Windows) can tell you.
If there is something listening there, can it respond to whatever is ping-ing 
it ?
(routing, firewall, ..)



On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal  wrote:

Guys, we are facing the below errors in Tomcat cluster, please advise


06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
  org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
at 
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at 
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at 
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at 
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at 
org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
at 
org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)


06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
process request in NioReceiver
  java.io.IOException: A non-blocking socket operation could not be
completed immediately
at sun.nio.ch.SocketDispatcher.close0(Native Method)
at sun.nio.ch.SocketDispatcher.close(Unknown Source)
at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at 
org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
at 
org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
at java.lang.Thread.run(Unknown Source)

Regards,
Fady


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Operation has timed out

[Fady Haikal]

Plz can i get some help here?
This issue is still occurring and it's filling the log file in the
Production server

Regards,
Fady

On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal  wrote:
> Guys, we are facing the below errors in Tomcat cluster, please advise
>
>
> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
>  org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> at 
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
> at 
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> at 
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
> at 
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
> at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at 
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
> at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
> at 
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
> at 
> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
> at 
> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>
>
> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
> process request in NioReceiver
>  java.io.IOException: A non-blocking socket operation could not be
> completed immediately
> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at 
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
> at 
> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
> at java.lang.Thread.run(Unknown Source)
>
> Regards,
> Fady

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Operation has timed out

[Fady Haikal]

Guys, we are facing the below errors in Tomcat cluster, please advise


06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
 org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
at 
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at 
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at 
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at 
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at 
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at 
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
at 
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at 
org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
at 
org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)


06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
process request in NioReceiver
 java.io.IOException: A non-blocking socket operation could not be
completed immediately
at sun.nio.ch.SocketDispatcher.close0(Native Method)
at sun.nio.ch.SocketDispatcher.close(Unknown Source)
at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at 
org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
at 
org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
at java.lang.Thread.run(Unknown Source)

Regards,
Fady

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;

[Mark Eggers]

My replies are buried in the middle. I'll surround them with = Question 
===

/mde/

- Original Message -

> From: Dhaval Jaiswal 
> To: Tomcat Users List 
> Cc: 
> Sent: Monday, March 26, 2012 3:33 AM
> Subject: org.apache.catalina.tribes.ChannelException: Operation has timed 
> out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;
> 
> Hi List,
> 
> Below is my server.xml configuration for session replication.
> 
> 
>  channelSendOptions="6"/>
>         className="org.apache.catalina.ha.session.BackupManager"
> expireSessionsOnShutdown="false"
>                    notifyListenersOnReplication="true" 
> mapSendOptions="6"/>
>              className="org.apache.catalina.tribes.group.GroupChannel">
>               className="org.apache.catalina.tribes.membership.McastService"
>                         address="192.168.3.14"
>                         port="45564"
>                         frequency="500"
>                         dropTime="3"/>
>               className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>                       address="auto"
>                       port="5000"
>                       selectorTimeout="5000"
>                       timeout="6"
>                       maxThreads="6"/>
> 
>               className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>                 className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="6" keepAliveCount="0"/>
>              
>               className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>               className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>               className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
>           
>           className="org.apache.catalina.ha.tcp.ReplicationValve"
> 
> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt"/>

= Question ===

Don't you need another Valve at this point? All of my (admittedly small, 
default) configurations have the following Valve in the Cluster configuration:



Each engine then gets a unique jvmRoute="someName" which matches the AJP worker 
name that the particular Tomcat talks to. The worker name is in 
workers.properties (/etc/httpd/conf.d/workers.properties on my Fedora 15 and 
CentOS 5.8 systems).

I don't know for certain since I've not worked with the backup manager.

= Question ===

>         className="org.apache.catalina.ha.session.ClusterSessionListener"/>

= Question ===

Don't you need another ClusterListener at this point? All of my (admittedly 
small, default) configurations have the following ClusterListener at the end:




I will also assume that you missed copying the closing  tag from your 
server.xml.

= Question ===

> 
> 
> We are continuously getting following logging in our catalina.out. Can any
> one explain me why it is saying Operation has timed out(3000 ms.)
> How do i resolve it. I have searched the archive mail & change the setting
> according to it. However, no luck. Server is getting high load after adding
> session replication for backup manager. Your valuable response will be
> highly appreciated.
> 
> 
> 
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
> ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;
>         at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
>         at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
>         at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>         at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>         at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>         at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>         at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>         at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
>         at

org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;

[Dhaval Jaiswal]

Hi List,

Below is my server.xml configuration for session replication.



   

 
 

 
   
 
 
 
 
  
 
   


We are continuously getting following logging in our catalina.out. Can any
one explain me why it is saying Operation has timed out(3000 ms.)
How do i resolve it. I have searched the archive mail & change the setting
according to it. However, no luck. Server is getting high load after adding
session replication for backup manager. Your valuable response will be
highly appreciated.



SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:816)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:794)
at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:209)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:419)
at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:378)
at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1509)
at java.lang.Thread.run(Thread.java:619)
Mar 26, 2012 3:30:56 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 3,
10}:4000,{-64, -88, 3, 10},4000, alive=11339404,id={45 15 -8 5 54 -73 68
116 -124 21 124 -128 -34 -127 73 21 }, payload={}, command={}, domain={}, ]
Mar 26, 2012 3:30:58 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNING: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88,
3, 16}:4000,{-64, -88, 3, 16},4000, alive=415653,id={-74 -55 5 -64 -44 -111
72 -63 -115 -105 -102 3 34 45 -109 -7 }, payload={}, command={}, domain={},
]
Mar 26, 2012 3:30:58 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 3,
16}:4000,{-64, -88, 3, 16},4000, alive=415653,id={-74 -55 5 -64 -44 -111 72
-63 -115 -105 -102 3 34 45 -109 -7 }, payload={}, command={}, domain={}, ]
Mar 26, 2012 3:30:58 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64,
-88, 3, 10}:4000,{-64, -88, 3, 10},4000, alive=11335305,id={45 15 -8 5 54
-73 68 116 -124 21 124 -128 -34 -127 73 21 }, payload={}, command={},
domain={}, ]] message. Will verify.
Mar 26, 2012 3:30:58 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 3,
10}:4000,{-64, -88, 3, 10},4000, alive=11335305,id={45 15 -8 5 54 -73 68
116 

Re: Operation has timed out(3000 ms.).;

[Dhaval Jaiswal]

I have set it as below. However, still it is showing the same error. Please
let me know what to do as our all tomcats are going on high load because of
this kind of continuous logging. If there is any change required let us
know.


ERROR logging:
*SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
ms.).; Faulty members:tcp://{-64, -88, 3, 9}:4000; tcp://{-64, -88, 3,
15}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
*



 
   

 
 

 
   
 
 
 
 
  
 
   




On Thu, Mar 22, 2012 at 9:37 PM, Filip Hanik Mailing Lists <
devli...@hanik.com> wrote:

> take a look at the  attribute
>
> http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-sender.html
>
>
>
> - Original Message -
> > From: "Dhaval Jaiswal" 
> > To: "Tomcat Users List" 
> > Sent: Wednesday, March 21, 2012 10:35:14 PM
> > Subject: Operation has timed out(3000 ms.).;
> >
> > Receiving below errors in catalina log. I have changed the backup
> > manager
> > value of dropTime="3000" to "1" with restart of tomcats. However,
> > still
> > it is not taking effect & notifying that Operation has timed out(3000
> > ms.)
> > Please guide me where should i look to stop this messages.
> >
> >
> > SEVERE: Unable to send message through cluster sender.
> > org.apache.catalina.tribes.ChannelException: Operation has timed
> > out(3000
> > ms.).; Faulty members:tcp://{-64, -88, 3, 6}:4000;
> > at
> >
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
> > at
> >
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
> > at
> >
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
> > at
> >
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
> > at
> >
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> > at
> >
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
> >
> >
> >
> >
> > --
> > *
> >   Dhaval Jaiswal
> > Database & System
> >  E: dhaval.jais...@via.com
> > T: +91-80-4043 3000
> > M: +91-8095397843
> >  www.via.com
> >  <http://www.admin.viaworld.in/callcenter?action1=ADD_EPR_LOG>
> >
> >
> >  *
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Operation has timed out(3000 ms.).;

[Filip Hanik Mailing Lists]

take a look at the  attribute

http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-sender.html



- Original Message -
> From: "Dhaval Jaiswal" 
> To: "Tomcat Users List" 
> Sent: Wednesday, March 21, 2012 10:35:14 PM
> Subject: Operation has timed out(3000 ms.).;
> 
> Receiving below errors in catalina log. I have changed the backup
> manager
> value of dropTime="3000" to "1" with restart of tomcats. However,
> still
> it is not taking effect & notifying that Operation has timed out(3000
> ms.)
> Please guide me where should i look to stop this messages.
> 
> 
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000
> ms.).; Faulty members:tcp://{-64, -88, 3, 6}:4000;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
> 
> 
> 
> 
> --
> *
>   Dhaval Jaiswal
> Database & System
>  E: dhaval.jais...@via.com
> T: +91-80-4043 3000
> M: +91-8095397843
>  www.via.com
>  <http://www.admin.viaworld.in/callcenter?action1=ADD_EPR_LOG>
> 
> 
>  *
> 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Operation has timed out(3000 ms.).;

[Dhaval Jaiswal]

Receiving below errors in catalina log. I have changed the backup manager
value of dropTime="3000" to "1" with restart of tomcats. However, still
it is not taking effect & notifying that Operation has timed out(3000 ms.)
Please guide me where should i look to stop this messages.


SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
ms.).; Faulty members:tcp://{-64, -88, 3, 6}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)




-- 
*
  Dhaval Jaiswal
Database & System
 E: dhaval.jais...@via.com
T: +91-80-4043 3000
M: +91-8095397843
 www.via.com <http://www.admin.viaworld.in/callcenter?action1=ADD_EPR_LOG>


 *

Re: Tomcat Cluster Error - Operation has timed out

[Filip Hanik - Dev Lists]

0606 messages
Sent:156,13 MB (total)
Sent:156,13 MB (application)
Time:61,55 seconds
Tx Speed:2,54 MB/sec (total)
TxSpeed:2,54 MB/sec (application)
Error Msg:9
Rx Msg:21000 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:150,14 MB]

and now we're receiving and sending message but also Error messages
and then:

[CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
Channel key is registered, but has had no interest ops for the last 3000 ms.
(canc
elled:false):[EMAIL PROTECTED] last access:2008-02-12
17:34:17.957
and finally:

[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={},
]] mess
age. Will verify.
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.Me
mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
payload={}, command={}, dom
ain={}, ]]
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={}, ]
[CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(6
ms.).; Faulty members:tcp://223.99.114.21:14022;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:595)
[CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
at
or

Tomcat Cluster Error - Operation has timed out

[gp456]

   TxSpeed:2,54 MB/sec (application)
Error Msg:9
Rx Msg:21000 messages
Rx Speed:0,01 MB/sec (since 1st msg)
Received:150,14 MB]

and now we're receiving and sending message but also Error messages
and then:

[CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
Channel key is registered, but has had no interest ops for the last 3000 ms.
(canc
elled:false):[EMAIL PROTECTED] last access:2008-02-12
17:34:17.957
and finally:

[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={},
]] mess
age. Will verify.
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.Me
mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
payload={}, command={}, dom
ain={}, ]]
[CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
-125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={}, domain={}, ]
[CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out(6
ms.).; Faulty members:tcp://223.99.114.21:14022;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
at
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
at
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
at java.lang.Thread.run(Thread.java:595)
[CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: No destination given; No faulty
members identified.
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:1