NullPointerException when using NIO connector

[João Sávio]

Hello guys

I'm using Tomcat 7.0.53 and I've changed it to use the NIO connector
recently. So, the following error start appearing on the logs (few per day)

Jan 25, 2016 3:10:16 PM org.apache.tomcat.util.net.NioEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.NullPointerException


Any guesses? Is it a bug already fixed on newest versions?

Thanks
João

Synchronous Tomcat cluster

[João Sávio]

Hello guys

I'm trying to setup a Tomcat cluster to be synchronous (default uses
channelSendOptions=8, which is asynchronous). Some time ago, I had issues
with session replication, I asked on this list, and using the configuration
channelSendOptions="6" solved my issues. However, now, during load tests I
can observe that the session is not replicated on all cases during load
tests.

Here is the only change I made from the default configuration:




On logs, I can see errors on "Error Msg":


Jan 25, 2016 2:52:10 PM
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
INFO: ThroughputInterceptor Report[
Tx Msg:63301 messages
Sent:227.38 MB (total)
Sent:75.83 MB (application)
Time:5.67 seconds
Tx Speed:40.11 MB/sec (total)
TxSpeed:13.38 MB/sec (application)
Error Msg:39
Rx Msg:20 messages
Rx Speed:0.00 MB/sec (since 1st msg)
Received:227.72 MB]

So, do you know other configuration that I should change in order to
mitigate this?

Thanks
João

Re: ids of all active sessions

[João Sávio]

I think you can also retrieve them via JMX

Regards

2016-01-23 9:05 GMT-02:00 Kaouthar Ghorbel :

> thanks Thomas ☺
>
> 2016-01-18 10:02 GMT+01:00 Thomas Scheffler  >:
>
> > Am 18.01.16 um 09:35 schrieb Kaouthar Ghorbel:
> >
> >> Hello everyone I want to retrieve the ids of all active sessions in the
> >> servlet.
> >>
> >> thanks for your help :)
> >>
> >>
> > Hi Kaouthar,
> >
> > look for the HttpSessionListener and collect that information on your
> own:
> >
> >
> >
> https://tomcat.apache.org/tomcat-8.0-doc/servletapi/javax/servlet/http/HttpSessionListener.html
> >
> > regards,
> >
> > Thomas
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>

HTTP session change is not been replicated when changing it in event processing

[João Sávio]

Hi guys

I have a Tomcat cluster with 4 nodes that is set to replicate HTTP session
across all nodes. Common session operations work fine (e.g., user perform a
request, and something is saved on the session). However, I have a
situation where the session is not replicated.

My application works with ActiveMQ events, and depending on the event,
something should be saved in the HTTP session. In these cases, the session
only changes in that node which receives the event (at the beginning only
one node receives events).

Is there a way to solve this issue without workarounds in my application?

My server.xml if you'd like to see: http://pastebin.com/FNgeemwH

Thanks
João
-- 
http://joaosavio.wordpress.com

Re: HTTP session change is not been replicated when changing it in event processing

[João Sávio]

I found the solution:

I was updating an attribute in a session object. In that case, I need to
explicitly call session.setAttribute() with the updated object

Thanks

Re: HTTP session change is not been replicated when changing it in event processing

[João Sávio]

Thanks!

2014-11-19 13:48 GMT-02:00 Mark Thomas ma...@apache.org:

 On 19/11/2014 16:32, João Sávio wrote:
  I found the solution:
 
  I was updating an attribute in a session object. In that case, I need to
  explicitly call session.setAttribute() with the updated object

 https://issues.apache.org/bugzilla/show_bug.cgi?id=43866

 Mark

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org




-- 
http://joaosavio.wordpress.com

Re: Tomcat7 cluster don't replicates session

[João Sávio]

Rapaz, sugiro vc perguntar na lista do Tomcat

Ab
Em 29/10/2014 15:06, Nilson Uehara nilueh...@gmail.com escreveu:

 I have a Ubuntu with 2 Tomcat7 with a cluster configurations.

 The balancer is Apache MOD_JK.

 *SERVER.XML:*

 Engine name=Catalina defaultHost=localhost jvmRoute=no0
 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster
  channelSendOptions=8
   Manager className=org.apache.catalina.ha.session.DeltaManager
expireSessionsOnShutdown=false
notifyListenersOnReplication=true/
   Channel
 className=org.apache.catalina.tribes.group.GroupChannel
 Membership
 className=org.apache.catalina.tribes.membership.McastService
 address=228.0.0.4
 port=45564
 frequency=500
 dropTime=3/
 Receiver
 className=org.apache.catalina.tribes.transport.nio.NioReceiver
   address=10.0.2.15
   port=4001
   autoBind=100
   selectorTimeout=5000
   maxThreads=6/
 Sender
 className=org.apache.catalina.tribes.transport.ReplicationTransmitter
   Transport
 className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/
 /Sender
 Interceptor

 className=org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor/
 Interceptor

 className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/
 Interceptor

 className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/
   /Channel
   Valve className=org.apache.catalina.ha.tcp.ReplicationValve
 filter=/
   Valve
 className=org.apache.catalina.ha.session.JvmRouteBinderValve/
   !--Deployer
 className=org.apache.catalina.ha.deploy.FarmWarDeployer
 tempDir=/home/nilson/war-temp/
 deployDir=/home/nilson/war-deploy/
 watchDir=/home/nilson/war-listen/
 watchEnabled=false/--
   ClusterListener

 className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/
   ClusterListener
 className=org.apache.catalina.ha.session.ClusterSessionListener/
 /Cluster


 In server.xml of tomcat2, the ports are 8105, 8109 and 8180 and
 jvmRoute=no1 and port=4002



 *WORKERS.PROPERTIES:*
 worker.list=no0, no1, cluster

 worker.cluster.type=lb
 worker.cluster.balance_workers=no0, no1
 worker.cluster.sticky_session=1

 worker.no0.type=ajp13
 worker.no0.host=localhost
 worker.no0.port=8009
 worker.no0.lbfactor=1

 worker.no1.type=ajp13
 worker.no1.host=localhost
 worker.no1.port=8109
 worker.no1.lbfactor=1


 *$ route*
 Kernel IP routing table
 Destination Gateway Genmask Flags Metric RefUse
 Iface
 default 10.0.2.20.0.0.0 UG0  00
 eth0
 10.0.2.0*   255.255.255.0   U 1  00
 eth0
 224.0.0.0   *   240.0.0.0   U 0  00
 eth0


 *$ ifconfig*
 eth0  Link encap:Ethernet  HWaddr 08:00:27:d2:22:ac
   inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
   inet6 addr: fe80::a00:27ff:fed2:22ac/64 Scope:Link
   UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
   RX packets:112 errors:0 dropped:0 overruns:0 frame:0
   TX packets:164 errors:0 dropped:0 overruns:0 carrier:0
   collisions:0 txqueuelen:1000
   RX bytes:54502 (54.5 KB)  TX bytes:19774 (19.7 KB)


 *$sudo iptables -L*
 Chain INPUT (policy ACCEPT)
 target prot opt source   destination

 Chain FORWARD (policy ACCEPT)
 target prot opt source   destination

 Chain OUTPUT (policy ACCEPT)
 target prot opt source   destination


 Clusters log

 *TOMCAT1-CLUSTER.LOG:* -http://pastebin.com/w1TV73Qk

 *TOMCAT2-CLUSTER.LOG:* -http://pastebin.com/V7C35LTL

 One point I found interesting in the log is:

1. FINE: Received a failure detector

  
 packet:ClusterData[src=org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
0, 2, 15}:4002,{10, 0, 2, 15},4002, alive=1414598894083, securePort=-1,
 UDP
Port=-1, id={-127 -84 -80 -93 43 11 68 -77 -65 -49 47 -119 -59 -120 40
 19
}, payload={}, command={}, domain={}, ]; id={84 35 -110 45 -81 -92 66 86
-65 -60 -61 68 -70 -31 -59 -92 }; sent=2014-10-29 14:08:14.095]
2. Oct 29, 2014 2:08:14 PM
org.apache.catalina.tribes.transport.nio.NioReplicationTask remoteEof
3. FINE: Channel closed on the remote end, disconnecting


 What I understood is that the tomcats are able to communicate and when one
 falls the other receives the request with the session of the first
 normally. It seems the problem is that tomcat is not able to receive or
 read session data, then just 

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello Filip

You solved my issue! Thank you very much!

Thanks everyone
João

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello Mark

Thanks for your answer. I put the new information and cluster works (maybe
because I reboot my machine)

Indeed, I'm trying BIO because NIO is causing session replication issues on
load tests. Using NIO, I've already tried to increase maxThreads, and this
minimizes the problem, but don't solve it.

Thanks


2014-07-02 13:28 GMT-03:00 Mark Eggers its_toas...@yahoo.com.invalid:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 7/2/2014 6:37 AM, João Sávio wrote:
  ?
 
 
  2014-06-30 18:38 GMT-03:00 João Sávio joaosa...@gmail.com:
 
  Hello people
 
  This is my first message on this group! I'm trying to set up a
  Tomcat clustering using BIO receiver but I've been receiving the
  following error when I started two nodes and tried to enter on my
  application:
 
  Jun 30, 2014 9:25:12 PM
  org.apache.catalina.tribes.transport.bio.BioReplicationT ask run
  SEVERE: Unable to service bio socket
  java.net.SocketTimeoutException: Read timed out at
  java.net.SocketInputStream.socketRead0(Native Method) at
  java.net.SocketInputStream.read(SocketInputStream.java:152) at
  java.net.SocketInputStream.read(SocketInputStream.java:122) at
  java.net.SocketInputStream.read(SocketInputStream.java:108) at
  org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock
 
 
 et(BioReplicationTask.java:146)
  at
  org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe
 
 
 plicationTask.java:64)
  at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
 
 
 java:1145)
  at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
 
 
 .java:615)
  at java.lang.Thread.run(Thread.java:744)
 
  Jun 30, 2014 9:25:15 PM
  org.apache.catalina.ha.session.DeltaManager startInternal INFO:
  Register manager localhost#/myApp to cluster element Engine with
  name Catalina Jun 30, 2014 9:25:15 PM
  org.apache.catalina.ha.session.DeltaManager startInternal INFO:
  Starting clustering manager at localhost#/myApp Jun 30, 2014
  9:25:15 PM org.apache.catalina.ha.session.DeltaManager
  getAllClusterSessions INFO: Manager [localhost#/myApp],
  requesting session state from
  org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212,
  191, 209}:4000,{10, 212, 191, 2 09},4000, alive=32096,
  securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 -12 714 -75 -2
  74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={}, ].
  This operation will timeout if no session state has been received
  within 60 seconds.
 
 
  Attached are my two server.xml configuration. I've taken the
  default configuration (worked on my computer) and replaced the
  following things:
 
 
  Receiver
  className=org.apache.catalina.tribes.transport.nio.NioReceiver
  by Receiver
  className=org.apache.catalina.tribes.transport.bio.BioReceiver
 
  and
 
  Transport
 
 className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/
 
 
 by
  Transport
  className=org.apache.catalina.tribes.transport.bio.PooledMultiSender/
 
 
 
 
 I don't know if I'm missing something. I'll be very thankful if someone
  can take a look on my server.xml files or send me an example
  configuration using BIO receiver (I couldn't find anyone).
 
  Thanks João -- http://joaosavio.wordpress.com

 João,

 Welcome to the list.

 I've only used NIO, and it's been a while since I've used clustering.

 A few questions to ask:

 1. Is there a firewall blocking port 4000 (if the Tomcats are on
separate machines)?
 2. Multicast enabled and routed?
 3. Environment (Tomcat version, java version, platforms)?
 4. Why BIO?

 What are you trying to accomplish by using the BIO connector rather
 than the NIO connector?

 You can turn on more logging by adding something like the following to
 logging.properties:

 # at the end of the handlers line:
 ,5cluster.org.apache.juli.FileHandler

 # in the handler-specific properties section:
 5cluster.org.apache.juli.FileHandler.level = FINER
 5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
 5cluster.org.apache.juli.FileHandler.prefix = cluster.

 # in the facility specific properties section
 # be aware of line wrapping
 org.apache.catalina.tribes.MESSAGES.level = FINE
 org.apache.catalina.tribes.MESSAGES.handlers =
 5cluster.org.apache.juli.FileHandler

 org.apache.catalina.tribes.level = FINE
 org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler

 org.apache.catalina.ha.level = FINE
 org.apache.catalina.ha.handlers = 5cluster.org.apache.juli.FileHander

 org.apache.catalina.ha.deploy.level = INFO
 org.apache.catalina.ha.deploy.handlers =
 5cluster.org.apache.juli.FileHandler

 Last note - we're normally a pretty helpful bunch of folks here, but
 we are all volunteers. Things like work impact participation :-p. Also
 as noted above, I've only used NIO so I don't know why this would not
 work with BIO.

 . . . just my two cents
 /mde/
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.13 (MingW32)
 Comment: Using GnuPG with Thunderbird

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Unfortunately it's not working yet

I increased the log level as you suggested. Log attached

Thanks



2014-07-03 11:10 GMT-03:00 João Sávio joaosa...@gmail.com:

 Hello Mark

 Thanks for your answer. I put the new information and cluster works (maybe
 because I reboot my machine)

 Indeed, I'm trying BIO because NIO is causing session replication issues
 on load tests. Using NIO, I've already tried to increase maxThreads, and
 this minimizes the problem, but don't solve it.

 Thanks


 2014-07-02 13:28 GMT-03:00 Mark Eggers its_toas...@yahoo.com.invalid:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 7/2/2014 6:37 AM, João Sávio wrote:
  ?
 
 
  2014-06-30 18:38 GMT-03:00 João Sávio joaosa...@gmail.com:
 
  Hello people
 
  This is my first message on this group! I'm trying to set up a
  Tomcat clustering using BIO receiver but I've been receiving the
  following error when I started two nodes and tried to enter on my
  application:
 
  Jun 30, 2014 9:25:12 PM
  org.apache.catalina.tribes.transport.bio.BioReplicationT ask run
  SEVERE: Unable to service bio socket
  java.net.SocketTimeoutException: Read timed out at
  java.net.SocketInputStream.socketRead0(Native Method) at
  java.net.SocketInputStream.read(SocketInputStream.java:152) at
  java.net.SocketInputStream.read(SocketInputStream.java:122) at
  java.net.SocketInputStream.read(SocketInputStream.java:108) at
  org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock
 
 
 et(BioReplicationTask.java:146)
  at
  org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe
 
 
 plicationTask.java:64)
  at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
 
 
 java:1145)
  at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
 
 
 .java:615)
  at java.lang.Thread.run(Thread.java:744)
 
  Jun 30, 2014 9:25:15 PM
  org.apache.catalina.ha.session.DeltaManager startInternal INFO:
  Register manager localhost#/myApp to cluster element Engine with
  name Catalina Jun 30, 2014 9:25:15 PM
  org.apache.catalina.ha.session.DeltaManager startInternal INFO:
  Starting clustering manager at localhost#/myApp Jun 30, 2014
  9:25:15 PM org.apache.catalina.ha.session.DeltaManager
  getAllClusterSessions INFO: Manager [localhost#/myApp],
  requesting session state from
  org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212,
  191, 209}:4000,{10, 212, 191, 2 09},4000, alive=32096,
  securePort=-1, UDP Port=-1, id={-58 95 -116 89 86 -12 714 -75 -2
  74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={}, ].
  This operation will timeout if no session state has been received
  within 60 seconds.
 
 
  Attached are my two server.xml configuration. I've taken the
  default configuration (worked on my computer) and replaced the
  following things:
 
 
  Receiver
  className=org.apache.catalina.tribes.transport.nio.NioReceiver
  by Receiver
  className=org.apache.catalina.tribes.transport.bio.BioReceiver
 
  and
 
  Transport
 
 className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/
 
 
 by
  Transport
 
 className=org.apache.catalina.tribes.transport.bio.PooledMultiSender/
 
 
 
 
 I don't know if I'm missing something. I'll be very thankful if someone
  can take a look on my server.xml files or send me an example
  configuration using BIO receiver (I couldn't find anyone).
 
  Thanks João -- http://joaosavio.wordpress.com

 João,

 Welcome to the list.

 I've only used NIO, and it's been a while since I've used clustering.

 A few questions to ask:

 1. Is there a firewall blocking port 4000 (if the Tomcats are on
separate machines)?
 2. Multicast enabled and routed?
 3. Environment (Tomcat version, java version, platforms)?
 4. Why BIO?

 What are you trying to accomplish by using the BIO connector rather
 than the NIO connector?

 You can turn on more logging by adding something like the following to
 logging.properties:

 # at the end of the handlers line:
 ,5cluster.org.apache.juli.FileHandler

 # in the handler-specific properties section:
 5cluster.org.apache.juli.FileHandler.level = FINER
 5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
 5cluster.org.apache.juli.FileHandler.prefix = cluster.

 # in the facility specific properties section
 # be aware of line wrapping
 org.apache.catalina.tribes.MESSAGES.level = FINE
 org.apache.catalina.tribes.MESSAGES.handlers =
 5cluster.org.apache.juli.FileHandler

 org.apache.catalina.tribes.level = FINE
 org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler

 org.apache.catalina.ha.level = FINE
 org.apache.catalina.ha.handlers = 5cluster.org.apache.juli.FileHander

 org.apache.catalina.ha.deploy.level = INFO
 org.apache.catalina.ha.deploy.handlers =
 5cluster.org.apache.juli.FileHandler

 Last note - we're normally a pretty helpful bunch of folks here, but
 we are all volunteers. Things like work impact participation :-p. Also
 as noted above, I've only used NIO so I don't know

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello!

Using NIO (with channelSendOptions=4, i.e., synchronous), with lightly
load, my tests pass 100%. But, on heavy load, not all sessions are
replicated on time, and I have about 20% of errors. If I increase
maxThreads to 400, I have about 15% of errors.

More information:
* I am not performing parallel requests with same session
* my cluster has 4 nodes (all in one machine - for test purpose only)
* Java 7 64 bits, Tomcat 7.0.52, windows 7 64 bits
* using default NIO configuration, but with maxThreads=400
* VM options:
-Xms512M   - on real environment this value is 1024
-Xmx512M  - on real environment this value is 1024
-XX:NewSize=450M
-XX:MaxNewSize=450M
-XX:PermSize=128M
-XX:MaxPermSize=245M
-XX:SurvivorRatio=8
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=15
-XX:+UseBiasedLocking
-XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+UseParNewGC
-XX:+DisableExplicitGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log

Moreover, I'm trying to attach the logs again.

Thanks
João


cluster.rar
Description: application/rar

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

cluster.log - http://pastebin.com/c98WhnmG


2014-07-03 13:04 GMT-03:00 João Sávio joaosa...@gmail.com:

 Hello!

 Using NIO (with channelSendOptions=4, i.e., synchronous), with lightly
 load, my tests pass 100%. But, on heavy load, not all sessions are
 replicated on time, and I have about 20% of errors. If I increase
 maxThreads to 400, I have about 15% of errors.

 More information:
 * I am not performing parallel requests with same session
 * my cluster has 4 nodes (all in one machine - for test purpose only)
 * Java 7 64 bits, Tomcat 7.0.52, windows 7 64 bits
 * using default NIO configuration, but with maxThreads=400
 * VM options:
 -Xms512M   - on real environment this value is 1024
 -Xmx512M  - on real environment this value is 1024
 -XX:NewSize=450M
 -XX:MaxNewSize=450M
 -XX:PermSize=128M
 -XX:MaxPermSize=245M
 -XX:SurvivorRatio=8
 -XX:TargetSurvivorRatio=90
 -XX:MaxTenuringThreshold=15
 -XX:+UseBiasedLocking
 -XX:CMSInitiatingOccupancyFraction=60
 -XX:+UseCMSInitiatingOccupancyOnly
 -XX:+CMSClassUnloadingEnabled
 -XX:+UseConcMarkSweepGC
 -XX:+CMSIncrementalMode
 -XX:+UseParNewGC
 -XX:+DisableExplicitGC
 -XX:+PrintGCDateStamps
 -XX:+PrintGCDetails
 -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log

 Moreover, I'm trying to attach the logs again.

 Thanks
 João




-- 
http://joaosavio.wordpress.com


cluster.rar
Description: application/rar

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello

Some points below:

** What is on time?*
In my application, a group of users should always hit the same node after
the first request. So, in first request each group of users will receive an
specific cookie, and LB will perform the load balancing based on this
cookie. In first request, a user can hit any node, but from the second, he
or she should hit the same node.

** What are the errors? Test result errors?*
For this test, I simplified the code of my application:
- first request: store one object in session
- second request: verify if the object is in session. If it's not - ERROR

** How big are are the sessions that you're trying to replicate?*
- I'm using Spring MVC, and I have 3 additional objects in session. They
are not big (15 attributes each one)

** What's the load like on the box when you're running the tests that
you get errors on?*
- I've experiencing this issue on BIO even without load

** It is preferred to use the non blocking receiver to be able to grow your
cluster without running into thread starvation.*
- That's why I've tried NIO first, but I'd like to see if BIO solve my
issue and if using BIO my system doesn't get too slow.


Now, I'll try to run my tests using NIO, default VM configuration and FINER
logs.

Thanks a lot
João


2014-07-03 14:07 GMT-03:00 Mark Eggers its_toas...@yahoo.com.invalid:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 7/3/2014 9:12 AM, João Sávio wrote:
  cluster.log - http://pastebin.com/c98WhnmG
 
 
  2014-07-03 13:04 GMT-03:00 João Sávio joaosa...@gmail.com:
 
  Hello!
 
  Using NIO (with channelSendOptions=4, i.e., synchronous), with
  lightly load, my tests pass 100%. But, on heavy load, not all
  sessions are replicated on time, and I have about 20% of errors.
  If I increase maxThreads to 400, I have about 15% of errors.
 
  More information: * I am not performing parallel requests with
  same session * my cluster has 4 nodes (all in one machine - for
  test purpose only) * Java 7 64 bits, Tomcat 7.0.52, windows 7 64
  bits * using default NIO configuration, but with maxThreads=400 *
  VM options: -Xms512M   - on real environment this value is 1024
  -Xmx512M  - on real environment this value is 1024
  -XX:NewSize=450M -XX:MaxNewSize=450M -XX:PermSize=128M
  -XX:MaxPermSize=245M -XX:SurvivorRatio=8
  -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15
  -XX:+UseBiasedLocking -XX:CMSInitiatingOccupancyFraction=60
  -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled
  -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
  -XX:+DisableExplicitGC -XX:+PrintGCDateStamps
  -XX:+PrintGCDetails -Xloggc:%CATALINA_BASE%/logs/tomcat-gc.log
 
  Moreover, I'm trying to attach the logs again.
 
  Thanks João

 João,

 I took a look at the log. This is the BIO attempt and you do run out
 of threads. See the following:

 Jul 03, 2014 11:41:21 AM
 org.apache.catalina.tribes.transport.bio.BioReceiver listen
 WARNING: All BIO server replication threads are busy, unable to handle
 more requests until a thread is freed up.

 What's the load like on the box when you're running the tests that you
 get errors on?

 As Dan asks in his message:

 What is on time?
 What are the errors? Test result errors?

 How big are are the sessions that you're trying to replicate?

 My guess is that something else is going on, since the following log
 entry doesn't show much in the way of cluster traffic.

 INFO: ThroughputInterceptor Report[
 Tx Msg:1 messages
 Sent:0.00 MB (total)
 Sent:0.00 MB (application)
 Time:0.01 seconds
 Tx Speed:0.04 MB/sec (total)
 TxSpeed:0.04 MB/sec (application)
 Error Msg:0
 Rx Msg:13 messages
 Rx Speed:0.00 MB/sec (since 1st msg)
 Received:0.00 MB]

 It would also be interesting to see the logs when you use the NIO
 connector. According to the documentation:

 It is preferred to use the non blocking receiver to be able to grow
 your cluster without running into thread starvation.

 Also from the documentation:

 Usually the rule is to use 1 thread per node in the cluster for small
 clusters, and then depending on your message frequency and your
 hardware, you'll find an optimal number of threads peak out at a
 certain number.

 We might need a little more background on your application and your
 test environment to figure out why clustering is not behaving for you.

 . . . just my two cents
 /mde/

 PS - you have some errors in your server.xml (see the log). While they
 won't impact this problem, it might be a good idea to address them.

 /mde/
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.13 (MingW32)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iQEcBAEBAgAGBQJTtY28AAoJEEFGbsYNeTwtbk4H/1ehs00fmOLGfpcDxKkbfJJc
 B2T3FEYmW2scV/W3Z0+z4uhBgVwRqPHgEZHotdRFhkadymCKz0d5RjjEgnTMv5vH
 eP1u35NjmtteeLg+EcZU9XP1HOR+oxcx9fFic9NULtUb1lQOd9pIV9SWO82vFSI5
 0ERzCxMr/ysiOZHPXPwl6SCe9TWGwYAWJh1QrH+3tqaD+EV7mYdZk7P/MOSWnSxn

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hi everyone

I ran my test (total of 1k requests, total of 100 threads) against two
nodes with default VM settings. I've just set heap size. I had about 15% of
errors.

cluster.log - node1 - http://pastebin.com/cpX900Qw
cluster.log - node2 - http://pastebin.com/qCSzMaU6

Running for a long time (total of 500k requests, total of 100 threads) I
had about 11% of errors. In this case we can see the statistics:

Jul 03, 2014 5:53:28 PM
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
INFO: ThroughputInterceptor Report[
Tx Msg:1 messages
Sent:12.36 MB (total)
 Sent:12.36 MB (application)
Time:7.82 seconds
Tx Speed:1.58 MB/sec (total)
 TxSpeed:1.58 MB/sec (application)
Error Msg:0
Rx Msg:10198 messages
 Rx Speed:0.08 MB/sec (since 1st msg)
Received:12.38 MB]


All session attributes are Serializable, and it's a session replication
issue because if I ran my test with just one node, I had 0% of errors.

Regarding on time, just a correction:

1. first request, pick a random server and store a session object
2. second request, pick *ANY* server (chose by LB based on the cookie - it
can be the same, but not necessarily) and ask for the session object.

To be more clean, I've been working with a conference system. Each
conference should occur in one node. So, the first request can hit any
server, and from the second request should hit the node where the
conference is.


Thanks a lot
João


2014-07-03 15:40 GMT-03:00 Filip Hanik fi...@hanik.com:

 you mention NIO and say maxThreads, that sounds like the Connector
 configuration, but the BIO receiver is on the cluster, and it a completely
 different component that also has an applicable NIO configuration.

 are you confusing the two?
 I'm saying that you should use the NIO receiver on the cluster component,
 and if you do, what kind of errors do you get?


 On Thu, Jul 3, 2014 at 12:19 PM, Mark Eggers its_toas...@yahoo.com.invalid
 
 wrote:

  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA1
 
  João,
 
  This list has a convention of posting either inline or at the end of
  the message you're replying to.
 
  See here for mailing list notes:
 
  http://tomcat.apache.org/lists.html#tomcat-users
 
  On 7/3/2014 10:24 AM, João Sávio wrote:
   Hello
  
   Some points below:
  
   ** What is on time?* In my application, a group of users should
   always hit the same node after the first request. So, in first
   request each group of users will receive an specific cookie, and
   LB will perform the load balancing based on this cookie. In first
   request, a user can hit any node, but from the second, he or she
   should hit the same node.
 
  Hmm, so 'on time' really means that subsequent requests should hit the
  same server.
 
  If you're using sessions, Tomcat has an attribute on the Engine
  element called jvmRoute. So depending on your load balancer (and if
  you use AJP), you can use Tomcat and AJP to route traffic. In that
  case, there's no need to write a special cookie.
 
  At any rate, this doesn't sound like a clustering error per se.
 
  
   ** What are the errors? Test result errors?* For this test, I
   simplified the code of my application: - first request: store one
   object in session - second request: verify if the object is in
   session. If it's not - ERROR
  
 
  So looking at the information from 'on time', the scenario should be:
 
  1. first request, pick a random server and store a session object
  2. second request, pick the SAME server and ask for the session object
 
  Again, I'm not seeing where this is a clustering issue per se.
 
   ** How big are are the sessions that you're trying to replicate?*
   - I'm using Spring MVC, and I have 3 additional objects in
   session. They are not big (15 attributes each one)
  
 
  And all attributes are serializable? The objects are also marked as
  serializable?
 
   ** What's the load like on the box when you're running the tests
   that you get errors on?* - I've experiencing this issue on BIO
   even without load
  
 
  I may have not phrased my question carefully. What is the CPU and
  memory situation on your test box while running the 4 Tomcat servers?
 
  I know you've trimmed down your Xms and Xmx (presumably to fit in your
  test environment), but in combination with your other JVM parameters
  could this be causing some issues?
 
  I would follow Dan's recommendation of maybe just setting Xms, Xmx, GC
  logging to see what happens. Ah, I see you're going to do that below.
 
   ** It is preferred to use the non blocking receiver to be able to
   grow your cluster without running into thread starvation.* -
   That's why I've tried NIO first, but I'd like to see if BIO solve
   my issue and if using BIO my system doesn't get too slow.
 
  I don't think speed is so much an issue here, but scalability is. NIO
  can handle multiple requests per thread, BIO cannot.
 
  
  
   Now, I'll try to run my tests using NIO, default VM configuration
   and FINER logs.
 
  Post

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello Filip

The nodes are in the same machine!

Regards
João

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello Mark

In fact, I'm not explicit invalidating session on this two requests.

Regards
João

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello Filip

I'm using channelSendOptions=4, which is supposed to be synchronous

Regards
João

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

I don't think so.

Here it is: http://pastebin.com/qYCzmECb  (server.xml - node1)

Regards
João

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

If I set channelSendOptions=8 (default value = asynchronous), the % of
errors increase (as expected)

Regards
João


2014-07-03 19:43 GMT-03:00 João Sávio joaosa...@gmail.com:

 I don't think so.

 Here it is: http://pastebin.com/qYCzmECb  (server.xml - node1)

 Regards
 João




-- 
http://joaosavio.wordpress.com

Re: Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

?


2014-06-30 18:38 GMT-03:00 João Sávio joaosa...@gmail.com:

 Hello people

 This is my first message on this group! I'm trying to set up a Tomcat
 clustering using BIO receiver but I've been receiving the following error
 when I started two nodes and tried to enter on my application:

 Jun 30, 2014 9:25:12 PM
 org.apache.catalina.tribes.transport.bio.BioReplicationT
 ask run
 SEVERE: Unable to service bio socket
 java.net.SocketTimeoutException: Read timed out
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:152)
 at java.net.SocketInputStream.read(SocketInputStream.java:122)
 at java.net.SocketInputStream.read(SocketInputStream.java:108)
 at
 org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock
 et(BioReplicationTask.java:146)
 at
 org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe
 plicationTask.java:64)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
 java:1145)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
 .java:615)
 at java.lang.Thread.run(Thread.java:744)

 Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager
 startInternal
 INFO: Register manager localhost#/myApp to cluster element Engine with
 name Catalina
 Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager
 startInternal
 INFO: Starting clustering manager at localhost#/myApp
 Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager
 getAllClusterSessions
 INFO: Manager [localhost#/myApp], requesting session state from
 org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, 191,
 209}:4000,{10, 212, 191, 2
 09},4000, alive=32096, securePort=-1, UDP Port=-1, id={-58 95 -116 89 86
 -12 714 -75 -2 74 -45 -54 -82 40 -114 }, payload={}, command={},
 domain={}, ]. This operation will timeout if no session state has been
 received within 60 seconds.


 Attached are my two server.xml configuration. I've taken the default
 configuration (worked on my computer) and replaced the following things:


 Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver
 by
 Receiver className=org.apache.catalina.tribes.transport.bio.BioReceiver

 and

 Transport
 className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/
 by
 Transport
 className=org.apache.catalina.tribes.transport.bio.PooledMultiSender/


 I don't know if I'm missing something. I'll be very thankful if someone
 can take a look on my server.xml files or send me an example configuration
 using BIO receiver (I couldn't find anyone).

 Thanks
 João
 --
 http://joaosavio.wordpress.com





-- 
http://joaosavio.wordpress.com

Help with Tomcat 7 clustering using BIO receiver

[João Sávio]

Hello people

This is my first message on this group! I'm trying to set up a Tomcat
clustering using BIO receiver but I've been receiving the following error
when I started two nodes and tried to enter on my application:

Jun 30, 2014 9:25:12 PM
org.apache.catalina.tribes.transport.bio.BioReplicationT
ask run
SEVERE: Unable to service bio socket
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:108)
at
org.apache.catalina.tribes.transport.bio.BioReplicationTask.drainSock
et(BioReplicationTask.java:146)
at
org.apache.catalina.tribes.transport.bio.BioReplicationTask.run(BioRe
plicationTask.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:744)

Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager
startInternal
INFO: Register manager localhost#/myApp to cluster element Engine with name
Catalina
Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager
startInternal
INFO: Starting clustering manager at localhost#/myApp
Jun 30, 2014 9:25:15 PM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
INFO: Manager [localhost#/myApp], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 212, 191,
209}:4000,{10, 212, 191, 2
09},4000, alive=32096, securePort=-1, UDP Port=-1, id={-58 95 -116 89 86
-12 714 -75 -2 74 -45 -54 -82 40 -114 }, payload={}, command={}, domain={},
]. This operation will timeout if no session state has been received within
60 seconds.


Attached are my two server.xml configuration. I've taken the default
configuration (worked on my computer) and replaced the following things:


Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver
by
Receiver className=org.apache.catalina.tribes.transport.bio.BioReceiver

and

Transport
className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/
by
Transport
className=org.apache.catalina.tribes.transport.bio.PooledMultiSender/


I don't know if I'm missing something. I'll be very thankful if someone can
take a look on my server.xml files or send me an example configuration
using BIO receiver (I couldn't find anyone).

Thanks
João
-- 
http://joaosavio.wordpress.com
?xml version='1.0' encoding='utf-8'?
!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the License); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an AS IS BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
--
!-- Note:  A Server is not itself a Container, so you may not
 define subcomponents such as Valves at this level.
 Documentation at /docs/config/server.html
 --
Server port=8005 shutdown=SHUTDOWN
  !-- Security listener. Documentation at /docs/config/listeners.html
  Listener className=org.apache.catalina.security.SecurityListener /
  --
  !--APR library loader. Documentation at /docs/apr.html --
  Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on /
  !--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html --
  Listener className=org.apache.catalina.core.JasperListener /
  !-- Prevent memory leaks due to use of particular java/javax APIs--
  Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener /
  Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener /
  Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener /

  !-- Global JNDI resources
   Documentation at /docs/jndi-resources-howto.html
  --
  GlobalNamingResources
!-- Editable user database that can also be used by
 UserDatabaseRealm to authenticate users
--
Resource name=UserDatabase auth=Container
  type=org.apache.catalina.UserDatabase
  description=User database that can be updated and saved
  factory=org.apache.catalina.users.MemoryUserDatabaseFactory
  pathname=conf/tomcat-users.xml /
  /GlobalNamingResources
  
  !-- A Service is a collection of one or more