Re: seamless restart

[Mark Eggers]

Chris,

On 5/12/2020 1:25 PM, Christopher Schultz wrote:
> Mark,
> 
> On 5/12/20 16:14, Mark Eggers wrote:
>> Chris,
> 
>> On 5/12/2020 12:55 PM, Christopher Schultz wrote:
>>> Jonathan,
>>>
>>> On 5/12/20 11:20, Jonathan Yom-Tov wrote:
 The problem is that my application is running on AWS which
 apparently doesn't support multicasting so I can't use
 Tomcat's DeltaManager.
>>>
>>> The membership-manager is separate from the replication-manager,
>>> so this has nothing to do with e.g. DeltaManager.
>>>
>>> You don't have to use multicast. You can use static membership if
>>> you know your node IP addresses.
>>>
>>> Rémy recently added a cloud membership service that uses
>>> Kubernetes as its default membership service. It looks like he
>>> hasn't written any documentation for it, but it exists in Tomcat
>>> 9 and 10.[1]
>>>
> 
>> This sounds interesting. I wonder how this will play using
>> multiple availability zones for high availability. It still won't
>> handle region outages, but there are other approaches for that.
> 
> I have no idea. There doesn't seem to me to be any reason why
> Kubernetes could not be used across regions. Maybe you wouldn't be
> able to use AWS-kube and might have to do it yourself. I have zero
> experience with Kubernetes, and zero experience with complex AWS
> deployments.
> 
>> I'll read the link you sent, and maybe play with that locally with
>> a Kubernetes setup. If I have questions about the set up, would
>> here or the dev list be the place to ask?
> 
> I think here would be better, since the answers will be visible to a
> wider group of people.
> 

That sounds reasonable. I know a lot of the devs read this group, so
hopefully (once I get started) there will be some answers.

> I'd love to see a writeup about this, including "how to set up
> Kubernetes from scratch to manage your Tomcat cluster" because I know
> literally nothing practical about it.

That and tying it into a cloud environment should be interesting. I'll
need to think about this before writing stuff on a mailing list, let
alone a document.

I'm just starting my journey through Docker / Kubernetes. I've put
together some simple images locally, and even have a private repository
set up running on Nexus 3. My experience with Kubernetes clusters is
limited to running "canned" environments.

It looks like an interesting road.

. . . just my two cents.
/mde/

> 
> -chris
> 
 I thought of using one of the Store implementations for
 PersistentManager but that has the issues which I mentioned
 earlier. My aim is to get to the point where I can add or take
 away servers from the cluster without impacting user
 experience.
>>>
>>> See above. Sounds like the cloud membership service is what you
>>> are looking for because it (a) handles dynamic membership and (b)
>>> doesn't use multicast.
>>>
 Ideally all state would be stored in a central location (e.g.
 Redis). But, since this is difficult because of the way the
 application is built I thought of using one server and only
 persisting the sessions when the server goes down. But I still
 have to solve the issues I mentioned.
>>> I would avoid single points of failure if possible. A "central
>>> location" tends to be a single point of failure. Tomcat clustered
>>> with e.g. BackupManager and dynamic membership will (a) achieve
>>> your goals and (b) not require additional products.
>>>
>>> Hope that helps, -chris
>>>
>>> [1]
>>> https://github.com/apache/tomcat/blob/master/java/org/apache/catalina
> /tr
>>>
>>>
> ibes/membership/cloud/CloudMembershipService.java#L34
>>>
 On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
 ch...@christopherschultz.net> wrote:
>>>
 Jonathan,
>>>
 On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>> I have an application which changes the state of user
>>> sessions in lots of places in the code. Is it possible to
>>> do a seamless switch of Tomcat servers, preserving all
>>> sessions?
>>>
>>> I know I can use PersistentManager to persist sessions
>>> and load them. I can think of two strategies:
>>>
>>> 1. Persist sessions periodically. This is more robust as
>>> I might not have control of when the server shuts down.
>>> 2. Persist sessions on server shutdown.
>>>
>>>
>>> The problem with the first approach is that I might lose
>>> the latest changes when the new server comes up. The
>>> problem with the second is that I'll have to lock access
>>> to the session until the old server is done saving it,
>>> which may make response times very slow.
>>>
>>> Is there a good solution to this that I might have
>>> overlooked?
>>>
 If you want to solve these problems:
>>>
 1. Seamless (uninterrupted) restarts 2. Always up-to-date
 (well, as much as possible) 3. No downtime
>>>
 Then you really need a cluster where the sessions are being
 replicated around the cluster.
>>>
 

Re: seamless restart

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark,

On 5/12/20 16:14, Mark Eggers wrote:
> Chris,
>
> On 5/12/2020 12:55 PM, Christopher Schultz wrote:
>> Jonathan,
>>
>> On 5/12/20 11:20, Jonathan Yom-Tov wrote:
>>> The problem is that my application is running on AWS which
>>> apparently doesn't support multicasting so I can't use
>>> Tomcat's DeltaManager.
>>
>> The membership-manager is separate from the replication-manager,
>> so this has nothing to do with e.g. DeltaManager.
>>
>> You don't have to use multicast. You can use static membership if
>> you know your node IP addresses.
>>
>> Rémy recently added a cloud membership service that uses
>> Kubernetes as its default membership service. It looks like he
>> hasn't written any documentation for it, but it exists in Tomcat
>> 9 and 10.[1]
>>
>
> This sounds interesting. I wonder how this will play using
> multiple availability zones for high availability. It still won't
> handle region outages, but there are other approaches for that.

I have no idea. There doesn't seem to me to be any reason why
Kubernetes could not be used across regions. Maybe you wouldn't be
able to use AWS-kube and might have to do it yourself. I have zero
experience with Kubernetes, and zero experience with complex AWS
deployments.

> I'll read the link you sent, and maybe play with that locally with
> a Kubernetes setup. If I have questions about the set up, would
> here or the dev list be the place to ask?

I think here would be better, since the answers will be visible to a
wider group of people.

I'd love to see a writeup about this, including "how to set up
Kubernetes from scratch to manage your Tomcat cluster" because I know
literally nothing practical about it.

- -chris

>>> I thought of using one of the Store implementations for
>>> PersistentManager but that has the issues which I mentioned
>>> earlier. My aim is to get to the point where I can add or take
>>> away servers from the cluster without impacting user
>>> experience.
>>
>> See above. Sounds like the cloud membership service is what you
>> are looking for because it (a) handles dynamic membership and (b)
>> doesn't use multicast.
>>
>>> Ideally all state would be stored in a central location (e.g.
>>> Redis). But, since this is difficult because of the way the
>>> application is built I thought of using one server and only
>>> persisting the sessions when the server goes down. But I still
>>> have to solve the issues I mentioned.
>> I would avoid single points of failure if possible. A "central
>> location" tends to be a single point of failure. Tomcat clustered
>> with e.g. BackupManager and dynamic membership will (a) achieve
>> your goals and (b) not require additional products.
>>
>> Hope that helps, -chris
>>
>> [1]
>> https://github.com/apache/tomcat/blob/master/java/org/apache/catalina
/tr
>>
>>
ibes/membership/cloud/CloudMembershipService.java#L34
>>
>>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>
>>> Jonathan,
>>
>>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>> I have an application which changes the state of user
>> sessions in lots of places in the code. Is it possible to
>> do a seamless switch of Tomcat servers, preserving all
>> sessions?
>>
>> I know I can use PersistentManager to persist sessions
>> and load them. I can think of two strategies:
>>
>> 1. Persist sessions periodically. This is more robust as
>> I might not have control of when the server shuts down.
>> 2. Persist sessions on server shutdown.
>>
>>
>> The problem with the first approach is that I might lose
>> the latest changes when the new server comes up. The
>> problem with the second is that I'll have to lock access
>> to the session until the old server is done saving it,
>> which may make response times very slow.
>>
>> Is there a good solution to this that I might have
>> overlooked?
>>
>>> If you want to solve these problems:
>>
>>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date
>>> (well, as much as possible) 3. No downtime
>>
>>> Then you really need a cluster where the sessions are being
>>> replicated around the cluster.
>>
>>> This will solve some other problems as well:
>>
>>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>>> Unexpected downtime (network outage, hardware fault) 6.
>>> Scaling-out (either manually or automatically)
>>
>>> You can do it with as little as two Tomcat instances. If you
>>> only care about being able to restart your application (and not
>>> the whole server, for example), then you can even run them
>>> side-by-side on the same server. You won't get protection
>>> against OS upgrades and unexpected downtime in that case, but
>>> you can get familiar with the setup without a whole lot of
>>> infrastructure.
>>
>>> -chris

 ---
- --


>>

To

Re: seamless restart

[Mark Eggers]

Chris,

On 5/12/2020 12:55 PM, Christopher Schultz wrote:
> Jonathan,
> 
> On 5/12/20 11:20, Jonathan Yom-Tov wrote:
>> The problem is that my application is running on AWS which
>> apparently doesn't support multicasting so I can't use Tomcat's
>> DeltaManager.
> 
> The membership-manager is separate from the replication-manager, so
> this has nothing to do with e.g. DeltaManager.
> 
> You don't have to use multicast. You can use static membership if you
> know your node IP addresses.
> 
> Rémy recently added a cloud membership service that uses Kubernetes as
> its default membership service. It looks like he hasn't written any
> documentation for it, but it exists in Tomcat 9 and 10.[1]
> 

This sounds interesting. I wonder how this will play using multiple
availability zones for high availability. It still won't handle region
outages, but there are other approaches for that.

I'll read the link you sent, and maybe play with that locally with a
Kubernetes setup. If I have questions about the set up, would here or
the dev list be the place to ask?

Thanks!

. . . just my two cents
/mde/

>> I thought of using one of the Store implementations for
>> PersistentManager but that has the issues which I mentioned
>> earlier. My aim is to get to the point where I can add or take away
>> servers from the cluster without impacting user experience.
> 
> See above. Sounds like the cloud membership service is what you are
> looking for because it (a) handles dynamic membership and (b) doesn't
> use multicast.
> 
>> Ideally all state would be stored in a central location (e.g.
>> Redis). But, since this is difficult because of the way the
>> application is built I thought of using one server and only
>> persisting the sessions when the server goes down. But I still have
>> to solve the issues I mentioned.
> I would avoid single points of failure if possible. A "central
> location" tends to be a single point of failure. Tomcat clustered with
> e.g. BackupManager and dynamic membership will (a) achieve your goals
> and (b) not require additional products.
> 
> Hope that helps,
> -chris
> 
> [1]
> https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/tr
> ibes/membership/cloud/CloudMembershipService.java#L34
> 
>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>> ch...@christopherschultz.net> wrote:
> 
>> Jonathan,
> 
>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
> I have an application which changes the state of user
> sessions in lots of places in the code. Is it possible to do
> a seamless switch of Tomcat servers, preserving all
> sessions?
>
> I know I can use PersistentManager to persist sessions and
> load them. I can think of two strategies:
>
> 1. Persist sessions periodically. This is more robust as I
> might not have control of when the server shuts down. 2.
> Persist sessions on server shutdown.
>
>
> The problem with the first approach is that I might lose the
> latest changes when the new server comes up. The problem with
> the second is that I'll have to lock access to the session
> until the old server is done saving it, which may make
> response times very slow.
>
> Is there a good solution to this that I might have
> overlooked?
> 
>> If you want to solve these problems:
> 
>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date (well, as
>> much as possible) 3. No downtime
> 
>> Then you really need a cluster where the sessions are being
>> replicated around the cluster.
> 
>> This will solve some other problems as well:
> 
>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>> Unexpected downtime (network outage, hardware fault) 6. Scaling-out
>> (either manually or automatically)
> 
>> You can do it with as little as two Tomcat instances. If you only
>> care about being able to restart your application (and not the
>> whole server, for example), then you can even run them side-by-side
>> on the same server. You won't get protection against OS upgrades
>> and unexpected downtime in that case, but you can get familiar with
>> the setup without a whole lot of infrastructure.
> 
>> -chris
>>>
>>> -
>>>
>>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
> 
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>




signature.asc
Description: OpenPGP digital signature

Re: seamless restart

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jonathan,

On 5/12/20 14:19, Jonathan Yom-Tov wrote:
> Thanks Mark. I've tried to use Redisson, it would've been the
> perfect solution for this except for the fact that my session
> object is a deep tree which is mutated in many areas of the code.
> So what happens is that as one request is changing the session
> state another will persist its session to Redis and overwrite the
> session of the first request. I can't think of an easy way to get
> around this.

This will be a problem with everything you try to do.

To make your code more "distributable", you might have to change
things so that you make a local copy (clone?), mutate the clone, then
write it back to the session.

Unless you call session.setAttribute (or similar), Tomcat won't
distribute anything to the other nodes in a cluster. For example:

((Transaction)session.getAttribute("tx")).setStatus(TransactionStaus.COM
PLETE);

Will not cause the cluster to replicate, but this will:

Transaction tx = ((Transaction)session.getAttribute("tx"));
tx.setStatus(TransactionStaus.COMPLETE);
session.setAttribute("tx", tx);

I think you will find many session managers work this way, otherwise
they end up being hideously inefficient by replicating the entire
session across the cluster (or to the session-store) after every request
.

- -chris

> On Tue, May 12, 2020 at 8:39 PM Mark Eggers
>  wrote:
>
>> Jonathan,
>>
>> On 5/12/2020 8:20 AM, Jonathan Yom-Tov wrote:
>>> The problem is that my application is running on AWS which
>>> apparently doesn't support multicasting so I can't use Tomcat's
>>> DeltaManager. I thought of using one of the Store
>>> implementations for
>> PersistentManager but
>>> that has the issues which I mentioned earlier. My aim is to get
>>> to the point where I can add or take away servers from the
>>> cluster without impacting user experience. Ideally all state
>>> would be stored in a central location (e.g. Redis). But, since
>>> this is difficult because of the way
>> the
>>> application is built I thought of using one server and only
>>> persisting
>> the
>>> sessions when the server goes down. But I still have to solve
>>> the issues
>> I
>>> mentioned.
>>>
>>>
>>>
>>>
>>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>>
>>> Jonathan,
>>>
>>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>> I have an application which changes the state of user
>> sessions in lots of places in the code. Is it possible to
>> do a seamless switch of Tomcat servers, preserving all
>> sessions?
>>
>> I know I can use PersistentManager to persist sessions
>> and load them. I can think of two strategies:
>>
>> 1. Persist sessions periodically. This is more robust as
>> I might not have control of when the server shuts down.
>> 2. Persist sessions on server shutdown.
>>
>>
>> The problem with the first approach is that I might lose
>> the latest changes when the new server comes up. The
>> problem with the second is that I'll have to lock access
>> to the session until the old server is done saving it,
>> which may make response times very slow.
>>
>> Is there a good solution to this that I might have
>> overlooked?
>>>
>>> If you want to solve these problems:
>>>
>>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date
>>> (well, as much as possible) 3. No downtime
>>>
>>> Then you really need a cluster where the sessions are being
>>> replicated around the cluster.
>>>
>>> This will solve some other problems as well:
>>>
>>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>>> Unexpected downtime (network outage, hardware fault) 6.
>>> Scaling-out (either manually or automatically)
>>>
>>> You can do it with as little as two Tomcat instances. If you
>>> only care about being able to restart your application (and not
>>> the whole server, for example), then you can even run them
>>> side-by-side on the same server. You won't get protection
>>> against OS upgrades and unexpected downtime in that case, but
>>> you can get familiar with the setup without a whole lot of
>>> infrastructure.
>>>
>>> -chris
>>
>> Could you use the RedissonSessionManager and an AWS - distributed
>> Redis server?
>>
>> You could put all of your Tomcat servers in an elastic group, and
>> let AWS manage that.
>>
>> The real problem with this approach is deployment. How do you
>> deploy across an elastic group of Tomcat servers when you may not
>> know the IP addresses of the servers or how many you have?
>>
>> I can think of some really kludgy ways to do this with S3 and
>> AWS events, but I've not worked out the details.
>>
>> Another way to approach this is to run Docker on AWS (along with
>> Redis), and then deploy a new version by deploying a new Docker
>> image in a rolling fashion.
>>
>> If your session interface changes a lot, that could create
>> issues.
>>
>> That's one of the adv

Re: seamless restart

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jonathan,

On 5/12/20 11:20, Jonathan Yom-Tov wrote:
> The problem is that my application is running on AWS which
> apparently doesn't support multicasting so I can't use Tomcat's
> DeltaManager.

The membership-manager is separate from the replication-manager, so
this has nothing to do with e.g. DeltaManager.

You don't have to use multicast. You can use static membership if you
know your node IP addresses.

Rémy recently added a cloud membership service that uses Kubernetes as
its default membership service. It looks like he hasn't written any
documentation for it, but it exists in Tomcat 9 and 10.[1]

> I thought of using one of the Store implementations for
> PersistentManager but that has the issues which I mentioned
> earlier. My aim is to get to the point where I can add or take away
> servers from the cluster without impacting user experience.

See above. Sounds like the cloud membership service is what you are
looking for because it (a) handles dynamic membership and (b) doesn't
use multicast.

> Ideally all state would be stored in a central location (e.g.
> Redis). But, since this is difficult because of the way the
> application is built I thought of using one server and only
> persisting the sessions when the server goes down. But I still have
> to solve the issues I mentioned.
I would avoid single points of failure if possible. A "central
location" tends to be a single point of failure. Tomcat clustered with
e.g. BackupManager and dynamic membership will (a) achieve your goals
and (b) not require additional products.

Hope that helps,
- -chris

[1]
https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/tr
ibes/membership/cloud/CloudMembershipService.java#L34

> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Jonathan,
>
> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
 I have an application which changes the state of user
 sessions in lots of places in the code. Is it possible to do
 a seamless switch of Tomcat servers, preserving all
 sessions?

 I know I can use PersistentManager to persist sessions and
 load them. I can think of two strategies:

 1. Persist sessions periodically. This is more robust as I
 might not have control of when the server shuts down. 2.
 Persist sessions on server shutdown.


 The problem with the first approach is that I might lose the
 latest changes when the new server comes up. The problem with
 the second is that I'll have to lock access to the session
 until the old server is done saving it, which may make
 response times very slow.

 Is there a good solution to this that I might have
 overlooked?
>
> If you want to solve these problems:
>
> 1. Seamless (uninterrupted) restarts 2. Always up-to-date (well, as
> much as possible) 3. No downtime
>
> Then you really need a cluster where the sessions are being
> replicated around the cluster.
>
> This will solve some other problems as well:
>
> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
> Unexpected downtime (network outage, hardware fault) 6. Scaling-out
> (either manually or automatically)
>
> You can do it with as little as two Tomcat instances. If you only
> care about being able to restart your application (and not the
> whole server, for example), then you can even run them side-by-side
> on the same server. You won't get protection against OS upgrades
> and unexpected downtime in that case, but you can get familiar with
> the setup without a whole lot of infrastructure.
>
> -chris
>>
>> -
>>
>>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl66/x4ACgkQHPApP6U8
pFhd8g//fAo+sX47/md+l9ZxwJnSLWic4c7idAOv1tm3C4PqkQacdfUmaq1jVurc
g9ZMh5NucdM65iubtkXtQrGOyO99kz2ZT9zrz3aFGyF3LY/F0YT3+i8bRtHyDTPp
ATEIJ4Bk+TkoHRRWTzJU3R6SMxa055KLBaEUWKUx8mEaXWNT+/bVtmxigKE0z2Ja
6qI6S1IzAdELRHUd2oVT5IjyBV/b2mjDG7N5YAJGNeCQ2obNrKBuQHdSElcA0Bjn
kXQo+ZHO1pvoMutgV49dvlAeT6u87KnNTPfwJcb0uEO4Eof7NbTswy40lASBRfp2
kRNrdFal/IE9wYUZF8GjVRU1kHOQB0+T8ZeW9hvpCvEsBWKwP74sPHVp71uudW/5
Ggxo5JkoIRvsi+2b6wPhRlQX4YG3awoBoZrGRSDGRaQttdh732hUVbvzSv9b2Z6F
JD0xSy8G5fqPUwouwrF3HIneszrZif1ZoDB6M/Py5i9Fs4F4UxagIEfb59/lTgy+
7GI/fgNV28rjR9QQJXWRp4j8uRreagSDK1Aehfq7cbAPdTpnh+yQlIkeCgYS3A05
wEzFcOlllLb6XW6UBle9GtY72opF/nAZGtToGeGiZvlP+t2x49aBdy6rN21oSyVC
BazreimVfhfS2FrYxYMOJJ8Fj52bSJgIj3dM39vvWpAdnjckg5k=
=8cBm
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: seamless restart

[Jonathan Yom-Tov]

Thanks Mark. I've tried to use Redisson, it would've been the perfect
solution for this except for the fact that my session object is a deep tree
which is mutated in many areas of the code. So what happens is that as one
request is changing the session state another will persist its session to
Redis and overwrite the session of the first request. I can't think of an
easy way to get around this.




On Tue, May 12, 2020 at 8:39 PM Mark Eggers 
wrote:

> Jonathan,
>
> On 5/12/2020 8:20 AM, Jonathan Yom-Tov wrote:
> > The problem is that my application is running on AWS which apparently
> > doesn't support multicasting so I can't use Tomcat's DeltaManager. I
> > thought of using one of the Store implementations for
> PersistentManager but
> > that has the issues which I mentioned earlier. My aim is to get to the
> > point where I can add or take away servers from the cluster without
> > impacting user experience. Ideally all state would be stored in a central
> > location (e.g. Redis). But, since this is difficult because of the way
> the
> > application is built I thought of using one server and only persisting
> the
> > sessions when the server goes down. But I still have to solve the issues
> I
> > mentioned.
> >
> >
> >
> >
> > On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Jonathan,
> >
> > On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>  I have an application which changes the state of user sessions in
>  lots of places in the code. Is it possible to do a seamless switch
>  of Tomcat servers, preserving all sessions?
> 
>  I know I can use PersistentManager to persist sessions and load
>  them. I can think of two strategies:
> 
>  1. Persist sessions periodically. This is more robust as I might
>  not have control of when the server shuts down. 2. Persist sessions
>  on server shutdown.
> 
> 
>  The problem with the first approach is that I might lose the latest
>  changes when the new server comes up. The problem with the second
>  is that I'll have to lock access to the session until the old
>  server is done saving it, which may make response times very slow.
> 
>  Is there a good solution to this that I might have overlooked?
> >
> > If you want to solve these problems:
> >
> > 1. Seamless (uninterrupted) restarts
> > 2. Always up-to-date (well, as much as possible)
> > 3. No downtime
> >
> > Then you really need a cluster where the sessions are being replicated
> > around the cluster.
> >
> > This will solve some other problems as well:
> >
> > 4. Expected downtime (e.g. OS/Tomcat/application upgrade)
> > 5. Unexpected downtime (network outage, hardware fault)
> > 6. Scaling-out (either manually or automatically)
> >
> > You can do it with as little as two Tomcat instances. If you only care
> > about being able to restart your application (and not the whole
> > server, for example), then you can even run them side-by-side on the
> > same server. You won't get protection against OS upgrades and
> > unexpected downtime in that case, but you can get familiar with the
> > setup without a whole lot of infrastructure.
> >
> > -chris
>
> Could you use the RedissonSessionManager and an AWS - distributed Redis
> server?
>
> You could put all of your Tomcat servers in an elastic group, and let
> AWS manage that.
>
> The real problem with this approach is deployment. How do you deploy
> across an elastic group of Tomcat servers when you may not know the IP
> addresses of the servers or how many you have?
>
> I can think of some really kludgy ways to do this with S3 and AWS
> events, but I've not worked out the details.
>
> Another way to approach this is to run Docker on AWS (along with Redis),
> and then deploy a new version by deploying a new Docker image in a
> rolling fashion.
>
> If your session interface changes a lot, that could create issues.
>
> That's one of the advantages of using versioned deployment
> (app.war##nnn) with a cluster. Old apps stay around until the session
> expires, while new sessions get the new version.
>
> Maybe -- just thinking out loud -- you could use an elastic group, AWS
> events, Redis (RedisSessionManager), and numbered WAR files to simulate
> a Tomcat cluster.
>
> Another question: Is the database-backed session manager provided with
> Tomcat slow? You could use that instead of the third party
> RedissonSessionManager.
>
> You should be able to test everything but the deployment locally. Just
> run a Docker implementation on your development machine, and then test
> either RedissonSessionManager or the JDBC backed session store. Docker
> will (can) be set up to mimic AWS elastic group behavior (expansion /
> contraction of containers), so the only question will be updates.
>
> Use something like JMeter to test sessions and hammer your Docker
> cluster. By default, Docker routes every request to a new container in a
> multi-container group. You'll kn

Re: seamless restart

[Mark Eggers]

Jonathan,

On 5/12/2020 8:20 AM, Jonathan Yom-Tov wrote:
> The problem is that my application is running on AWS which apparently
> doesn't support multicasting so I can't use Tomcat's DeltaManager. I
> thought of using one of the Store implementations for
PersistentManager but
> that has the issues which I mentioned earlier. My aim is to get to the
> point where I can add or take away servers from the cluster without
> impacting user experience. Ideally all state would be stored in a central
> location (e.g. Redis). But, since this is difficult because of the way the
> application is built I thought of using one server and only persisting the
> sessions when the server goes down. But I still have to solve the issues I
> mentioned.
>
>
>
>
> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Jonathan,
> 
> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
 I have an application which changes the state of user sessions in
 lots of places in the code. Is it possible to do a seamless switch
 of Tomcat servers, preserving all sessions?

 I know I can use PersistentManager to persist sessions and load
 them. I can think of two strategies:

 1. Persist sessions periodically. This is more robust as I might
 not have control of when the server shuts down. 2. Persist sessions
 on server shutdown.


 The problem with the first approach is that I might lose the latest
 changes when the new server comes up. The problem with the second
 is that I'll have to lock access to the session until the old
 server is done saving it, which may make response times very slow.

 Is there a good solution to this that I might have overlooked?
> 
> If you want to solve these problems:
> 
> 1. Seamless (uninterrupted) restarts
> 2. Always up-to-date (well, as much as possible)
> 3. No downtime
> 
> Then you really need a cluster where the sessions are being replicated
> around the cluster.
> 
> This will solve some other problems as well:
> 
> 4. Expected downtime (e.g. OS/Tomcat/application upgrade)
> 5. Unexpected downtime (network outage, hardware fault)
> 6. Scaling-out (either manually or automatically)
> 
> You can do it with as little as two Tomcat instances. If you only care
> about being able to restart your application (and not the whole
> server, for example), then you can even run them side-by-side on the
> same server. You won't get protection against OS upgrades and
> unexpected downtime in that case, but you can get familiar with the
> setup without a whole lot of infrastructure.
> 
> -chris

Could you use the RedissonSessionManager and an AWS - distributed Redis
server?

You could put all of your Tomcat servers in an elastic group, and let
AWS manage that.

The real problem with this approach is deployment. How do you deploy
across an elastic group of Tomcat servers when you may not know the IP
addresses of the servers or how many you have?

I can think of some really kludgy ways to do this with S3 and AWS
events, but I've not worked out the details.

Another way to approach this is to run Docker on AWS (along with Redis),
and then deploy a new version by deploying a new Docker image in a
rolling fashion.

If your session interface changes a lot, that could create issues.

That's one of the advantages of using versioned deployment
(app.war##nnn) with a cluster. Old apps stay around until the session
expires, while new sessions get the new version.

Maybe -- just thinking out loud -- you could use an elastic group, AWS
events, Redis (RedisSessionManager), and numbered WAR files to simulate
a Tomcat cluster.

Another question: Is the database-backed session manager provided with
Tomcat slow? You could use that instead of the third party
RedissonSessionManager.

You should be able to test everything but the deployment locally. Just
run a Docker implementation on your development machine, and then test
either RedissonSessionManager or the JDBC backed session store. Docker
will (can) be set up to mimic AWS elastic group behavior (expansion /
contraction of containers), so the only question will be updates.

Use something like JMeter to test sessions and hammer your Docker
cluster. By default, Docker routes every request to a new container in a
multi-container group. You'll know really quickly if distributed
sessions aren't working.

I need to get back to this for $work, but I've been getting yanked
around a bit. Hopefully, I'll be able to start testing all of these
ideas in the next month or so.

. . . just my two cents
/mde/



signature.asc
Description: OpenPGP digital signature

Re: seamless restart

[Jonathan Yom-Tov]

The problem is that my application is running on AWS which apparently
doesn't support multicasting so I can't use Tomcat's DeltaManager. I
thought of using one of the Store implementations for PersistentManager but
that has the issues which I mentioned earlier. My aim is to get to the
point where I can add or take away servers from the cluster without
impacting user experience. Ideally all state would be stored in a central
location (e.g. Redis). But, since this is difficult because of the way the
application is built I thought of using one server and only persisting the
sessions when the server goes down. But I still have to solve the issues I
mentioned.




On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Jonathan,
>
> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
> > I have an application which changes the state of user sessions in
> > lots of places in the code. Is it possible to do a seamless switch
> > of Tomcat servers, preserving all sessions?
> >
> > I know I can use PersistentManager to persist sessions and load
> > them. I can think of two strategies:
> >
> > 1. Persist sessions periodically. This is more robust as I might
> > not have control of when the server shuts down. 2. Persist sessions
> > on server shutdown.
> >
> >
> > The problem with the first approach is that I might lose the latest
> > changes when the new server comes up. The problem with the second
> > is that I'll have to lock access to the session until the old
> > server is done saving it, which may make response times very slow.
> >
> > Is there a good solution to this that I might have overlooked?
>
> If you want to solve these problems:
>
> 1. Seamless (uninterrupted) restarts
> 2. Always up-to-date (well, as much as possible)
> 3. No downtime
>
> Then you really need a cluster where the sessions are being replicated
> around the cluster.
>
> This will solve some other problems as well:
>
> 4. Expected downtime (e.g. OS/Tomcat/application upgrade)
> 5. Unexpected downtime (network outage, hardware fault)
> 6. Scaling-out (either manually or automatically)
>
> You can do it with as little as two Tomcat instances. If you only care
> about being able to restart your application (and not the whole
> server, for example), then you can even run them side-by-side on the
> same server. You won't get protection against OS upgrades and
> unexpected downtime in that case, but you can get familiar with the
> setup without a whole lot of infrastructure.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl66rKgACgkQHPApP6U8
> pFjsChAAtKdo2Ow+X+Lb0JQLAOxijryxlsRtO3rvdRPJLUxVG4ttVGXuJ86cPX9g
> uRoyOrtgSw3N7mPMgL51B39Sp8GtB6zcjAiB4CQzOTXA3FiRN99Jna6dEVpCUcQx
> xAUBNuJh6Q5N/l1tpLYIMgKXEmhGcexplNTEVl8IAb5tppNy1LtjUVMG3C4NolI1
> eKUVqNhtSmclBH9aDC94jlWLVH+OiKJhAfc965oCRv4E3vIcj5HGoKfK65iqFYPx
> hzOMUzvJfsHBfNMo0UyKvRiDIFyBUgeUEIAJZWAJojhTSHymYpQozYBDIjCfNHYa
> II6NYhYb9OuiwV9PEXFN2yZejBsgJvZzlbooWOg2z4nMmCItaTiUhOMyKNIY5wlz
> iXnh85yx/GMRSsiQrTD6X4cAwCsXusHwoF/WC0JncYr7hZw/mqwcXDnMhUAccOCc
> Z1l2x53yoVSH+RyoGgdG/DSWqLHZfiK/xr8UOtRbMD1WNfGOPo+GhpJBiSNSBeeS
> LMNiQbvZEcTKOsxUFgUDPwB62kbDyLnt765tgrSOWubBIaHXBctTNLtv9sjp6ARG
> DnNTmAtdk89UuhDA63iimDxx9jxDKsWKPz8Srf6RS7z8yUrlvFkfKQcIEaqHSLTr
> LnbrTHK2k92bJonJua5hDcuTgU7axEBlWmXxChyZWmYm2r1R76M=
> =suSa
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-- 
[image: SysAid Technologies]

Jonathan Yom-Tov
Senior Architect
jonathan.yom...@sysaid.com
Phone (IL): +972 (3) 533-3675 Ext. 932
[image: SysAid Technologies]

  [image: SysAid on Facebook]    [image:
SysAid on Twitter]    [image: SysAid on
Linked-in]    [image:
SysAid on YouTube]    [image: SysAid
on Instagram] 
[image: Banner] 

Re: seamless restart

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jonathan,

On 5/12/20 05:51, Jonathan Yom-Tov wrote:
> I have an application which changes the state of user sessions in
> lots of places in the code. Is it possible to do a seamless switch
> of Tomcat servers, preserving all sessions?
>
> I know I can use PersistentManager to persist sessions and load
> them. I can think of two strategies:
>
> 1. Persist sessions periodically. This is more robust as I might
> not have control of when the server shuts down. 2. Persist sessions
> on server shutdown.
>
>
> The problem with the first approach is that I might lose the latest
> changes when the new server comes up. The problem with the second
> is that I'll have to lock access to the session until the old
> server is done saving it, which may make response times very slow.
>
> Is there a good solution to this that I might have overlooked?

If you want to solve these problems:

1. Seamless (uninterrupted) restarts
2. Always up-to-date (well, as much as possible)
3. No downtime

Then you really need a cluster where the sessions are being replicated
around the cluster.

This will solve some other problems as well:

4. Expected downtime (e.g. OS/Tomcat/application upgrade)
5. Unexpected downtime (network outage, hardware fault)
6. Scaling-out (either manually or automatically)

You can do it with as little as two Tomcat instances. If you only care
about being able to restart your application (and not the whole
server, for example), then you can even run them side-by-side on the
same server. You won't get protection against OS upgrades and
unexpected downtime in that case, but you can get familiar with the
setup without a whole lot of infrastructure.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl66rKgACgkQHPApP6U8
pFjsChAAtKdo2Ow+X+Lb0JQLAOxijryxlsRtO3rvdRPJLUxVG4ttVGXuJ86cPX9g
uRoyOrtgSw3N7mPMgL51B39Sp8GtB6zcjAiB4CQzOTXA3FiRN99Jna6dEVpCUcQx
xAUBNuJh6Q5N/l1tpLYIMgKXEmhGcexplNTEVl8IAb5tppNy1LtjUVMG3C4NolI1
eKUVqNhtSmclBH9aDC94jlWLVH+OiKJhAfc965oCRv4E3vIcj5HGoKfK65iqFYPx
hzOMUzvJfsHBfNMo0UyKvRiDIFyBUgeUEIAJZWAJojhTSHymYpQozYBDIjCfNHYa
II6NYhYb9OuiwV9PEXFN2yZejBsgJvZzlbooWOg2z4nMmCItaTiUhOMyKNIY5wlz
iXnh85yx/GMRSsiQrTD6X4cAwCsXusHwoF/WC0JncYr7hZw/mqwcXDnMhUAccOCc
Z1l2x53yoVSH+RyoGgdG/DSWqLHZfiK/xr8UOtRbMD1WNfGOPo+GhpJBiSNSBeeS
LMNiQbvZEcTKOsxUFgUDPwB62kbDyLnt765tgrSOWubBIaHXBctTNLtv9sjp6ARG
DnNTmAtdk89UuhDA63iimDxx9jxDKsWKPz8Srf6RS7z8yUrlvFkfKQcIEaqHSLTr
LnbrTHK2k92bJonJua5hDcuTgU7axEBlWmXxChyZWmYm2r1R76M=
=suSa
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

seamless restart

[Jonathan Yom-Tov]

I have an application which changes the state of user sessions in lots of
places in the code. Is it possible to do a seamless switch of Tomcat
servers, preserving all sessions?

I know I can use PersistentManager to persist sessions and load them. I can
think of two strategies:

   1. Persist sessions periodically. This is more robust as I might not
   have control of when the server shuts down.
   2. Persist sessions on server shutdown.


The problem with the first approach is that I might lose the latest changes
when the new server comes up. The problem with the second is that I'll have
to lock access to the session until the old server is done saving it, which
may make response times very slow.

Is there a good solution to this that I might have overlooked?