Re: Zero downtime deployments

[Christopher Schultz]

Jason,

On 12/22/15 11:46 PM, Jason Britton wrote:
> On Tue, Dec 22, 2015 at 4:01 PM, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>> So mod_jk routes a request to the node which is going down, and then
>> decides to re-route because the connection times-out? Just making sure I
>> have that all in my head (it's an awfully long sentence).
>
> my understanding, could be wrong, that if the connect_timeout is reached by
> mod_jk that it would try a different node.

I can't remember if it will auto-retry; it might depend upon your
configuration.

>>> A different active tomcat node now receives the request
>>> and tries to reconstitute the user's session from the same shared
>>> JDBCStore, but what if the first tomcat node is not finished shutting
>> down
>>> and has not finished writing out this particular user's session data yet?
>>> How can we ensure that session data will be there?
>>
>> Although I'm not entirely sure of the behavior of Tomcat's clustering
>> features in that particular case, you are mostly asking the following:
>> "what happens if two nodes are essentially sharing a session? how do I
>> make sure their view of the session is consistent?"
>>
> 
> I should have clarified the environment I was envisioning was one with
> sticky sessions, no clustering.  And the situation I'm looking to wrap my
> brain around is one where the once active node is triggered to write out
> session data due to the impending shut down of the particular node.
> 
> At about the same time this shutdown is occurring a request comes in for a
> client that has a session cookie value routing it to the node being shut
> down.  mod_jk times out trying to connect to this node and reroutes the
> request to an active node.  I see a race condition where depending on how
> long it takes the first tomcat being shutdown to write out session data to
> the JDBCStore, that the subsequent tomcat node trying to service the
> request may not find this user's session data in the shared JDBCStore.  I'd
> really like to know if this is truly something to be concerned about, and
> if so, how to account for it.

This scenario really isn't any different than a clustered environment
with two requests going to two different nodes (for any reason, really).
I think this answer still stands:

>> I think the answer is: you can't. Make your requests as idempotent as
>> possible and, when possible, execute the other kinds of requests such a
>> way that the first one to execute "wins" and the others fail gracefully.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Zero downtime deployments

[Frederik Nosi]

Hi Christopher,

Il 23/12/2015 19:12, Christopher Schultz ha scritto:

Jason,

On 12/22/15 11:46 PM, Jason Britton wrote:

On Tue, Dec 22, 2015 at 4:01 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

So mod_jk routes a request to the node which is going down, and then
decides to re-route because the connection times-out? Just making sure I
have that all in my head (it's an awfully long sentence).

my understanding, could be wrong, that if the connect_timeout is reached by
mod_jk that it would try a different node.

I can't remember if it will auto-retry; it might depend upon your
configuration.


This is configurable, usually i set:

worker.$WORKER_NAME.recovery_options=19

that is from memory, retry only GET and HEAD, not POST.


This because it have occured to me, that a POST timeouts to some app 
running in a tomcat backend but the tomcat side processing thread does 
not know this and it continues running. JK retries the same POST in 
another tomcat backend. In this case you get your POST request executed 
at least twice which usually is not nice.



BTW this thread reminds me about a patch i sent but never followed on 
being busy with other stuff:


https://mail-archives.apache.org/mod_mbox/tomcat-users/201404.mbox/%3c533ec184.7070...@postecom.it%3E


The problem i tried to solve was that when you have N backends and your 
site get's slashdotted so that backends are all busy, mod_jk amplifies 
the load even more, as it retries at least all N backends. With that 
patch i added an option for thelling mod_jk to retry only X backends, 
usually only once.





A different active tomcat node now receives the request
and tries to reconstitute the user's session from the same shared
JDBCStore, but what if the first tomcat node is not finished shutting

down

and has not finished writing out this particular user's session data yet?
How can we ensure that session data will be there?

Although I'm not entirely sure of the behavior of Tomcat's clustering
features in that particular case, you are mostly asking the following:
"what happens if two nodes are essentially sharing a session? how do I
make sure their view of the session is consistent?"


I should have clarified the environment I was envisioning was one with
sticky sessions, no clustering.  And the situation I'm looking to wrap my
brain around is one where the once active node is triggered to write out
session data due to the impending shut down of the particular node.

At about the same time this shutdown is occurring a request comes in for a
client that has a session cookie value routing it to the node being shut
down.  mod_jk times out trying to connect to this node and reroutes the
request to an active node.  I see a race condition where depending on how
long it takes the first tomcat being shutdown to write out session data to
the JDBCStore, that the subsequent tomcat node trying to service the
request may not find this user's session data in the shared JDBCStore.  I'd
really like to know if this is truly something to be concerned about, and
if so, how to account for it.

This scenario really isn't any different than a clustered environment
with two requests going to two different nodes (for any reason, really).
I think this answer still stands:


I think the answer is: you can't. Make your requests as idempotent as
possible and, when possible, execute the other kinds of requests such a
way that the first one to execute "wins" and the others fail gracefully.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Regards,
Frederik

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Zero downtime deployments

[Christopher Schultz]

Jason,

On 12/21/15 12:22 PM, Jason Britton wrote:
> Following back up after perusing Chris' very helpful presentation (
> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf)
> on
> load balancing tomcats with mod_jk.
> 
> One part not mentioned was session fail over.

Correct. I lump that in with "clustering", which was explicitly
out-of-score for the 50-minute presentation at ApacheCon. Mark covered
it a bit in his 3-part presentation at the same conference.

Have a look at the 2015-04-15* presentations here:
http://home.apache.org/~markt/presentations/

Also possibly here:
http://home.apache.org/~markt/presentations/2014-11-20-Tomcat-load-balancing-and-clustering/

> Which my current plan is to
> use JDBCStore for persisting to a database table shared by all tomcats
> powering apps.  But this has brought up a couple concerns...  If while a
> tomcat node is being shut down and triggered by this shut down tomcat is in
> the middle of writing out session data to the JDBCStore, a request from a
> client that was using that node comes in, mod_jk tries to route the request
> to the node that is shutting down but mod_jk's configured connect_timeout
> times out waiting and mod_jk then routes to another node (hopefully that's
> how this works).

So mod_jk routes a request to the node which is going down, and then
decides to re-route because the connection times-out? Just making sure I
have that all in my head (it's an awfully long sentence).

> A different active tomcat node now receives the request
> and tries to reconstitute the user's session from the same shared
> JDBCStore, but what if the first tomcat node is not finished shutting down
> and has not finished writing out this particular user's session data yet?
> How can we ensure that session data will be there?

Although I'm not entirely sure of the behavior of Tomcat's clustering
features in that particular case, you are mostly asking the following:
"what happens if two nodes are essentially sharing a session? how do I
make sure their view of the session is consistent?"

I think the answer is: you can't. Make your requests as idempotent as
possible and, when possible, execute the other kinds of requestssuch a
way that the first one to execute "wins" and the others fail gracefully.


> From looking at the config of the Manager, there is a maxIdleBackup
> parameter that I imagine could be set to 0 but I'm a little wary of the
> possible performance implications.  Its description: The time interval (in
> seconds) since the last access to a session before it is eligible for being
> persisted to the session store, or -1 to disable this feature. By default,
> this feature is disabled.  Even if this was ok from a performance
> perspective, once a session was initially persisted, would its session data
> be updated in the session store on subsequent changes/requests?

Even if you update every time the background thread runs, that thread is
doing a handful of different things and you might not get that real-time
feeling you are looking for.

Session data will be updated if you call session.setAttribute() or
session.removeAttribute (or equivalent) during the request. If you have
a HashMap in the session and you call
((Map)session.getAttribute("myMap")).put("foo", "bar") then your session
will effectively change only on the node where that request was being
served. If you want to persist those changes to other nodes (e.g. using
BackupManager) or to a Store, then you'll need to do the equivalent of:

session.setAttribute("myMap", session.getAttribute("myMap"));

This will poke the cluster manager to do whatever is appropriate to push
the session to other places (db, other Tomcat server, memcached, etc.).

> I've read about some folks configuring valves for shared in-memory caches
> (redis), I'm not sure I want to go that route at this point.

You only want to do this kind of thing if two items apply to you:

1. You need better performance than a disk-based db can provide
2. You can afford to lose everything if a process dies

Uaually #2 is a deal-breaker for people considering "real" clustering;
that is, using distributable sessions and not sticky-sessions without
distribution or storage of any kind (since otherwise, you would just let
people's sessions die when they fail-over).

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Zero downtime deployments

[Jason Britton]

On Tue, Dec 22, 2015 at 4:01 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Jason,
>
> On 12/21/15 12:22 PM, Jason Britton wrote:
> > Following back up after perusing Chris' very helpful presentation (
> >
> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf
> )
> > on
> > load balancing tomcats with mod_jk.
> >
> > One part not mentioned was session fail over.
>
> Correct. I lump that in with "clustering", which was explicitly
> out-of-score for the 50-minute presentation at ApacheCon. Mark covered
> it a bit in his 3-part presentation at the same conference.
>
>
Understandable


> Have a look at the 2015-04-15* presentations here:
> http://home.apache.org/~markt/presentations/
>
> Also possibly here:
>
> http://home.apache.org/~markt/presentations/2014-11-20-Tomcat-load-balancing-and-clustering/
>
>
Thanks for these links.


> > Which my current plan is to
> > use JDBCStore for persisting to a database table shared by all tomcats
> > powering apps.  But this has brought up a couple concerns...  If while a
> > tomcat node is being shut down and triggered by this shut down tomcat is
> in
> > the middle of writing out session data to the JDBCStore, a request from a
> > client that was using that node comes in, mod_jk tries to route the
> request
> > to the node that is shutting down but mod_jk's configured connect_timeout
> > times out waiting and mod_jk then routes to another node (hopefully
> that's
> > how this works).
>
> So mod_jk routes a request to the node which is going down, and then
> decides to re-route because the connection times-out? Just making sure I
> have that all in my head (it's an awfully long sentence).
>
>
my understanding, could be wrong, that if the connect_timeout is reached by
mod_jk that it would try a different node.



> > A different active tomcat node now receives the request
> > and tries to reconstitute the user's session from the same shared
> > JDBCStore, but what if the first tomcat node is not finished shutting
> down
> > and has not finished writing out this particular user's session data yet?
> > How can we ensure that session data will be there?
>
> Although I'm not entirely sure of the behavior of Tomcat's clustering
> features in that particular case, you are mostly asking the following:
> "what happens if two nodes are essentially sharing a session? how do I
> make sure their view of the session is consistent?"
>

I should have clarified the environment I was envisioning was one with
sticky sessions, no clustering.  And the situation I'm looking to wrap my
brain around is one where the once active node is triggered to write out
session data due to the impending shut down of the particular node.

At about the same time this shutdown is occurring a request comes in for a
client that has a session cookie value routing it to the node being shut
down.  mod_jk times out trying to connect to this node and reroutes the
request to an active node.  I see a race condition where depending on how
long it takes the first tomcat being shutdown to write out session data to
the JDBCStore, that the subsequent tomcat node trying to service the
request may not find this user's session data in the shared JDBCStore.  I'd
really like to know if this is truly something to be concerned about, and
if so, how to account for it.



> I think the answer is: you can't. Make your requests as idempotent as
> possible and, when possible, execute the other kinds of requestssuch a
> way that the first one to execute "wins" and the others fail gracefully.
>
>
> > From looking at the config of the Manager, there is a maxIdleBackup
> > parameter that I imagine could be set to 0 but I'm a little wary of the
> > possible performance implications.  Its description: The time interval
> (in
> > seconds) since the last access to a session before it is eligible for
> being
> > persisted to the session store, or -1 to disable this feature. By
> default,
> > this feature is disabled.  Even if this was ok from a performance
> > perspective, once a session was initially persisted, would its session
> data
> > be updated in the session store on subsequent changes/requests?
>
> Even if you update every time the background thread runs, that thread is
> doing a handful of different things and you might not get that real-time
> feeling you are looking for.
>
> Session data will be updated if you call session.setAttribute() or
> session.removeAttribute (or equivalent) during the request. If you have
> a HashMap in the session and you call
> ((Map)session.getAttribute("myMap")).put("foo", "bar") then your session
> will effectively change only on the node where that request was being
> served. If you want to persist those changes to other nodes (e.g. using
> BackupManager) or to a Store, then you'll need to do the equivalent of:
>
> session.setAttribute("myMap", session.getAttribute("myMap"));
>
> This will poke the cluster manager to do whatever is 

Re: Zero downtime deployments

[Jason Britton]

Following back up after perusing Chris' very helpful presentation (
http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf)
on
load balancing tomcats with mod_jk.

One part not mentioned was session fail over.  Which my current plan is to
use JDBCStore for persisting to a database table shared by all tomcats
powering apps.  But this has brought up a couple concerns...  If while a
tomcat node is being shut down and triggered by this shut down tomcat is in
the middle of writing out session data to the JDBCStore, a request from a
client that was using that node comes in, mod_jk tries to route the request
to the node that is shutting down but mod_jk's configured connect_timeout
times out waiting and mod_jk then routes to another node (hopefully that's
how this works).  A different active tomcat node now receives the request
and tries to reconstitute the user's session from the same shared
JDBCStore, but what if the first tomcat node is not finished shutting down
and has not finished writing out this particular user's session data yet?
How can we ensure that session data will be there?

>From looking at the config of the Manager, there is a maxIdleBackup
parameter that I imagine could be set to 0 but I'm a little wary of the
possible performance implications.  Its description: The time interval (in
seconds) since the last access to a session before it is eligible for being
persisted to the session store, or -1 to disable this feature. By default,
this feature is disabled.  Even if this was ok from a performance
perspective, once a session was initially persisted, would its session data
be updated in the session store on subsequent changes/requests?

I've read about some folks configuring valves for shared in-memory caches
(redis), I'm not sure I want to go that route at this point.

Any thoughts would be much appreciated,

Jason


On Tue, Dec 8, 2015 at 10:43 AM, Kevin Hale Boyes  wrote:

> On 4 December 2015 at 11:19, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Kevin,
> >
> > On 12/3/15 2:21 PM, Kevin Hale Boyes wrote:
> > > Thanks for this link to the presentation.
> > > How do you all deal with some of the other dependencies that the web
> > > application has?
> > >
> > > For example, if v2 of my application needs new database columns or
> > worse, a
> > > change to an existing column how can I have v1 and v2 running at the
> same
> > > time?  We use Oracle as our database though the problem exists for many
> > > database servers.
> >
> > That's not a deployment problem. That's a design problem.
> >
> > In short, you need to make sure that your web application is both
> > forward-compatible and backward-compatible at the same time.
> >
> > That's rubbish, or course. Here's a more concrete suggestion, and it
> > requires a LOT of work, but it's worth it.
> >
> > First of all, you can't think of a release as a single event any more.
> > Instead, there are 4 events.
> >
>
>
> We already do this sort of development when needed.  In particular, when we
> have database changes that will take a long time like back populating new
> columns or creating new indexes on large tables, we run then in a
> pre-release phase against live production.  We time the activities to
> finish just before our release window.  Sometimes we also have post-release
> database activities as well that follow shortly after our release outage.
>
> I guess I hadn't extended my thoughts far enough and seen the full
> potential of the idea so thanks.
> One drawback for us when we use this mechanism is related to our release
> schedule.  We release on a quarterly basis so having a process that spans
> four releases pushes it out for an entire year.  But that's our problem and
> we're making headway there too.
>
> Thanks Christopher
> Kevin.
>

Re: Zero downtime deployments

[Kevin Hale Boyes]

On 4 December 2015 at 11:19, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Kevin,
>
> On 12/3/15 2:21 PM, Kevin Hale Boyes wrote:
> > Thanks for this link to the presentation.
> > How do you all deal with some of the other dependencies that the web
> > application has?
> >
> > For example, if v2 of my application needs new database columns or
> worse, a
> > change to an existing column how can I have v1 and v2 running at the same
> > time?  We use Oracle as our database though the problem exists for many
> > database servers.
>
> That's not a deployment problem. That's a design problem.
>
> In short, you need to make sure that your web application is both
> forward-compatible and backward-compatible at the same time.
>
> That's rubbish, or course. Here's a more concrete suggestion, and it
> requires a LOT of work, but it's worth it.
>
> First of all, you can't think of a release as a single event any more.
> Instead, there are 4 events.
>


We already do this sort of development when needed.  In particular, when we
have database changes that will take a long time like back populating new
columns or creating new indexes on large tables, we run then in a
pre-release phase against live production.  We time the activities to
finish just before our release window.  Sometimes we also have post-release
database activities as well that follow shortly after our release outage.

I guess I hadn't extended my thoughts far enough and seen the full
potential of the idea so thanks.
One drawback for us when we use this mechanism is related to our release
schedule.  We release on a quarterly basis so having a process that spans
four releases pushes it out for an entire year.  But that's our problem and
we're making headway there too.

Thanks Christopher
Kevin.

Re: Zero downtime deployments

[Kevin Hale Boyes]

This is exactly what I'm after.  Thanks to both of you.

On 3 December 2015 at 12:54, Tauzell, Dave <dave.tauz...@surescripts.com>
wrote:

> If you cannot use a tool like flyway you can do it by hand, too.  The key
> is that all database changes need to be backwards compatible.  For example:
>
> So, if you want to drop a column:
> 1. Deploy new version of app that doesn't require column X
> 2. Shutdown version which does
> 3. drop column X
>
> If you want to rename a column:
> 1. Deploy new version of app that writes data to old column name + new name
> 2. shutdown old version
> 3. drop old column
>
> We generally test out our old version of an app running against the new
> database schema to make sure this process will work.
>
> -Dave
>
> -Original Message-
> From: jieryn [mailto:jie...@gmail.com]
> Sent: Thursday, December 03, 2015 1:32 PM
> To: Tomcat Users List
> Subject: Re: Zero downtime deployments
>
> Use http://flywaydb.org/ to perform database migrations. You will need at
> least 3 versions in order to perform an incompatible database change. v1 is
> existing behavior, v2 is a shim that bridges v1 and v3, and then v3 cleans
> up the shim and removes all the unnecessary hacks.
> When you have v1 rolled out against N images, then you can start rolling
> out compat shim v2 which adds some temporary work for the db and coexists,
> then when all instances are at v2 you can move to v3 which uses all the new
> stuff.
>
> I haven't found anything better or more efficient than the 2-step
> deployment with live code.
>
> On Thu, Dec 3, 2015 at 2:21 PM, Kevin Hale Boyes <kcbo...@gmail.com>
> wrote:
> > Thanks for this link to the presentation.
> > How do you all deal with some of the other dependencies that the web
> > application has?
> >
> > For example, if v2 of my application needs new database columns or
> > worse, a change to an existing column how can I have v1 and v2 running
> > at the same time?  We use Oracle as our database though the problem
> > exists for many database servers.
> >
> > Thanks,
> > Kevin.
> >
> > On 3 December 2015 at 01:31, Neill Lima <neill.l...@visual-meta.com>
> wrote:
> >
> >> Hello Jason,
> >>
> >> This approach of using httpd in front of 2+ Tomcats via AJP works
> >> well in my company. There is a bit of config necessary at httpd level
> >> so httpd is aware of all the Tomcats and also Tomcat config needs to
> >> be set to listen to AJP port instead of default port but it is not
> rocket science.
> >>
> >> This facilitates the deployment of nodes sequentially with no
> >> downtime. Of course, there is a shared session server to take care
> >> the sessions are not lost when Tomcats flip up and down.
> >>
> >> Reply in pvt if you need help setting up this.
> >>
> >> Thanks,
> >>
> >> Neill
> >>
> >> On Thu, Dec 3, 2015 at 12:08 AM, Jason Britton <jbritto...@gmail.com>
> >> wrote:
> >>
> >> > Thank you Christopher, reading now and we'll see if I can swing the
> >> > conference :)
> >> >
> >> > On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
> >> > ch...@christopherschultz.net> wrote:
> >> >
> >> > > Jason,
> >> > >
> >> > > On 12/2/15 4:07 PM, Jason Britton wrote:
> >> > > > I was looking for information for how those on the list achieve
> >> > > > zero downtime deployments of their tomcat hosted web
> >> > > > applications.  I
> >> > imagine
> >> > > > this can be achieved in a variety of ways, but would love to
> >> > > > hear
> >> what
> >> > > > works for you.  In our current environment we front multiple
> >> > > > tomcat instances with apache httpd, each tomcat instance
> >> > > > hosting one or more unique web apps.  In order to support this
> >> > > > effort we do have the
> >> > > resources
> >> > > > where we could spin up multiple tomcat instances to serve
> >> > > > requests
> >> for
> >> > a
> >> > > > single application.  I know there is mod_proxy_balancer
> >> > > > available for httpd, and I understand starting with tomcat 7
> >> > > > there is support for parallel deployment of versioned wars, and
> >> > > > tomcat also supports clustering.  I'm 

Re: Zero downtime deployments

[Christopher Schultz]

Kevin,

On 12/3/15 2:21 PM, Kevin Hale Boyes wrote:
> Thanks for this link to the presentation.
> How do you all deal with some of the other dependencies that the web
> application has?
> 
> For example, if v2 of my application needs new database columns or worse, a
> change to an existing column how can I have v1 and v2 running at the same
> time?  We use Oracle as our database though the problem exists for many
> database servers.

That's not a deployment problem. That's a design problem.

In short, you need to make sure that your web application is both
forward-compatible and backward-compatible at the same time.

That's rubbish, or course. Here's a more concrete suggestion, and it
requires a LOT of work, but it's worth it.

First of all, you can't think of a release as a single event any more.
Instead, there are 4 events.

First, there is the pre-release release. That's where you release a
small upgrade to your previous version that is forward-compatible with
the release you really want to perform. For instance, you make sure that
none of your database queries will bomb if you have additional fields in
your database tables. You also have to make sure that the release won't
bomb if the /new/ structure is *not* in place. That may require you to
do a few temporary hacks.

Second, you perform your database upgrade. No code release is involved,
here. Remember that if you want zero-downtime, you have to be careful
about modifying tables that contain data. Most databases can handle
online ALTER statements, but they can take a long time if you have a
lot of data. During that lng time, your users may be blocked
(waiting!) for access to the data in those tables. If you can't afford
for that to happen, you'll have to make other arrangements.

Third, you release your new version.

Fourth, you clean-up any hacks you may have left in from the Second
Step. This may require another small release, and it may also require
another small database modification (for instance, removing a now-unused
field from a table). If you are doing a small-enough release in the
Third Step, then this may not even be necessary at all.

Hope that helps,
-chris

> On 3 December 2015 at 01:31, Neill Lima  wrote:
> 
>> Hello Jason,
>>
>> This approach of using httpd in front of 2+ Tomcats via AJP works well in
>> my company. There is a bit of config necessary at httpd level so httpd is
>> aware of all the Tomcats and also Tomcat config needs to be set to listen
>> to AJP port instead of default port but it is not rocket science.
>>
>> This facilitates the deployment of nodes sequentially with no downtime. Of
>> course, there is a shared session server to take care the sessions are not
>> lost when Tomcats flip up and down.
>>
>> Reply in pvt if you need help setting up this.
>>
>> Thanks,
>>
>> Neill
>>
>> On Thu, Dec 3, 2015 at 12:08 AM, Jason Britton 
>> wrote:
>>
>>> Thank you Christopher, reading now and we'll see if I can swing the
>>> conference :)
>>>
>>> On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>>
 Jason,

 On 12/2/15 4:07 PM, Jason Britton wrote:
> I was looking for information for how those on the list achieve zero
> downtime deployments of their tomcat hosted web applications.  I
>>> imagine
> this can be achieved in a variety of ways, but would love to hear
>> what
> works for you.  In our current environment we front multiple tomcat
> instances with apache httpd, each tomcat instance hosting one or more
> unique web apps.  In order to support this effort we do have the
 resources
> where we could spin up multiple tomcat instances to serve requests
>> for
>>> a
> single application.  I know there is mod_proxy_balancer available for
> httpd, and I understand starting with tomcat 7 there is support for
> parallel deployment of versioned wars, and tomcat also supports
> clustering.  I'm just unsure of what approach I should start digging
>>> into
> and would very much appreciate any of your experiences.  The servers
 we'll
> be rolling out will be using the latest versions of tomcat 8 and
>> apache
> httpd 2.4.  Thanks for any insights!

 Check this out:


>>>
>> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf

 Start on slide/page 41.

 Then come to ApacheCon NA 2016 and discuss it!

 -chris

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org


>>>
>>
> 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Zero downtime deployments

[Neill Lima]

Hello Jason,

This approach of using httpd in front of 2+ Tomcats via AJP works well in
my company. There is a bit of config necessary at httpd level so httpd is
aware of all the Tomcats and also Tomcat config needs to be set to listen
to AJP port instead of default port but it is not rocket science.

This facilitates the deployment of nodes sequentially with no downtime. Of
course, there is a shared session server to take care the sessions are not
lost when Tomcats flip up and down.

Reply in pvt if you need help setting up this.

Thanks,

Neill

On Thu, Dec 3, 2015 at 12:08 AM, Jason Britton  wrote:

> Thank you Christopher, reading now and we'll see if I can swing the
> conference :)
>
> On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Jason,
> >
> > On 12/2/15 4:07 PM, Jason Britton wrote:
> > > I was looking for information for how those on the list achieve zero
> > > downtime deployments of their tomcat hosted web applications.  I
> imagine
> > > this can be achieved in a variety of ways, but would love to hear what
> > > works for you.  In our current environment we front multiple tomcat
> > > instances with apache httpd, each tomcat instance hosting one or more
> > > unique web apps.  In order to support this effort we do have the
> > resources
> > > where we could spin up multiple tomcat instances to serve requests for
> a
> > > single application.  I know there is mod_proxy_balancer available for
> > > httpd, and I understand starting with tomcat 7 there is support for
> > > parallel deployment of versioned wars, and tomcat also supports
> > > clustering.  I'm just unsure of what approach I should start digging
> into
> > > and would very much appreciate any of your experiences.  The servers
> > we'll
> > > be rolling out will be using the latest versions of tomcat 8 and apache
> > > httpd 2.4.  Thanks for any insights!
> >
> > Check this out:
> >
> >
> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf
> >
> > Start on slide/page 41.
> >
> > Then come to ApacheCon NA 2016 and discuss it!
> >
> > -chris
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>

Re: Zero downtime deployments

[Kevin Hale Boyes]

Thanks for this link to the presentation.
How do you all deal with some of the other dependencies that the web
application has?

For example, if v2 of my application needs new database columns or worse, a
change to an existing column how can I have v1 and v2 running at the same
time?  We use Oracle as our database though the problem exists for many
database servers.

Thanks,
Kevin.

On 3 December 2015 at 01:31, Neill Lima  wrote:

> Hello Jason,
>
> This approach of using httpd in front of 2+ Tomcats via AJP works well in
> my company. There is a bit of config necessary at httpd level so httpd is
> aware of all the Tomcats and also Tomcat config needs to be set to listen
> to AJP port instead of default port but it is not rocket science.
>
> This facilitates the deployment of nodes sequentially with no downtime. Of
> course, there is a shared session server to take care the sessions are not
> lost when Tomcats flip up and down.
>
> Reply in pvt if you need help setting up this.
>
> Thanks,
>
> Neill
>
> On Thu, Dec 3, 2015 at 12:08 AM, Jason Britton 
> wrote:
>
> > Thank you Christopher, reading now and we'll see if I can swing the
> > conference :)
> >
> > On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > > Jason,
> > >
> > > On 12/2/15 4:07 PM, Jason Britton wrote:
> > > > I was looking for information for how those on the list achieve zero
> > > > downtime deployments of their tomcat hosted web applications.  I
> > imagine
> > > > this can be achieved in a variety of ways, but would love to hear
> what
> > > > works for you.  In our current environment we front multiple tomcat
> > > > instances with apache httpd, each tomcat instance hosting one or more
> > > > unique web apps.  In order to support this effort we do have the
> > > resources
> > > > where we could spin up multiple tomcat instances to serve requests
> for
> > a
> > > > single application.  I know there is mod_proxy_balancer available for
> > > > httpd, and I understand starting with tomcat 7 there is support for
> > > > parallel deployment of versioned wars, and tomcat also supports
> > > > clustering.  I'm just unsure of what approach I should start digging
> > into
> > > > and would very much appreciate any of your experiences.  The servers
> > > we'll
> > > > be rolling out will be using the latest versions of tomcat 8 and
> apache
> > > > httpd 2.4.  Thanks for any insights!
> > >
> > > Check this out:
> > >
> > >
> >
> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf
> > >
> > > Start on slide/page 41.
> > >
> > > Then come to ApacheCon NA 2016 and discuss it!
> > >
> > > -chris
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
> > >
> >
>

Re: Zero downtime deployments

[jieryn]

Use http://flywaydb.org/ to perform database migrations. You will need
at least 3 versions in order to perform an incompatible database
change. v1 is existing behavior, v2 is a shim that bridges v1 and v3,
and then v3 cleans up the shim and removes all the unnecessary hacks.
When you have v1 rolled out against N images, then you can start
rolling out compat shim v2 which adds some temporary work for the db
and coexists, then when all instances are at v2 you can move to v3
which uses all the new stuff.

I haven't found anything better or more efficient than the 2-step
deployment with live code.

On Thu, Dec 3, 2015 at 2:21 PM, Kevin Hale Boyes  wrote:
> Thanks for this link to the presentation.
> How do you all deal with some of the other dependencies that the web
> application has?
>
> For example, if v2 of my application needs new database columns or worse, a
> change to an existing column how can I have v1 and v2 running at the same
> time?  We use Oracle as our database though the problem exists for many
> database servers.
>
> Thanks,
> Kevin.
>
> On 3 December 2015 at 01:31, Neill Lima  wrote:
>
>> Hello Jason,
>>
>> This approach of using httpd in front of 2+ Tomcats via AJP works well in
>> my company. There is a bit of config necessary at httpd level so httpd is
>> aware of all the Tomcats and also Tomcat config needs to be set to listen
>> to AJP port instead of default port but it is not rocket science.
>>
>> This facilitates the deployment of nodes sequentially with no downtime. Of
>> course, there is a shared session server to take care the sessions are not
>> lost when Tomcats flip up and down.
>>
>> Reply in pvt if you need help setting up this.
>>
>> Thanks,
>>
>> Neill
>>
>> On Thu, Dec 3, 2015 at 12:08 AM, Jason Britton 
>> wrote:
>>
>> > Thank you Christopher, reading now and we'll see if I can swing the
>> > conference :)
>> >
>> > On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
>> > ch...@christopherschultz.net> wrote:
>> >
>> > > Jason,
>> > >
>> > > On 12/2/15 4:07 PM, Jason Britton wrote:
>> > > > I was looking for information for how those on the list achieve zero
>> > > > downtime deployments of their tomcat hosted web applications.  I
>> > imagine
>> > > > this can be achieved in a variety of ways, but would love to hear
>> what
>> > > > works for you.  In our current environment we front multiple tomcat
>> > > > instances with apache httpd, each tomcat instance hosting one or more
>> > > > unique web apps.  In order to support this effort we do have the
>> > > resources
>> > > > where we could spin up multiple tomcat instances to serve requests
>> for
>> > a
>> > > > single application.  I know there is mod_proxy_balancer available for
>> > > > httpd, and I understand starting with tomcat 7 there is support for
>> > > > parallel deployment of versioned wars, and tomcat also supports
>> > > > clustering.  I'm just unsure of what approach I should start digging
>> > into
>> > > > and would very much appreciate any of your experiences.  The servers
>> > > we'll
>> > > > be rolling out will be using the latest versions of tomcat 8 and
>> apache
>> > > > httpd 2.4.  Thanks for any insights!
>> > >
>> > > Check this out:
>> > >
>> > >
>> >
>> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf
>> > >
>> > > Start on slide/page 41.
>> > >
>> > > Then come to ApacheCon NA 2016 and discuss it!
>> > >
>> > > -chris
>> > >
>> > > -
>> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> > > For additional commands, e-mail: users-h...@tomcat.apache.org
>> > >
>> > >
>> >
>>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Zero downtime deployments

[Tauzell, Dave]

If you cannot use a tool like flyway you can do it by hand, too.  The key is 
that all database changes need to be backwards compatible.  For example:

So, if you want to drop a column:
1. Deploy new version of app that doesn't require column X
2. Shutdown version which does
3. drop column X

If you want to rename a column:
1. Deploy new version of app that writes data to old column name + new name
2. shutdown old version
3. drop old column

We generally test out our old version of an app running against the new 
database schema to make sure this process will work.

-Dave

-Original Message-
From: jieryn [mailto:jie...@gmail.com]
Sent: Thursday, December 03, 2015 1:32 PM
To: Tomcat Users List
Subject: Re: Zero downtime deployments

Use http://flywaydb.org/ to perform database migrations. You will need at least 
3 versions in order to perform an incompatible database change. v1 is existing 
behavior, v2 is a shim that bridges v1 and v3, and then v3 cleans up the shim 
and removes all the unnecessary hacks.
When you have v1 rolled out against N images, then you can start rolling out 
compat shim v2 which adds some temporary work for the db and coexists, then 
when all instances are at v2 you can move to v3 which uses all the new stuff.

I haven't found anything better or more efficient than the 2-step deployment 
with live code.

On Thu, Dec 3, 2015 at 2:21 PM, Kevin Hale Boyes <kcbo...@gmail.com> wrote:
> Thanks for this link to the presentation.
> How do you all deal with some of the other dependencies that the web
> application has?
>
> For example, if v2 of my application needs new database columns or
> worse, a change to an existing column how can I have v1 and v2 running
> at the same time?  We use Oracle as our database though the problem
> exists for many database servers.
>
> Thanks,
> Kevin.
>
> On 3 December 2015 at 01:31, Neill Lima <neill.l...@visual-meta.com> wrote:
>
>> Hello Jason,
>>
>> This approach of using httpd in front of 2+ Tomcats via AJP works
>> well in my company. There is a bit of config necessary at httpd level
>> so httpd is aware of all the Tomcats and also Tomcat config needs to
>> be set to listen to AJP port instead of default port but it is not rocket 
>> science.
>>
>> This facilitates the deployment of nodes sequentially with no
>> downtime. Of course, there is a shared session server to take care
>> the sessions are not lost when Tomcats flip up and down.
>>
>> Reply in pvt if you need help setting up this.
>>
>> Thanks,
>>
>> Neill
>>
>> On Thu, Dec 3, 2015 at 12:08 AM, Jason Britton <jbritto...@gmail.com>
>> wrote:
>>
>> > Thank you Christopher, reading now and we'll see if I can swing the
>> > conference :)
>> >
>> > On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
>> > ch...@christopherschultz.net> wrote:
>> >
>> > > Jason,
>> > >
>> > > On 12/2/15 4:07 PM, Jason Britton wrote:
>> > > > I was looking for information for how those on the list achieve
>> > > > zero downtime deployments of their tomcat hosted web
>> > > > applications.  I
>> > imagine
>> > > > this can be achieved in a variety of ways, but would love to
>> > > > hear
>> what
>> > > > works for you.  In our current environment we front multiple
>> > > > tomcat instances with apache httpd, each tomcat instance
>> > > > hosting one or more unique web apps.  In order to support this
>> > > > effort we do have the
>> > > resources
>> > > > where we could spin up multiple tomcat instances to serve
>> > > > requests
>> for
>> > a
>> > > > single application.  I know there is mod_proxy_balancer
>> > > > available for httpd, and I understand starting with tomcat 7
>> > > > there is support for parallel deployment of versioned wars, and
>> > > > tomcat also supports clustering.  I'm just unsure of what
>> > > > approach I should start digging
>> > into
>> > > > and would very much appreciate any of your experiences.  The
>> > > > servers
>> > > we'll
>> > > > be rolling out will be using the latest versions of tomcat 8
>> > > > and
>> apache
>> > > > httpd 2.4.  Thanks for any insights!
>> > >
>> > > Check this out:
>> > >
>> > >
>> >
>> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancin
>> g%20Tomcat%20with%20mod_jk.pdf
>> > >
>> > &g

Re: Zero downtime deployments

[Christopher Schultz]

Jason,

On 12/2/15 4:07 PM, Jason Britton wrote:
> I was looking for information for how those on the list achieve zero
> downtime deployments of their tomcat hosted web applications.  I imagine
> this can be achieved in a variety of ways, but would love to hear what
> works for you.  In our current environment we front multiple tomcat
> instances with apache httpd, each tomcat instance hosting one or more
> unique web apps.  In order to support this effort we do have the resources
> where we could spin up multiple tomcat instances to serve requests for a
> single application.  I know there is mod_proxy_balancer available for
> httpd, and I understand starting with tomcat 7 there is support for
> parallel deployment of versioned wars, and tomcat also supports
> clustering.  I'm just unsure of what approach I should start digging into
> and would very much appreciate any of your experiences.  The servers we'll
> be rolling out will be using the latest versions of tomcat 8 and apache
> httpd 2.4.  Thanks for any insights!

Check this out:
http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf

Start on slide/page 41.

Then come to ApacheCon NA 2016 and discuss it!

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Zero downtime deployments

[Jason Britton]

Thank you Christopher, reading now and we'll see if I can swing the
conference :)

On Wed, Dec 2, 2015 at 4:00 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Jason,
>
> On 12/2/15 4:07 PM, Jason Britton wrote:
> > I was looking for information for how those on the list achieve zero
> > downtime deployments of their tomcat hosted web applications.  I imagine
> > this can be achieved in a variety of ways, but would love to hear what
> > works for you.  In our current environment we front multiple tomcat
> > instances with apache httpd, each tomcat instance hosting one or more
> > unique web apps.  In order to support this effort we do have the
> resources
> > where we could spin up multiple tomcat instances to serve requests for a
> > single application.  I know there is mod_proxy_balancer available for
> > httpd, and I understand starting with tomcat 7 there is support for
> > parallel deployment of versioned wars, and tomcat also supports
> > clustering.  I'm just unsure of what approach I should start digging into
> > and would very much appreciate any of your experiences.  The servers
> we'll
> > be rolling out will be using the latest versions of tomcat 8 and apache
> > httpd 2.4.  Thanks for any insights!
>
> Check this out:
>
> http://people.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20Tomcat%20with%20mod_jk.pdf
>
> Start on slide/page 41.
>
> Then come to ApacheCon NA 2016 and discuss it!
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>