On 12/12/2025 01:11, Linus Kamb wrote:
Mark and Chris,
Thank you again for your replies. Responses inline below.
TLDR; I have answers and comments below, but I feel this has taken up a
fair amount of your time.
If you wish to dig further into this, I'd certainly like to understand what
(at least may have) happened.
If not, I don't think this is very likely to have been a security incident,
and I suppose I'll just chalk it up to
being related to the disk being full.
That seems the most likely cause at this point.
<snip/>
Yes. All tomcats are running on one physical machine.
The tomcat instances are all separate, complete installs with their own
webapps directories, and each distinct webapp has its own data directory on
a separate local (logical - /dev/mapper ) disk.
OK. That rules out quite a few possibilities.
Are any files shared between the instances?
There should not be.
As does that.
How is the web application deployed? WAR file that Tomcat then expands?
Yes.
autoDeploy had been turned on. We are turning that off.
Good decision.
I want to make sure I understand the architecture before I spend too
much time thinking about what might be going on.
Understandably. And I don't want to take too much of your time.
It isn't that I don't want to get to the bottom of this. More that there
are lots possibilities and it is better to narrow down the options a bit
before speculating.
Could the fact that there was no space left on the disk be treated as
being unavailable?
Seems unlikely. Tomcat has a list of files it checks for each webapp and
looks to see if the modification time has changed.
Is there a list of those files?
I think you want this:
https://tomcat.apache.org/tomcat-11.0-doc/config/automatic-deployment.html
I think you have the WAR and the DIR present and both are in
$CATALINA_BASE/webapps (i.e. not external) with no XMl file.
From that, the things that could trigger redeployment are:
- deleting the DIR
- modified WAR file
Which really narrows down what we need to look at.
And why only the one webapp? Perhaps because it was the one
Full disclosure, in case it makes a difference:
I missed one of the tomcats -- it did not have the one particular webapp.
In that tomcat, *all* of its webapps were redeployed.
However, there was a tomcat with the main webapp that had one additional
webapp. That additional webapp was *not* redeployed.
>
FWIW.
Hmm.
Web applications in webapps are checked for modification one at a time.
It does look like something (and the disk filling up is a strong
candidate) either made Tomcat think the DIRs had been deleted or that
the WARs had a different (including invalid) last modified time. And
that triggered the redeploy.
Interesting that, despite the disk being full, the redeploy succeeded.
attempting to
write to the disk?
Again, seems unlikely.
Might some external process changed the last modification time on any
files? A back-up process maybe? Those times should not be sensitive to
clock / time zone changes but I'd check if there were any changes of
that nature just in case.
I am not aware of any external process that would touch files in the webapp
directories.
Looking at the webapp directories now, nothing has changed since
redeployment.
All the webapps were undeployed (as noted in the logs) and then redeployed,
presumably unpacking the war. The webapp deployment directory was restored
to its
state as in the war. The timestamps on all the files in the deployment
directory are
equal to their timestamps in the war, but all the directories had
timestamps
~equal to the time of redeployment.
That is as expected for redeployment.
Unfortunately, since the webapps were all undeployed, I cannot evaluate the
contents and
file timestamps prior to undeployment.
No problem. The timestamps of the files in the DIR aren't the
interesting ones since a change doesn't trigger a redeploy.
The interesting ones are those on the WAR files. Did those change? Thta
would trigger a redeploy.
(I am assuming that when the logs say "Undeploying context," that means it
removes the
webapp directory. Either way, everything was restored as if unpacked from
the war.)
Yes. Delete the DIR. Unpack the WAR. Set the timestamps of the unpacked
files to match those of the files in the WAR.
Is the web application configured to auto-reload? Tomcat will watch more
than just the timestamp on the application's deployment directory for
changes. There are other files that can cause a reload. (But generally
not a re-deployment.)
I don't believe the application or server were configured to auto-reload.
I find no mention of reloadable.
See the autodeployment page I referenced above for what changes can
trigger a reload. Also look at $CATALINA_BASE/conf/context.xml for other
resources that might be checked.
The logs (that were able to be written) stated "Undeploying <webapp>
context." (Or maybe context <webapp>)
And followed later by indicating the redeployent of the webapp.
Yep. That is what I'd expect to see for a deletion of the DIR followed
by a re-expansion of the WAR.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
