One thing I just tested was to undeploy the ROOT context, which is how we run anyways, and this causes request.getContext() to return null, which with the code, as is, results in a null pointer and a 500 being thrown--which inadvertently would cause mod_jk to retry on another node. I don't like letting code knowingly throw null pointers, so I was thinking of just checking if the context is null and throwing a 503. The only problem is that the valve would only work when the ROOT context wasn't deployed, so your two other suggestions would be the only options.
Mark, I've been considering opening an official enhancement request to the clustering implementation in Tomcat that would state the following... Currently, when an application within a clustered environment is unavailable or stopped, Tomcat returns an HTTP 404 (Not Found) status code. While this behavior is generally acceptable in a non-clustered environment, it can lead to less than optimal routing decisions by load balancers within a clustered setup. Most load balancers, including mod_jk, do not interpret a 404 status code as an indication of application unavailability warranting a failover. Moreover, reconfiguring load balancers to treat 404 codes as triggers for failover could potentially expose systems to DOS attacks, as malicious users could generate unnecessary failovers by requesting non-existent resources. While there are workarounds to this issue, such as creating a custom valve to check the application status and modifying the 404 to a 503, or using root context and servlet mappings to return a 503, these solutions require custom implementations by the end user. This adds complexity and is not an ideal solution. In light of this, I propose that Tomcat should return an HTTP 503 (Service Unavailable) status code when an application is not available in a clustered environment. The 503 code, which signifies temporary unavailability of the application, would align more accurately with the circumstances and could enable load balancers to make more informed and effective routing decisions. Thoughts? -- Thanks, Dan -- Thanks, Dan McLaughlin Robert Clay Vineyards Proprietor/Vigneron d...@robertclayvineyards.com mobile: 512.633.8086 main: 325.261.0075 https://robertclayvineyards.com ________________________________ Facebook | Instagram On Tue, Jun 20, 2023 at 10:28 AM Mark Thomas <ma...@apache.org> wrote: > > On 20/06/2023 15:41, Dan McLaughlin wrote: > > So I tried to create a Valve to check to see if the application is stopped > > and convert the 404 response to a 503, but I haven't had any luck getting > > it to work. Is there another internal API that I should be using? > > context.getState().isAvailable > > ways seems to report the app is available even though it's stopped. > > The code is looking at the wrong Context. Since the web application has > been stopped the request won't be mapped to it. I'm guessing the request > has been mapped to the root context which is available. > > You'll need to do something like: > > Container[] containers = request.getHost().findChildren(); > for (Container container : containers) { > if (container.getState().isAvailable()) { > continue; > } > Context context = (Context) container; > if (request.getDecodedRequestURI().equals(context.getPath()) || > request.getDecodedRequestURI().startsWith( > context.getPath() + '/')) { > response.sendError(HttpServletResponse.SC_SERVICE_UNAVAILABLE); > } > } > > I haven't optimised this at all. It isn't particularly efficient. It is > just to give you an idea. > > Actually. I have just had a much better idea. It works by taking > advantage of the Servlet specification mapping rules which require the > longest context path match. > > Lets assume you have /app1 /app2 and /app3 > > In your ROOT web application create a maintenance Servlet that just > returns a 503 and map it to "/app1/*" "/app2/*" and /app3/*". > > If app1 is running, the longest context path match rule means it will be > mapped to /app1 and the application will handle it. If the web > application is stopped, the request will be mapped to ROOT where it will > match the maintenance Servlet and return a 503. > > The only thing that this won't work for is if you want to take the RROT > web application out of service. > > Mark > > > > import org.apache.catalina.*; > > import org.apache.catalina.connector.Request; > > import org.apache.catalina.connector.Response; > > import org.apache.catalina.valves.ValveBase; > > > > import jakarta.servlet.ServletException; > > import java.io.IOException; > > import java.util.logging.Logger; > > import java.util.logging.Level; > > > > public class DownForMaintenanceValve extends ValveBase { > > > > // Create a Logger > > private static final Logger log = Logger.getLogger(DownForMaintenanceValve. > > class.getName()); > > > > public DownForMaintenanceValve() { > > log.info("DownForMaintenanceValve started"); > > } > > > > @Override > > public void invoke(Request request, Response response) throws > > IOException, ServletException > > { > > Context context = request.getContext(); > > if (!context.getState().isAvailable()) { > > log.info("Application is not available, sending 503"); > > response.sendError(503); > > } else { > > log.fine("Application is available, passing to next valve"); > > getNext().invoke(request, response); > > } > > } > > } > > > > > > -- > > > > Thanks, > > Dan > > > > On Wed, Jun 14, 2023 at 2:32 PM Mark Thomas <ma...@apache.org> wrote: > > > >> On 14/06/2023 19:49, Dan McLaughlin wrote: > >>> Hello, > >>> > >>> This is probably a question that would be better suited for the dev list, > >>> but I thought I'd start here first. > >> > >> That depends. It is generally better to start on the users list. > >> > >>> Does anyone understand the reasoning behind why Tomcat, when clustered, > >>> throws an HTTP status 404 and not a 503 when you have an application > >>> deployed but stopped or paused? > >> > >> The issue you describe only affects stopped applications. If an > >> application is paused then any requests to that application should be > >> held until the application is unpaused (or the client timeouts out). > >> > >> The current Tomcat Mapper dates back to at least Tomcat 4. It might be > >> earlier but I don't know the Tomcat 3 code well enough to find the > >> Tomcat 3 mapping code in the web interface and I'm not curious enough to > >> check the code out so I can use grep. > >> > >> The clustering implementation dates back to Tomcat 5. > >> > >> You'll need to dig through the archives to see if this topic was ever > >> raised and, if it was, the result of that discussion. Probably around > >> the time clustering was added. > >> > >>> I think I understand that my only option is to > >>> failover for 404s considering the current implementation. > >> > >> That might cause problems. If the node returning 404 is marked as down > >> you'll have a DoS vulnerability that is trivial to exploit. > >> > >>> I've looked to > >>> see if there was a configuration setting related to clustering that would > >>> allow me to change the behavior, and I couldn't find one; the only > >> solution > >>> seems to be to write a custom listener that detects that an application > >> is > >>> deployed but stopped or paused, and then throw a 503 instead. > >> > >> That would be a better short-term solution and fairly simple to write. > >> I'd probably do it as a Valve as you'll get access to Tomcat's internals > >> that way. > >> > >> The clustering implementation generally assumes that all applications > >> are available on all nodes. If that isn't the case I wouldn't be > >> surprised to see log messages indicating issues with replication. > >> > >> What is the use case for stopping one (or more) web applications on a node? > >> > >> Mark > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- *NOTICE:* This e-mail message and all attachments transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is strictly prohibited. The contents of this e-mail are confidential and may be subject to work product privileges. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
