Rob,
On 12/9/20 23:58, Rob Sargent wrote:
My apologies if this is too vague to warrant consideration.
It is vague, but we can always ask questions :)
In the recent past I managed a naked port, a Selector, a
ThreadPoolExecutor and friends (and it worked well enough...) but a dear
and knowledgeable friend suggested embedding tomcat and using http.[3]
Managing that yourself can be a pain. The downside to using Tomcat is
that you have to use HTTP. But maybe that can work in your favor in
certain cases, especially if you have other options (e.g. upgrade from
HTTP to Websocket after connection.)
> I have that working, one request at a time.
Great.
Now to get to "production level": I have two distinct types of
requests which might be problematic: one is rather large[1] and the
other rather prolific[2]. Both types are handled by the same servlet.
The data structure is identical, just much less in case[2].
With respect to your footnotes, you can pretty much ignore anything that
requires more than 1 host, so lets talk about the individual requests a
single instance of the server would expect. To confirm, you have some
requests that are huge and others that are ... not huge? The contents
don't really matter, honestly.
Tomcat can handle both huge and non-huge requests without a problem.
When you implement your servlet, you simply get an InputStream and do
whatever you want. Same thing with responses to the client: get an
OutputStream and write away.
Is it advisable, practical to (re)establish a ThreadPoolExecutor, queue
etc as a tomcat accessible "Resource" with JDNI lookup, and have my
servlet pass the work off to the Executor's queue?
I don't really understand this at all. Are you asking how to mitigate a
self-inflected DOS because you have so many incoming connections?
If you have enough hardware to satisfy the requests, and your usage
pattern is as you suggest, then you will mostly have one or two huge
requests in-flight at any time, and some large number of smaller
(faster?) requests also in-flight at the same time.
Again, no problem. You are constrained only by the resources you have
available:
1. Memory
2. Maximum connections
3. Maximum threads (in your executor / request-processor thread pool)
If you have data that doesn't fix into byte[MAXINT] then maybe you don't
want to try to handle it all at once. That's an application design
decision and if gzipping helps you in the short term, then great. My
recommendation would be to look at ways of handling that request in a
streaming-fashion instead of buffering everything up in memory. The
overall performance of your application will likely improve because of
that change.
[2] Given infinite EC2 capacity there would be tens of thousands of
jobs started at once. Realistic AWS capacity constraints limit this
to hundreds of instances from a queue of thousands. The duration of
any instance varies from hours to days. But the payload is simple,
under 5K bytes.
If you are using AWS, then you can load-balance between any number of
back-end Tomcat instances. The lb just has to decide which back-end
instance to use. Sometimes lbs make bad decisions and you get all the
load on a single node. That's bad because (a) one node is overloaded and
(b) the other nodes are under-utilized. It's up to you to figure out how
to get your load distributed in an equitable way.
Back to the problem you are actually trying to solve.
Are these "small requests" in any way a problem? Do they arrive
frequently and are they handled quickly? If so, then you can probably
mostly just ignore them. It's the huge requests that are (likely) the
problem.
If you want to hand-off control of a request to another thread for
processing, there are a few ways to do that I can think of:
1. new Thread(new Runnable() { /* your stuff */ }).start();
This is bad for a few of reasons:
1a. Unlimited threads created by remote clients? Bad.
1b. Call to Thread.start() returns immediately and the servlet's
execution ends. There is no way to reply to the client's, and if you
haven't read all their input, Bad Things will happen in Tomcat. (Like,
your request and response objects will be re-used and you'll observe
mass-chaos).
2. sharedExecutor.submit(new Runnable() { /* your stuff */ });
This is bad for the same reason as 1b above, but it does not suffer from
1a. 1a is now replaced by:
2a. Unlimited jobs submitted by remote clients? Bad.
3. Use servlet async processing.
I think this is probably ideal for your use-case. When you go into
asynchronous mode, the request-processing thread is allowed to go back
and service other requests, so you get a more responsive server, at
least from your clients' perspectives.
The bad news is that asynchronous mode requires that you completely
change the way you think about communicating with a client. Instead of
reading the input until you are satisfied, performing your
business-logic, then writing to the client until you are done, you have
to subscribe to I/O events and handle them all appropriately. If you get
it wrong, you can make a mess of things.
It would help to understand the nature of what has to happen with the
data once it's received by your servlet. Does the processing take a long
time, or is it mostly the data-upload that takes forever? Is there a way
for the client to cut-up the work in any way? Are there other ways for
the client to get a response? Making an HTTP connection and then waiting
18 hours for a response is ... not how HTTP is supposed to work.
Long ago I worked on a product where we had to perform some long
operation on the server and client browsers would time-out. (This was a
web browser, so they have timeouts of like 60 seconds or so. They aren't
custom clients where you say just say "wait 18 hours for a response.").
Our solution was to accept the job via HTTP request and respond saying
"okay, not done yet". The expectation was that the client would them
poll to see if the job was done. We even provided a response saying "not
done yet, 50% done" or something like that.
On the server-side, we implemented it like this (roughly):
servlet {
get {
if(isNewJob(request)) {
Job job = new Job();
job.populate(request.getInputStream());
if(job.isOkay()) {
request.getSession().setAttribute("job", job);
sharedThreadPool.submit(job);
} else {
return "500 Internal Server Error";
}
} else if(isJobCheck(request)) {
Job job = (Job)request.getSession().getAttribute("job");
if(job.isDone()) {
return "200 Job all done";
} else {
return "200 Job running, " + job.getPercentDone() + " done.";
}
}
}
}
And then the job would just do this:
Job {
private volatile boolean done = false;
public void run() {
/* do long-running stuff */
setDone(true);
}
public boolean getDone() { return done; }
protected void setDone(boolean done) { this.done = done; }
public int getPercentDone() { return whatever; }
}
The above is super psuedo-code and ignores a whole bunch of details
(e.g. multiple jobs per client, error handling, etc.). The client is
required to maintain session-state using either cookies or jsessionid
path parameters. (At least, that's how we implemented it.) Instead of
the session, you could store your jobs someplace accessible to all
requests such as the application (ServletContext) and give them all
unique (and difficult to guess) ids.
This allows the server to make progress even if the client times out, or
loses a network connection, or power, or whatever. It also allows the
server to use that network connection used to submit the initial job for
accepting other connections. And you don't have to use servlet async and
re-write your whole process.
I don't know if any of this helps, but I thik it will get you thinking
in a more HTTP-style way rather than a connection-oriented service like
the one you had originally implemented.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
