Thanks Chris
[snip, snip, snippety-snip]
CS> What's the database? And the driver?
Oracle 19, oracle.jdbc.OracleDriver - jdbc:oracle:thin.
CS> MySQL Connector/J used to (still does?) read 100% of the results
CS> into the heap before Statement.executeQuery() returns unless you
CS> specifically tell it not to. So if your query returns 1M rows, you
CS> might bust your heap.
CS> It's entirely possible that other drivers do similar things.
The JSON has all the rows, so it appears no pagination is being used on the DB
level.
cb> Multiple TC instances
cb> (3) because multiple copies of the apps don't play nice with each
cb> other. That is, we can't just rename the WAR files and expect the
cb> deployed apps to stay inside that context name (I think).
CS> You might want to look into that, eventually. If they aren't playing
CS> together nicely, they are not "good" servlet citizens. Solving those
CS> issues may improve other things. *shrug*
Yeah, I was working on that previously, but attention spans are short, and I
got pulled off that task onto - SQUIRREL!
cb> StringBuilder - 264MB for the supporting byte array and 264MB for the
cb> returned String, about 790MB total for that piece of the pie.
cb> Contents were simply the JSON query results returned to the client.
cb> No mystery there.
Also, I noticed that the SB internal memory usage is about 2x the size of the
actual contents. Is that because each char is stored as 2 bytes for Unicode?
(Not the char array to string conversion, which is different.)
CS> Yep: runaway string concatenation. This is a devolution of the
CS> "Connector/J reads the whole result set into memory before
CS> returning" thing I mentioned above. Most JSON endpoints
CS> return arbitrarily large JSON responses and most client
CS> applications just go "duh, read the JSON, then process it".
CS> If your JSON is big, well, then you need a lot of memory to
CS> store it all if that' who you do things.
Looking at the contents of the JSON, it's not normalized - a lot of redundant
metadata. Hand-editing the JSON for analysis reduced it from 135 MB to 26 MB.
Maybe the code that generates it can be improved.
CS> If you want to deal with JSON at scale, you need to process
CS> it in a streaming fashion. The only library I know that can do
CS> streaming JSON is Noggit, which was developed for use with
CS> Solr (I think, maybe it came from elsewhere before that).
CS> Anyway, it's ... not for the faint of heart. But if you can figure
CS> out out, you can handle petabytes of JSON with a tiny heap.
I don't think we need to serve up that much data, but I'm guessing we can do
better with what we do serve. Interesting nonetheless.
CS> You might want to throttle/serialize queries you expect to
CS> have big responses so that only e.g. 2 of them can be running
CS> at a time. Maybe all is well when they come one-at-a-time,
CS> but if you try to handle 5 concurrent "big responses" you bust
CS> your heap.
Hmm... I had not thought of throttling that way, restricting the number of
concurrent queries. I was thinking about restricting the number of records
returned. Not sure how to handle lots of users connected but only a few able
to query concurrently. Different DB connection pool with fewer connections for
queries?
cb> (At least StringBuilder is being
cb> used instead of plus-sign String concatenation.)
CS> In Java "..." + "..." uses a StringBuilder
I did not know that. Or I forgot, in which case I can't tell the diff. :-P
CS> In some code, "..." + "..." is just fine
Often it's run-on sentences of plus-sign concatenation with nested quotes,
almost unreadable and even worse for editing. I like to replace with SB for
readability and maintainability.
CS> I hate it when someone replaces it with:
CS> String foo = new StringBuilder("bar").append("baz").toString();
CS> because the compiler does the _exact same thing_ and you've
CS> just made the code more difficult to read.
Ahhh, the classic train wreck. :-)
CS> in a *loop*, then replacing it with a StringBuilder is pretty
CS> important for performance, otherwise the compiler will
CS> do something stupid
I believe the technical term for that is "stoopid". :-) Yeah, I like to be
strategic about SB's and loops.
CS> You might actually have to start reading some code (shiver!).
"You're ... mocking me." :-) Actually, I might be able to pass it off onto
the guy who wrote the library. *phew*
--
Cris Berneburg
CACI Senior Software Engineer
This electronic message contains information from CACI International Inc or
subsidiary companies, which may be company sensitive, proprietary, privileged
or otherwise protected from disclosure. The information is intended to be used
solely by the recipient(s) named above. If you are not an intended recipient,
be aware that any review, disclosure, copying, distribution or use of this
transmission or its contents is prohibited. If you have received
