Ortwin Glück wrote:
Gordon Mohr wrote:
If I understand the HttpCore code properly, there is no direct
facility for protecting against the OOME in the code -- just a chance
to hook in a theoretical alternate implementation that would address
the problem.
Is that correct?
Yes.
To use the HttpCore-4.0 facility, it appears I would create my own
HttpDataReceiver implementation which keeps a count of the bytes it
shovels & throws an IO or HTTP exception when some count is exceeded;
create a factory that makes such receivers; install that factory into
each HttpClientConnection instance before it begins receiving data.
Correct.
OK, thanks.
This could work, but seems a roundabout and obscure approach. The
really valuable feature would be for OOME-resistance -- and friendly,
usable indicators that extreme content has been encountered -- to be
features of the library. It's require a switch or paramter to enable,
rather than patching in custom/third-party code.
An OOM Exception is exactly that: a friendly usable indicator that an
extreme condition has been encountered. I am personally against another
mechanism that tries to somehow monitor heap memory.
I'd have to strongly disagree that OOMEs are friendly and usable. Among
the reasons I consider it unfriendly:
(1) OOMEs can occur anywhere, making it hard to catch and recover from
(2) pre-JDK 1.6, there's no stack trace info
(3) OOMEs only occasionally occur near the 'culpable' code
(3) is most troublesome for high-volume multi-threaded apps. If thread A
is behaving nicely, using 1MB and freeing it, endlessly -- but then
thread B comes along and grabs 1000MB, it's very possible that
well-behaved A, not gluttonous B, receives the OOME. Given that A can be
a crucial task whose unrecoverable error is fatal to your application,
this is a serious and "unfriendly" problem.
(It's bitten my project constantly -- from HttpClient most recently, but
several other open-source libraries as well.)
Thus it's nice to have libraries that at least have a mode where they
use a practically bounded amount of memory. Some libraries do this well;
I would like HttpClient(/HttpCore) to do whis well too, without
requiring custom alternate implementations to be hooked in. It would
make it a more robust library for many purposes.
I agree that by sending large header information a malicious server
could DOS a spider based on HttpCore. So it makes sense for this use
case to include a protection parameter that sets an upper limit to the
information in headers. A default of 100 KB should be enough for the
real world. This is the easy solution.
I agree. It wouldn't even need to be a default -- the default could be
"take your chances", because the HTTP spec has no limits. But it'd be
nice if setting such a limit were a built-in option.
My application currently requires a fix for this in the 3.x line, so
I'll either use Christian's approach or work up my own. It'd be great to
see this capability in 3.1, especially if it's HttpClient-3.1 is still
pre-alpha and release ~4 months away.
If it can be opened as an RFE against 4.0, just to track that it's
pending and a priority, I'd also be happy to contribute an
implementation when my project gets around to using the 4.0 line (where
we'll also need this).
Another possibility is to use a header stream like we do for the message
body. This would mean a refactoring and probably cause painful client
code. We would still have to protect ourselves against long header lines
which is again not easy.
Thought about this too, but it'd be quite a contortion for a need --
truly handling arbitrarily long headers -- that's impractical if not
completely mythical.
This all boils down to a missing feature: byte counting in the connection.
Agreed, that'd be one simple way to do it.
Is there a summary of expected dates of Core-4.0/Client-4.0 release
somewhere, or any assessments of how the 4.0 codebases match up
against 3.0 features? (Is it reasonable for an HttpClient-3.0-using
project to consider transitioning to the 4.0 codebase(s)?)
This is the only plan there is:
http://wiki.apache.org/jakarta-httpclient/HttpComponentsProjectRoadmap
Great, thanks.
- Gordon @ IA
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
