Hello Simon,
I am also just a user here, but I certainly agree with you that the sample Code
should be able to communicate with (well behaved) public web Servers.
So if no TLS or htttp2 is involved, I dont think this is the Problem here.
The following is a test Server I used (baed on yours), it does produce a
correct response (i.e. no exception) if the Parameter is fail=false and a
aborted exception if the Parameter fail=true (because it writes one Byte less
than the Content-length would announce and therefore the Client reads past the
payload)
In the fail case it does indeed not return a Body result (but that is IMHO ok
the sample Code must not deal with web Servers which are defect)
For the httpclient code, the following improvements are IMHO possible:
a) As you mentioned the EOF should contain the callsite and not be transorted
from a worker thread context. This can either be done by rethrwing the EOF
Exception or by actually constructing them in the read(). The same (or arguably
even worse) Problem is a „connection refused“ type of exception which also has
no clear callsite.
b) Instead of throwing an EOF exception when the read Buffer exceeds the Body
lenth I would return a short read till the last available Byte and only throw
at the next read. This way the handler can get all of the partial Body. This is
however not a good optimization for BodyHander.asString().
It is still unlear why the node.js expresss Server has Problems. For that I
think its a good idea to trace it either with Wireshark/tcpdump or Maybe one of
the web app Debugging reverse proxies.
Gruss
Bernd
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.net.ServerSocket;
import java.net.Socket;
public class FakeSever
{
static final boolean fail = true;
static final String body = "<html><body><h1>Heading</h1><p>Some
Text</p></body></html>";
public static final String[] response = {
"HTTP/1.1 200 OK",
"X-Powered-By: Expressd",
"Content-Type: text/html; charset=utf-8",
"Content-Length: " + (body.length()+(fail?3:2)),
"ETag: W/\"3c-CQHFqoSATSxoI5iZHLfu5OeEG3k\"",
"Date: Tue, 15 May 2018 20:34:49 GMT",
"Connection: keep-alive",
"",
body,
};
public static void main(String[] args) throws Throwable {
try (ServerSocket ss = new ServerSocket(8080);) {
System.out.println("Listening...");
try (Socket s = ss.accept();) {
System.out.println("Accepted " + s);
BufferedReader in = new BufferedReader(new
InputStreamReader(s.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
System.out.println("< " + line);
if ("".equals(line)) break;
}
Writer out = new OutputStreamWriter(s.getOutputStream());
for (String st : response) {
out.write(st + "\r\n");
System.out.println("> " + st);
}
out.flush();
}
}
}
}
--
http://bernd.eckenfels.net
Von: Simon Roberts
Gesendet: Mittwoch, 16. Mai 2018 00:02
An: [email protected]
Betreff: Re: EOF excption in HTTP 1.1 server interaction
No, HTTP 2.0 is not supported by nodejs (... current LTS release,
eight-dot-something) and by my fake server. I believe that there might be
something about the way that node is doing that ignoring that's causing the
problem.
If there's an easy way for me to dump the entire transaction, and anyone cares
why the send request fails (this thing **never calls the BodyHandler** in case
I've not made that clear) then I'm happy to run one more test, if it helps. I
guess if anyone would like to see it, I can install wireshark. Let me know if
you want me to do that. (And to be clear, this is now for the benefit of anyone
*else* who wants to pursue this, not for me, I'm gong to continue learning this
API against a server that it likes.
On Tue, May 15, 2018 at 3:17 PM Bernd Eckenfels <[email protected]> wrote:
When you talk about HTT/2.0 Upgrades, do you also deal with TLS? You can use a
network tracer instead of curl to debug the whole exchange.
But you might need to turn on TLS debug to dump the session key (so the
protocol analyser like wireshark can actually decrypt it)
BTW you added one empty line too much in your Simulator Server now, so it’s 62
bytes.
Gruss
Bernd
--
http://bernd.eckenfels.net
From: net-dev <[email protected]> on behalf of Simon Roberts
<[email protected]>
Sent: Tuesday, May 15, 2018 10:47:35 PM
To: [email protected]
Subject: Re: EOF excption in HTTP 1.1 server interaction
Well, I give up. I'ts something to do with nodejs (which would seem like a
popular enough server to be of interest, but whatever) and perhaps the way that
node responds when asked to perform an HTTP2.0 upgrade.
Node generated a response that caused httpClient to fail. I used curl to try to
extract that response and pasted it into my "fake server". The response from my
fake server works. Of course, the one remaining difference is that I cannot
tell how node might be reacting to the upgrade request, since curl didn't issue
that part of the request. So, I think the problem is there.
public static final String[] response = {
"HTTP/1.1 200 OK",
"X-Powered-By: Expressd",
"Content-Type: text/html; charset=utf-8",
"Content-Length: 60",
"ETag: W/\"3c-CQHFqoSATSxoI5iZHLfu5OeEG3k\"",
"Date: Tue, 15 May 2018 20:34:49 GMT",
"Connection: keep-alive",
"",
"",
"<html><body><h1>Heading</h1><p>Some Text</p></body></html>",
""
};
public static void main(String[] args) throws Throwable {
var ss = new ServerSocket(8080);
var s = ss.accept();
var in = new BufferedReader(new InputStreamReader(s.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
System.out.println("< " + line);
if ("".equals(line)) break;
}
var out = new PrintWriter(new OutputStreamWriter(s.getOutputStream()));
for (var st : response) {
out.print(st + "\r\n");
System.out.println("> " + st);
}
out.flush();
s.close();
}
}
On Tue, May 15, 2018 at 1:55 PM Simon Roberts <[email protected]>
wrote:
Wooops, bran failure. JLS 3.10.4 :( Will fix to use \r\n and get back...
On Tue, May 15, 2018 at 1:43 PM Bernd Eckenfels <[email protected]> wrote:
Try using out.print(st+“\n\r“); instead. (And Account for the extra bytes in
the body as well or output the last string without the EOLs.
Gruss
Bernd
--
http://bernd.eckenfels.net
From: net-dev <[email protected]> on behalf of Simon Roberts
<[email protected]>
Sent: Tuesday, May 15, 2018 8:44:08 PM
To: [email protected]
Subject: Re: EOF excption in HTTP 1.1 server interaction
Thanks for the clarification; as I mentioned, I tried a number of variations,
with and without the "excess" empty lines. I also copied the output from a curl
session with a server that provides a successful interaction with the client
(so the content length etc were all correct). In every case the *send* failed
regardless. I'm finding myself inclined to believe that something about the way
the upgrade to 2.0 request is being handled is relevant. That said, I'm unsure
what line endings I'm sending in my fake server, but I'm running on Linux, not
windows (perhaps that's your point).
So that we can finally put to bed one way or the other the suggestion that this
is the server-side's fault, perhaps you could indicate an exact (minimal)
string, that you believe should work in my fake-server? I'm not an expert in
the HTTP specification, so it could be pretty inefficient to keep cycling round
this loop ;)
On Tue, May 15, 2018 at 12:16 PM Bernd Eckenfels <[email protected]> wrote:
Your example code writes 2 (emp5) lines more than the Content-Length includes.
You should also use crlf for the end of http headers (Sample works only on
Windows)ö
Gruss
Bernd
Gruss
Bernd
--
http://bernd.eckenfels.net
From: net-dev <[email protected]> on behalf of Simon Roberts
<[email protected]>
Sent: Tuesday, May 15, 2018 7:22:27 PM
To: [email protected]
Subject: EOF excption in HTTP 1.1 server interaction
(Added subject line, sorry, not sure how I missed that in the first place!)
I can pretty much confirm this has nothing to do with content length. I wrote
the code below to allow experimentation and even as it stands, which I believe
has a correct content length, and a bunch of other stuff removed from the
response) the client still fails. I also tried it with various combinations of
the response lines commented out, and all of them present, failure every time.
If you'd like to suggest the combination that "should" work, I'd be happy to
try it of course.
public final class Main {
public static final String[] response = {
"HTTP/1.1 200 OK",
// "X-Powered-By: Express",
// "Content-Type: text/html; charset=utf-8",
"Content-Length: 58",
// "ETag: W/\"3a-EwoPOQKsJivlqZA3z/ulngzMv9U\"",
// "Date: Tue, 15 May 2018 00:18:47 GMT",
// "Connection: keep-alive",
"",
"<html><body><h1>Heading</h1><p>Some Text</p></body></html>",
"",
""
};
public static void main(String[] args) throws Throwable {
var ss = new ServerSocket(8080);
var s = ss.accept();
var in = new BufferedReader(new InputStreamReader(s.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
System.out.println("< " + line);
if ("".equals(line)) break;
}
var out = new PrintWriter(new OutputStreamWriter(s.getOutputStream()));
for (var st : response) {
out.println(st);
System.out.println("> " + st);
}
out.flush();
s.close();
}
}
On Tue, May 15, 2018 at 9:46 AM Chris Hegarty <[email protected]> wrote:
Simon,
Only a partial reply, I’ll reply with other details later.
> On 15 May 2018, at 16:10, Simon Roberts <[email protected]>
> wrote:
>
> If I understand you correctly, you are saying that the "simple" version of
> the code-code by the way that the project's main web page presents as it's
> first example--is so simple that it's actually unusable in any production
> scenario.
That is not what I am saying. What I am saying is that the String handler is a
convenience handler that buffers all the response data and returns it once
decoded. In some circumstances it is just not possible to return, as a String,
the response data, if the server behaves incorrectly. In the scenario you are
encountering it appears that the server is returning too little data. It may
not be possible to always decode this partial data. And even if you do decode
this partial data, something further up the stack is likely to fail, if parsing
JSON for example.
My point is that if you want fault tolerance in the case of a misbehaving
server there are other ways to achieve that.
> I know I cannot use code for production that fails to read any data from a
> (presumably) merely mis-configured server (a server from which all other
> tools successfully read data.
What do you hope to do with partial data read from the server? If it’s JSON can
you decode it, or a gif can you display it?
> Did I interpret correctly, or did I miss something? If correctly, I'd be
> surprised if that doesn't strike you as sub-optimal to the point your pride
> in your work would want to make it more usable.
We do have pride in our work. I am engaging here to try to help you.
> It doesn't seem inappropriate to report a "warning" situation using an
> exception? Would not an aggregate return that includes data, headers, status
> code, ... and warnings be more helpful in this case?
There are other ways to achieve that, but IMO doing so for the String handler
would not be helpful to the vast majority of Java developers, that would not
check the carried warning.
> Mis-configured servers (assuming that's the cause) are not uncommon around
> the web.
Sure, but many clients will not be able to operate correctly with such, it’s a
matter of where and how they fail.
> But perhaps more importantly, whatever the problem is, it is not fixed by
> your code. The error is actually thrown by the *send* call, as I've now
> determined as a result of trying your code. (I modified it very slightly so
> as to complete it, and catch the exception.)
D’oh! Apologies, that’s what I get for sending something without testing it
more carefully, but you seem to have gotten past it.
-Chris
--
Simon Roberts
(303) 249 3613
--
Simon Roberts
(303) 249 3613
--
Simon Roberts
(303) 249 3613
--
Simon Roberts
(303) 249 3613
--
Simon Roberts
(303) 249 3613
