On 13/10/15 10:41, Billig, Andreas wrote:
Dear Andy, great, thanks a lot.
- HttpOp.setDefaultHttpClient(HttpClient);
works perfectly (therefore I used it directly without taking care of my
environment; If you still want my environment info, let me know). Again, very
good framework with anticipatory API, very good support.
I've also discussed the problem with colleagues and they said it's an mature
http/1 issue (on every machine (windows or linux)) that will be obsolete with
http/2 (->SPDY).
The default behaviour of the Jena code is to open a new connection each
time. It does not, by default, use cached connections; that leads to a
different problem where the cached connection is kept open in the client
and leads to a different cause of exhaustion of resources. And, as you
point out, concurrency is the issue with cached connections. The Jena
code is robust in the common case - hitting the server hard repeatedly
from a single thread is not the common case :-).
Yes, http/2 would multiplex the connection and make it much easier to
get request connection reuse and threading all working.
What do you think: Will it be possible to initially set a http client
"globally" separately for each thread (via ThreadLocal) ?
I don't know any reason why that could not be added if someone sent a
pull request.
It will have to deal with the issue about thread locals that you have to
explicitly delete them or else you get a build up of state.
Personally, rather than ThreadLocals in HttpOp, I'd look at allowing the
HttpClient for the execution to be provided externally HttpQuery not
just use the HttpOp.getDefaultHttpClient() all the time. Maybe add
constructors in QueryEngineHTTP, or setter style, to take a HttpClient
and pass to a new HttpQuery(string, HttpClient).
Andy
Ciao, Andreas
-----Ursprüngliche Nachricht-----
Von: Andy Seaborne [mailto:[email protected]]
Gesendet: Sonntag, 11. Oktober 2015 12:32
An: [email protected]
Betreff: Re: java.net.BindException during query evaluation
Hi,
It's not clear what's happening - could you say some more about your
environment?
Is it always at the same point?
Which OS? (Windows?)
Which version of Jena?
Which version of Java?
What's the hardware? How many cores?
Lots on stackoverflow but this looks close:
http://stackoverflow.com/questions/4708649/java-net-bindexception-address-already-in-use-when-trying-to-do-rapid-socket
On Linux:
The Jena tests provoked a similar situation where we saw TCP sockets not
getting recycled fast enough when the CPUs are all busy.
Programs run into problems when opening/closing sockets at high frequency for a
sustained period of time when the server is on the same system as the client,
as it is here.
If so, put a short wait in your loop.
Run "netstat -t" ; if you see a lot of connections and they go away after a
couple of minutes, then its zombie connections.
It may help to reconfigure the low level HTTP setup with
HttpOp.setDefaultHttpClient(HttpClient); Remote SPARQL uses the Jena default
but that may not fix the issue.
TDB in the same JVM does not use network connections.
Andy
On 11/10/15 10:40, Billig, Andreas wrote:
Hello team,
we are developing a constraints checker (schema-check under Closed World
Assumption (OWL, RDFS, etc.)) for a quad store and implement it via SPARQL/HTTP.
We ran into java.net.BindException during query evaluation.
I have isolated the code and made an abstraction. So now with the
following code
-----
for (int i = 0; i < 20000; i++) { // TODO (i:~16300)
QueryExecution qe =
QueryExecutionFactory.sparqlService("http://localhost:3030/Yyy/query";,
"select * {?x ?y ?z}"); // only two solutions to keep the result set
small
System.out.println(i);
ResultSet rs = qe.execSelect();
while (rs.hasNext()) {
System.out.println(rs.next());
}
qe.close();
}
-----
I got an exception around i > 16000 (so may be some queuing is
involved) with trace
------
HttpException: java.net.BindException: Address already in use: connect:
Unexpected error making the query: java.net.BindException: Address already in
use: connect
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.rewrap(HttpQuery.java:417)
at
com.hp.hpl.jena.sparql.engine.http.HttpQuery.execGet(HttpQuery.java:358)
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.exec(HttpQuery.java:295)
at
com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execResultSetInner(QueryEngineHTTP.java:346)
at
com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execSelect(QueryEngineHTTP.java:338)
<own trace items>
...
Caused by: java.net.BindException: Address already in use: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
...
------
Is this a known problem? It would be nice to have some solution because I
prefer not to go back to embedded TDB, where this error probably does not occur
(?).
thanks in advance, Andreas
