Andy, speaking of nginx - maybe you happen to know if is possible to limit request content size without a Content-Length header?
Martynas On Tue, Dec 10, 2013 at 6:53 PM, Andy Seaborne <[email protected]> wrote: > On 10/12/13 16:26, Martynas Jusevičius wrote: >> >> I'm upgrading from an old version of DatasetGraphAccessorHTTP (that I >> had tweaked to support authentication): >> >> https://github.com/Graphity/graphity-ldp/blob/master/src/main/java/org/graphity/update/DatasetGraphAccessorHTTP.java > > > HttpClient must be doing it because of the HttpEntity that is set. > >> >> The server is Dydra and the log follows. If getting the server to >> stream is not possible at the moment, what are my options? > > > 1/ Call HttpOps.execHttpPost(String.HttpEntity) yourself. > > 2/ Change DatasetGraphAccessorHTTP.graphToHttpEntity to use the same sort of > HttpEntity as before (which will send a single Conent-Length set > > 3/ ?? Use a different HttpClient from the default > See HttpOps.setDefaultHttpClient > No idea what the right setting are I'm afraid. > > 4/ Get Dydra to support streaming - I guess they don't support chunk > encoding and only support simple modes with Content-Length: - but it looks > like nginx proxied to the backend so hard to be sure. > > It's http-client that's doing that and as a very well used piece of > software, I'm mildly confident it is doing the right thing. > > Maybe it's just a matter of tuning that off (see 3). > > http://en.wikipedia.org/wiki/Chunked_transfer_encoding > > """ > Chunked encoding has the benefit that it is not necessary to generate the > full content before writing the header, as it allows streaming of content as > chunks and explicitly signaling the end of the content, making the > connection available for the next HTTP request/response. > """ > > > I suppose we could have an option on DatasetGraphAccessorHTTP for > "inefficient" mode (several of us are rather pro streaming :-) but not the > default. > > Andy > > >> >> 15:17:05,340 DEBUG wire:63 - >> "POST >> >> /graphity/some-test/service?graph=http%3A%2F%2Flocalhost%3A8080%2Fdanske_aviser%2Fgraphs%2Fceb6ec92-3b71-4bc8-a438-a327ba889ecf%23this >> HTTP/1.1[\r][\n]" >> 15:17:05,340 DEBUG wire:63 - >> "Transfer-Encoding: chunked[\r][\n]" >> 15:17:05,341 DEBUG wire:63 - >> "Content-Type: >> application/rdf+xml[\r][\n]" >> 15:17:05,341 DEBUG wire:63 - >> "Host: xx.dydra.com[\r][\n]" >> 15:17:05,341 DEBUG wire:63 - >> "Connection: Keep-Alive[\r][\n]" >> 15:17:05,342 DEBUG wire:63 - >> "User-Agent: Apache-HttpClient/4.2.3 >> (java 1.5)[\r][\n]" >> 15:17:05,342 DEBUG wire:63 - >> "Authorization: Basic >> .......[\r][\n]" >> >> 15:17:05,342 DEBUG wire:63 - >> "[\r][\n]" >> 15:17:05,342 DEBUG headers:273 - >> POST >> >> /graphity/some-test/service?graph=http%3A%2F%2Flocalhost%3A8080%2Fdanske_aviser%2Fgraphs%2Fceb6ec92-3b71-4bc8-a438-a327ba889ecf%23this >> HTTP/1.1 >> 15:17:05,343 DEBUG headers:276 - >> Transfer-Encoding: chunked >> 15:17:05,343 DEBUG headers:276 - >> Content-Type: application/rdf+xml >> 15:17:05,343 DEBUG headers:276 - >> Host: xx.dydra.com >> 15:17:05,343 DEBUG headers:276 - >> Connection: Keep-Alive >> 15:17:05,343 DEBUG headers:276 - >> User-Agent: >> Apache-HttpClient/4.2.3 (java 1.5) >> ... >> 15:17:05,406 DEBUG DefaultClientConnection:254 - Receiving response: >> HTTP/1.1 411 Length Required >> 15:17:05,406 DEBUG headers:257 - << HTTP/1.1 411 Length Required >> 15:17:05,407 DEBUG headers:260 - << Server: nginx/1.2.1 >> 15:17:05,407 DEBUG headers:260 - << Date: Tue, 10 Dec 2013 14:17:07 GMT >> 15:17:05,407 DEBUG headers:260 - << Content-Type: text/html >> 15:17:05,407 DEBUG headers:260 - << Content-Length: 180 >> 15:17:05,408 DEBUG headers:260 - << Connection: close >> 15:17:05,409 DEBUG SystemDefaultHttpClient:73 - Authentication succeeded >> 15:17:05,409 DEBUG HttpOp:1017 - [8] 411 Length Required >> 15:17:05,410 DEBUG DefaultClientConnection:169 - Connection >> 0.0.0.0:51690<->176.9.131.227:80 closed >> 15:17:05,410 DEBUG PoolingClientConnectionManager:281 - Connection >> released: [id: 7][route: {}->http://xx.dydra.com][total kept alive: >> >> On Tue, Dec 10, 2013 at 5:18 PM, Andy Seaborne <[email protected]> wrote: >>> >>> On 10/12/13 14:33, Martynas Jusevičius wrote: >>>> >>>> >>>> Hey, >>>> >>>> after upgrade to 2.11.0 I get this exception during Graph Store >>>> Protocol request: >>>> >>>> org.apache.jena.atlas.web.HttpException: 411 - Length Required >>>> org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1020) >>>> org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:633) >>>> org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:601) >>>> >>>> >>>> org.apache.jena.web.DatasetGraphAccessorHTTP.doPost(DatasetGraphAccessorHTTP.java:184) >>>> >>>> >>>> org.apache.jena.web.DatasetGraphAccessorHTTP.httpPost(DatasetGraphAccessorHTTP.java:179) >>>> org.apache.jena.web.DatasetAdapter.add(DatasetAdapter.java:98) >>>> org.graphity.server.util.DataManager.addModel(DataManager.java:440) >>>> >>>> Has something changed wrt to Content-Length? I guess I'll need to >>>> extend DatasetGraphAccessorHTTP... Or is there an easier way to work >>>> around this? >>>> >>>> Martynas >>>> graphityhq.com >>>> >>> >>> Looks like the server not liking the client request - what did the server >>> say about the request? (and what is the server?) >>> >>> ARQ like to stream - and that means no content length. >>> >>> What does the Jena code from the version you are upgarding for under >>> DatasetGraphAccessorHTTP look like? >>> >>> Andy > >
