Re: vibe.d maxRequestSize
On Monday, 19 September 2016 at 17:54:05 UTC, Steven Schveighoffer wrote: On 9/19/16 1:34 PM, Chris wrote: [...] Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve Thanks a million! I'll look into it tomorrow.
Re: vibe.d maxRequestSize
On 9/19/16 1:34 PM, Chris wrote: On Monday, 19 September 2016 at 16:55:05 UTC, Steven Schveighoffer wrote: Hm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -Steve I didn't want to litter the thread, here it is: 400 - Bad Request Bad Request Internal error information: object.Exception@../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:405 pure @safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x8f11a7] /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:363 pure @safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8f112a] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:348 void vibe.stream.operations.readUntilSmall!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntilSmall(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06b00] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:137 void vibe.stream.operations.readUntil!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntil(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06a1f] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:53 void vibe.stream.operations.readLine!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readLine(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, ulong, immutable(char)[]) [0xb069c8] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:39 ubyte[] vibe.stream.operations.readLine!().readLine(vibe.core.stream.InputStream, ulong, immutable(char)[], vibe.utils.memory.Allocator) [0xb06984] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1861 void vibe.http.server.parseRequestHeader(vibe.http.server.HTTPServerRequest, vibe.core.stream.InputStream, vibe.utils.memory.Allocator, ulong) [0xb5e827] Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve
Re: vibe.d maxRequestSize
On Monday, 19 September 2016 at 16:55:05 UTC, Steven Schveighoffer wrote: Hm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -Steve I didn't want to litter the thread, here it is: 400 - Bad Request Bad Request Internal error information: object.Exception@../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:405 pure @safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x8f11a7] /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:363 pure @safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8f112a] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:348 void vibe.stream.operations.readUntilSmall!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntilSmall(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06b00] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:137 void vibe.stream.operations.readUntil!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntil(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06a1f] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:53 void vibe.stream.operations.readLine!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readLine(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, ulong, immutable(char)[]) [0xb069c8] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:39 ubyte[] vibe.stream.operations.readLine!().readLine(vibe.core.stream.InputStream, ulong, immutable(char)[], vibe.utils.memory.Allocator) [0xb06984] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1861 void vibe.http.server.parseRequestHeader(vibe.http.server.HTTPServerRequest, vibe.core.stream.InputStream, vibe.utils.memory.Allocator, ulong) [0xb5e827] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1659 bool vibe.http.server.handleRequest(vibe.core.stream.Stream, vibe.core.net.TCPConnection, vibe.http.server.HTTPListenInfo, ref vibe.http.server.HTTPServerSettings, ref bool) [0xb5c5d0] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1552 void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPListenInfo) [0xb5be4e] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1433 void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings).doListen(vibe.http.server.HTTPListenInfo, bool, bool).__lambda4(vibe.core.net.TCPConnection) [0xb5b814] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/drivers/libevent2_tcp.d:610 void vibe.core.drivers.libevent2_tcp.ClientTask.execute() [0xbe8715] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/core.d:488 void vibe.core.core.makeTaskFuncInfo!(void delegate()).makeTaskFuncInfo(ref void delegate()).callDelegate(vibe.core.core.TaskFuncInfo*) [0xafea75] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/core.d:1119 void vibe.core.core.CoreTask.run() [0xb8dd79] ??:? void core.thread.Fiber.run() [0xc70ea5] ??:? fiber_entryPoint [0xc70c27] ??:? [0x]
Re: vibe.d maxRequestSize
On 9/19/16 11:59 AM, Chris wrote: On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote: On 9/15/16 9:11 AM, Chris wrote: On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote: Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 It doesn't seem to make any difference in my case. I wonder what could be wrong Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 Finally, I could find where it happens. If I use "get" in a HTML form (instead of "post"), vibe.d complains. I set the max request and request header size to 25MB. ` 400 - Bad Request Bad Request Internal error information: object.Exception@../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. ` https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/stream/operations.d#L349 Hm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -Steve
Re: vibe.d maxRequestSize
On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote: On 9/15/16 9:11 AM, Chris wrote: On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote: Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve It doesn't seem to make any difference in my case. I wonder what could be wrong Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -Steve Finally, I could find where it happens. If I use "get" in a HTML form (instead of "post"), vibe.d complains. I set the max request and request header size to 25MB. ` 400 - Bad Request Bad Request Internal error information: object.Exception@../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. ` https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/stream/operations.d#L349
Re: vibe.d maxRequestSize
On Friday, 16 September 2016 at 00:35:25 UTC, sarn wrote: I hope this isn't too obvious, but I have to ask because it's such a common gotcha: Are you reverse proxying through a server like nginx by any chance? There are default request size limits there. (For nginx specifically, it's this one: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) No. Atm I'm running the server locally on my machine (for development), so I access vibe.d directly. But later it will run on Apache, and query string size will inevitably be an issue.
Re: vibe.d maxRequestSize
I hope this isn't too obvious, but I have to ask because it's such a common gotcha: Are you reverse proxying through a server like nginx by any chance? There are default request size limits there. (For nginx specifically, it's this one: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size)
Re: vibe.d maxRequestSize
On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote: On 9/15/16 9:11 AM, Chris wrote: On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote: Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve It doesn't seem to make any difference in my case. I wonder what could be wrong Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -Steve Hm, maybe the browsers are limiting the the length of the query string themselves, although that shouldn't be an issue with POST. I can send very long strings, if I enable HTTPServerOption.parseFormBody, but then I can no longer parse the URI at the same time (as far as I know). The browser always sends `Content-Length: 0` in the request header, which doesn't affect vibe.d (short strings work).
Re: vibe.d maxRequestSize
On 9/15/16 9:11 AM, Chris wrote: On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote: Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve It doesn't seem to make any difference in my case. I wonder what could be wrong Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -Steve
Re: vibe.d maxRequestSize
On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer wrote: Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve It doesn't seem to make any difference in my case. I wonder what could be wrong
Re: vibe.d maxRequestSize
On 9/14/16 9:58 AM, Chris wrote: The vibe.d server rejects `XMLHttpRequest`s that are too long (in the eyes of the server). In the docs it says "maxRequestSize ulong Maximum number of transferred bytes per request after which the connection is closed with [sic!]" However, when you go to http://vibed.org/api/vibe.http.server/HTTPServerSettings.maxRequestSize it says "Maximum number of transferred bytes per request after which the connection is closed with an error; not supported yet" "not supported yet" and I think it is not supported yet. Can I still change the server settings so that it accepts longer query strings? Atm, the server rejects strings that are > ~2,500 characters, which is not much. Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve
