I refer to the size of the request.
There is a test "it should echo a large input" sending a 1MB big request
({"arg":"aaaa.....aaaa"})
The runtime failed the test (now I fixed it).
The problem was I was using golang scanner.Scan() and I discovered it is using
a fixed size buffer, 64kb large by default.
Go is about efficiency and this kind of limits are spread all around the apis
(I found others like implicit timeouts etc).
I enlarged the buffer and passed the test, but I would like to know if there is
some parameter passed to the runtime so I can size this buffer not with an
hardcoded value but with some knowledge of the biggest request I can receive.
Otherwise I should implement my own scanner using a dynamic array...
--
Michele Sciabarra
[email protected]
----- Original message -----
From: Rodric Rabbah <[email protected]>
To: [email protected]
Subject: Re: How to pass size the max size of the request to the runtime?
Date: Tue, 11 Sep 2018 06:43:23 -0700
I assume Michele is referring to a buffer to eventually hold the function’s
input (which is limited to 1MB by default though some deployments use 6MB) -
not a transient/intermediate buffer. We’ll see...
-r
> On Sep 11, 2018, at 6:30 AM, Markus Thömmes <[email protected]> wrote:
>
> Heya,
>
> I tend to agree with Felix. Buffers are usually just used as a transport
> mechanism and they shouldn't need to fit the whole response at once (which
> doesn't imply we don't load the response in memory, but at least that input
> buffer doesn't need to be == content-length). Could you give a bit more
> context on what the scan buffer does and why 64k isn't enough space to do
> what we need?
>
> Moreover, I'd rather not set any hard limits on the runtimes themselves.
> The limits should be imposed by the surrounding system, the user-containers
> could accept however much payload as they can fit. That reduces the amount
> of configuration and testing needed. If we start imposing these limits on
> every layer, we'll need to test every layer to correctly impose the limits.
>
> Cheers,
> Markus
>
> Am Di., 11. Sep. 2018 um 15:08 Uhr schrieb Felix Meschberger
> <[email protected]>:
>
>> Hi
>>
>> Holding the complete input in memory ? Sounds like a good DoS surface -
>> unless you limit the input size....
>>
>> Regards
>> Felix
>>
>> --
>> Creative typing support courtesy of my iPhone
>>
>> Am 11.09.2018 um 15:02 schrieb Rodric Rabbah <[email protected]<mailto:
>> [email protected]>>:
>>
>> The http post will have a content length is that useful?
>>
>> -r
>>
>> On Sep 10, 2018, at 7:45 AM, Michele Sciabarra <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hello, I am in the process of running the mandatory tests against the Go
>> Runtime.
>> In the process, I fixed a lot of bugs, because those tests revealed a
>> number of details about encoding, env variables and other things that were
>> not obvious to me in the first place.
>>
>> Now I have a problem: I am trying to pass the test that tried to send a
>> one-megabyte big request to the runtime.
>> Currently, it does not work because I discovered the "scan" buffer has in
>> Golang a fixed size of 64k.
>>
>> Of course, I can increase it but I need to know how big it must be. I know
>> that you can set some parameters at OpenWhisk level but I am not aware how
>> a runtime can know those parameters. Most notably I need to be able to read
>> the maximum size of the requests because I need to allocate a buffer at
>> init time. Any hints?
>>
>>
>> --
>> Michele Sciabarra
>> [email protected]<mailto:[email protected]>
>>
