There is a trade off involving invoker performance vs user debugging info.
I would for starter implement 3.2 behind a configuration flag turned off by
default that can be turn on in ansible then downstream/operator can choose
if they want their invokers to suffer by always trying parse/process the
very large response into a json object into memory and the n inspecting the
object to pick up fields to slim it down to fit the limit of 1mb
Iterating over each field and check and try to come up with all the
fields that summing them together will be close to 1mb and marking the
fields that don't fit the 1mb with some value like null
If the debugging flag is off, I would just make invoker create and store
the reason message and then the first 1MB of raw (no JSON) data string
Something like
{
message: "Action result of X MB exceeds system limit of Y MB"
payload:"{ a:1, b:"c", ..., z:"hell" "
}
payload above will be 1mb worth of data and it will be truncated z was
suppose to be hello :-)
basically invoker takes the payload text from the response of the action
proxy checks against the limit for the exception and then takes the first
1mb and stores it straig without trying to parse into JSON and try to find
what are the largest fields to nuke to come up with a small 1mb JSON to
store
After writing this email I not sure if this should be here in dev list or
Github issue. My personal preference would be Github issue, I will copy it
there now :-)
Next time I will just answer here with a the first sentence and put the
details over on the issue
-- Carlos
On Fri, Jan 27, 2017 at 10:37 AM Rodric Rabbah <[email protected]> wrote:
Currently, the size of an action result is not checked by the system and
can exceed the assumed limit of 1MB. This will manifest itself in blocking
invokes will have at least a 30s delay on getting their results. Similarly
larger than allowed invoke time parameters are not rejected as early as
they should be in the pipeline. This is a standing open defect you can read
more about here [1] along with my proposed solution [2] which I include
below for convenience.
1. Response is sized by the invoker.
1. If it exceeds the system limit, the invoker will reject it.
2. If it is within the limits, it will be processed as usual.
2. When the invoker rejects an action response, it will do the
following:
1. Construct an error response object with a message explaining the
failure
2. In addition, it will provide as a debugging aid a partial result
that contains as many of the top level fields from the
dictionary response
into the error object.
3. The error response for large responses shall include:
1. "Action result of X MB exceeds system limit of Y MB".
2. The number of top level fields contained in the response and how
many are actually included in the partial response.
As part of doing this work, I will also apply the same technique to reject
large incoming payloads which currently get rejected a bit later in the
pipeline than they should.
Comments and feedback welcomed and appreciated.
-r
[1] https://github.com/openwhisk/openwhisk/issues/566
[2] https://github.com/openwhisk/openwhisk/issues/566#issuecomment-275687757
