shreemaan-abhishek opened a new pull request, #14016: URL: https://github.com/apache/apisix/pull/14016
### Description Adds a `request-decompress` plugin, so a client can send a request body with `Content-Encoding: gzip` or `Content-Encoding: deflate`, which [RFC 9110 §8.4](https://www.rfc-editor.org/rfc/rfc9110#section-8.4) allows. Nothing on the request path inflates a request body today. The `gzip` plugin works on responses through `header_filter`, and `apisix/utils/content-decode.lua` is reached only from `response-rewrite` and `log-util`. So a plugin that reads the request body sees compressed bytes: `oas-validator` validates them against the schema and fails, `request-validation` rejects them, `chaitin-waf` inspects them, and `body-transformer` cannot parse them. ### How **Core.** `core.request.get_body(max_size, ctx, opts)` gains `opts.decompress`, off by default, so the roughly 35 existing callers keep the bytes exactly as they arrived. `hmac-auth`'s `Digest` check is the one that matters most. On failure the third return value names the cause, one of `unsupported_encoding`, `decompress_failed` or `too_large`, so each caller picks its own status code. - `gzip` and `deflate` both work from a single `inflate_gzip` call, since both are zlib streams and ffi-zlib's default windowBits of 15+32 auto-detects the header. - Codings are inflated in the reverse of the order the client applied them, so a chained `Content-Encoding: gzip, gzip` works, and `identity` is a no-op. - `max_size` bounds the inflated data as well as the raw read, which is what makes a small payload that expands to a very large one safe. The cap is enforced mid-stream: a falsy return from the ffi-zlib output callback stops the stream, so an oversized body is never buffered in full. - `get_body` still returns a single value on success. Some callers splat the result straight into varargs, `ngx.print(core.request.get_body())` in `t/plugin/body-transformer.t` among them, so a trailing `nil` would corrupt them. **The inflate helper moves to `apisix/utils/gzip.lua`**, which also exposes a deflate for the plugin's re-compress path. `apisix/utils/content-decode.lua` now requires it instead of carrying its own copy, so the repository keeps one inflate implementation rather than two. **The plugin** runs in `rewrite`. When the request carries a `Content-Encoding` it reads the body through the core path, writes the inflated bytes back with `ngx.req.set_body_data`, and removes the `Content-Encoding` header. Every later reader sees plain bytes with no change of its own. A request without a `Content-Encoding` returns immediately and its body is never read. | field | type | default | meaning | |---|---|---|---| | `max_req_body_size` | integer, min 1 | 1048576 | bounds the decompressed size | | `forward_compressed` | boolean | `false` | the upstream receives a compressed body | `forward_compressed` restores a compressed body in `before_proxy`, the last hook before proxying, so the plugins on the route still read the plain body while the upstream gets compressed bytes. When the body is still byte-identical to the plaintext the plugin wrote, the bytes as they arrived are restored verbatim along with the original `Content-Encoding` and nothing is compressed. Only a body another plugin rewrote is really deflated, re-applying the client's codings in the order the client applied them, and the header then names the codings actually applied rather than the list as sent. Rejections are `415` with an `Accept-Encoding: gzip, deflate` response header for an unsupported coding, per [RFC 9110 §12.5.3](https://www.rfc-editor.org/rfc/rfc9110#section-12.5.3), `400` for a malformed body, and `413` for a body inflating past `max_req_body_size`. ### Priority 2850 `ip-restriction` holds 3000, so the plugin takes 2850. That puts it after the plugins that reject a request cheaply (`ua-restriction` 2999, `referer-restriction` 2990, `csrf` 2980, `uri-blocker` 2900), so a blocked request is never inflated, and before every plugin that reads the body: `request-validation` 2800, `chaitin-waf` 2700, the auth plugins, `body-transformer` 1080, `proxy-rewrite` 1008, `oas-validator` 512, `degraphql` 509, `grpc-transcode` 506. Running ahead of the WAF is deliberate, since a WAF inspecting compressed bytes can be walked past by gzipping the payload. Two consequences are documented, and `_meta.priority` changes either: - `hmac-auth` (2530) with `validate_request_body` hashes the inflated body on a route that also enables this plugin. A client that signs the compressed bytes needs `hmac-auth` above 2850. - Decompression happens before authentication. `max_req_body_size` and `client_max_body_size` bound the work an unauthenticated client can cause. `br` is deliberately left out for now, even though `content-decode.lua` can decode it when `lua-brotli` is present, so that the supported set is the same whether or not that optional dependency is installed. It can be added behind the same `is_br_libs_loaded` check if there is demand. ### Tests - `t/plugin/request-decompress.t`, 25 blocks: schema defaults and validation; gzip, deflate, chained `gzip, gzip`, `identity`, and a request with no `Content-Encoding`; the rewritten `Content-Length`; a malformed body, an unsupported `br`, and an over-cap body; a `Content-Encoding` on a bodyless GET; stacking with `oas-validator` for a body that passes the spec and one that fails it; and for `forward_compressed`, the upstream receiving the bytes as they arrived for gzip, deflate and chained codings, validation still rejecting an invalid body, an uncompressed request passing through, and a `body-transformer` route exercising the real compress path, where the forwarded body differs from the original and inflates to the transformed JSON. - `t/core/request.t` gains 4 blocks: raw bytes without `opts.decompress`, and each of the three failure kinds. - `t/utils/gzip.t` covers the new helper, including the mid-stream `max_output` abort. ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [x] I have added tests corresponding to this change - [x] I have updated the documentation to reflect this change - [x] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
