pjfanning opened a new pull request, #1402: URL: https://github.com/apache/pekko-http/pull/1402
Stacked on #1391, whose commits are included here because a PR in this repo cannot target a branch on a fork. Only the last commit is new. Merge #1391 first, then rebase this. ### Motivation The `zstd` content coding from #1391 cannot use a dictionary. [RFC 9842](https://www.rfc-editor.org/rfc/rfc9842) defines `dcz` for that, which compresses small or frequently changing resources much better using a dictionary that both sides have. ### Modification - **Model:** `HttpEncodings.dcz`, and modeled `Available-Dictionary`, `Dictionary-ID` and `Use-As-Dictionary` headers with parser rules, in scaladsl and javadsl. Parsing reuses the structured field rules of `Accept-Query`. Members of `Use-As-Dictionary` that are not known are ignored; a known member with the wrong type makes the header invalid, so it stays a `RawHeader`. - **Coding:** `Coders.Dcz(dictionary, ...)` and javadsl `DictionaryCoder`. They write and check the 40 byte header, and a hash mismatch fails decoding. `decodeRequestWith` and `Coders.decodeMessage` work with them unchanged. Like `zstd`, `dcz` needs zstd-jni, so it is not one of the default coders. - **Window:** the decoder allows the RFC 9842 limit, which is 8 MB or 1.25 times the dictionary, up to 128 MB. The encoder always sets the window: 2 MiB, or enough to reach the dictionary plus 25% growth, staying within the limit. - **Server:** `encodeResponseWithDictionary(lookup, [encoders])`. It uses `dcz` if `Accept-Encoding` explicitly lists it and the lookup finds the hash from `Available-Dictionary`, and otherwise falls back to normal negotiation. All responses get `Vary: Accept-Encoding, Available-Dictionary`. The application owns dictionary storage. The cross-origin rules of RFC 9842 section 9.3.3 are applied. - **Client:** no dictionary store. Users keep the dictionary, send `Available-Dictionary` themselves and decode with the coder. - **Docs:** a compression dictionaries section in the encoding page, and a page for the directive. ### Decisions to review - **Dictionaries that start with the zstd dictionary magic number are rejected.** This answers the open question in #1392. RFC 9842 requires raw content, but pekko-stream's `loadDict` uses libzstd's auto-detection and zstd-jni has no option to force raw content, so these would be parsed as formatted dictionaries. Failing early seemed better than output that other implementations cannot decode. Supporting them needs a change upstream. - **HTTPS is not enforced.** RFC 9842 requires a secure context, but the directive may be behind a proxy that terminates TLS, so this is documented instead. - **A directive that adds `Access-Control-Allow-Origin` has to be inside this one.** Otherwise cross-origin requests do not use the dictionary. - **No rejection when a dictionary was found.** If the dictionary cannot be used for a response and the client accepts none of the fallback encoders, the response is sent unencoded. - **`dcb` is out of scope**, as it needs a Brotli implementation. No `Link` helper is added for `rel="compression-dictionary"`. ### Result Servers can encode responses with a dictionary that the client has, and both sides can decode `dcz` messages given the dictionary. ### Tests - New `DczSpec`: the shared `CoderSpec` suite against plain zstd-jni streams, plus the header bytes with a SHA-256 test vector, hash mismatch, truncated input and window limits. - New cases in `CodingDirectivesSpec`, javadsl `CodingDirectivesTest` and `HttpHeaderSpec`. - `DczSpec`, `ZstdSpec`, `CodingDirectivesSpec` and `CodingDirectivesTest` pass locally on Scala 2.13. - `HttpHeaderSpec` passed locally before its whitespace cases were added. Those cases have not been run. - `headerCreateAll`, `javafmtAll` and `scalafmtAll` were run. - Not run locally, left to CI: the docs examples, MiMa, Scala 3 and the full test suite. This is why the PR is a draft. ### References Fixes #1392 Refs #1391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
