Ilia Lazebnik created YUNIKORN-3341:
---------------------------------------
Summary: gzip middleware double-compresses responses that handlers
already encoded
Key: YUNIKORN-3341
URL: https://issues.apache.org/jira/browse/YUNIKORN-3341
Project: Apache YuniKorn
Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Ilia Lazebnik
The gzip middleware on the REST router compresses any response whose buffered
body exceeds \{{minCompressionSize}}, without checking whether the wrapped
handler had already encoded the body itself.
{\{/ws/v1/metrics}} is served by \{{promhttp.Handler()}}, which performs its
own gzip content negotiation: when the client sends \{{Accept-Encoding: gzip}}
it compresses the payload and sets \{{Content-Encoding: gzip}}. The middleware
then sees the same \{{Accept-Encoding}}, buffers promhttp's already-gzipped
bytes, finds them over the threshold, and compresses a second time.
\{{switchToGzip}} uses \{{Header().Set}}, so the response still advertises a
single \{{Content-Encoding: gzip}} while carrying a doubly encoded body.
Clients decode once and get gzip framing instead of the payload. For Prometheus
this means every scrape of \{{/ws/v1/metrics}} fails to parse and the target is
marked down, so scheduler metrics are lost entirely for any scraper that
advertises gzip support — which is the default.
Reproduced against a running 1.9.0 scheduler:
{code}
$ curl -s -D- -o body.bin http://scheduler:9080/ws/v1/metrics
200 OK, 69714 bytes, "# HELP go_gc_duration_seconds ..."
$ curl -s -D- -o body.gz -H 'Accept-Encoding: gzip'
http://scheduler:9080/ws/v1/metrics
200 OK, Content-Encoding: gzip, 8839 bytes
$ gunzip -c body.gz | file -
/dev/stdin: gzip compressed data # still gzip after one decode
$ gunzip -c body.gz | gunzip -c | head -1
# HELP go_gc_duration_seconds ... # payload only after two
{code}
This is a regression in 1.9.0: \{{pkg/webservice/gzip.go}} is new in that
release and \{{StartWebApp}} changed from \{{Handler: router}} to \{{Handler:
compressResponse(router)}}. The route itself and port 9080 are unchanged from
1.8.0.
*Proposed fix*
Treat a \{{Content-Encoding}} set by the handler as a signal that the body is
already encoded, and pass the response through untouched. Metrics responses
stay compressed — promhttp still gzips them, just once — and handlers that do
not set \{{Content-Encoding}} are unaffected.
Fixing \{{getMetrics}} alone (via \{{promhttp.HandlerOpts{DisableCompression:
true}}}) would patch only the one endpoint that self-encodes today; the
middleware would still corrupt any future handler that does the same.
PR: https://github.com/apache/yunikorn-core/pull/1115
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
