GitHub user attilapiros opened a pull request:
https://github.com/apache/spark/pull/23090
[SPARK-26118][UI] Introducing spark.ui.requestHeaderSize for setting HTTP
requestHeaderSize
## What changes were proposed in this pull request?
Introducing spark.ui.requestHeaderSize for configuring Jetty's HTTP
requestHeaderSize.
This way long authorization field does not lead to HTTP 413.
## How was this patch tested?
Manually with curl (which version must be at least 7.55).
With the original default value (8k limit):
```bash
# Starting history server with default requestHeaderSize
$ ./sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
# Creating huge header
$ echo -n "X-Custom-Header: " > cookie
$ printf 'A%.0s' {1..9500} >> cookie
# HTTP GET with huge header fails with 431
$ curl -H @cookie http://[email protected]:18080/
<h1>Bad Message 431</h1><pre>reason: Request Header Fields Too Large</pre>
# The log contains the error
$ tail -1
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
18/11/19 21:24:28 WARN HttpParser: Header is too large 8193>8192
```
After:
```bash
# Creating the history properties file with the increased requestHeaderSize
$ echo spark.ui.requestHeaderSize=10000 > history.properties
# Starting Spark History Server with the settings
$ ./sbin/start-history-server.sh --properties-file history.properties
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
# HTTP GET with huge header gives back HTML5 (I have added here only just a
part of the response)
$ curl -H @cookie http://[email protected]:18080/
<!DOCTYPE html><html>
<head>...
<link rel="shortcut icon"
href="/static/spark-logo-77x50px-hd.png"></link>
<title>History Server</title>
</head>
<body>
...
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/attilapiros/spark JettyHeaderSize
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23090.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23090
----
commit c6278becc29f5605254786ad5070ab2666ffad47
Author: âattilapirosâ <piros.attila.zsolt@...>
Date: 2018-11-19T19:55:28Z
Introducing spark.ui.requestHeaderSize
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
