Repository: spark
Updated Branches:
refs/heads/branch-2.3 8b6504e33 -> 62010d648
[SPARK-26118][BACKPORT-2.3][WEB UI] Introducing spark.ui.requestHeaderSize for
setting HTTP requestHeaderSize
## What changes were proposed in this pull request?
Introducing spark.ui.requestHeaderSize for configuring Jetty's HTTP
requestHeaderSize.
This way long authorization field does not lead to HTTP 413.
## How was this patch tested?
Manually with curl (which version must be at least 7.55).
With the original default value (8k limit):
```bash
$ ./sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
$ echo -n "X-Custom-Header: " > cookie
$ printf 'A%.0s' {1..9500} >> cookie
$ curl -H cookie http://458apiros-MBP.lan:18080/
<h1>Bad Message 431</h1><pre>reason: Request Header Fields Too Large</pre>
$ tail -1
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
18/11/19 21:24:28 WARN HttpParser: Header is too large 8193>8192
```
After:
```bash
$ echo spark.ui.requestHeaderSize=10000 > history.properties
$ ./sbin/start-history-server.sh --properties-file history.properties
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
$ curl -H cookie http://458apiros-MBP.lan:18080/
<!DOCTYPE html><html>
<head>...
<link rel="shortcut icon"
href="/static/spark-logo-77x50px-hd.png"></link>
<title>History Server</title>
</head>
<body>
...
```
(cherry picked from commit ab61ddb34d58ab5701191c8fd3a24a62f6ebf37b)
Closes #23114 from attilapiros/julianOffByDays-2.3.
Authored-by: âattilapirosâ <piros.attila.zs...@gmail.com>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/62010d64
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/62010d64
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/62010d64
Branch: refs/heads/branch-2.3
Commit: 62010d64826d91a30d52cdc94467a45d211bcfd2
Parents: 8b6504e
Author: âattilapirosâ <piros.attila.zs...@gmail.com>
Authored: Wed Nov 21 22:58:30 2018 -0800
Committer: Dongjoon Hyun <dongj...@apache.org>
Committed: Wed Nov 21 22:58:30 2018 -0800
----------------------------------------------------------------------
.../scala/org/apache/spark/internal/config/package.scala | 6 ++++++
core/src/main/scala/org/apache/spark/ui/JettyUtils.scala | 6 ++++--
docs/configuration.md | 8 ++++++++
3 files changed, 18 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/62010d64/core/src/main/scala/org/apache/spark/internal/config/package.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 11cb5d3..a07e43a 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -499,6 +499,12 @@ package object config {
.stringConf
.createOptional
+ private[spark] val UI_REQUEST_HEADER_SIZE =
+ ConfigBuilder("spark.ui.requestHeaderSize")
+ .doc("Value for HTTP request header size in bytes.")
+ .bytesConf(ByteUnit.BYTE)
+ .createWithDefaultString("8k")
+
private[spark] val EXTRA_LISTENERS = ConfigBuilder("spark.extraListeners")
.doc("Class names of listeners to add to SparkContext during
initialization.")
.stringConf
http://git-wip-us.apache.org/repos/asf/spark/blob/62010d64/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
index ba98fa1..8412cda 100644
--- a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
@@ -355,13 +355,15 @@ private[spark] object JettyUtils extends Logging {
(connector, connector.getLocalPort())
}
+ val httpConfig = new HttpConfiguration()
+ httpConfig.setRequestHeaderSize(conf.get(UI_REQUEST_HEADER_SIZE).toInt)
// If SSL is configured, create the secure connector first.
val securePort = sslOptions.createJettySslContextFactory().map { factory
=>
val securePort = sslOptions.port.getOrElse(if (port > 0)
Utils.userPort(port, 400) else 0)
val secureServerName = if (serverName.nonEmpty) s"$serverName (HTTPS)"
else serverName
val connectionFactories =
AbstractConnectionFactory.getFactories(factory,
- new HttpConnectionFactory())
+ new HttpConnectionFactory(httpConfig))
def sslConnect(currentPort: Int): (ServerConnector, Int) = {
newConnector(connectionFactories, currentPort)
@@ -376,7 +378,7 @@ private[spark] object JettyUtils extends Logging {
// Bind the HTTP port.
def httpConnect(currentPort: Int): (ServerConnector, Int) = {
- newConnector(Array(new HttpConnectionFactory()), currentPort)
+ newConnector(Array(new HttpConnectionFactory(httpConfig)), currentPort)
}
val (httpConnector, httpPort) =
Utils.startServiceOnPort[ServerConnector](port, httpConnect,
http://git-wip-us.apache.org/repos/asf/spark/blob/62010d64/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 153066b..2c18fa9 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -909,6 +909,14 @@ Apart from these, the following properties are also
available, and may be useful
How many dead executors the Spark UI and status APIs remember before
garbage collecting.
</td>
</tr>
+<tr>
+ <td><code>spark.ui.requestHeaderSize</code></td>
+ <td>8k</td>
+ <td>
+ The maximum allowed size for a HTTP request header, in bytes unless
otherwise specified.
+ This setting applies for the Spark History Server too.
+ <td>
+</tr>
</table>
### Compression and Serialization
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
