Repository: spark
Updated Branches:
refs/heads/branch-2.2 1d8a4a2aa -> 9ceee6f18
[SPARK-26118][BACKPORT-2.2][WEB UI] Introducing spark.ui.requestHeaderSize for
setting HTTP requestHeaderSize
## What changes were proposed in this pull request?
Introducing spark.ui.requestHeaderSize for configuring Jetty's HTTP
requestHeaderSize.
This way long authorization field does not lead to HTTP 413.
## How was this patch tested?
Manually with curl (which version must be at least 7.55).
With the original default value (8k limit):
```bash
$ ./sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
$ echo -n "X-Custom-Header: " > cookie
$ printf 'A%.0s' {1..9500} >> cookie
$ curl -H cookie http://458apiros-MBP.lan:18080/
<h1>Bad Message 431</h1><pre>reason: Request Header Fields Too Large</pre>
$ tail -1
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
18/11/19 21:24:28 WARN HttpParser: Header is too large 8193>8192
```
After:
```bash
$ echo spark.ui.requestHeaderSize=10000 > history.properties
$ ./sbin/start-history-server.sh --properties-file history.properties
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
$ curl -H cookie http://458apiros-MBP.lan:18080/
<!DOCTYPE html><html>
<head>...
<link rel="shortcut icon"
href="/static/spark-logo-77x50px-hd.png"></link>
<title>History Server</title>
</head>
<body>
...
```
(cherry picked from commit ab61ddb34d58ab5701191c8fd3a24a62f6ebf37b)
Closes #23115 from attilapiros/julianOffByDays-2.2.
Authored-by: âattilapirosâ <piros.attila.zs...@gmail.com>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9ceee6f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9ceee6f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9ceee6f1
Branch: refs/heads/branch-2.2
Commit: 9ceee6f188e6c3794d31ce15cc61d29f907bebf7
Parents: 1d8a4a2
Author: âattilapirosâ <piros.attila.zs...@gmail.com>
Authored: Wed Nov 21 23:06:46 2018 -0800
Committer: Dongjoon Hyun <dongj...@apache.org>
Committed: Wed Nov 21 23:06:46 2018 -0800
----------------------------------------------------------------------
.../scala/org/apache/spark/internal/config/package.scala | 7 +++++++
core/src/main/scala/org/apache/spark/ui/JettyUtils.scala | 7 +++++--
docs/configuration.md | 8 ++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9ceee6f1/core/src/main/scala/org/apache/spark/internal/config/package.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index f65a9d7..d41fc49 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -312,4 +312,11 @@ package object config {
" service is disabled.")
.bytesConf(ByteUnit.BYTE)
.createWithDefault(Long.MaxValue)
+
+ private[spark] val UI_REQUEST_HEADER_SIZE =
+ ConfigBuilder("spark.ui.requestHeaderSize")
+ .doc("Value for HTTP request header size in bytes.")
+ .bytesConf(ByteUnit.BYTE)
+ .createWithDefaultString("8k")
+
}
http://git-wip-us.apache.org/repos/asf/spark/blob/9ceee6f1/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
index 3e0b62d..711fdcc 100644
--- a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
@@ -40,6 +40,7 @@ import org.json4s.jackson.JsonMethods.{pretty, render}
import org.apache.spark.{SecurityManager, SparkConf, SSLOptions}
import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
import org.apache.spark.util.Utils
/**
@@ -342,13 +343,15 @@ private[spark] object JettyUtils extends Logging {
(connector, connector.getLocalPort())
}
+ val httpConfig = new HttpConfiguration()
+ httpConfig.setRequestHeaderSize(conf.get(UI_REQUEST_HEADER_SIZE).toInt)
// If SSL is configured, create the secure connector first.
val securePort = sslOptions.createJettySslContextFactory().map { factory
=>
val securePort = sslOptions.port.getOrElse(if (port > 0)
Utils.userPort(port, 400) else 0)
val secureServerName = if (serverName.nonEmpty) s"$serverName (HTTPS)"
else serverName
val connectionFactories =
AbstractConnectionFactory.getFactories(factory,
- new HttpConnectionFactory())
+ new HttpConnectionFactory(httpConfig))
def sslConnect(currentPort: Int): (ServerConnector, Int) = {
newConnector(connectionFactories, currentPort)
@@ -363,7 +366,7 @@ private[spark] object JettyUtils extends Logging {
// Bind the HTTP port.
def httpConnect(currentPort: Int): (ServerConnector, Int) = {
- newConnector(Array(new HttpConnectionFactory()), currentPort)
+ newConnector(Array(new HttpConnectionFactory(httpConfig)), currentPort)
}
val (httpConnector, httpPort) =
Utils.startServiceOnPort[ServerConnector](port, httpConnect,
http://git-wip-us.apache.org/repos/asf/spark/blob/9ceee6f1/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index c39c758..bc230e6 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -801,6 +801,14 @@ Apart from these, the following properties are also
available, and may be useful
How many dead executors the Spark UI and status APIs remember before
garbage collecting.
</td>
</tr>
+<tr>
+ <td><code>spark.ui.requestHeaderSize</code></td>
+ <td>8k</td>
+ <td>
+ The maximum allowed size for a HTTP request header, in bytes unless
otherwise specified.
+ This setting applies for the Spark History Server too.
+ <td>
+</tr>
</table>
### Compression and Serialization
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
