Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4944#discussion_r26260907
--- Diff:
repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala ---
@@ -71,30 +73,54 @@ class ExecutorClassLoader(conf: SparkConf, classUri:
String, parent: ClassLoader
}
}
+ private def getClassFileInputStreamFromHttpServer(pathInDirectory:
String): InputStream = {
+ val url = if (SparkEnv.get.securityManager.isAuthenticationEnabled()) {
+ val uri = new URI(classUri + "/" + urlEncode(pathInDirectory))
+ val newuri = Utils.constructURIForAuthentication(uri,
SparkEnv.get.securityManager)
+ newuri.toURL
+ } else {
+ new URL(classUri + "/" + urlEncode(pathInDirectory))
+ }
+ val connection: HttpURLConnection =
Utils.setupSecureURLConnection(url.openConnection(),
+ SparkEnv.get.securityManager).asInstanceOf[HttpURLConnection]
+ // Set the connection timeouts (for testing purposes)
+ if (httpUrlConnectionTimeoutMillis != -1) {
+ connection.setConnectTimeout(httpUrlConnectionTimeoutMillis)
+ connection.setReadTimeout(httpUrlConnectionTimeoutMillis)
+ }
+ connection.connect()
--- End diff --
Per the [HttpURLConnection
Javadoc](http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html):
> Each HttpURLConnection instance is used to make a single request but the
underlying network connection to the HTTP server may be transparently shared by
other instances. Calling the close() methods on the InputStream or OutputStream
of an HttpURLConnection after a request may free network resources associated
with this instance but has no effect on any shared persistent connection.
Calling the disconnect() method may close the underlying socket if a persistent
connection is otherwise idle at that time.
This can cause problems in practice:
https://scotte.github.io/2015/01/httpurlconnection-socket-leak/.
Actually, that blog post reminds me that I should probably call
`getErrorStream` and read that stream in the next error-handling block prior to
calling disconnect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
