yellowflash commented on a change in pull request #33849:
URL: https://github.com/apache/spark/pull/33849#discussion_r698178363
##########
File path: repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala
##########
@@ -160,7 +155,7 @@ class ExecutorClassLoader(
private def getClassFileInputStreamFromFileSystem(fileSystem: FileSystem)(
pathInDirectory: String): InputStream = {
- val path = new Path(directory, pathInDirectory)
+ val path = new Path(new Path(uri), pathInDirectory)
Review comment:
Hadoop Http filesystem require the paths to be fully qualified URLs.
It does `path.toUri.toUrl` which fails in our case because the Path is not
fully qualified.
So the class loader doesn't work if the class uri is `http://..../` I raised
a PR on hadoop too,
https://github.com/apache/hadoop/pull/3338
But this is a regression on spark, as it used to work with a very specific
implementation for http based class servers and now since it uses the
Filesystem api for everything and it doesn't work anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
