yellowflash commented on a change in pull request #33849:
URL: https://github.com/apache/spark/pull/33849#discussion_r698178363
##########
File path: repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala
##########
@@ -160,7 +155,7 @@ class ExecutorClassLoader(
private def getClassFileInputStreamFromFileSystem(fileSystem: FileSystem)(
pathInDirectory: String): InputStream = {
- val path = new Path(directory, pathInDirectory)
+ val path = new Path(new Path(uri), pathInDirectory)
Review comment:
Hadoop Http filesystem doesn't take Path as in it need to have
authority, scheme etc..
As in it does `path.toUri.toUrl` which fails in our case because the Path is
not fully qualified.
So the class loader doesn't work if the class uri is `http://..../` I raised
a PR on hadoop too,
https://github.com/apache/hadoop/pull/3338
But this is a regression on spark, as it used to work with a very specific
implementation for http based class servers and now since it uses the
Filesystem api for everything it doesn't anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
