Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/19130#discussion_r138151499
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -367,6 +368,52 @@ object SparkSubmit extends CommandLineUtils with
Logging {
}.orNull
}
+ // Using a dummy http URI to check if HTTP(s) FileSystem is available,
it returns true in
+ // Hadoop 2.9+, otherwise it returns false.
+ val isHttpFsAvailable = Try {
FileSystem.get(Utils.resolveURI("http://foo/bar";), hadoopConf) }
--- End diff --
There is a Filesystem.getFileSystemClass function we could use here instead
of calling dummy uri
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
