igozali commented on a change in pull request #25910: [SPARK-28762][CORE] Read
JAR main class if JAR is not located in local file system
URL: https://github.com/apache/spark/pull/25910#discussion_r329149462
##########
File path:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
##########
@@ -216,20 +218,27 @@ private[deploy] class SparkSubmitArguments(args:
Seq[String], env: Map[String, S
val uri = new URI(primaryResource)
val uriScheme = uri.getScheme()
- uriScheme match {
- case "file" =>
- try {
+ try {
+ uriScheme match {
+ case "file" =>
Utils.tryWithResource(new JarFile(uri.getPath)) { jar =>
// Note that this might still return null if no main-class is
set; we catch that later
mainClass =
jar.getManifest.getMainAttributes.getValue("Main-Class")
}
- } catch {
- case _: Exception =>
- error(s"Cannot load main class from JAR $primaryResource")
- }
- case _ =>
+ case _ =>
+ val sparkConf = new SparkConf()
+ sparkConf.setAll(sparkProperties)
+ val fs = FileSystem.get(uri,
SparkHadoopUtil.newConfiguration(sparkConf))
+
+ Utils.tryWithResource(new JarInputStream(fs.open(new
Path(primaryResource)))) { jar =>
Review comment:
Thanks, that's really valuable input. I'm not a big fan of my own approach
of having Hadoop classes being used in `SparkSubmitArguments` anyway. I'll take
a look into this, but probably won't be able to get into it until this weekend.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
