Yun Tang created SPARK-17523:
--------------------------------
Summary: Cannot get Spark build info from spark-core package which
built in Windows
Key: SPARK-17523
URL: https://issues.apache.org/jira/browse/SPARK-17523
Project: Spark
Issue Type: Bug
Components: Build, Spark Core
Affects Versions: 2.0.0
Reporter: Yun Tang
Fix For: 2.0.1
Currently, if we build Spark, it will generate a
'spark-version-info.properties' and merged into spark-core_2.11-*.jar. However,
the script 'build/spark-build-info' which generates this file can only be
executed with bash environment.
Without this file, errors like below will happen when submitting Spark
application, which break the whole submitting phrase at beginning.
{code:java}
ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194)
at
org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:394)
at
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:247)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:759)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:757)
at
org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at
scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at
scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:648)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.spark.package$.<init>(package.scala:91)
at org.apache.spark.package$.<clinit>(package.scala)
at
org.apache.spark.SparkContext$$anonfun$3.apply(SparkContext.scala:187)
at
org.apache.spark.SparkContext$$anonfun$3.apply(SparkContext.scala:187)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
at org.apache.spark.SparkContext.logInfo(SparkContext.scala:76)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:187)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2287)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:822)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:814)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:814)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:630)
Caused by: org.apache.spark.SparkException: Error while locating file
spark-version-info.properties
at
org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:75)
at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:61)
at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala)
... 19 more
Caused by: java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:434)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at
org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:64)
... 21 more
{code}
We need to provide method to generate 'spark-version-info.properties' in
Windows environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
