Hi, I've written a short scala app to perform word counts on a text file and am getting the following exception as the program completes (after it prints out all of the word counts).
Exception in thread "delete Spark temp dir C:\Users\Josh\AppData\Local\Temp\spark-0fdd0b79-7329-4690-a093-0fdb0d21e32c" java.io.IOException: Failed to delete: C:\Users\Josh\AppData\Local\Temp\spark-0fdd0b79-7329-4690-a093-0fdb0d21e32c\word-count_2.10-1.0.jar at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:692) at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:686) at org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:685) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:685) at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:281) A few details: Spark version 1.1.0 built from source on Windows 8. Scala version 2.11.4, JRE version 7 Update 71. Everything is local on a single machine. I can run the code in the scala shell just fine, this appears to be an issue with not being able to delete the temporary JAR file after running the application via spark-submit. I've checked the location of the JAR file and it is indeed left behind, so it is not being deleted. I guess my question is why is it unable to be deleted and is there anything I can do differently to fix this? I do not see this exception when I run the SimpleApp example taken from here: http://spark.apache.org/docs/latest/quick-start.html#standalone-applications Here is the code for my WordCount app that is producing this exception: import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkConf object WordCount { def main(args: Array[String]) { val filename = "data/rideshare.txt" val conf = new SparkConf().setAppName("Word Count").setMaster("local") val sc = new SparkContext(conf) val file = sc.textFile(filename, 2).cache() val wordCounts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b) wordCounts.collect().foreach(println) } } Thanks, Josh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deleting-temp-dir-Exception-tp18006.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org