[
https://issues.apache.org/jira/browse/SPARK-12216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744675#comment-16744675
]
Kingsley Jones commented on SPARK-12216:
----------------------------------------
Okay, so I think we have a candidate for what is actually causing the problem.
There is an open bug on the scala language site for a class within the scala
REPL IMain.scala
[https://github.com/scala/bug/issues/10045]
scala.tools.nsc.interpreter.IMain.TranslatingClassLoader calls a
non-thread-safe method {{translateSimpleResource}} (this method calls
{{SymbolTable.enteringPhase}}), which makes it become non-thread-safe.
However, a ClassLoader must be thread-safe since the class can be loaded in
arbitrary thread.
In my REPL reflection experiment above the relevant class is:
{color:#333333}scala> val loader =
Thread.currentThread.getContextClassLoader(){color}
{color:#333333} loader: ClassLoader =
scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@3a1a20f{color}
{color:#333333}Ergo ... the very class in the above bug being marked as non
threadsafe.{color}
{color:#333333}See this stack overflow for a discussion of thread safety:{color}
{color:#333333}{color:#006000}https://stackoverflow.com/questions/46258558/scala-objects-and-thread-safety{color}{color}
The scala REPL code has some internal classloaders which are used to compile
and execute any code entered into the REPL.
On Windows 10, if you simply start a spark-shell from the command line, do
nothing, and then :quit the REPL will barf with a stacktrace to this particular
class reference (namely {color:#24292e}TranslatingClassLoader) which is
identified as an open bug in the scala language issues marked "non
threadsafe".{color}
{color:#24292e}I am gonna try and contact the person who raised the bug on the
scala issues thread and get some input.{color}
It seemed like he could only produce it with a complicated SQL script.
Here we have with Apache Spark a simple, and on my tests 100% reproducible,
instance of the bug on Windows 10 and in my tests Windows Server 2016.
That fits the perps modus operandi in my book... marked non threadsafe and
causes a sensitive operating system like Windows to barf.
> Spark failed to delete temp directory
> --------------------------------------
>
> Key: SPARK-12216
> URL: https://issues.apache.org/jira/browse/SPARK-12216
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Environment: windows 7 64 bit
> Spark 1.52
> Java 1.8.0.65
> PATH includes:
> C:\Users\Stefan\spark-1.5.2-bin-hadoop2.6\bin
> C:\ProgramData\Oracle\Java\javapath
> C:\Users\Stefan\scala\bin
> SYSTEM variables set are:
> JAVA_HOME=C:\Program Files\Java\jre1.8.0_65
> HADOOP_HOME=C:\Users\Stefan\hadoop-2.6.0\bin
> (where the bin\winutils resides)
> both \tmp and \tmp\hive have permissions
> drwxrwxrwx as detected by winutils ls
> Reporter: stefan
> Priority: Minor
>
> The mailing list archives have no obvious solution to this:
> scala> :q
> Stopping spark context.
> 15/12/08 16:24:22 ERROR ShutdownHookManager: Exception while deleting Spark
> temp dir:
> C:\Users\Stefan\AppData\Local\Temp\spark-18f2a418-e02f-458b-8325-60642868fdff
> java.io.IOException: Failed to delete:
> C:\Users\Stefan\AppData\Local\Temp\spark-18f2a418-e02f-458b-8325-60642868fdff
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:884)
> at
> org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:63)
> at
> org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:60)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at
> org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:60)
> at
> org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:264)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:234)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234)
> at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:234)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234)
> at scala.util.Try$.apply(Try.scala:161)
> at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:234)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
