[
https://issues.apache.org/jira/browse/SPARK-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604935#comment-14604935
]
koert kuipers commented on SPARK-8577:
--------------------------------------
i do not have a way to reproduce this in non-test code. i do have a thread
dump but can not share all of it.
see below. both threads are blocked by ScalaReflectionLock.synchronized
Thread 2942: (state = BLOCKED)
-
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(org.apache.spark.sql.catalyst.ScalaReflection,
scala.reflect.api.Types$TypeApi) @bci=2611, line=140 (Interpret\
ed frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(scala.reflect.api.Types$TypeApi)
@bci=2, line=28 (Interpreted frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(scala.reflect.api.Symbols$SymbolApi)
@bci=21, line=123 (Interpreted frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(java.lang.Object)
@bci=5, line=121 (Interpreted frame)
- scala.collection.immutable.List.map(scala.Function1,
scala.collection.generic.CanBuildFrom) @bci=74, line=277 (Compiled frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(org.apache.spark.sql.catalyst.ScalaReflection,
scala.reflect.api.Types$TypeApi) @bci=1699, line=121 (Interpret\
ed frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(scala.reflect.api.Types$TypeApi)
@bci=2, line=28 (Interpreted frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(org.apache.spark.sql.catalyst.ScalaReflection,
scala.reflect.api.TypeTags$TypeTag) @bci=12, line=59 (Interpret\
ed frame)
-
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(scala.reflect.api.TypeTags$TypeTag)
@bci=2, line=28 (Interpreted frame)
Thread 2941: (state = BLOCKED)
- org.apache.spark.sql.types.AtomicType.<init>() @bci=11, line=95
(Interpreted frame)
- org.apache.spark.sql.types.NumericType.<init>() @bci=1, line=107
(Interpreted frame)
- org.apache.spark.sql.types.IntegralType.<init>() @bci=1, line=141
(Interpreted frame)
- org.apache.spark.sql.types.IntegerType.<init>() @bci=1, line=34
(Interpreted frame)
- org.apache.spark.sql.types.IntegerType$.<init>() @bci=1, line=54
(Interpreted frame)
- org.apache.spark.sql.types.IntegerType$.<clinit>() @bci=3 (Interpreted
frame)
> ScalaReflectionLock.synchronized can cause deadlock
> ---------------------------------------------------
>
> Key: SPARK-8577
> URL: https://issues.apache.org/jira/browse/SPARK-8577
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: koert kuipers
> Priority: Minor
>
> Just a heads up, i was doing some basic coding using DataFrame, Row,
> StructType, etc. in my own project and i ended up with deadlocks in my sbt
> tests due to the usage of ScalaReflectionLock.synchronized in the spark sql
> code.
> the issue went away when i changed my build to have:
> parallelExecution in Test := false
> so that the tests run consecutively...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
