[
https://issues.apache.org/jira/browse/SPARK-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-10407:
------------------------------
Component/s: Spark Core
> Possible Stack-overflow using InheritableThreadLocal nested-properties for
> SparkContext.localProperties
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-10407
> URL: https://issues.apache.org/jira/browse/SPARK-10407
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Matt Cheah
> Priority: Minor
>
> In my long-running web server that eventually uses a SparkContext, I
> eventually came across some stack overflow errors that could only be cleared
> by restarting my server.
> {code}
> java.lang.StackOverflowError: null
> at
> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2307)
> ~[na:1.7.0_45]
> at
> java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2718)
> ~[na:1.7.0_45]
> at
> java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2742)
> ~[na:1.7.0_45]
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1979)
> ~[na:1.7.0_45]
> at
> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500)
> ~[na:1.7.0_45]
> ...
> ...
> at
> org.apache.commons.lang3.SerializationUtils.clone(SerializationUtils.java:96)
> ~[commons-lang3-3.3.jar:3.3]
> at
> org.apache.spark.scheduler.DAGScheduler.submitJob(DAGScheduler.scala:516)
> ~[spark-core_2.10-1.4.1-palantir1.jar:1.4.1-palantir1]
> at
> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:529)
> ~[spark-core_2.10-1.4.1-palantir1.jar:1.4.1-palantir1]
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1770)
> ~[spark-core_2.10-1.4.1-palantir1.jar:1.4.1-palantir1]
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1788)
> ~[spark-core_2.10-1.4.1-palantir1.jar:1.4.1-palantir1]
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1803)
> ~[spark-core_2.10-1.4.1-palantir1.jar:1.4.1-palantir1]
> at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1276)
> ~[spark-core_2.10-1.4.1-palantir1.jar:1.4.1-palantir1]
> ...
> {code}
> The bottom of the trace indicates that serializing a properties object is
> part of the stack when the overflow happens. I checked the origin of the
> properties, and it turns out it's coming from SparkContext.localProperties,
> an InheritableThreadLocal field.
> When I debugged further, I found that localProperties.childValue() wraps its
> parent properties object in another properties object, and returns the
> wrapper properties. The problem is that every time childValue was being
> called, I was seeing the properties that was passed in from the parent have a
> deeper and deeper nesting of wrapped properties. This doesn't make any sense
> since my application doesn't create threads recursively or anything like
> that, so I'm marking this issue as a minor one since it shouldn't affect the
> average application.
> On the other hand, there shouldn't really be any reason to be creating the
> properties in childValue using nesting. Instead, the properties returned by
> childValue should be flattened, and more importantly, a deep copy of the
> parent.I'm also concerned about the parent thread possibly modifying the
> wrapped properties object while it's being used by the child thread, creating
> possible race conditions since Properties is not thread-safe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
