[
https://issues.apache.org/jira/browse/SPARK-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacek Lewandowski updated SPARK-5425:
-------------------------------------
Description:
This fragment of code:
{code}
if (loadDefaults) {
// Load any spark.* system properties
for ((k, v) <- System.getProperties.asScala if k.startsWith("spark.")) {
settings(k) = v
}
}
{code}
causes
{noformat}
ERROR 09:43:15 SparkMaster service caused error in state
STARTINGjava.util.ConcurrentModificationException: null
at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
~[na:1.7.0_60]
at
scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:458)
~[scala-library-2.10.4.jar:na]
at
scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:454)
~[scala-library-2.10.4.jar:na]
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
~[scala-library-2.10.4.jar:na]
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
~[scala-library-2.10.4.jar:na]
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
~[scala-library-2.10.4.jar:na]
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
~[scala-library-2.10.4.jar:na]
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
~[scala-library-2.10.4.jar:na]
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
at org.apache.spark.SparkConf.<init>(SparkConf.scala:47)
~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
{noformat}
when there is another thread which modifies system properties at the same time.
This bug https://issues.scala-lang.org/browse/SI-7775 is somehow related to the
issue and shows that the problem has been also found elsewhere.
was:
This fragment of code:
{code}
if (loadDefaults) {
// Load any spark.* system properties
for ((k, v) <- System.getProperties.asScala if k.startsWith("spark.")) {
settings(k) = v
}
}
{code}
causes
{noformat}
ERROR 09:43:15 SparkMaster service caused error in state
STARTINGjava.util.ConcurrentModificationException: null
at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
~[na:1.7.0_60]
at
scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:458)
~[scala-library-2.10.4.jar:na]
at
scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:454)
~[scala-library-2.10.4.jar:na]
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
~[scala-library-2.10.4.jar:na]
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
~[scala-library-2.10.4.jar:na]
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
~[scala-library-2.10.4.jar:na]
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
~[scala-library-2.10.4.jar:na]
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
~[scala-library-2.10.4.jar:na]
at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
at org.apache.spark.SparkConf.<init>(SparkConf.scala:47)
~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
{noformat}
when there is another thread which modifies system properties at the same time.
The problem actually lies in Scala wrapper over Java Properties which uses
thread-unsafe access to properties when traversing them.
This bug https://issues.scala-lang.org/browse/SI-7775 is somehow related to the
issue and shows that the problem has been also found elsewhere.
> ConcurrentModificationException during SparkConf creation
> ---------------------------------------------------------
>
> Key: SPARK-5425
> URL: https://issues.apache.org/jira/browse/SPARK-5425
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.1, 1.2.0
> Reporter: Jacek Lewandowski
>
> This fragment of code:
> {code}
> if (loadDefaults) {
> // Load any spark.* system properties
> for ((k, v) <- System.getProperties.asScala if k.startsWith("spark.")) {
> settings(k) = v
> }
> }
> {code}
> causes
> {noformat}
> ERROR 09:43:15 SparkMaster service caused error in state
> STARTINGjava.util.ConcurrentModificationException: null
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
> ~[na:1.7.0_60]
> at
> scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:458)
> ~[scala-library-2.10.4.jar:na]
> at
> scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:454)
> ~[scala-library-2.10.4.jar:na]
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> ~[scala-library-2.10.4.jar:na]
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> ~[scala-library-2.10.4.jar:na]
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> ~[scala-library-2.10.4.jar:na]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> ~[scala-library-2.10.4.jar:na]
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> ~[scala-library-2.10.4.jar:na]
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> ~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:47)
> ~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
> {noformat}
> when there is another thread which modifies system properties at the same
> time.
> This bug https://issues.scala-lang.org/browse/SI-7775 is somehow related to
> the issue and shows that the problem has been also found elsewhere.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
