MaxGekk commented on a change in pull request #25951: [SPARK-28917][CORE] Synchronize access to RDD mutable state. URL: https://github.com/apache/spark/pull/25951#discussion_r372812088
########## File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala ########## @@ -225,10 +225,24 @@ abstract class RDD[T: ClassTag]( /** Get the RDD's current storage level, or StorageLevel.NONE if none is set. */ def getStorageLevel: StorageLevel = storageLevel + /** + * Lock for all mutable state of this RDD (persistence, partitions, dependencies, etc.). We do + * not use `this` because RDDs are user-visible, so users might have added their own locking on + * RDDs; sharing that could lead to a deadlock. + * + * One thread might hold the lock on many of these, for a chain of RDD dependencies; but + * because DAGs are acyclic, and we only ever hold locks for one path in that DAG, there is no + * chance of deadlock. + * + * The use of Integer is simply so this is serializable -- executors may reference the shared + * fields (though they should never mutate them, that only happens on the driver). + */ + private val stateLock = new Integer(0) Review comment: I tried to eliminate the warning in https://github.com/apache/spark/pull/27399 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
