Repository: spark Updated Branches: refs/heads/master d7ae36a81 -> 075dd620e
[SPARK-25586][CORE] Remove outer objects from logdebug statements in ClosureCleaner ## What changes were proposed in this pull request? Cause: Recently test_glr_summary failed for PR of SPARK-25118, which enables spark-shell to run with default log level. It failed because this logdebug was called for GeneralizedLinearRegressionTrainingSummary which invoked its toString method, which started a Spark Job and ended up running into an infinite loop. Fix: Remove logDebug statement for outer objects as closures aren't implemented with outerclasses in Scala 2.12 and this debug statement looses its purpose ## How was this patch tested? Ran python pyspark-ml tests on top of PR for SPARK-25118 and ClosureCleaner unit tests Closes #22616 from ankuriitg/ankur/SPARK-25586. Authored-by: ankurgupta <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/075dd620 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/075dd620 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/075dd620 Branch: refs/heads/master Commit: 075dd620e32872b5d90a2fa7d09b43b15502182b Parents: d7ae36a Author: ankurgupta <[email protected]> Authored: Wed Oct 3 16:18:36 2018 -0700 Committer: Marcelo Vanzin <[email protected]> Committed: Wed Oct 3 16:18:36 2018 -0700 ---------------------------------------------------------------------- .../scala/org/apache/spark/util/ClosureCleaner.scala | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/075dd620/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala b/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala index 43d6256..6c4740c 100644 --- a/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala +++ b/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala @@ -285,8 +285,6 @@ private[spark] object ClosureCleaner extends Logging { innerClasses.foreach { c => logDebug(s" ${c.getName}") } logDebug(s" + outer classes: ${outerClasses.size}" ) outerClasses.foreach { c => logDebug(s" ${c.getName}") } - logDebug(s" + outer objects: ${outerObjects.size}") - outerObjects.foreach { o => logDebug(s" $o") } } // Fail fast if we detect return statements in closures @@ -318,19 +316,20 @@ private[spark] object ClosureCleaner extends Logging { if (outerPairs.nonEmpty) { val (outermostClass, outermostObject) = outerPairs.head if (isClosure(outermostClass)) { - logDebug(s" + outermost object is a closure, so we clone it: ${outerPairs.head}") + logDebug(s" + outermost object is a closure, so we clone it: ${outermostClass}") } else if (outermostClass.getName.startsWith("$line")) { // SPARK-14558: if the outermost object is a REPL line object, we should clone // and clean it as it may carray a lot of unnecessary information, // e.g. hadoop conf, spark conf, etc. - logDebug(s" + outermost object is a REPL line object, so we clone it: ${outerPairs.head}") + logDebug(s" + outermost object is a REPL line object, so we clone it:" + + s" ${outermostClass}") } else { // The closure is ultimately nested inside a class; keep the object of that // class without cloning it since we don't want to clone the user's objects. // Note that we still need to keep around the outermost object itself because // we need it to clone its child closure later (see below). - logDebug(" + outermost object is not a closure or REPL line object," + - "so do not clone it: " + outerPairs.head) + logDebug(s" + outermost object is not a closure or REPL line object," + + s" so do not clone it: ${outermostClass}") parent = outermostObject // e.g. SparkContext outerPairs = outerPairs.tail } @@ -341,7 +340,7 @@ private[spark] object ClosureCleaner extends Logging { // Clone the closure objects themselves, nulling out any fields that are not // used in the closure we're working on or any of its inner closures. for ((cls, obj) <- outerPairs) { - logDebug(s" + cloning the object $obj of class ${cls.getName}") + logDebug(s" + cloning instance of class ${cls.getName}") // We null out these unused references by cloning each object and then filling in all // required fields from the original object. We need the parent here because the Java // language specification requires the first constructor parameter of any closure to be @@ -351,7 +350,7 @@ private[spark] object ClosureCleaner extends Logging { // If transitive cleaning is enabled, we recursively clean any enclosing closure using // the already populated accessed fields map of the starting closure if (cleanTransitively && isClosure(clone.getClass)) { - logDebug(s" + cleaning cloned closure $clone recursively (${cls.getName})") + logDebug(s" + cleaning cloned closure recursively (${cls.getName})") // No need to check serializable here for the outer closures because we're // only interested in the serializability of the starting closure clean(clone, checkSerializable = false, cleanTransitively, accessedFields) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
