Ngone51 commented on a change in pull request #29228:
URL: https://github.com/apache/spark/pull/29228#discussion_r474396405
##########
File path: core/src/test/scala/org/apache/spark/LocalSparkContext.scala
##########
@@ -22,12 +22,39 @@ import org.scalatest.BeforeAndAfterAll
import org.scalatest.BeforeAndAfterEach
import org.scalatest.Suite
+import org.apache.spark.internal.Logging
import org.apache.spark.resource.ResourceProfile
/** Manages a local `sc` `SparkContext` variable, correctly stopping it after
each test. */
-trait LocalSparkContext extends BeforeAndAfterEach with BeforeAndAfterAll {
self: Suite =>
+trait LocalSparkContext extends BeforeAndAfterEach
+ with BeforeAndAfterAll with Logging{ self: Suite =>
+
+ private var _conf: SparkConf = defaultSparkConf
@transient var sc: SparkContext = _
+ @transient private var _sc: SparkContext = _
+
+ def getSparkConf: SparkConf = _conf
+
+ /**
+ * Currently, we are focusing on the reconstruction of LocalSparkContext, so
this method
+ * was created temporarily. When the migration work is completed, this
method will be
+ * renamed to `sc` and the variable `sc` will be deleted.
+ */
+ def sparkCtx: SparkContext = {
+ if (_sc == null) {
+ _sc = new SparkContext(_conf)
+ }
+ _sc
+ }
+
+ def setConf(pairs: (String, String)*): Unit = {
Review comment:
`setConf` just sounds like an attribute setting function which needs to
invoke like `conf.setConf`. After we exposing the `conf`, I think it's more
straightforward to use it like `conf.setConf(...)` or `conf.setAll(...)`.
Here, I think what we want is a helper method, e.g., `withConf`. And `withXXX`
is more like a common pattern in Spark. We can use it to work with other helper
methods like, `withTempDir`, `withListener`, and others in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
