mccheah commented on a change in pull request #25007:
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
URL: https://github.com/apache/spark/pull/25007#discussion_r305145209
##########
File path: core/src/test/scala/org/apache/spark/ShuffleSuite.scala
##########
@@ -383,13 +383,19 @@ abstract class ShuffleSuite extends SparkFunSuite with
Matchers with LocalSparkC
// simultaneously, and everything is still OK
def writeAndClose(
- writer: ShuffleWriter[Int, Int])(
+ writer: ShuffleWriter[Int, Int],
+ taskContext: TaskContext)(
iter: Iterator[(Int, Int)]): Option[MapStatus] = {
- val files = writer.write(iter)
- writer.stop(true)
+ TaskContext.setTaskContext(taskContext)
Review comment:
Basically I would prefer to minimize the API surface area as much as
possible. If we can consider `TaskContext.get()` as an implementation detail
and that the `TaskContext` isn't absolutely necessary for the API, I'd rather
not include it and only add it to the API in a follow-up. Especially since this
discussion is primarily centered around tests, which we have seen there are
other places in tests where `TaskContext.set` is used anyways 0- so we're not
going against existing precedent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
