Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21913#discussion_r206704050
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala
---
@@ -313,9 +313,7 @@ private[streaming] object FileBasedWriteAheadLog {
val taskSupport = new ExecutionContextTaskSupport(executionContext)
val groupSize = taskSupport.parallelismLevel.max(8)
source.grouped(groupSize).flatMap { group =>
- val parallelCollection = group.par
- parallelCollection.tasksupport = taskSupport
- parallelCollection.map(handler)
+ ThreadUtils.parmap(group)(handler)(executionContext)
--- End diff --
I would prefer to not change DStream codes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
