Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21913#discussion_r207525278
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala
---
@@ -313,9 +313,7 @@ private[streaming] object FileBasedWriteAheadLog {
val taskSupport = new ExecutionContextTaskSupport(executionContext)
val groupSize = taskSupport.parallelismLevel.max(8)
source.grouped(groupSize).flatMap { group =>
- val parallelCollection = group.par
- parallelCollection.tasksupport = taskSupport
- parallelCollection.map(handler)
+ ThreadUtils.parmap(group)(handler)(executionContext)
--- End diff --
Are there any specific reasons, or you think `parmap` is not reliable
enough?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
