Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21428#discussion_r194905420
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/RPCContinuousShuffleReader.scala
---
@@ -68,7 +66,7 @@ private[shuffle] class UnsafeRowReceiver(
}
override def receiveAndReply(context: RpcCallContext):
PartialFunction[Any, Unit] = {
- case r: UnsafeRowReceiverMessage =>
+ case r: RPCContinuousShuffleMessage =>
queues(r.writerId).put(r)
--- End diff --
This line may block PRC threads and cause some critical RPC messages
delayed. In addition, if the reader fails, this line may block forever if the
queue is full.
I'm okey with this right now since it's an experimental feature. Could you
create a SPARK ticket and add a TODO here to comment the potential issue so
that we won't forget this issue?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
