gaborgsomogyi commented on a change in pull request #23716: [SPARK-26734]
[STREAMING] Fix StackOverflowError with large block queue
URL: https://github.com/apache/spark/pull/23716#discussion_r253767186
##########
File path:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala
##########
@@ -111,8 +111,13 @@ private[streaming] class ReceivedBlockTracker(
*/
def allocateBlocksToBatch(batchTime: Time): Unit = synchronized {
if (lastAllocatedBatchTime == null || batchTime > lastAllocatedBatchTime) {
+ // We explicitly create an ArrayBuffer here because at least as of Scala
2.11 and 2.12
+ // a mutable.Queue fails serialization with a StackOverflow error if it
has more than
+ // a few thousand elements. So we explicitly allocate a collection for
serialization which
+ // we know doesn't have this issue. (See SPARK-26734).
val streamIdToBlocks = streamIds.map { streamId =>
- (streamId, getReceivedBlockQueue(streamId).clone())
+ (streamId,
+ mutable.ArrayBuffer(getReceivedBlockQueue(streamId).clone(): _*))
Review comment:
Nit: no line break required.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
