Burak Yavuz created SPARK-11141:
-----------------------------------
Summary: Batching of ReceivedBlockTrackerLogEvents for efficient
WAL writes
Key: SPARK-11141
URL: https://issues.apache.org/jira/browse/SPARK-11141
Project: Spark
Issue Type: Improvement
Components: Streaming
Reporter: Burak Yavuz
When using S3 as a directory for WALs, the writes take too long. The driver
gets very easily bottlenecked when multiple receivers send AddBlock events to
the ReceiverTracker. This PR adds batching of events in the
ReceivedBlockTracker so that receivers don't get blocked by the driver for too
long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
