François Garillot created SPARK-7398:
----------------------------------------
Summary: Add back-pressure to Spark Streaming
Key: SPARK-7398
URL: https://issues.apache.org/jira/browse/SPARK-7398
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: François Garillot
Spark Streaming has trouble dealing with situations where
batch processing time > batch interval
Meaning a high throughput of input data w.r.t. Spark's ability to remove data
from the queue.
If this throughput is sustained for long enough, it leads to an unstable
situation where the memory of the Receiver's Executor is overflowed. This aims
at transmitting a back-pressure signal back to data ingestion to help with
dealing with that high throughput, in a backwards-compatible way.
The design doc can be found here:
https://docs.google.com/document/d/15m8N717n4hwFya_pJHUbiv5s50Mq1SZZzL9ftOhms4E/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
