Mukul Murthy created SPARK-24525: ------------------------------------ Summary: Provide an option to limit MemorySink memory usage Key: SPARK-24525 URL: https://issues.apache.org/jira/browse/SPARK-24525 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 2.3.1 Reporter: Mukul Murthy
MemorySink stores stream results in memory and is mostly used for testing and displaying streams, but for large streams, this can OOM the driver. We should add an option to limit the number of rows and the total size of a memory sink and not add any new data once either limit is hit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org