[
https://issues.apache.org/jira/browse/SPARK-26302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718824#comment-16718824
]
Behroz Sikander commented on SPARK-26302:
-----------------------------------------
By code, i meant a warning in logs.
I will prepare a documentation commit.
> retainedBatches configuration can eat up memory on driver
> ---------------------------------------------------------
>
> Key: SPARK-26302
> URL: https://issues.apache.org/jira/browse/SPARK-26302
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, DStreams
> Affects Versions: 2.4.0
> Reporter: Behroz Sikander
> Priority: Minor
> Attachments: heap_dump_detail.png
>
>
> The documentation for configuration "spark.streaming.ui.retainedBatches" says
> "How many batches the Spark Streaming UI and status APIs remember before
> garbage collecting"
> The default for this configuration is 1000.
> From our experience, the documentation is incomplete and we found it the hard
> way.
> The size of a single BatchUIData is around 750KB. Increasing this value to
> something like 5000 increases the total size to ~4GB.
> If your driver heap is not big enough, the job starts to slow down, has
> frequent GCs and has long scheduling days. Once the heap is full, the job
> cannot be recovered.
> A note of caution should be added to the documentation to let users know the
> impact of this seemingly harmless configuration property.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
