[
https://issues.apache.org/jira/browse/HUDI-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516189#comment-17516189
]
Alexey Kudinkin commented on HUDI-3774:
---------------------------------------
There's also very apparent double-buffering (note double-wrapped
BufferedFSInputStream): where each buffered input stream allocates buffer of
16Mb, bringing total footprint of single log-reader to 32Mb
!Screen Shot 2022-04-01 at 6.24.31 PM.png!
> Fix memory leaks through shutdown hooks
> ---------------------------------------
>
> Key: HUDI-3774
> URL: https://issues.apache.org/jira/browse/HUDI-3774
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Blocker
> Fix For: 0.11.0
>
> Attachments: Screen Shot 2022-04-01 at 11.58.14 AM.png, Screen Shot
> 2022-04-01 at 6.24.31 PM.png, image-2022-04-01-18-20-40-163.png
>
>
> Our tests occasionally crash w/ OOM in Azure.
> I re-run the tests locally under the same conditions trying to reproduce it
> and took a few heap dumps.
> I see that out of about 1.1Gb about 272Mb are the HoodieLogFileReaders
> retained t/h shutdown hooks. This means that those HoodieLogFileReader has
> not been properly closed (potentially under conditions that there were some
> exceptions thrown)
>
> To make sure that all readers get closed upon reaching their end of life,
> we'd use try-resource-statement wherever possible, which will guarantee
> proper closing for the reader under any possible control-flow
> {code:java}
> try (Reader r = new Reader) {
> // ...
> }
> {code}
>
> !Screen Shot 2022-04-01 at 11.58.14 AM.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
