[
https://issues.apache.org/jira/browse/HDDS-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-6722:
-----------------------------------
Target Version/s: (was: 1.4.0)
> Memory leak after updating to RocksDB 7
> ---------------------------------------
>
> Key: HDDS-6722
> URL: https://issues.apache.org/jira/browse/HDDS-6722
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Wei-Chiu Chuang
> Assignee: Duong
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.3.0
>
> Attachments: om rocksdb crash.png
>
>
> After HDDS-6456 updated RocksDB version from 6.25.3 to 7.0.4, we started to
> experience OOM at Ozone Manager very quickly.
> In our test environment, the OM memory stayed at around 3GB. But after that,
> OM memory quickly increase to over 100GB and crashed within 2 hours.
> After some investigation and discussion with the RocksDB developers
> (https://github.com/facebook/rocksdb/issues/9962) we were able to isolate the
> issue to this particular breaking change:
> https://github.com/facebook/rocksdb/commit/99d86252b
> Prior to the change, RocksDB 6 automatically reclaims native memory when the
> corresponding Java object is garbage collected. After the change (RocksDB 7),
> applications must proactively close the objects unused.
> The biggest offenders are the RocksIterator. But there are many others. I'll
> file jira to clean up their usage. Unfortunately, SonarCloud doesn't seem to
> detect these unclosed objects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
