Ivan Andika created HDDS-8580:
---------------------------------
Summary: Reduce memory usage in ContainerKeyMapperTask#reprocess
Key: HDDS-8580
URL: https://issues.apache.org/jira/browse/HDDS-8580
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Recon
Affects Versions: 1.3.0
Reporter: Ivan Andika
Assignee: Ivan Andika
Fix For: 1.4.0
As OM performance test during Recon OM full snapshot, I removed the Recon DB
directory before restarting Recon to trigger full snapshot (essentially
bootstrapping a new Recon).
However, it is found after OM DB is successfully downloaded, during the
{{reprocess}} of {{{}ContainerKeyMapperTask{}}}, the Recon heap usage increased
significantly.
It is found that the issue was caused due to in-memory maps that store all the
OM keys during the reprocess. This is a regression introduced in
[HDDS-6783|https://jira.shopee.io/browse/HDDS-6783]. In essence, the patch is
to revert the implementation of
[HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] only for
{{{}ContainerKeyMapperTask#reprocess{}}}.
* {{ContainerKeyMapperTask#reprocess}} should not increase the heap memory
significantly since the number of delta updates are already limited by the
Recon configurations
* [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] aims for atomicity
during the Recon OM task updates. However since
{{ContainerKeyMapperTask#reprocess}} truncate all the Recon Container DB before
it starts and rebuilt the Recon Container DB, I think it's acceptable.
After the patch is applied, the Recon heap size stays stable during the full
snapshot.
Attached is Recon heap memory before and after the patch.
!https://ia.s3.sto.shopee.io/gitlabuploads-s3ia-sg-live/%40hashed/a3/58/a3586514e477d5e1700d579d73a073faeb99dd736ec62ed96c5992bd85027c9c/eec403a69b9974bdf465ed10101e4ab6/image.png?response-content-disposition=inline%3B%20filename%3D%22image.png%22%3B%20filename%2A%3DUTF-8%27%27image.png&response-content-type=image%2Fpng&X-Amz-Expires=600&X-Amz-Date=20230510T071733Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=D9J17WWO57F3HBGQT1DU%2F20230510%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5a0af4e7c060683ce514bdee9998b754b15abd1d9c3f5701bd5e11084bf063ca|width=562,height=429!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
