[
https://issues.apache.org/jira/browse/HDDS-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385409#comment-17385409
]
Janus Chow commented on HDDS-5472:
----------------------------------
Sure. [~adoroszlai] thanks for the review.
The user use the following code to write files, and the function is triggered
many times.
{code:java}
def writeFile(fileName: Path, content: String): Unit = {
val bytes = content.getBytes()
try {
val writeStream = ozoneBucket.createKey(fileName.toString, bytes.size)
writeStream.write(bytes)
writeStream.close()
}...
}{code}
I think it should be easy to reproduce to write the same file many times and
then check the file status using the following code.
{code:java}
OzoneFileStatus fileStatus = ozoneBucket.getFileStatus("keyname");
{code}
Since in https://issues.apache.org/jira/browse/HDDS-5243 we return the latest
version location only for mostly read operation, in this case, the OOM is
caused by user writing keys, so it isn't covered in HDDS-5243.
> Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
> -------------------------------------------------------------------
>
> Key: HDDS-5472
> URL: https://issues.apache.org/jira/browse/HDDS-5472
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Labels: pull-request-available
>
> Currently the keyLocation of different versions is stored redundantly in
> OmKeyInfo.
> * OmKeyInfo:
> ** keyLocationVersions: List<OmKeyLocationInfoGroup> stores different
> versions of information in a list.
> * OmKeyLocationInfoGroup:
> ** locationVersionMap: Map<Long, List<OmKeyLocationInfo>> stores different
> versions of location in a map.
> If the versions are large, the redundent location is causing the large GC
> overhead for OM, in our cluster, the OM even crashes because of OOM.
> This ticket is to remove the redundant location information to keep OM
> healthy.
> Attached the OmKeyInfo of a key with 549 versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
