[
https://issues.apache.org/jira/browse/HDDS-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17439318#comment-17439318
]
Tsz-wo Sze commented on HDDS-5946:
----------------------------------
[~micahzhao], thanks for testing it. You are right that we don't have a good
way to implement Retry without a bufferList. How about we make the size of the
buffer list configurable? However, we may need to block the client when the
buffer list is full.
{quote}a single client to write 400 files concurrently. and each file 128MB.
{quote}
This seems the worst case – writing many small files concurrently. For writing
larges files, the buffers will be released after the watch replied success.
> The 'Cannot allocate Memory 'problem occurs in client while test write
> -----------------------------------------------------------------------
>
> Key: HDDS-5946
> URL: https://issues.apache.org/jira/browse/HDDS-5946
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: mingchao zhao
> Priority: Major
> Attachments: screenshot-2.png, screenshot-3.png
>
>
> When I tested write performance with the new client, I got the following
> exception
> !screenshot-2.png!
> I found the problem started after we implemented retry in HDDS-5674. In the
> test, I used a single client to write 400 files concurrently. and each file
> 128MB. The client's memory will become much larger than before.
> !screenshot-3.png!
> The reason for this is that bufferList is used in Retry to maintain
> references to all byteBuffers currently being written. The client will hold
> those references to byteBuffer until ack. In the past, all Bytebuffers were
> released after they were sent, but now we need to wait for write successful
> before releasing. So now the client will accumulate more Bytebuffers and need
> more memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
