[
https://issues.apache.org/jira/browse/HDDS-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440192#comment-17440192
]
mingchao zhao edited comment on HDDS-5946 at 11/8/21, 6:31 AM:
---------------------------------------------------------------
bq. How about we make the size of the buffer list configurable? However, we may
need to block the client when the buffer list is full.
[~szetszwo] Yes, that's what I can think of so far. This very similar to what
we did in Async Write. This will reduce the memory of client。 I'm not sure if
it affects performance, so let me implement it and verify.
was (Author: micahzhao):
bq. How about we make the size of the buffer list configurable? However, we may
need to block the client when the buffer list is full.
{quote}How about we make the size of the buffer list configurable? However, we
may need to block the client when the buffer list is full.{quote}
> The 'Cannot allocate Memory 'problem occurs in client while test write
> -----------------------------------------------------------------------
>
> Key: HDDS-5946
> URL: https://issues.apache.org/jira/browse/HDDS-5946
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: mingchao zhao
> Assignee: mingchao zhao
> Priority: Major
> Attachments: screenshot-2.png, screenshot-3.png
>
>
> When I tested write performance with the new client, I got the following
> exception
> !screenshot-2.png!
> I found the problem started after we implemented retry in HDDS-5674. In the
> test, I used a single client to write 400 files concurrently. and each file
> 128MB. The client's memory will become much larger than before.
> !screenshot-3.png!
> The reason for this is that bufferList is used in Retry to maintain
> references to all byteBuffers currently being written. The client will hold
> those references to byteBuffer until ack. In the past, all Bytebuffers were
> released after they were sent, but now we need to wait for write successful
> before releasing. So now the client will accumulate more Bytebuffers and need
> more memory.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
