Sumit Agrawal created HDDS-8329:
-----------------------------------
Summary: Performance Impact during container close when still
Client is writting
Key: HDDS-8329
URL: https://issues.apache.org/jira/browse/HDDS-8329
Project: Apache Ozone
Issue Type: Bug
Reporter: Sumit Agrawal
Assignee: Sumit Agrawal
Fix For: 1.4.0
Steps for container closure:
# SCM allocates blocks to client for write
# Client perform write to blocks at DN pipeline
# DN when identify container is about to full (default 90% of 5GB container
size)
# Then DN send notification to SCM for close
# SCM move state from Open to closing and send notification to DN to close
# At DN on success close, notify SCM for closure
During Step 5, If client still writing to DN for blocks allocated before SCM
receive close, those write will be rejected when DN close container.
This causes retry for client to allocate new block and retry write.
As solution,
* SCM will move state to closing, so that it will not allocate new blocks to
client
* SCM will wait for a configured time before sending to DN close (as a buffer
time), where Client can perform write.
This buffer time is configurable and initially set as "2.5 minutes" which is 5
times DN notification interval (i.e. 30 sec * 5).
Problem not intended to solve:
* over subscription and container size - This is best approach to complete the
write from client within configured time. Container size can cross the 5GB
default limit, which does not much impact to system.
* Slow client writer - If client is writing slowly, it may cross buffer time
and can fail. This solution will provide optimal time as configured and will
not handle slow client writer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
