[
https://issues.apache.org/jira/browse/HDDS-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan resolved HDDS-1486.
-------------------------------------
Resolution: Cannot Reproduce
> Ozone write fails in allocateBlock while writing >10MB files in multiple
> threads.
> ---------------------------------------------------------------------------------
>
> Key: HDDS-1486
> URL: https://issues.apache.org/jira/browse/HDDS-1486
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Aravindan Vijayan
> Priority: Major
> Labels: intermittent
> Attachments: Datanode Logs.zip
>
>
> 15 node physical cluster. All Datanodes are up and running.
> Client using 16 threads attempting to write 16000 x 10MB+ files using the
> FsStress utility
> (https://github.com/arp7/FsPerfTest) fails with the following error.
> This is an intermittent issue.
> *Server side exceptions*
> {code}
> 19/04/22 10:13:32 ERROR io.KeyOutputStream: Try to allocate more blocks for
> write failed, already allocated 0 blocks for this write.
> 19/04/18 14:33:23 WARN io.KeyOutputStream: Encountered exception
> java.io.IOException: Unexpected Storage Container Exception:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client
> client-ADE7F801D3AD->RAFT is closed.. The last committed block length is 0,
> uncommitted data length is 10485760 retry count 0
> {code}
> *Client side exceptions*
> {code}
> FAILED org.apache.ratis.protocol.NotLeaderException: Server
> c6e64cc4-91e9-4b36-83e4-6d84a4e71b7f is not the leader
> (f44c1413-0847-45e3-982d-ac3aec15dffc:10.17.200.23:9858). Request must be
> sent to leader., logIndex=0,
> commits[c6e64cc4-91e9-4b36-83e4-6d84a4e71b7f:c131161,
> 287eccfb-8461-419a-8732-529d042380b3:c131161,
> f44c1413-0847-45e3-982d-ac3aec15dffc:c131161]
> {code}
> In the case of small key sizes (<1MB) and big key sizes with single thread,
> the above client side exceptions are infrequent. However, in the case of
> multithreaded 10MB+ size keys, the exceptions occur about 50% of the time and
> eventually cause write failures. I have attached one such failed pipeline
> logs.
> [^Datanode Logs.zip]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
