[
https://issues.apache.org/jira/browse/HDDS-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16981:
------------------------------
Attachment:
CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch
> Ratis key write waits for its own retry handling when the data written on
> retry fills the replacement block
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16981
> URL: https://issues.apache.org/jira/browse/HDDS-16981
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch
>
>
> h3. Mechanism
> Since HDDS-11239 (2.0.0) the block that {{KeyOutputStream.handleRetry}}
> allocates to take over the unacknowledged data of a failed block is created
> in retry handling mode ({{BlockOutputStreamEntry.isHandlingRetry}}). Calls
> that find the flag set wait in {{waitForRetryHandling}} until
> {{KeyOutputStream.handleException}} has finished the handling of the failed
> block and calls {{finishRetryHandling}}.
> {{handleRetry}} writes the data again through {{handleWrite(retry=true)}}.
> When that data fills the new block, {{handleWrite}} calls
> {{handleFlushOrClose(StreamAction.FULL)}} to close it. {{handleFlushOrClose}}
> waits for the retry handling of the current block like any other call, but
> here the caller is the retry handling, so the thread waits for a signal that
> only it can send.
> The Javadoc of {{handleFlushOrClose}} assumes that a block written on retry
> "will be at max half full". That holds when the block is at least twice
> {{ozone.client.stream.buffer.max.size}}, and nothing enforces it.
> h3. Trigger
> # {{ozone.scm.block.size}} at or below
> {{ozone.client.stream.buffer.max.size}}. The defaults are 256MB and 32MB, so
> the default block size is not affected. Nothing validates the two settings
> against each other.
> # A block of a Ratis key fails while a full block of its data is
> unacknowledged, and a new block is allocated for the retry. If a block
> preallocated at key creation takes the retry, it is not in retry handling
> mode and the write returns.
> Run on unmodified source with mock datanodes and default buffer sizes (blocks
> of 1MB, 4MB, 16MB and 32MB wait, blocks of 64MB and 256MB return), and on a
> mini cluster with a block size of 1MB and no failure injected, where SCM
> closed the container of the first block before the block was sent.
> h3. Impact
> * {{write}} does not return. There is no exception and no timeout, and only
> an interrupt ends the wait. The stream is closed then and the key is not
> committed.
> * No committed data is affected.
> * Workaround: a block size above {{ozone.client.stream.buffer.max.size}}.
> h3. Reproduction
> No standalone reproduction is attached to this issue. The regression test in
> the patch, {{TestOzoneClient.testPutKeyRatisWhenRetriedDataFillsBlock}},
> covers it with mock datanodes: without the change five of its six cases run
> into their timeout, the sixth is a control with a half full block.
> h3. Patch
> [^CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch],
> against 031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master
> at d0454e0cffd.
> {{KeyOutputStream.handleWrite}} no longer issues the {{FULL}} action when the
> write is the rewrite of a retry. The full block stays the current block and
> is closed once the handling of the failed block is finished, through the
> normal wait, by the action that was being retried, by the next {{write}} or
> by {{close}}.
> The change takes effect only when the data written again fills the block.
> Besides the case above that is a retry that lands on a preallocated block,
> which does not wait today: there the block is closed one step later with the
> same result (run with mock datanodes and on the mini cluster, with and
> without the patch: same block lengths, key reads back).
> Skipping the wait for {{FULL}} would also end the wait, but it is not safe
> with concurrent calls ({{ozone.client.key.write.concurrency}} other than 1):
> the writer closes the retry block and moves to a next block while the handoff
> of the failed block is still pending. In a mock datanode run of that variant
> with a pending {{hsync}}, {{close}} failed with "Expected offset: 2560
> expected len: 3072". With the attached patch the same run returns for both
> calls and the key reads back.
> The regression test checks the committed block lengths and reads the data
> back, for a key of exactly one block and a key of 2.5 blocks, with the
> failure noticed by the close of the full block or by the write that fills it,
> and with the replacement block failing as well. With the patch all tests of
> {{hadoop-ozone/client}} (165) pass, checkstyle is clean, and on a mini
> cluster {{TestHSync}} (42), {{TestBlockOutputStreamWithFailures}} (20),
> {{TestFailureHandlingByClient}} (6),
> {{TestFailureHandlingByClientFlushDelay}} (1),
> {{TestOzoneClientRetriesOnExceptions}} (2),
> {{TestOzoneClientRetriesOnExceptionFlushDelay}} (1) and
> {{TestMultiBlockWritesWithDnFailures}} (2) pass.
> The patch is independent of HDDS-16979 and HDDS-16980 and applies before or
> after their patches. It and the patch of HDDS-16975 both add imports and a
> test to {{TestOzoneClient}}, so the one merged second needs a rebase of that
> file.
> Found by code review of the Ratis key write path while validating fixes from
> the TLA+ verification effort under HDDS-15926, on commit
> 031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
