Siyao Meng created HDDS-16981:
---------------------------------
Summary: Ratis key write waits for its own retry handling when the
data written on retry fills the replacement block
Key: HDDS-16981
URL: https://issues.apache.org/jira/browse/HDDS-16981
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch
h3. Mechanism
Since HDDS-11239 (2.0.0) the block that {{KeyOutputStream.handleRetry}}
allocates to take over the unacknowledged data of a failed block is created in
retry handling mode ({{BlockOutputStreamEntry.isHandlingRetry}}). Calls that
find the flag set wait in {{waitForRetryHandling}} until
{{KeyOutputStream.handleException}} has finished the handling of the failed
block and calls {{finishRetryHandling}}.
{{handleRetry}} writes the data again through {{handleWrite(retry=true)}}. When
that data fills the new block, {{handleWrite}} calls
{{handleFlushOrClose(StreamAction.FULL)}} to close it. {{handleFlushOrClose}}
waits for the retry handling of the current block like any other call, but here
the caller is the retry handling, so the thread waits for a signal that only it
can send.
The Javadoc of {{handleFlushOrClose}} assumes that a block written on retry
"will be at max half full". That holds when the block is at least twice
{{ozone.client.stream.buffer.max.size}}, and nothing enforces it.
h3. Trigger
# {{ozone.scm.block.size}} at or below {{ozone.client.stream.buffer.max.size}}.
The defaults are 256MB and 32MB, so the default block size is not affected.
Nothing validates the two settings against each other.
# A block of a Ratis key fails while a full block of its data is
unacknowledged, and a new block is allocated for the retry. If a block
preallocated at key creation takes the retry, it is not in retry handling mode
and the write returns.
Run on unmodified source with mock datanodes and default buffer sizes (blocks
of 1MB, 4MB, 16MB and 32MB wait, blocks of 64MB and 256MB return), and on a
mini cluster with a block size of 1MB and no failure injected, where SCM closed
the container of the first block before the block was sent.
h3. Impact
* {{write}} does not return. There is no exception and no timeout, and only an
interrupt ends the wait. The stream is closed then and the key is not committed.
* No committed data is affected.
* Workaround: a block size above {{ozone.client.stream.buffer.max.size}}.
h3. Reproduction
No standalone reproduction is attached to this issue. The regression test in
the patch, {{TestOzoneClient.testPutKeyRatisWhenRetriedDataFillsBlock}}, covers
it with mock datanodes: without the change five of its six cases run into their
timeout, the sixth is a control with a half full block.
h3. Patch
[^CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch], against
031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master at
d0454e0cffd.
{{KeyOutputStream.handleWrite}} no longer issues the {{FULL}} action when the
write is the rewrite of a retry. The full block stays the current block and is
closed once the handling of the failed block is finished, through the normal
wait, by the action that was being retried, by the next {{write}} or by
{{close}}.
The change takes effect only when the data written again fills the block.
Besides the case above that is a retry that lands on a preallocated block,
which does not wait today: there the block is closed one step later with the
same result (run with mock datanodes and on the mini cluster, with and without
the patch: same block lengths, key reads back).
Skipping the wait for {{FULL}} would also end the wait, but it is not safe with
concurrent calls ({{ozone.client.key.write.concurrency}} other than 1): the
writer closes the retry block and moves to a next block while the handoff of
the failed block is still pending. In a mock datanode run of that variant with
a pending {{hsync}}, {{close}} failed with "Expected offset: 2560 expected len:
3072". With the attached patch the same run returns for both calls and the key
reads back.
The regression test checks the committed block lengths and reads the data back,
for a key of exactly one block and a key of 2.5 blocks, with the failure
noticed by the close of the full block or by the write that fills it, and with
the replacement block failing as well. With the patch all tests of
{{hadoop-ozone/client}} (165) pass, checkstyle is clean, and on a mini cluster
{{TestHSync}} (42), {{TestBlockOutputStreamWithFailures}} (20),
{{TestFailureHandlingByClient}} (6), {{TestFailureHandlingByClientFlushDelay}}
(1), {{TestOzoneClientRetriesOnExceptions}} (2),
{{TestOzoneClientRetriesOnExceptionFlushDelay}} (1) and
{{TestMultiBlockWritesWithDnFailures}} (2) pass.
The patch is independent of HDDS-16979 and HDDS-16980 and applies before or
after their patches. It and the patch of HDDS-16975 both add imports and a test
to {{TestOzoneClient}}, so the one merged second needs a rebase of that file.
Found by code review of the Ratis key write path while validating fixes from
the TLA+ verification effort under HDDS-15926, on commit
031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
