Siyao Meng created HDDS-16972:
---------------------------------
Summary: EC key write fails with NullPointerException instead of
retrying the stripe when a datanode client cannot be acquired for a new block
group
Key: HDDS-16972
URL: https://issues.apache.org/jira/browse/HDDS-16972
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-2-retry-ec-stripe-when-block-stream-creation-fails.patch,
TestBugCR2EcStreamSetupFailure.java
h3. Mechanism
{{ECBlockOutputStreamEntry.checkStream}} creates one {{ECBlockOutputStream}}
per node when the first cell is written to a block group, and each constructor
calls {{XceiverClientFactory.acquireClient}}. If that throws for one node, the
failure is recorded nowhere. The half built array is dropped without releasing
the clients already acquired, and {{ECKeyOutputStream.handleOutputStreamWrite}}
calls {{markFailed}}, which does nothing while the entry is uninitialized. The
next cell calls {{checkStream}} again and the loop starts at
{{currentStreamIdx}}, so the entry is initialized with the streams of the later
nodes only and the earlier slots stay null. {{streamsWithWriteFailure}} skips
null slots and reports no failure. {{commitStripeWrite}} then calls
{{calculateChecksum}}, which dereferences slot 0 and throws a
{{NullPointerException}} before the stripe can be rolled back and retried. If
the failing node is the last one of the block group, no array is ever assigned
and the same call throws {{IOException: Block Output Stream is null}}.
The flush thread ends with that exception, {{close}} (or the next {{write}})
rethrows it and the key is not committed, although the other nodes are healthy
and no stripe retry was used. When the same node fails one step later, at
WriteChunk, the stripe is rolled back, the pipeline and the node are excluded
and the stripe is written to a new block group.
h3. Trigger
# An EC write starts a new block group (the first stripe of a key or of a later
block group).
# {{acquireClient}} throws for one node of the group.
With {{XceiverClientManager}} step 2 means that creating the gRPC channel
failed. A datanode that is down or whose name does not resolve does not trigger
it: the channel connects lazily and the error shows at WriteChunk, which is
handled.
The trigger I ran with the real client needs
{{hdds.datanode.use.datanode.hostname=true}} (default false) and a datanode
whose hostname is not a valid URI host, for example one containing an
underscore. {{NettyChannelBuilder.forAddress}} rejects it
({{IllegalArgumentException: Invalid host or port}}), while SCM registers the
datanode as healthy and places it in EC pipelines.
In the default configuration the client connects to the IP address SCM recorded
for the datanode, and I found no node specific trigger there. Run against the
real {{XceiverClientManager}}: a closed manager, a client evicted from the
cache while referenced, an unresolvable host and an out of range port do not
make {{acquireClient}} throw. A local failure while the channel is built (for
example file descriptor exhaustion) would be a trigger in the default
configuration; this is argued from the code and not run.
h3. Impact
* The write fails with a {{NullPointerException}} or {{Block Output Stream is
null}} and the key is not committed. Committed data is not affected. On a mini
cluster with 7 datanodes, one of them with the hostname {{dn_1}}, 5 of 12
writes of a one stripe RS-3-2 key failed this way (the number varies with
pipeline placement).
* Datanode clients stay acquired. Every cell of the stripe repeats
{{checkStream}} and leaves the clients it acquired before the failing node
unreleased. Counted with a client factory that records acquire and release
calls, after {{close}}: 3 clients for a failing third node and 10 for a failing
fifth node, per stripe. {{XceiverClientManager}} closes a client only when its
reference count is zero, so with the real manager those channels would stay
open (argued from {{XceiverClientSpi.cleanup}}, not run).
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR2EcStreamSetupFailure.java]
({{hadoop-ozone/client}}, about one second) drives the unmodified write path
through {{OzoneBucket.createKey}} and {{OzoneOutputStream}} with the test
doubles of {{TestOzoneECClient}}. A subclass of {{MockXceiverClientFactory}}
makes {{acquireClient}} throw for chosen calls and counts acquired and released
clients. It writes one RS-3-2 stripe with the client of node 0, 2 or 4 of the
block group failing, and with a single failed call. In each case the write
fails as described, only the block allocation of {{createKey}} is made, the key
is not committed, and the counted clients remain. A control with node 0 failing
at WriteChunk commits the key in a second block group. A passing test means the
defect is present.
Output of the three cases with one failing node (node 0, 2 and 4, in this
order):
{noformat}
caller saw: java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.hdds.scm.storage.ECBlockOutputStream.getContainerBlockData()"
because "this.blockOutputStreams[0]" is null, acquireClient calls: 5, clients
never released: 0, block allocations: 1, key commits: 0
caller saw: java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.hdds.scm.storage.ECBlockOutputStream.getContainerBlockData()"
because "this.blockOutputStreams[0]" is null, acquireClient calls: 8, clients
never released: 3, block allocations: 1, key commits: 0
caller saw: java.io.IOException: Block Output Stream is null, acquireClient
calls: 15, clients never released: 10, block allocations: 1, key commits: 0
{noformat}
The mini cluster run mentioned under Impact used the real
{{XceiverClientManager}} and real datanodes. It is not attached.
h3. Patch
[^CR-2-retry-ec-stripe-when-block-stream-creation-fails.patch], against
031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master at
c5d4f28266d.
{{checkStream}} builds the streams from index 0 and, when one cannot be
created, records the node, logs the cause once, closes the streams built so far
so that their clients are released, and rethrows. The entry then counts as
closed and is not initialized later. {{commitStripeWrite}} checks for the
recorded node first, excludes the pipeline and the node and returns {{FAILED}},
so the stripe takes the existing rollback and retry path, as for a failed
WriteChunk. The nearest earlier change is HDDS-9659 (fixed in 1.4.0): a comment
there shows the same {{NullPointerException}} in {{calculateChecksum}} and
notes that it may also happen if the stream with index 0 is not created during
a retry. Its fix, assigning the array only when complete, is kept. A failure
caused by the stream closing (HDDS-7931) does not start a retry.
One visible change: if no datanode client can be created at all, the write now
fails after {{ozone.client.max.ec.stripe.write.retries}} further block
allocations with "Completed max allowed retries N on stripe failures." instead
of at once with "Block Output Stream is null", the same as when every
WriteChunk fails. Each attempt excludes one more datanode, so on a small
cluster the write fails earlier, with the block allocation error, as soon as
fewer than d+p datanodes are left outside the exclude list (with the mock block
allocator, RS-3-2 and the default 10 retries need 15 datanodes to reach the
retry limit; not run against SCM).
Covered by {{testStripeWriteRetriesOnClientAcquireFailure}} (failing node 0, 2
and 4) and {{testStripeWriteRetriesAreLimitedOnClientAcquireFailure}} in the
existing {{TestOzoneECClient}}. Without the change they fail with the two
exceptions above. With it the 18 test classes of {{hadoop-ozone/client}} (163
tests) and {{TestECKeyOutputStream}} on a mini cluster (14 tests) pass and
checkstyle is clean. On the mini cluster with the {{dn_1}} datanode all 12
writes commit and read back. With the patch the failure cases of the
reproduction fail, because the write succeeds; the control still passes.
Found by code review of the client EC key write and file checksum paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
