Siyao Meng created HDDS-17272:
---------------------------------
Summary: Interrupted S3 PUT to an encrypted bucket fails but
commits an empty object with the full body's ETag, replacing the previous
version
Key: HDDS-17272
URL: https://issues.apache.org/jira/browse/HDDS-17272
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-flush-encrypted-s3-write-before-commit.patch,
TestBugMC4TdeInterruptedPutCommit.java
h3. Mechanism
For a bucket with an encryption key the S3 gateway writes through
{{OzoneOutputStream(CryptoOutputStream(KeyOutputStream))}}.
{{CryptoOutputStream}} keeps up to its buffer size (8 KiB) of data and writes
it to the {{KeyOutputStream}} only on flush or close. {{S3ObjectWriteGuard}}
counts the bytes {{CryptoOutputStream}} accepted, so after {{copyFrom}} it sees
the full length and no failure.
If the request thread has been interrupted, the write of the buffered data at
close is rejected by {{KeyOutputStreamSemaphore.acquire}} with
{{InterruptedIOException}} ("Write aborted. Interrupted waiting for
KeyOutputStream semaphore"). Unlike the other write failures of
{{KeyOutputStream}}, this one does not mark the stream closed. Hadoop's
{{CryptoOutputStream.close}} closes the inner stream even when its flush
failed, so {{KeyOutputStream.closeInternal}} runs the gateway's pre-commits,
which pass, and calls {{commitKey}} with the bytes it actually received. The
ETag of the full body is already in the key metadata. The commit RPC returns
"Call interrupted" to the gateway, but OM applies it.
This is a regression from HDDS-15193. Up to 2.1.x,
{{KeyOutputStream.closeInternal}} checked for S3 gateway writes
({{atomicKeyCreation}}) that the bytes it received match the size given at
open, which blocked this commit. HDDS-15193 moved that check into the
endpoints, where it counts the bytes accepted by the outermost stream.
h3. Trigger
# PUT an object to a bucket with an encryption key (any size), it succeeds.
# PUT a new version of the same key with a body smaller than 8 KiB. The request
thread is interrupted after the gateway read the last body byte and before the
close. Jetty's {{QueuedThreadPool.doStop}} interrupts pool threads that are
still busy after half of the stop timeout, so an S3 gateway shutdown is a
possible source. That was not run end to end; the reproduction sets the
interrupt flag at that point.
# The PUT fails with an error.
An interrupt that lands during the body read, the open key RPC or a datanode
write fails the request without a commit. With a 16,400 byte body (16,384 bytes
already in the {{KeyOutputStream}}) the close failed writing the chunk and
nothing was committed, so I expect only bodies that are still entirely in the
crypto buffer to be affected. Unencrypted buckets are not affected: the
interrupted write fails inside {{copyFrom}} and the guard blocks the commit.
h3. Impact
The client gets an error, yet the key now has size 0 and the ETag (md5) of the
full new body, and the previous version is gone. A reader sees an empty object
whose ETag claims other content. Nothing repairs it; a retry with
{{If-None-Match: *}} would be rejected because the key exists (not run).
h3. Reproduction
[^TestBugMC4TdeInterruptedPutCommit.java]
({{hadoop-ozone/integration-test-s3}}) runs one MiniOzoneCluster with a
{{JavaKeyStoreProvider}} as KMS and drives the real {{ObjectEndpoint.put}} and
Ozone client; only the HTTP request is mocked, and the request body sets the
interrupt flag of the request thread once its last byte is read. On the
unmodified source both tests pass, which means the defect is present: on the
encrypted bucket the failed PUT leaves {{dataSize=0}} with the ETag of the new
16 byte body, on the unencrypted bucket the previous 45 byte version stays.
h3. Patch
[^MC-4-flush-encrypted-s3-write-before-commit.patch], against
b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at
e0de543cbe9.
{{S3ObjectWriteGuard.close}} flushes the stream first when it wraps a
{{CryptoOutputStream}}, records a flush failure as the transfer failure and
then closes as before, so the existing pre-commit refuses the commit and the
{{KeyOutputStream}} only cleans up. Unencrypted writes are unchanged. Encrypted
writes do no extra work, because {{CryptoOutputStream.close}} flushes before
closing anyway; the flush just happens before the commit decision now.
Covered by {{encryptedWriteFailureOnCloseBlocksCommit}} and
{{encryptedWriteCommitsAllContent}} in the existing {{TestS3ObjectWriteGuard}},
using a real {{CryptoOutputStream}}. Without the change the first one fails
because the key is committed. With it all s3gateway unit tests (1620) pass and
checkstyle is clean. With the patch the reproduction's encrypted case fails,
because the previous version stays.
Found by TLA+ model checking and code review of the S3 gateway object put and
key commit paths under HDDS-15926, on commit
b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
