[
https://issues.apache.org/jira/browse/HDDS-17272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17272:
------------------------------
Affects Version/s: 2.2.1
2.2.0
> Interrupted S3 PUT to an encrypted bucket fails but commits an empty object
> with the full body's ETag, replacing the previous version
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17272
> URL: https://issues.apache.org/jira/browse/HDDS-17272
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-4-flush-encrypted-s3-write-before-commit.patch,
> TestBugMC4TdeInterruptedPutCommit.java
>
>
> h3. Mechanism
> For a bucket with an encryption key the S3 gateway writes through
> {{OzoneOutputStream(CryptoOutputStream(KeyOutputStream))}}.
> {{CryptoOutputStream}} keeps up to its buffer size (8 KiB) of data and writes
> it to the {{KeyOutputStream}} only on flush or close. {{S3ObjectWriteGuard}}
> counts the bytes {{CryptoOutputStream}} accepted, so after {{copyFrom}} it
> sees the full length and no failure.
> If the request thread has been interrupted, the write of the buffered data at
> close is rejected by {{KeyOutputStreamSemaphore.acquire}} with
> {{InterruptedIOException}} ("Write aborted. Interrupted waiting for
> KeyOutputStream semaphore"). Unlike the other write failures of
> {{KeyOutputStream}}, this one does not mark the stream closed. Hadoop's
> {{CryptoOutputStream.close}} closes the inner stream even when its flush
> failed, so {{KeyOutputStream.closeInternal}} runs the gateway's pre-commits,
> which pass, and calls {{commitKey}} with the bytes it actually received. The
> ETag of the full body is already in the key metadata. The commit RPC returns
> "Call interrupted" to the gateway, but OM applies it.
> This is a regression from HDDS-15193. Up to 2.1.x,
> {{KeyOutputStream.closeInternal}} checked for S3 gateway writes
> ({{atomicKeyCreation}}) that the bytes it received match the size given at
> open, which blocked this commit. HDDS-15193 moved that check into the
> endpoints, where it counts the bytes accepted by the outermost stream.
> h3. Trigger
> # PUT an object to a bucket with an encryption key (any size), it succeeds.
> # PUT a new version of the same key with a body smaller than 8 KiB. The
> request thread is interrupted after the gateway read the last body byte and
> before the close. Jetty's {{QueuedThreadPool.doStop}} interrupts pool threads
> that are still busy after half of the stop timeout, so an S3 gateway shutdown
> is a possible source. That was not run end to end; the reproduction sets the
> interrupt flag at that point.
> # The PUT fails with an error.
> An interrupt that lands during the body read, the open key RPC or a datanode
> write fails the request without a commit. With a 16,400 byte body (16,384
> bytes already in the {{KeyOutputStream}}) the close failed writing the chunk
> and nothing was committed, so I expect only bodies that are still entirely in
> the crypto buffer to be affected. Unencrypted buckets are not affected: the
> interrupted write fails inside {{copyFrom}} and the guard blocks the commit.
> h3. Impact
> The client gets an error, yet the key now has size 0 and the ETag (md5) of
> the full new body, and the previous version is gone. A reader sees an empty
> object whose ETag claims other content. Nothing repairs it; a retry with
> {{If-None-Match: *}} would be rejected because the key exists (not run).
> h3. Reproduction
> [^TestBugMC4TdeInterruptedPutCommit.java]
> ({{hadoop-ozone/integration-test-s3}}) runs one MiniOzoneCluster with a
> {{JavaKeyStoreProvider}} as KMS and drives the real {{ObjectEndpoint.put}}
> and Ozone client; only the HTTP request is mocked, and the request body sets
> the interrupt flag of the request thread once its last byte is read. On the
> unmodified source both tests pass, which means the defect is present: on the
> encrypted bucket the failed PUT leaves {{dataSize=0}} with the ETag of the
> new 16 byte body, on the unencrypted bucket the previous 45 byte version
> stays.
> h3. Patch
> [^MC-4-flush-encrypted-s3-write-before-commit.patch], against
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at
> e0de543cbe9.
> {{S3ObjectWriteGuard.close}} flushes the stream first when it wraps a
> {{CryptoOutputStream}}, records a flush failure as the transfer failure and
> then closes as before, so the existing pre-commit refuses the commit and the
> {{KeyOutputStream}} only cleans up. Unencrypted writes are unchanged.
> Encrypted writes do no extra work, because {{CryptoOutputStream.close}}
> flushes before closing anyway; the flush just happens before the commit
> decision now.
> Covered by {{encryptedWriteFailureOnCloseBlocksCommit}} and
> {{encryptedWriteCommitsAllContent}} in the existing
> {{TestS3ObjectWriteGuard}}, using a real {{CryptoOutputStream}}. Without the
> change the first one fails because the key is committed. With it all
> s3gateway unit tests (1620) pass and checkstyle is clean. With the patch the
> reproduction's encrypted case fails, because the previous version stays.
> Found by TLA+ model checking and code review of the S3 gateway object put and
> key commit paths under HDDS-15926, on commit
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
