[
https://issues.apache.org/jira/browse/HDDS-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prince Raj updated HDDS-16363:
------------------------------
Description:
h2. Description
Recon's storage distribution information does not appear to account for the
storage consumed by parts of an incomplete S3 multipart upload.
When a multipart upload is initiated and one or more parts are uploaded without
completing the MPU, the uploaded parts remain as open/incomplete data and
should be reflected in Recon's {{multipartOpenKeyBytes}} metric.
Currently, {{multipartOpenKeyBytes}} can remain unchanged even after a
multipart upload part has been successfully written.
h3. Expected behavior
For an incomplete multipart upload:
# An MPU is initiated for an object.
# One or more parts are uploaded successfully.
# The MPU is intentionally left incomplete.
# Recon's storage distribution API should reflect the replicated storage
consumed by the uploaded MPU parts in {{{}multipartOpenKeyBytes{}}}.
# Once the corresponding MPU data is cleaned up, {{multipartOpenKeyBytes}}
should decrease accordingly.
For example, if a 16 MiB part is uploaded with RATIS replication factor 3, the
corresponding replicated storage contribution should be approximately:
{{16 MiB × 3 = 50,331,648 bytes}}
The exact value should account for the existing baseline and the configured
replication factor.
h3. Actual behavior
After successfully uploading a part to an incomplete MPU, Recon's
{{multipartOpenKeyBytes}} does not increase as expected and may continue
reporting the previous value.
This suggests that the storage associated with incomplete MPU parts may not be
correctly propagated into Recon's multipart open-key accounting.
h3.
was:
h2. Summary
Storage distribution QE test multipart_upload_openkeybytes validates that Recon
reports multipartOpenKeyBytes correctly for an {*}incomplete S3 multipart
upload (MPU){*}. After uploading *one 16MB part* and leaving the MPU open, the
test expects Recon to report baseline + 48MB (16MB logical × RATIS RF=3). On
upstream runs, multipartOpenKeyBytes {*}never reaches 50331648 bytes{*};
polling exhausts retries and actual *≠* expected.
h2. What this testcase does
h3. Purpose
Verify Recon *Capacity / storageDistribution* API correctly tracks bytes held
by {*}open (incomplete) multipart uploads{*}, and that those bytes are cleared
after MPU container cleanup.
h3. Scenario configuration
{{StorageScenario(
test_id="multipart_upload_openkeybytes",
num_keys=1,
key_sizes=['16MB'], # not used for MPU upload path
multipart_upload=True,
mpu_open_key_bytes=True, # leave MPU incomplete
mpu_part_sizes=['16MB', '16MB'], # 2 parts defined
mpu_parts_to_upload=1, # only part 1 uploaded → 16MB raw
keys_to_delete=None, # no normal delete workflow
)}}
h3. Data created
* *1 incomplete MPU* on volume s3v, key name mpu-open-key-test
* *Uploaded:* 1 part × *16MB* = *16,777,216 bytes* (part 2 never uploaded, MPU
never completed)
* *Expected Recon metric (replicated):* 16MB × *RF 3* = *50,331,648 bytes*
h3. Actual test flow
||Step||Action||
|1|Read multipartOpenKeyBytes *BEFORE* MPU (GET /api/v1/storageDistribution) —
baseline|
|2|Capture OM/SCM/DN Recon metrics baseline|
|3|Create incomplete MPU via S3 API (1 × 16MB part, no complete)|
|4|*Poll* until multipartOpenKeyBytes == baseline + 50331648 (up to 100 × 5s) ←
*FAILS here on upstream*|
|5|Resolve MPU container IDs from OM multipartInfoTable|
|6|closeContainer() on those containers|
|7|Poll until multipartOpenKeyBytes == 0|
|8|Verify SCM/DN pending-deletion metrics reflect cleanup|
h3.
How expected value is computed
{{uploaded_bytes = 16 * 1024 * 1024 # from created_keys[0]['size']
expected_open_delta = Ozone.calculate_total_repl_size(
sizes=[16777216],
replication_type="RATIS", # default RF=3
return_bytes=True,
) # → 50331648
expected_multipart_open = multipart_open_before + 50331648}}
h3. How actual value is read
{{GET Recon storageDistribution
→ usedSpaceBreakdown.openKeyBytes.multipartOpenKeyBytes}}
(Recon backend uses totalReplicatedDataSize from MPU open-key summary — see
StorageDistributionEndpoint.calculateOpenKeySizes().)
----
h2. Steps to reproduce
# Run storage distribution E2E with scenario multipart_upload_openkeybytes:
{{pytest
tests/ozone/ozone_18/storage_capacity_distribution/test_delete_workflow.py \
-k multipart_upload_openkeybytes}}
# Confirm S3 Gateway is up and kinit / S3 creds are configured.
3. Observe logs from poll_multipart_open_key_bytes_until_match:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Observed behavior (upstream)
* Incomplete MPU is created successfully (S3 upload part succeeds).
* multipartOpenKeyBytes *does not increase to expected 48MB replicated size*
(50331648).
* Poll loop runs full retries (max_retries=100, retry_interval=5s ≈ {*}8+
minutes{*}) without match.
* Example log pattern:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Expected behavior
After uploading *one 16MB MPU part* (incomplete):
* Recon multipartOpenKeyBytes should become baseline + 50,331,648 bytes (16MB
× RF 3).
* Test assertion at step 4 should pass within poll window.
----
h3. Root Cause:
This Change was causing the failure :
<https://github.com/apache/ozone/pull/10588>
> Recon does not report storage used by incomplete multipart uploads in
> multipartOpenKeyBytes
> -------------------------------------------------------------------------------------------
>
> Key: HDDS-16363
> URL: https://issues.apache.org/jira/browse/HDDS-16363
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Prince Raj
> Assignee: Priyesh Karatha
> Priority: Major
> Labels: pull-request-available
>
> h2. Description
> Recon's storage distribution information does not appear to account for the
> storage consumed by parts of an incomplete S3 multipart upload.
> When a multipart upload is initiated and one or more parts are uploaded
> without completing the MPU, the uploaded parts remain as open/incomplete data
> and should be reflected in Recon's {{multipartOpenKeyBytes}} metric.
> Currently, {{multipartOpenKeyBytes}} can remain unchanged even after a
> multipart upload part has been successfully written.
> h3. Expected behavior
> For an incomplete multipart upload:
> # An MPU is initiated for an object.
> # One or more parts are uploaded successfully.
> # The MPU is intentionally left incomplete.
> # Recon's storage distribution API should reflect the replicated storage
> consumed by the uploaded MPU parts in {{{}multipartOpenKeyBytes{}}}.
> # Once the corresponding MPU data is cleaned up, {{multipartOpenKeyBytes}}
> should decrease accordingly.
> For example, if a 16 MiB part is uploaded with RATIS replication factor 3,
> the corresponding replicated storage contribution should be approximately:
>
> {{16 MiB × 3 = 50,331,648 bytes}}
> The exact value should account for the existing baseline and the configured
> replication factor.
> h3. Actual behavior
> After successfully uploading a part to an incomplete MPU, Recon's
> {{multipartOpenKeyBytes}} does not increase as expected and may continue
> reporting the previous value.
> This suggests that the storage associated with incomplete MPU parts may not
> be correctly propagated into Recon's multipart open-key accounting.
> h3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
