Prince Raj created HDDS-16363:
---------------------------------
Summary: Storage-Capacity-Distribution:
multipart_upload_openkeybytes fails on upstream — Recon metric does not reach
expected 48MB
Key: HDDS-16363
URL: https://issues.apache.org/jira/browse/HDDS-16363
Project: Apache Ozone
Issue Type: Bug
Reporter: Prince Raj
h2. Summary
Storage distribution QE test multipart_upload_openkeybytes validates that Recon
reports multipartOpenKeyBytes correctly for an {*}incomplete S3 multipart
upload (MPU){*}. After uploading *one 16MB part* and leaving the MPU open, the
test expects Recon to report baseline + 48MB (16MB logical × RATIS RF=3). On
upstream runs, multipartOpenKeyBytes \{*}never reaches 50331648 bytes{*};
polling exhausts retries and actual *≠* expected.
h2. What this testcase does
h3. Purpose
Verify Recon *Capacity / storageDistribution* API correctly tracks bytes held
by {*}open (incomplete) multipart uploads{*}, and that those bytes are cleared
after MPU container cleanup.
h3. Scenario configuration
{{StorageScenario(
test_id="multipart_upload_openkeybytes",
num_keys=1,
key_sizes=['16MB'], # not used for MPU upload path
multipart_upload=True,
mpu_open_key_bytes=True, # leave MPU incomplete
mpu_part_sizes=['16MB', '16MB'], # 2 parts defined
mpu_parts_to_upload=1, # only part 1 uploaded → 16MB raw
keys_to_delete=None, # no normal delete workflow
)}}
h3. Data created
* *1 incomplete MPU* on volume s3v, key name mpu-open-key-test
* *Uploaded:* 1 part × *16MB* = *16,777,216 bytes* (part 2 never uploaded, MPU
never completed)
* *Expected Recon metric (replicated):* 16MB × *RF 3* = *50,331,648 bytes*
h3. Actual test flow
||Step||Action||
|1|Read multipartOpenKeyBytes *BEFORE* MPU (GET /api/v1/storageDistribution) —
baseline|
|2|Capture OM/SCM/DN Recon metrics baseline|
|3|Create incomplete MPU via S3 API (1 × 16MB part, no complete)|
|4|*Poll* until multipartOpenKeyBytes == baseline + 50331648 (up to 100 × 5s) ←
*FAILS here on upstream*|
|5|Resolve MPU container IDs from OM multipartInfoTable|
|6|closeContainer() on those containers|
|7|Poll until multipartOpenKeyBytes == 0|
|8|Verify SCM/DN pending-deletion metrics reflect cleanup|
h3. How expected value is computed
{{uploaded_bytes = 16 * 1024 * 1024 # from created_keys[0]['size']
expected_open_delta = Ozone.calculate_total_repl_size(
sizes=[16777216],
replication_type="RATIS", # default RF=3
return_bytes=True,
) # → 50331648
expected_multipart_open = multipart_open_before + 50331648}}
h3. How actual value is read
{{GET Recon storageDistribution
→ usedSpaceBreakdown.openKeyBytes.multipartOpenKeyBytes}}
(Recon backend uses totalReplicatedDataSize from MPU open-key summary — see
StorageDistributionEndpoint.calculateOpenKeySizes().)
----
h2. Steps to reproduce
# Run storage distribution E2E with scenario multipart_upload_openkeybytes:
{{pytest
tests/ozone/ozone_18/storage_capacity_distribution/test_delete_workflow.py \
-k multipart_upload_openkeybytes}}
# Confirm S3 Gateway is up and kinit / S3 creds are configured.
# Observe logs from poll_multipart_open_key_bytes_until_match:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Observed behavior (upstream)
* Incomplete MPU is created successfully (S3 upload part succeeds).
* multipartOpenKeyBytes *does not increase to expected 48MB replicated size*
(50331648).
* Poll loop runs full retries (max_retries=100, retry_interval=5s ≈ {*}8+
minutes{*}) without match.
* Example log pattern:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Expected behavior
After uploading *one 16MB MPU part* (incomplete):
* Recon multipartOpenKeyBytes should become baseline + 50,331,648 bytes (16MB
× RF 3).
* Test assertion at step 4 should pass within poll window.
----
h2. Possible root causes (for investigation)
# *Recon MPU open-key summary not updated* for incomplete S3 MPU on upstream
(getMPUKeySummary() / totalReplicatedDataSize stays 0).
# *Recon sync lag* beyond test poll window (less likely if 500s exhausted).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
