devmadhuu commented on PR #11685: URL: https://github.com/apache/ozone/pull/11685#issuecomment-6497177164
> @errose28 @sumitagrawl agreed on keeping open key and MPU stats separate. My original proposal for [HDDS-16364](https://issues.apache.org/jira/browse/HDDS-16364) follows that: > > Problem: at MPU initiate, OM writes a multipart open key with size 0 and no blocks, which stays until the upload is completed or aborted. Recon counts it as a regular open key, while committed parts are reported by /keys/open/mpu/summary. > > For example, start 2 multipart uploads and commit 2 parts of 5 MiB to each, without completing either upload. Recon then reports: > > /keys/open/summary: 2 open keys, 0 bytes /keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB) Showing these as 2 open keys with 0 bytes is misleading, since the uploads already hold 20 MiB of committed parts. > > Fix (Recon only): exclude multipart open keys (isMultipartKey=true) from /keys/open and /keys/open/summary, and report uploads only via /keys/open/mpu/summary. > > With the same 2 uploads, while a third part of one upload is still being written, Recon would report: > > /keys/open/summary: 1 open key (the part being written) /keys/open/mpu/summary: 2 uploads, 20971520 bytes (20 MiB) Behavior for parts being written is unchanged: they are counted as open keys until they commit, then move to the MPU total. > > This matches OM: getExpiredOpenKeys (used by OpenKeyCleanupService) skips these keys, and expired ones are cleaned up only by MultipartUploadCleanupService. The tradeoff is that the stored open key totals must be recomputed after upgrade, via a Recon upgrade action that queues a task reinitialization (as existing upgrade actions do). > > Would you prefer this, or leaving the open key view as is? > > CC: @devmadhuu @priyeshkaratha Thanks @lpavanvenkat for giving clarity and explanation. IMO above makes sense to filter out committed key parts from openkey summary using flag `isMultipartKey=true`. @sumitagrawl @errose28 any concerns on this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
