ivandika3 commented on PR #7566: URL: https://github.com/apache/ozone/pull/7566#issuecomment-2579502529
@sokui Ok, I was thinking a bit more about this. > why we missing these directories in the complete, abort and exprire phases? I can think one possible case, when between the initiate and complete / abort / expire, the parent directory is deleted and all the child directories and files are garbage collected by `DirectoryDeletingService` (runs every 60s). For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet, the `DIRECTORY_NOT_EMPTY` will not be thrown in `OMKeyDeleteRequestWithFSO#validateAndUpdateCache`. Therefore when `mpu_key` is orphaned and when it's completed / aborted, it will fail in `OMFileRequest#getParentID` since the parent directory has been deleted. Now I can understand that initiate and complete MPU will need to add the missing parent directories, since they will be accessed again. However, the abort (expired) MPUs, I don't think https://github.com/apache/ozone/pull/6496 applies in abort cases since we don't really want to add the missing parent directories for a key that is deleted instead. Even if we add the parent directories, adding it in cache only might result in unpredictable behavior where the missing directories created can suddenly disappear since they are not persisted in the OM DB. So, I believe either we should persist the missing parent directories or we should not create them in the first place. I think the original intention is that we want to ensure the MPU abort will succeed regardless whether the parent exists or not. I can think of one way, how about we first get the `OmMultipartKeyInfo` from the `multipartInfoTable` first since it does not require parentId information. Afterwards, we use the `OmMultipartKeyInfo#getParentId` to derive the MPU open key, without needing to called `OMFileRequest#getParentId` In the future, preventive measures, we can try to take into account incomplete MPUs for directory deletions. For example: - `OMFileRequest#hasChildren` might need to return true if there is a incomplete MPU key underneath it. - `DirectoryDeletingService` can also add some logic to abort the incomplete MPU However, these can be done in the future tickets. Note: I'm not too well-versed about FSO logic, so I could be wrong. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
