sokui commented on PR #7566: URL: https://github.com/apache/ozone/pull/7566#issuecomment-2584786432
> @sokui Ok, I was thinking a bit more about this. > > > why we missing these directories in the complete, abort and exprire phases? > > I can think one possible case, when between the initiate and complete / abort / expire, the parent directory is deleted and all the child directories and files are garbage collected by `DirectoryDeletingService` (runs every 60s). > > For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet and does not exist in fileTable, the `DIRECTORY_NOT_EMPTY` will not be thrown in `OMKeyDeleteRequestWithFSO#validateAndUpdateCache`. Therefore `mpu_key` is orphaned and when it's completed / aborted, it will fail in `OMFileRequest#getParentID` since the parent directory has been deleted. > > Now I can understand that initiate and complete MPU will need to add the missing parent directories, since the directories will be accessed again. However, for abort (expired) MPUs, I don't think #6496 applies since we don't really want to add the missing parent directories for a key that is going to be deleted anyway. Even if we add the parent directories, adding it in cache only might result in unpredictable behavior where the missing directories created can suddenly disappear since they are not persisted in the OM DB. So, I believe either we should persist the missing parent directories or we should not create them in the first place. > > I think the original intention is that we want to ensure the MPU abort will succeed regardless whether the parent exists or not. I can think of one way, how about we first get the `OmMultipartKeyInfo` from the `multipartInfoTable` since it does not require parentId information. Afterwards, we use the `OmMultipartKeyInfo#getParentId` to derive the MPU open key, without needing to call `OMFileRequest#getParentId`. That way, we don't need to create the missing parent directories. > > In the future, for preventive measures, we can try to take into account incomplete MPUs for directory deletions. For example: > > * `OMFileRequest#hasChildren` might need to return true if there is a incomplete MPU key underneath it. > * `DirectoryDeletingService` can also add some logic to abort the incomplete MPU > However, these can be done in the future tickets. > > Note: I'm not too well-versed about FSO logic, so I could be wrong. I am testing your idea. Will keep you updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
