peterxcli opened a new pull request, #11762: URL: https://github.com/apache/ozone/pull/11762
## What changes were proposed in this pull request? Recon keeps two indexes of the keys in each container, `containerKeyTable` and `keyContainerTable`, and a key count per container. Both indexes use the OM DB key: `/vol/bucket/key` for a `keyTable` entry and `/volumeId/bucketId/parentId/fileName` for a `fileTable` entry. When a key is overwritten, Recon gets an UPDATE event. `ContainerKeyMapperHelper#process` handles it in two steps. It deletes the key's old mappings, then adds the mappings of the new value. The delete step looked up `getOldValue().getKeyName()`. That is the key name (`logs/a.log`) or the file name (`a.log`), not the DB key, so the delete step never found anything. For example: 1. Key `/vol/bucket/a` has a block in container 1. 2. The key is overwritten, and the new block is in container 2. 3. Recon adds the mapping to container 2, and the mapping to container 1 stays. Container 1 still lists the key, and its key count stays one too high. A replay of 100K OM transactions on a 1.25M-key OM DB left 24,959 stale container-key rows. A replay of 1M transactions on a 100M-key OM DB left 230,478. Deleting by the DB key is not enough on its own. Recon collects the changes of a batch in memory and writes them to its DB at the end. The add step checks the DB to see whether a mapping already exists. But the DB still has the mappings that the delete step has just removed in memory. So when the new value keeps a block in the same container, the add step skips that mapping, and the batch then deletes it. This would affect every update that keeps a container: an overwrite whose new block lands in the same container, and every update that changes only metadata, such as ACLs or tags. With this change: - An UPDATE deletes the old mappings by the event's DB key, the same key that the add step uses. - The add step adds a mapping again if an earlier event in the same batch deleted it. - The delete step skips the mappings that an earlier event in the same batch deleted. Without this, a key that is overwritten twice in one batch would lower the old container's count twice. - The mappings deleted in a batch are kept in a `HashSet` instead of an `ArrayList`, because both steps now look them up. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17465 ## How was this patch tested? Two new tests in `TestContainerKeyMapperTask`, each run for an FSO file and an OBS key: - `testOverwriteMovesKeyToOtherContainers`: an overwrite moves the key from containers 1 and 2 to containers 2 and 3, while another key stays in container 1. The test checks that container 1 lists only the other key, that containers 2 and 3 list the overwritten key, and that all three key counts are 1. - `testTwoOverwritesInOneBatch`: one batch moves the key from container 1 to container 2, and then to container 3. The test checks that container 1 keeps a count of 1 for its other key, that container 2 ends empty with a count of 0, and that container 3 lists the key. Without the change, all four test runs fail. Without the re-add in the add step, the first test fails. Without the skip in the delete step, the second test fails. An end-to-end check replayed 100K OM transactions on a 1.25M-key OM DB through `OMDBUpdatesHandler` and the tasks' `process()`, in fetches of up to 50,000 updates. It then compared Recon's tables with a rebuild of the final OM DB. With this change, `containerKeyTable` and `keyContainerTable` match the rebuild. The container key counts match too when the tasks run one at a time. When the tasks run in parallel, as `ReconTaskControllerImpl` runs them, some counts of containers that hold both FSO and OBS keys still come out too low. The FSO and OBS tasks each read a container's count, change it and write it back, so one task can overwrite the other's change. That race is separate from this fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
