BewareMyPower opened a new pull request, #25246: URL: https://github.com/apache/pulsar/pull/25246
### Motivation https://github.com/apache/pulsar/pull/25187 introduces a regression on `MetadataCacheImpl#refresh`. When a path is created or modified, the `refresh` method will be called by updating `objCache` with the future returned by `readValueFromStore()`. However, there is a race: 1. [current thread] Acquire the lock of the node in the `ConcurrentMap` of `objCache`. Then call `readValueFromStore`, which calls `store.get` 2. [metadata store worker thread] In the callback of `store.get`, `objCache.getIfPresent` calls the `ConcurrentMap#get` to get the cached future and check if it's the same with the future inserted to the map 3. [current thread] Insert the future returned by to the `ConcurrentMap` of `objCache` and release the lock The updated future of step 3 is not guaranteed to be immediately visible because `ConcurrentMap#get` is lock-free, which means it does not need to wait the lock on `path` is released after step 3. https://github.com/apache/pulsar/blob/9db31cca40e3948182386b6930907daf449abd41/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java#L156-L162 The correctness of the code above is based on the fact that the `cachedFuture` must be the future of `readValueFromStore`, but not the existing cached future. This logic was added originally because `testCloneInReadModifyUpdateOrCreate` failed. I've thought it's caused by duplicated `refresh` calls but it's actually not. However, the root cause is that when it failed, the `create` method implementation was wrong: ```java return serialize(path, value).thenAcceptAsync(content -> store.put(path, content, Optional.of(-1L))) .thenApply(stat -> objCache.get(path)) /* ... */ .thenApply(__ -> null); ``` It should be `thenComposeAsync` rather than `thenAcceptAsync`, otherwise `objCache.get` might not see the updated value. This bug was accidentally fixed in my later commits. ### Modifications - Revert the `cachedFuture` related code. We might need a careful design to reduce unnecessary deserializations when `refresh` is called multiple times for an update (e.g. the callback of `MetadataCacheImpl#put` and the notification method of `AbstractMetadataStore#accept`) - Add `testRefreshRace` to verify this race disappears. ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: <!-- ENTER URL HERE --> <!-- After opening this PR, the build in apache/pulsar will fail and instructions will be provided for opening a PR in the PR author's forked repository. apache/pulsar pull requests should be first tested in your own fork since the apache/pulsar CI based on GitHub Actions has constrained resources and quota. GitHub Actions provides separate quota for pull requests that are executed in a forked repository. The tests will be run in the forked repository until all PR review comments have been handled, the tests pass and the PR is approved by a reviewer. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
