lhotari opened a new pull request #13069:
URL: https://github.com/apache/pulsar/pull/13069
### Motivation
This PR depends on #13066 .
When a lookup is made to a topic that isn't currently loaded, the decision
will be made in a distributed fashion on the follower brokers since the
information about the leader broker is missing (because
`LeaderElectionService.getCurrentLeader()` always returned Optional.empty()).
This leads to races when assigning the topic ownership to a broker, since the
decision isn't made centrally on the leader broker.
Besides the problem fixed by #13066 , there's another race condition in
metadata operations.
```
2021-12-01T15:19:16,659+0200 [metadata-store-266-1] WARN
com.github.benmanes.caffeine.cache.LocalAsyncCache - Exception thrown during
asynchronous load
java.util.concurrent.CompletionException:
org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyExistsException:
org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException:
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /admin/local-policies/public/default
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$create$12(MetadataCacheImpl.java:232)
~[classes/:?]
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$storePut$9(ZKMetadataStore.java:222)
~[classes/:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[netty-common-4.1.68.Final.jar:4.1.68.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by:
org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyExistsException:
org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException:
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /admin/local-policies/public/default
```
This problem hasn't yet been fixed, but the goal is to fix it to complete
this PR. The PR will be kept in draft state while the fix is in progress.
### Modifications
- builds upon changes from #13066
- add failing test case
`MultiBrokerLeaderElectionTest.shouldProvideConsistentAnswerToTopicLookup`
- adds more logging to topic ownership assignment decision made in
`NamespaceService.searchForCandidateBroker`
- optimizes performance of the topic ownership assignment by using
`LoadManager.getAvailableBrokers` instead of reading the list every time from
Zookeeper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
