TakaHiR07 opened a new pull request, #20944: URL: https://github.com/apache/pulsar/pull/20944
### Motivation https://github.com/apache/pulsar/pull/14248 fix rackaware placement policy does not take effect after delete rack configuration. But after https://github.com/apache/pulsar/pull/16825, this fix is ineffective when delete zk rack info after bkclient initialize. The reason is "register available bookie" is after "BookieRackAffinityMapping#setConf" in bookieClient constructor. So updateRacksWithHost(racksWithHost) would throw BookieIdNotResolvedException, make "racksWithHost" become null. "racksWithHost" is updated until "watchAvailableBookies()" listener is trigger. I add some log of unittest testRackUpdate() to show the order of bookieClient constructor: ``` 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - ZkBookieRackAffinityMapping setConf(), racksWithHost:{default={10.0.0.1:56962=BookieInfoImpl(rack=rack-0, hostname=bookie-1), 10.0.0.2:56964=BookieInfoImpl(rack=rack-0, hostname=bookie-2), 10.0.0.3:56966=BookieInfoImpl(rack=rack-0, hostname=bookie-3), 10.0.0.4:56969=BookieInfoImpl(rack=rack-1, hostname=bookie-4), 10.0.0.5:56971=BookieInfoImpl(rack=rack-1, hostname=bookie-5), 10.0.0.6:56973=BookieInfoImpl(rack=rack-1, hostname=bookie-6)}} 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.1:56962, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.1:56962 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.1:56962, bookie does not exist or it is not running 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.2:56964, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.2:56964 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.2:56964, bookie does not exist or it is not running 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.3:56966, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.3:56966 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.3:56966, bookie does not exist or it is not running 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.4:56969, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-08-07T15:47:23,540+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.4:56969 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.4:56969, bookie does not exist or it is not running 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.5:56971, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.5:56971 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.5:56971, bookie does not exist or it is not running 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.6:56973, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.6:56973 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.6:56973, bookie does not exist or it is not running 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - ZkBookieRackAffinityMapping setConf() after updateRacksWithHost(), racksWithHost:{} 2023-08-07T15:47:23,541+0800 [main] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 127.0.0.1, using default rack for it : /default-rack. 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:127.0.0.1:0> @ /default-rack : org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping. 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted 2023-08-07T15:47:23,541+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.5:56971 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.5:56971, port=56971, host=10.0.0.5, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.5:56971 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.5:56971, port=56971, host=10.0.0.5, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.2:56964 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.2:56964, port=56964, host=10.0.0.2, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.2:56964 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.2:56964, port=56964, host=10.0.0.2, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.6:56973 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.6:56973, port=56973, host=10.0.0.6, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.6:56973 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.6:56973, port=56973, host=10.0.0.6, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.4:56969 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.4:56969, port=56969, host=10.0.0.4, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.4:56969 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.4:56969, port=56969, host=10.0.0.4, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.3:56966 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.3:56966, port=56966, host=10.0.0.3, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.3:56966 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.3:56966, port=56966, host=10.0.0.3, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.1:56962 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.1:56962, port=56962, host=10.0.0.1, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.1:56962 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.1:56962, port=56962, host=10.0.0.1, protocol=bookie-rpc, auth=[], extensions=[]}]} 2023-08-07T15:47:23,545+0800 [pulsar-registration-client-209-1] INFO org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - watchAvailableBookies() trigger racksWithHost:{default={10.0.0.1:56962=BookieInfoImpl(rack=rack-0, hostname=bookie-1), 10.0.0.2:56964=BookieInfoImpl(rack=rack-0, hostname=bookie-2), 10.0.0.3:56966=BookieInfoImpl(rack=rack-0, hostname=bookie-3), 10.0.0.4:56969=BookieInfoImpl(rack=rack-1, hostname=bookie-4), 10.0.0.5:56971=BookieInfoImpl(rack=rack-1, hostname=bookie-5), 10.0.0.6:56973=BookieInfoImpl(rack=rack-1, hostname=bookie-6)}} 2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-1/10.0.0.5:56971 2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-0/10.0.0.2:56964 2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-1/10.0.0.6:56973 2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-1/10.0.0.4:56969 2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-0/10.0.0.3:56966 2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-0/10.0.0.1:56962 ``` ### Modifications 1. modify the execute order in BookieRackAffinityMapping#setConf 2. improve the testRackUpdate() to test this case. ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: <!-- ENTER URL HERE --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
