rolish commented on issue #2080: Geo-replication setup on K8s cluster URL: https://github.com/apache/bookkeeper/issues/2080#issuecomment-489463036 @sijie, thanks for this, very useful. Using ``BK_metadataServiceUri`` instead of ``BK_zkServers`` (which is completely excluded from configmap) works both in case bookies and bookie-autorecovery. BK successfully connects to all global ZK nodes distributed across two data centers. We are using statefulsets with ``hostNetwork: true`` for both ZK and BK nodes. Both Azure data centers are interconnected by VPN. Bookie hosts are identified by host IP (``BK_useHostNameAsBookieID: "false"``). Statefulset has defined anti affinity so no two bookies can run on single node (which would cause bookie ID conflict). ``` affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "component" operator: In values: - bookie topologyKey: "kubernetes.io/hostname" ``` This is however not ideal, as each pod in statefulset has it's own PVC attached and when killed it can start on another node attaching the same PVC. This will result in exception ``` 2019-05-05 18:34:56,501 - ERROR - [main:Main@223] - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "172.16.189.66:3181" journalDir: "/bookkeeper/data/journal" ledgerDirs: "1\t/bookkeeper/data/ledgers" instanceId: "038629cb-93b8-46ad-a469-5b9bb070dcb8" ] is not matching with [4 bookieHost: "172.16.189.35:3181" journalDir: "/bookkeeper/data/journal" ledgerDirs: "1\t/bookkeeper/data/ledgers" instanceId: "038629cb-93b8-46ad-a469-5b9bb070dcb8" ] at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:136) ``` It looks like there is no other option than to use daemonset using host attached disks for bookies.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
