[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-591283710 > cool! @hari819 are you willing to contribute the documentation for this part? @sijie yes , but any guide lines , any document format which i need to follow and prepare This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-590789617 @sijie ,actually the problem was at myside , the shell `"bin/bookkeeper shell metaformat --nonInteractive || true;"` was missing the zookeeper tls settings , i am able to complete SSL/TLS/SASL settings on all the three Zookeeper,Bookkeeper and Pulsar and my pulsar cluster is working fine . Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-590180883 @sijie , i have now enabled the hierarchical type , 16:39:52.631 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+hierarchical://zookeeper/ledgers : zkServers = zookeeper, ledgersRootPath = /ledgers. i am already running the metaformat , using `bin/bookkeeper shell metaformat --nonInteractive || true;` , before running the bookie, But still bookkeeper metadata is not available , 16:39:52.761 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - **ZooKeeper client is connected now.** 16:39:52.797 [main] ERROR org.apache.bookkeeper.discover.ZKRegistrationManager - **BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat** 16:39:52.798 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 16:39:52.807 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:392) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] **Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata** at **org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] ... 10 more** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-588344528 @sijie , After enabling ssl on zookeepr , i have enabled TLS in bookkeeper also , but i saw some weird error , org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service **uri zk+null://zookeeper/ledgers : zkServers** = zookeeper, ledgersRootPath = /ledgers. why it is , metadata service uri zk+null://zookeeper/ledgers : zkServers = zookeeper, ledgersRootPath = /ledgers i think the metaformat command is unable to create the folder "/ledgers" ,and finally ends up with bookkeeper going down , 06:15:35.976 [main] ERROR org.apache.bookkeeper.discover.ZKRegistrationManager - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat 06:15:35.977 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 06:15:35.982 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: **Failed to get cluster instance id at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:392) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]** at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] **Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata at** org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] ... 10 more This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-586102838 > > parallel i am also trying to setup tls authentication for bookkeeper , is it supported via pulsar ?. > > Yes. it is supported. > > > but could not get past this error when calling bin/pulsar bookie, > > Which version of Pulsar are you using? Try to set `BOOKIE_MEM` instead of `PULSAR_MEM` for running `bin/pulsar bookie`. BOOKIE_MEM solved my problem , thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-586099263 @sijie , i am using the latest version of pulsar via the docker image , apachepulsar\pulsar-all:latest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-585897882 @sijie , i could get moving on the zookeeper security part , parallel i am also trying to setup tls authentication for bookkeeper , is it supported via pulsar ?. i followed the docs , [https://bookkeeper.apache.org/docs/4.9.0/reference/config/](url) and [https://bookkeeper.apache.org/docs/4.9.0/security/tls/](url) , but could not get past this error when calling bin/pulsar bookie, `[conf/pulsar_env.sh] Applying config PULSAR_MEM = " -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g -Dpulsar.root.logger=DEBUG,FILE " Error: Could not find or load main class "` will you be able to check this ? Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-584014285 @sijie , i can start but need some info on the below , i will start by having the zookeeper.conf outside of the the arguments provided for the "pulsar-all" container `args: - > bin/wait-for-all-dns.sh && bin/wait-for-my-dns.sh && bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/generate-zookeeper-config.sh conf/zookeeper.conf && cat conf/zookeeper.conf && bin/pulsar zookeeper` there is a configMap for zookeeper already with `kind: ConfigMap metadata: name: zookeeper-config data: PULSAR_MEM: "\" -Xms100m -Xmx256m \"" PULSAR_GC: "\" -XX:+UseG1GC -XX:MaxGCPauseMillis=10\""` , can i start adding it here ? also this `" bin/apply-config-from-env.py conf/zookeeper.conf &&"` is the guy who is generating the zookeeper.conf , so where to add the ssl properties ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-583752009 @sijie , yes that would be a good idea , but if you have the content already with you could you please provide the same here ,? thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf
hari819 commented on issue #6236: add ssl/tls configuration to zookeeper via the zookeeper.conf URL: https://github.com/apache/pulsar/issues/6236#issuecomment-583246362 thanks @jiazhai @sijie , i could get rid of that error now . if i have to add some jvm arguments to zookeeper.conf like , `-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass" ` where should i add them to add ssl/tls authentication to zookeeper? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
