Bug#1025012: zookeeper: starts but is completely unusable
Hey Pierre. On Tue, 2022-12-06 at 23:08 +0100, Pierre Gruet wrote: > Thanks for the bug report (and the follow-up precisions you sent)! > > Yet I fail to reproduce it on testing. I installed zookeeper and > zookeeperd on testing, then ran > > $ /usr/share/zookeeper/bin/zkCli.sh > specifying nothing about the classpath (it is already handled in the > MANIFEST.MF file of the zookeeper jar), and I could enter commands > (though with no prompt, as you say). When you've started your zookeeper (I assume you also use the sysvinit script, respectively the virtual systemd unit generated from that?), what does your process' command line show, e.g.: # ps ax | grep org.apache.zookeeper.server.quorum.QuorumPeerMain 3156 ?Sl 167:08 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/log4j-1.2.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=true -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg Especially, which CLASSPATH (seems to be the -cp)? Is your test system clustered (i.e. more than one ZK node?)? > (However, I also get the hanging issue if I do not install > zookeeperd.) Well I guess that's clear, cause then nothing would have started zookeeper?! > I guess there is some difference in the versions of the dependencies > between stable and testing, as you underline that the dependencies > come > from stable. That would be quite surprising, IMO,.. cause the issue seems to be quite clearly the missing CLASSPATH stuff. I mean if there would be some other package, that is used to start the java process... and that other package would have a different version and thus be incompatible, I could imagine this. But it seems as if zookeeper's sysvinit script does all on it's own and simply execute java in the end. So it's hard to imagine, that something interferes, and somehow breaks the CLASSPATH there?! > Concerning the SLF4J warning: in the past I already stumbled upon > this > and visited the URI > https://www.slf4j.org/codes.html#StaticLoggerBinder > which is in the warning message. Yeah, also stumbled over that via the error message. > Because of the last paragraph in the > relevant section therein, I was unsure I should choose a SLF4J > binding > as this would impose my choice to the end user. What do you think? Well since that two infamous security holes, log4j has a somewhat damaged reputation ;-) ... but apart from that I think this would make the most sense here. As I've said in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025012#20 : /usr/share/java/slf4j-log4j12.jar wasn't enough for me and I needed to add /usr/share/java/log4j-1.2.jar to the CLASSPATH instead, in order to get output to /var/log/zookeeper > Finally, as: > - I think there is a mismatch somewhere in the versions of the > dependencies between stable and testing; > - I cannot reproduce on a system with dependencies fetched from > testing > only, > I would tend to discuss only the SLF4J issue and ignore the rest. > > Please tell me what you think, I think we should look into both, and at best split things apart. I mean I can try to reproduce the whole thing on a sid or testing only systemd an see whether it still fails to start, but right now I'd be quite surprised why it wouldn't start because of that... and even *if* that would be the reason, then bullseye to bookwork upgrades need to be supported, and therefore the package (zookeeper) would need to have correct versioned dependencies (*if* it's because of a incompatible version of some dependency). Thanks, Chris.
Bug#1025012: zookeeper: starts but is completely unusable
And here we go: CLASSPATH="/etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/log4j-1.2.jar" Seems to do the trick to get logging to /var/log/zookeeper/foobar . The zkCli shows still no prompt, though. It also needs the /usr/share/java/log4j-1.2.jar, /usr/share/java/slf4j-log4j12.jar alone ain't enough. I wonder a bit whether the classpath is just completely mess up? When installing, I needed the following dependencies: [INSTALL, DEPENDENCIES] junit4:amd64 4.13.1-2 [INSTALL, DEPENDENCIES] libactivation-java:amd64 1.2.0-2 [INSTALL, DEPENDENCIES] libapache-pom-java:amd64 18-1 [INSTALL, DEPENDENCIES] libasm-java:amd64 9.1-1 [INSTALL, DEPENDENCIES] libatinject-jsr330-api-java:amd64 1.0+ds1-5 [INSTALL, DEPENDENCIES] libcommons-cli-java:amd64 1.4-2 [INSTALL, DEPENDENCIES] libcommons-io-java:amd64 2.8.0-1 [INSTALL, DEPENDENCIES] libcommons-logging-java:amd64 1.2-2 [INSTALL, DEPENDENCIES] libcommons-parent-java:amd64 43-1 [INSTALL, DEPENDENCIES] libdropwizard-metrics-java:amd64 3.2.6-1 [INSTALL, DEPENDENCIES] libeclipse-jdt-core-java:amd64 3.24.0+eclipse4.18-1 [INSTALL, DEPENDENCIES] libel-api-java:amd64 3.0.0-3 [INSTALL, DEPENDENCIES] libfindbugs-annotations-java:amd64 3.1.0~preview2-3 [INSTALL, DEPENDENCIES] libguava-java:amd64 29.0-6 [INSTALL, DEPENDENCIES] libhamcrest-java:amd64 1.3-9 [INSTALL, DEPENDENCIES] libjackson2-annotations-java:amd64 2.12.1-1 [INSTALL, DEPENDENCIES] libjackson2-core-java:amd64 2.12.1-1 [INSTALL, DEPENDENCIES] libjackson2-databind-java:amd64 2.12.1-1+deb11u1 [INSTALL, DEPENDENCIES] libjaxb-api-java:amd64 2.3.1-1 [INSTALL, DEPENDENCIES] libjctools-java:amd64 2.0.2-1 [INSTALL, DEPENDENCIES] libjetty9-extra-java:amd64 9.4.39-3+deb11u1 [INSTALL, DEPENDENCIES] libjetty9-java:amd64 9.4.39-3+deb11u1 [INSTALL, DEPENDENCIES] libjffi-java:amd64 1.2.7-11 [INSTALL, DEPENDENCIES] libjffi-jni:amd64 1.2.7-11+b1 [INSTALL, DEPENDENCIES] libjnr-constants-java:amd64 0.10.1-1 [INSTALL, DEPENDENCIES] libjnr-enxio-java:amd64 0.32.3-2 [INSTALL, DEPENDENCIES] libjnr-ffi-java:amd64 2.1.7-1 [INSTALL, DEPENDENCIES] libjnr-posix-java:amd64 3.0.45-2 [INSTALL, DEPENDENCIES] libjnr-unixsocket-java:amd64 0.18-4 [INSTALL, DEPENDENCIES] libjnr-x86asm-java:amd64 1.0.2-5.1 [INSTALL, DEPENDENCIES] libjsp-api-java:amd64 2.3.4-3 [INSTALL, DEPENDENCIES] libjsr305-java:amd64 0.1~+svn49-11 [INSTALL, DEPENDENCIES] liblog4j1.2-java:amd64 1.2.17-10+deb11u1 [INSTALL, DEPENDENCIES] libmail-java:amd64 1.6.5-1 [INSTALL, DEPENDENCIES] libnetty-java:amd64 1:4.1.48-4 [INSTALL, DEPENDENCIES] libnetty-tcnative-java:amd64 2.0.28-1 [INSTALL, DEPENDENCIES] libnetty-tcnative-jni:amd64 2.0.28-1 [INSTALL, DEPENDENCIES] libservlet-api-java:amd64 4.0.1-2 [INSTALL, DEPENDENCIES] libservlet3.1-java:amd64 1:4.0.1-2 [INSTALL, DEPENDENCIES] libslf4j-java:amd64 1.7.30-1 [INSTALL, DEPENDENCIES] libsnappy-java:amd64 1.1.8.3-1 [INSTALL, DEPENDENCIES] libsnappy-jni:amd64 1.1.8.3-1 [INSTALL, DEPENDENCIES] libsnappy1v5:amd64 1.1.8-1 [INSTALL, DEPENDENCIES] libspring-beans-java:amd64 4.3.30-1 [INSTALL, DEPENDENCIES] libspring-core-java:amd64 4.3.30-1 [INSTALL, DEPENDENCIES] libtaglibs-standard-impl-java:amd64 1.2.5-2.1 [INSTALL, DEPENDENCIES] libtaglibs-standard-spec-java:amd64 1.2.5-2.1 [INSTALL, DEPENDENCIES] libtomcat9-java:amd64 9.0.43-2~deb11u4 [INSTALL, DEPENDENCIES] libwebsocket-api-java:amd64 1.1-2 [INSTALL, DEPENDENCIES] libzookeeper-java:amd64 3.8.0-10 [INSTALL, DEPENDENCIES] zookeeperd:amd64 3.8.0-10 [INSTALL] zookeeper:amd64 3.8.0-10 Wouldn't there need to be some CLASSPATH settings for more or less all of them? I guess there's at least a problem with that: 2022-11-28 21:41:34,760 - WARN [main:AdminServerFactory@58] - Unable to load jetty, not starting JettyAdminServer java.lang.NoClassDefFoundError: javax/servlet/Servlet at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) at org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43) at org.apache.zookeeper.server.quorum.QuorumPeer.(QuorumPeer.java:1070) at org.apache.zookeeper.server.quorum.QuorumPeerMain.getQuorumPeer(QuorumPeerMain.java:246) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:177) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91) Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 8 more Thanks, Chris.
Bug#1025012: zookeeper: starts but is completely unusable
I got a bit further: Setting: CLASSPATH="/etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-simple.jar" i.e. adding the ":/usr/share/java/slf4j-simple.jar" helps a bit... The server seems to start now, and via zkCli, I can `ls` my paths and `get` values. But there's still no logging (neither to file nor to systemd)... ...and zkCli used to show a "prompt" like: [zk: localhost:2181(CONNECTED) 3] commands go here but now, there is nothing, though I can enter commands. May be unrelated though
Bug#1025012: zookeeper: starts but is completely unusable
I should perhaps add, that I have installed the zookeeper packages (zookeeper zookeeperd libzookeeper-java) from testing into stable (bullseye), all other dependencies were already met with bullseye versions. Also, according to https://www.slf4j.org/codes.html#StaticLoggerBinder and there the question "Failed to load class org.slf4j.impl.StaticLoggerBinder"... the files "slf4j-nop.jar slf4j- simple.jar, slf4j-log4j12.jar, slf4j-jdk14.jar or logback-classic.jar" are needed,... but any except for the latter is in place. All are in /usr/share/java. Though manually adding this to the CLASSPATH in /etc/zookeeper/conf/environment doesn't seem to help.
Bug#1025012: zookeeper: starts but is completely unusable
Package: zookeeper Version: 3.8.0-10 Severity: grave Justification: renders package unusable Hey. I've tried the new packagin, but while all my config and data files are in place, and while the server "runs", there is no logging (neither to stdout/err for systemd nor /var/log/zookeeper .. not even any files get created in there). Connecting via zkCli gives: # /usr/share/zookeeper/bin/zkCli.sh SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is disabled SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder". SLF4J: Defaulting to no-operation MDCAdapter implementation. SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details. Perhaps some missing dependency? Thanks, Chris.
