On 2021-09-27 17:45:14, micah wrote: > All of our puppet runs were having this same issue. > > Once we downgraded libjetty9, the problem went away.
I can confirm we are also having this issue, and that the downgrade fixes the issue. Oddly, however, the problem *only* started after the recent buster point release. It's strange, because we did upgrade the jetty9 package all the way back when it was published, in August 2021. And we did restart PuppetDB (reboot the host even) since then, so I'm pretty sure we ran PuppetDB with the new jetty9 package without problem. Therefore it seems the downgrade fix might point at another issue, unrelated to jetty9? I'm tempted to think an upgrade of the PuppetDB package might help here though. Why are we even stuck in 6.2 land? There's been 18 upstream releases on 6.x since... :) > It appears this updated version came from this security update: > https://lists.debian.org/debian-security-announce/2021/msg00132.html > > I tried also to use the backport package, but it also had this problem. I tried to use the backport package but it utterly failed to even start PuppetDB at all. It would crash with: Mar 29 15:58:33 pauli/pauli java[5522]: 15:58:32.770 [main] DEBUG puppetlabs.puppetdb.http - The v1 API has been retired; please use v4 Caught HTTP processing exception Mar 29 15:58:33 pauli/pauli java[5522]: 15:58:32.778 [main] DEBUG puppetlabs.puppetdb.http - The v2 API has been retired; please use v4 Caught HTTP processing exception Mar 29 15:58:33 pauli/pauli java[5522]: 15:58:32.779 [main] DEBUG puppetlabs.puppetdb.http - The v3 API has been retired; please use v4 Caught HTTP processing exception Mar 29 15:58:33 pauli/pauli java[5522]: 15:58:33.358 [main] DEBUG puppetlabs.trapperkeeper.bootstrap - Loading bootstrap config from classpath: 'jar:file:/usr/share/puppetdb/puppetdb.jar!/bootstrap.cfg' Mar 29 15:58:35 pauli/pauli java[5522]: Exception in thread "main" java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1283) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1265) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:372) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.server.Server.doStart(Server.java:401) Mar 29 15:58:35 pauli/pauli java[5522]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$eval43528$start_webserver_BANG___43533$fn__43534$fn__43535.invoke(jetty9_core.clj:685) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$eval43528$start_webserver_BANG___43533$fn__43534.invoke(jetty9_core.clj:684) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$eval43528$start_webserver_BANG___43533.invoke(jetty9_core.clj:677) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$start_server_single_default.invokeStatic(jetty9_core.clj:929) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$start_server_single_default.invoke(jetty9_core.clj:926) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$eval43947$start_BANG___43952$fn__43953.invoke(jetty9_core.clj:1008) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_core$eval43947$start_BANG___43952.invoke(jetty9_core.clj:1003) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services.webserver.jetty9_service$reify__44354$service_fnk__23931__auto___positional$reify__44361.start(jetty9_service.clj:44) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services$eval23729$fn__23743$G__23719__23746.invoke(services.clj:8) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.services$eval23729$fn__23743$G__23718__23750.invoke(services.clj:8) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24312$run_lifecycle_fn_BANG___24319$fn__24320.invoke(internal.clj:204) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24312$run_lifecycle_fn_BANG___24319.invoke(internal.clj:187) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24341$run_lifecycle_fns__24346$fn__24347.invoke(internal.clj:238) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24341$run_lifecycle_fns__24346.invoke(internal.clj:215) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24899$build_app_STAR___24908$fn$reify__24920.start(internal.clj:591) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24947$boot_services_for_app_STAR__STAR___24954$fn__24955$fn__24957.invoke(internal.clj:617) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24947$boot_services_for_app_STAR__STAR___24954$fn__24955.invoke(internal.clj:615) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24947$boot_services_for_app_STAR__STAR___24954.invoke(internal.clj:609) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core$partial$fn__5826.invoke(core.clj:2630) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24383$initialize_lifecycle_worker__24394$fn__24395$fn__24534$state_machine__12865__auto____24559$fn__24562.invoke(internal.clj:255) Mar 29 15:58:35 pauli/pauli java[5522]: at puppetlabs.trapperkeeper.internal$eval24383$initialize_lifecycle_worker__24394$fn__24395$fn__24534$state_machine__12865__auto____24559.invoke(internal.clj:255) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async$ioc_alts_BANG_$fn__13094.invoke(async.clj:384) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async$do_alts$fn__13026$fn__13029.invoke(async.clj:253) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.core.async.impl.channels.ManyToManyChannel$fn__7046$fn__7047.invoke(channels.clj:95) Mar 29 15:58:35 pauli/pauli java[5522]: at clojure.lang.AFn.run(AFn.java:22) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) Mar 29 15:58:35 pauli/pauli java[5522]: at java.base/java.lang.Thread.run(Thread.java:829) Mar 29 15:58:35 pauli/pauli systemd[1]: puppetdb.service: Main process exited, code=exited, status=1/FAILURE Mar 29 15:58:35 pauli/pauli systemd[1]: puppetdb.service: Failed with result 'exit-code'. -- Only after disaster can we be resurrected. It's only after you've lost everything that you're free to doanything. Nothing is static, everything is evolving, everything is falling apart. - Chuck Palahniuk, Fight Club