GBM-tamerm commented on issue #3292:
URL: https://github.com/apache/bookkeeper/issues/3292#issuecomment-1137930267
> > But it is causing issue as shown the above excpetion trace
> > the auto-recovery is failing when the leader ZK stopped and new election
start , and when it fail , it still shut down bookies nodes that has
auto-recovery , although i manually stopped auto recovery before shut down the
ZK leader .
> > what is the solution?
>
> @GBM-tamerm In bookies you need to disable auto-recovery by setting in
`bookkeeper.conf`:
>
> ```
> autoRecoveryDaemonEnabled=false
> ```
>
> Then you can run auto-recovery as a separate stateless service:
>
> ```
> bin/bookkeeper autorecovery
> ```
i tried that now , but autorecovery is failing with below excpetion
2022-05-25T19:02:14,298-0400 [main] ERROR
org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Calling
uncaughtExceptionHandler
2022-05-25T19:02:14,299-0400 [main] ERROR
org.apache.bookkeeper.common.component.ComponentStarter - Triggered
exceptionHandler of Component: autorecovery-server because of Exception in
Thread: Thread[main,5,main]
java.lang.RuntimeException: java.io.IOException: Failed to bind to
/0.0.0.0:8000
at
org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:114)
~[org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.server.service.StatsProviderService.doStart(StatsProviderService.java:51)
~[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406)
[com.google.guava-guava-30.1-jre.jar:?]
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144)
[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85)
[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.replication.AutoRecoveryMain.doMain(AutoRecoveryMain.java:334)
[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.replication.AutoRecoveryMain.main(AutoRecoveryMain.java:308)
[org.apache.bookkeeper-bookkeeper-server-4.14.4.jar:4.14.4]
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:8000
at
org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:111)
~[org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.4.jar:4.14.4]
... 8 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_332]
at sun.nio.ch.Net.bind(Net.java:461) ~[?:1.8.0_332]
at sun.nio.ch.Net.bind(Net.java:453) ~[?:1.8.0_332]
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
~[?:1.8.0_332]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
~[?:1.8.0_332]
at
org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
~[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
at
org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:111)
~[org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.4.jar:4.14.4]
... 8 more
2022-05-25T19:02:14,299-0400 [component-shutdown-thread] INFO
org.apache.bookkeeper.common.component.ComponentStarter - Closing component
autorecovery-server in shutdown hook.
2022-05-25T19:02:14,301-0400 [main] INFO
org.apache.bookkeeper.common.component.ComponentStarter - Started component
autorecovery-server.
2022-05-25T19:02:14,301-0400 [component-shutdown-thread] ERROR
org.apache.bookkeeper.common.component.ComponentStarter - Failed to close
component autorecovery-server in shutdown hook gracefully, Exiting anyway
java.lang.IllegalStateException: Can't move to closed before moving to
stopped mode
at
org.apache.bookkeeper.common.component.Lifecycle.moveToClosed(Lifecycle.java:185)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:121)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$6(LifecycleComponentStack.java:154)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406)
~[com.google.guava-guava-30.1-jre.jar:?]
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:154)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.ComponentStarter$ComponentShutdownHook.run(ComponentStarter.java:47)
[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
2022-05-25T19:02:14,303-0400 [main] ERROR
org.apache.bookkeeper.replication.AutoRecoveryMain - Error in bookie shutdown
java.lang.IllegalStateException: Can't move to closed before moving to
stopped mode
at
org.apache.bookkeeper.common.component.Lifecycle.moveToClosed(Lifecycle.java:185)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:121)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$6(LifecycleComponentStack.java:154)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406)
~[com.google.guava-guava-30.1-jre.jar:?]
at
org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:154)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.common.component.ComponentStarter$ComponentShutdownHook.run(ComponentStarter.java:47)
~[org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_332]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
