dzmitryk opened a new issue #6330: Bug in netty 4.1.43 prevents from using Linux native Epoll transport URL: https://github.com/apache/pulsar/issues/6330 **Describe the bug** netty 4.1.43 has a bug preventing it from using Linux native Epoll transport - https://github.com/netty/netty/issues/9758 This results in pulsar brokers failing over to `NioEventLoopGroup` even when running on Linux. The bug seems to be fixed in netty releases 4.1.44+ **To Reproduce** 1. Run pulsar docker container: ``` docker run -it apachepulsar/pulsar:latest bin/pulsar standalone ``` 2. Observe the logs. Following errors appears on startup ``` 23:56:05.428 [main] WARN org.apache.distributedlog.impl.BKNamespaceDriver - Could not use Netty Epoll event loop for bookie server: java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.EpollEventLoop at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:257) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:268) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:206) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:239) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doStart(DLNamespaceProviderService.java:120) ~[org.apache.bookkeeper-stream-storage-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407) [com.google.guava-guava-25.1-jre.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doStart(StreamStorageLifecycleComponent.java:51) [org.apache.bookkeeper-stream-storage-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runStreamStorage(LocalBookkeeperEnsemble.java:323) [org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:260) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:119) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] ``` With DEBUG logging enabled following error appears as well: ``` 22:31:13.747 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism. java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) ~[?:1.8.0_232] at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[?:1.8.0_232] at java.lang.System.loadLibrary(System.java:1122) ~[?:1.8.0_232] at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232] at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_232] at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.Native.<clinit>(Native.java:57) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:52) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final] at org.apache.bookkeeper.util.EventLoopUtil.getEventLoopGroup(EventLoopUtil.java:59) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.util.EventLoopUtil.getClientEventLoopGroup(EventLoopUtil.java:43) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:458) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:296) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:70) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at org.apache.pulsar.broker.ManagedLedgerClientFactory.<init>(ManagedLedgerClientFactory.java:61) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:394) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:264) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:329) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] ``` **Expected behavior** Netty `EpollEventLoopGroup` is initialized correctly.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
