We also just encountered this same issue as originally reported by Raivis. I'm not sure what is going on either. It just happened all of a sudden when a worker restarted.
From: pwozniak <[email protected]> Sent: Tuesday, February 15, 2022 8:28 AM To: [email protected]; [email protected] Subject: [External] Re: Address already bound issue This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments. ________________________________ Hi, we encountered similar problem lately. In our case it was related with JMX port that we use for our worker. Quite strange in here is that it happened some time after worker startup (not during worker startup). Our Storm version is 2.3.0 (we lately upgraded it from 1.1.3). Any Ideas would be appreciated :) 2022-01-26 19:43:09.655 e.e.c.s.d.s.AbstractDpsBolt Thread-103-EDMObjectProcessorBolt-executor[17, 17] [INFO] Attempt number 2 to process this message. No cleaning done here. 2022-01-27 09:43:35.558 STDERR Thread-0 [INFO] Error: JMX connector server communication error: service:jmx:rmi://storm-supervisor2.novalocal:16700 2022-01-27 09:43:35.583 STDERR Thread-0 [INFO] jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 16700; nested exception is: 2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] java.net.BindException: Address already in use (Bind failed) 2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:820) 2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479) 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] Caused by: java.rmi.server.ExportException: Port already in use: 16700; nested exception is: 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] java.net.BindException: Address already in use (Bind failed) 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335) 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243) 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412) 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234) 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:203) 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:153) 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:138) 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:473) 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816) 2022-01-27 09:43:35.588 STDERR Thread-0 [INFO] ... 3 more Regards, Pawel On 12/22/21 11:16 AM, Raivis Strogonovs wrote: Hello, I was wondering if anybody could help me out with the issue I'm having for the past couple months. For some reason our storm instances are running fine for couple weeks, and then suddenly it get's address bound issue, which can only be resolved by restarting the respective machine. And it's not like it only happens one machine, but randomly on any of them We are still using 2.2.0 at the moment. The instances are running in docker. So far I haven't had luck at finding or figuring out, why would it happen... See logs here 2021-12-22 10:03:39.057 o.a.s.m.n.Server main [INFO] Create Netty Server Netty-server-localhost-6701, buffer_size: 5242880, maxWorkers: 1 2021-12-22 10:03:39.062 STDERR Thread-0 [INFO] WARNING: An illegal reflective access operation has occurred 2021-12-22 10:03:39.075 STDERR Thread-0 [INFO] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar<file://apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar>) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object) 2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil 2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations 2021-12-22 10:03:39.082 STDERR Thread-0 [INFO] WARNING: All illegal access operations will be denied in a future release 2021-12-22 10:03:39.144 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server... java.lang.Error: java.security.PrivilegedActionException: java.net.BindException: Address already in use at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:655) ~[storm-client-2.2.0.jar:2.2.0] at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:634) ~[storm-client-2.2.0.jar:2.2.0] at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1016) ~[storm-client-2.2.0.jar:2.2.0] at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?] at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?] at java.lang.Thread.dispatchUncaughtException(Unknown Source) [?:?] Caused by: java.security.PrivilegedActionException at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at javax.security.auth.Subject.doAs(Unknown Source) ~[?:?] at org.apache.storm.daemon.worker.Worker.start(Worker.java:188) ~[storm-client-2.2.0.jar:2.2.0] at org.apache.storm.daemon.worker.Worker.main(Worker.java:153) ~[storm-client-2.2.0.jar:2.2.0] Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) ~[?:?] at sun.nio.ch.Net.bind(Unknown Source) ~[?:?] at sun.nio.ch.Net.bind(Unknown Source) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:?] at org.apache.storm.shade.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[storm-shaded-deps-2.2.0.jar:2.2.0] at java.lang.Thread.run(Unknown Source) ~[?:?] Any Ideas would be appreciated. Thank You, Raivis Disclaimer: The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. The security of customers and suppliers as a priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email. No employee has the authority to conclude any binding contract without an explicit written consent of their supervisor. Therefore, any will to enter into an agreement must be confirmed by a second manager or director. Any quotation contained is sent to compare available offers and does not imply entering into a legally binding contract. Please do not print this email unless it is necessary. Every unprinted email helps the environment. The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorise infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails. ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy. ______________________________________________________________________________________ www.accenture.com
