[jira] [Commented] (FLINK-8334) Elasticsearch Connector throwing java.lang.ClassNotFoundException: org.jboss.netty.channel.socket.nio.SocketSendBufferPool$GatheringSendBuffer
[ https://issues.apache.org/jira/browse/FLINK-8334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308330#comment-16308330 ] Bhaskar Divya commented on FLINK-8334: -- Thanks [~aljoscha]. I think I got it working now.(no more failures on start of the Job) My POM was messed up trying to fix this issue. So, I re-created a fresh POM from the maven archetype of 1.4.0. I put the dependencies I required one by one. Also, There were issues with the Elasticsearch docker environment. For anybody looking at this, Please go through [this link|https://stackoverflow.com/questions/41192680/update-max-map-count-for-elasticsearch-docker-container-mac-host#41251595] to set vm.max_map_count And, set the following environment variables in docker container as below : http.host = 0.0.0.0 transport.host = 0.0.0.0 xpack.security.enabled = false Also, The POM which is working for me is 1) Add dependency in the outer dependencies block {noformat} org.apache.flink flink-connector-elasticsearch5_${scala.binary.version} ${flink.version} {noformat} 2) Follow the guidelines from [here|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/linking.html]. Add the provided configuration. Inside the I added the following for elasticsearch {noformat} org.apache.flink flink-connector-elasticsearch5_${scala.binary.version} 1.4.0 jar false ${project.build.directory}/classes org/apache/flink/** org.elasticsearch elasticsearch 5.1.2 jar false ${project.build.directory}/classes ** org.elasticsearch.plugin transport-netty3-client 5.1.2 jar false ${project.build.directory}/classes ** io.netty netty 3.10.6.Final jar false ${project.build.directory}/classes ** {noformat} Few of them are definitely extras and maybe not actually required. Hope it helps anyone trying the Elasticsearch connector with Docker. > Elasticsearch Connector throwing java.lang.ClassNotFoundException: > org.jboss.netty.channel.socket.nio.SocketSendBufferPool$GatheringSendBuffer > -- > > Key: FLINK-8334 > URL: https://issues.apache.org/jira/browse/FLINK-8334 > Project: Flink > Issue Type: Bug > Components: ElasticSearch Connector >Affects Versions: 1.4.0 > Environment: Using Elasticsearch 5.1.2 in a docker environment > Flink is deployed on a different docker >Reporter: Bhaskar Divya > Labels: elasticsearch, netty > > I have a Elasticsearch sink configured. When a job is submitted, It goes into > fail status in a few seconds. > Following is the Exception from the Job screen: > {code:java} > java.lang.RuntimeException: Elasticsearch client is not connected to any > Elasticsearch nodes! > at > org.apache.flink.streaming.connectors.elasticsearch5.Elasticsearch5ApiCallBridge.createClient(Elasticsearch5ApiCallBridge.java:80) > at > org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.open(ElasticsearchSinkBase.java:281) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at > org.apache.flink.streaming.
[jira] [Commented] (FLINK-8334) Elasticsearch Connector throwing java.lang.ClassNotFoundException: org.jboss.netty.channel.socket.nio.SocketSendBufferPool$GatheringSendBuffer
[ https://issues.apache.org/jira/browse/FLINK-8334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308168#comment-16308168 ] Aljoscha Krettek commented on FLINK-8334: - Could you check if your jar files contain any netty classes? This could be a clash in dependencies or Maven might be excluding stuff since it "thinks" it comes provided with Flink. > Elasticsearch Connector throwing java.lang.ClassNotFoundException: > org.jboss.netty.channel.socket.nio.SocketSendBufferPool$GatheringSendBuffer > -- > > Key: FLINK-8334 > URL: https://issues.apache.org/jira/browse/FLINK-8334 > Project: Flink > Issue Type: Bug > Components: ElasticSearch Connector >Affects Versions: 1.4.0 > Environment: Using Elasticsearch 5.1.2 in a docker environment > Flink is deployed on a different docker >Reporter: Bhaskar Divya > Labels: elasticsearch, netty > > I have a Elasticsearch sink configured. When a job is submitted, It goes into > fail status in a few seconds. > Following is the Exception from the Job screen: > {code:java} > java.lang.RuntimeException: Elasticsearch client is not connected to any > Elasticsearch nodes! > at > org.apache.flink.streaming.connectors.elasticsearch5.Elasticsearch5ApiCallBridge.createClient(Elasticsearch5ApiCallBridge.java:80) > at > org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.open(ElasticsearchSinkBase.java:281) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) > at > org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:393) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) > at java.lang.Thread.run(Thread.java:748) > {code} > In the logs, Following stack trace is shown. > {code} > 2018-01-01 12:15:14,432 INFO > org.elasticsearch.client.transport.TransportClientNodesService - failed to > get node info for > {#transport#-1}{8IZTMPcSRCyKRynhfyN2fA}{192.168.99.100}{192.168.99.100:9300}, > disconnecting... > NodeDisconnectedException[[][192.168.99.100:9300][cluster:monitor/nodes/liveness] > disconnected] > 2018-01-01 12:15:19,433 ERROR org.elasticsearch.transport.netty3.Netty3Utils > - fatal error on the network layer > at > org.elasticsearch.transport.netty3.Netty3Utils.maybeDie(Netty3Utils.java:195) > at > org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:82) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:291) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) > at > org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:292) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto