[ https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956265#comment-15956265 ]
tycho_yang edited comment on FLUME-3021 at 4/5/17 3:59 AM: ----------------------------------------------------------- HI Yonghao Zou Have you tested flume Kafka channels with your sink to elasticsearch5.x,The next one I'm testing is wrong。Can determine the elasticsearch cluster is normal 2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:82)] Kafka version : 0.9.0.1 2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:83)] Kafka commitId : 23c69d62a0cabf06 2017-04-05 11:49:38,908 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.KafkaChannel.createConsumerAndRecords(KafkaChannel.java:296)] Created new consumer to connect to Kafka 2017-04-05 11:49:39,202 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.kafka.KafkaSource.doStart(KafkaSource.java:520)] Kafka source s1 started. 2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: s1: Successfully registered new MBean. 2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: s1 started 2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatCompletionHandler.handle(AbstractCoordinator.java:623)] Attempt to heart beat failed since the group is rebalancing, try to re-join group. 2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 0 revoked. 2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 1 revoked. 2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 0 assigned. 2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 1 assigned. 2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 0 assigned. 2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 1 assigned. 2017-04-05 11:49:42,769 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:222)] Failed to commit transaction. Transaction rolled back. org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200) at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106) at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:167) at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:370) at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:166) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:179) at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:206) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) 2017-04-05 11:49:42,771 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows. org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] was (Author: ytc301): HI Yonghao Zou Have you tested flume Kafka channels with your sink to elasticsearch5.x,The next one I'm testing is wrong。 2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:82)] Kafka version : 0.9.0.1 2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:83)] Kafka commitId : 23c69d62a0cabf06 2017-04-05 11:49:38,908 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.KafkaChannel.createConsumerAndRecords(KafkaChannel.java:296)] Created new consumer to connect to Kafka 2017-04-05 11:49:39,202 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.kafka.KafkaSource.doStart(KafkaSource.java:520)] Kafka source s1 started. 2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: s1: Successfully registered new MBean. 2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: s1 started 2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatCompletionHandler.handle(AbstractCoordinator.java:623)] Attempt to heart beat failed since the group is rebalancing, try to re-join group. 2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 0 revoked. 2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 1 revoked. 2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 0 assigned. 2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 1 assigned. 2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 0 assigned. 2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 1 assigned. 2017-04-05 11:49:42,769 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:222)] Failed to commit transaction. Transaction rolled back. org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200) at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106) at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:167) at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:370) at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:166) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:179) at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:206) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) 2017-04-05 11:49:42,771 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows. org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] Can determine the elasticsearch cluster is normal > flume Elasticsearch 5.0 not support > ----------------------------------- > > Key: FLUME-3021 > URL: https://issues.apache.org/jira/browse/FLUME-3021 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources > Affects Versions: 1.7.0 > Reporter: tycho_yang > Assignee: Yonghao Zou > Fix For: 1.8.0 > > Attachments: FLUME-3021-0.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)