There is another approach which is listed below. kafka -> logstash consumer (which is logstash-kafka) -> es you can check logstash-kafka at "https://github.com/joekiller/logstash-kafka"; And, I tried the approach which you mentioned above. However, for me, es sink of flume is kind of unstable. Anyway, you can try it. Hope it helps.
On Tue, Dec 2, 2014 at 3:25 AM, ES_MacTalk <[email protected]> wrote: > Hi, > We are trying to move data from a Kafka Topic to ElasticSearch.We are > getting data in JSON format in the Kafka Topic.We are planning to move this > data to Elastic Search and then finally visualize using kibana. We are right > now using Flume Elastic search sink using the default serializer for the > same.But we are not able to visualize the data in Kibana.. > > In the elastic search ,the > { > "_index": "handleindex6-2014-12-01", > "_type": "bar_type", > "_id": "AUoGQAHfOA15NKJPcqsy", > "_score": 1, > "_source": { > *"@message": > "{\"handle\":\"speed\",\"mac\":\"WIN1223\",\"owner\":\"Sam\",\"schema > ID\":\"12B007\",\"data\":\"19.66\",\"timestamp\":1366150981}",* > "@timestamp": "2014-12-01T14:27:45.509Z", > "@fields": { > "topic": "mtalk", > "timestamp": "1417444065509" > } > We are seeing that the data is getting embedded withing the message part of > the json saved in elastic search.Is this the right approach > Kafka-->Flume-->ES-->Kibana? > > > > > > > -- > View this message in context: > http://elasticsearch-users.115913.n3.nabble.com/Kafka-to-ElasticSearch-tp4066982.html > Sent from the ElasticSearch Users mailing list archive at Nabble.com. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/1417458311296-4066982.post%40n3.nabble.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADQPeWz%2B7YtaL9rSFqsKfs-dvVDte0RMp%2BnFyu1Q%3Do%2BCUVwjqw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
