[ 
https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932389#comment-15932389
 ] 

Helmut Wahrman edited comment on FLUME-3021 at 3/20/17 9:39 AM:
----------------------------------------------------------------

I already corrected all the dependencies. I have jackson 2.8.6, the es jar and 
all the transport jars in flume lib.

My conf looks like this. I am reading an avro file and my custom serializer 
reads the schema from the avro file.
This works pretty fine in pre 2.0 versions of ES.

This is my config:

a1.sources = avrofld
a1.sinks = k1
a1.channels = c1

a1.sources.avrofld.type = spooldir
a1.sources.avrofld.channels = c1
a1.sources.avrofld.spoolDir = /var/export
a1.sources.avrofld.fileHeader = true
a1.sources.avrofld.deserializer = avro

a1.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
a1.sinks.k1.hostNames = 192.168.1.50:9300
a1.sinks.k1.indexName = sa
a1.sinks.k1.indexType = data
a1.sinks.k1.clusterName = elasticsearch
a1.sinks.k1.batchSize = 1000
a1.sinks.k1.ttl = 2d
a1.sinks.k1.serializer = com.rsa.flume.serialization.FlumeAvroEventDeserializer

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 100000

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


Maybe you can post your config that works fine for you and i will try on my side



was (Author: hwahrmann):
I already corrected all the dependencies. I have jackson 2.8.6, the es jar and 
all the transport jars in flume lib.

My conf looks like this. I am reading an avro file and my custom serializer 
reads the schema from the avro file.
This works pretty fine in pre 2.0 versions of ES.

This is my config:

# Name the components on this agent
a1.sources = avrofld
a1.sinks = k1
a1.channels = c1

# Describe/configure the folder with the avro files from Warehouse Connector
a1.sources.avrofld.type = spooldir
a1.sources.avrofld.channels = c1
a1.sources.avrofld.spoolDir = /var/export
a1.sources.avrofld.fileHeader = true
a1.sources.avrofld.deserializer = avro

# Describe the elasticsearch sink
a1.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
a1.sinks.k1.hostNames = 192.168.1.50:9300
a1.sinks.k1.indexName = sa
a1.sinks.k1.indexType = data
a1.sinks.k1.clusterName = elasticsearch
a1.sinks.k1.batchSize = 1000
a1.sinks.k1.ttl = 2d
a1.sinks.k1.serializer = com.rsa.flume.serialization.FlumeAvroEventDeserializer

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 100000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


Maybe you can post your config that works fine for you and i will try on my side


> flume Elasticsearch 5.0 not support
> -----------------------------------
>
>                 Key: FLUME-3021
>                 URL: https://issues.apache.org/jira/browse/FLUME-3021
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.7.0
>            Reporter: tycho_yang
>            Assignee: Yonghao Zou
>             Fix For: 1.8.0
>
>         Attachments: FLUME-3021-0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to