[
https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932726#comment-15932726
]
Yonghao Zou commented on FLUME-3021:
------------------------------------
Hi, [~ytc301]
I tested with following configuration:
--------------------------------------------------------------------------------------------------------------------
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# source interceptors
a1.sources.r1.interceptors.i0.regex = ([^
]*)\\s+-\\s+(.+?|-)\\s+\\[(.*)\\]\\s+\"([^ ]*)\\s+([^ ]*)\\s+([^
|\"]*)\"\\s+(-|[0-9]*)\\s+(-|[0-9]*)\\s+\"(.+?|-)\"\\s+\"(.+?|-)\"\\s+\"(.+?|-)\"\\s+\"(.+?|-)\"\\s+(.+?|-)\\s+(.+?|-)\\s+(.+?|-)\\s+\"(.+?|-)\"\\s+(.+?|-)\\s+(.*)
a1.sources.r1.interceptors=i0 i3
a1.sources.r1.interceptors.i0.type=regex_extractor
a1.sources.r1.interceptors.i0.serializers = s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11
s12 s13 s14 s15 s16 s17 s18
a1.sources.r1.interceptors.i0.serializers.s1.name = remote_addr
a1.sources.r1.interceptors.i0.serializers.s2.name = user
a1.sources.r1.interceptors.i0.serializers.s3.type =
org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
a1.sources.r1.interceptors.i0.serializers.s3.name = timestamp
a1.sources.r1.interceptors.i0.serializers.s3.pattern= yyyy-MM-dd'T'HH:mm:ssZ
a1.sources.r1.interceptors.i0.serializers.s4.name = request_methond
a1.sources.r1.interceptors.i0.serializers.s5.name = request_url
a1.sources.r1.interceptors.i0.serializers.s6.name = request_http
a1.sources.r1.interceptors.i0.serializers.s7.name = http_status
a1.sources.r1.interceptors.i0.serializers.s8.name = body_bytes_sent
a1.sources.r1.interceptors.i0.serializers.s9.name = referer
a1.sources.r1.interceptors.i0.serializers.s10.name = user_Agent
a1.sources.r1.interceptors.i0.serializers.s11.name = http_x_forwarded_for
a1.sources.r1.interceptors.i0.serializers.s12.name = gzip_ratio
a1.sources.r1.interceptors.i0.serializers.s13.name = request_time
a1.sources.r1.interceptors.i0.serializers.s14.name = bytes_sent
a1.sources.r1.interceptors.i0.serializers.s15.name = request_length
a1.sources.r1.interceptors.i0.serializers.s16.name = upstream_addr
a1.sources.r1.interceptors.i0.serializers.s17.name = upstream_status
a1.sources.r1.interceptors.i0.serializers.s18.name = upstream_response_time
a1.sources.r1.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder
a1.sources.r1.interceptors.i3.hostHeader = host
# Describe the sink
# a1.sinks.k1.type = logger
a1.sinks.k1.type = elasticsearch
a1.sinks.k1.hostNames = 127.0.0.1:9300
a1.sinks.k1.indexName = foo_index
a1.sinks.k1.indexType = bar_type
a1.sinks.k1.clusterName = elasticsearch
a1.sinks.k1.batchSize = 500
a1.sinks.k1.ttl = 5d
a1.sinks.k1.serializer.outputCharset = UTF-8
a1.sinks.k1.serializer =
org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer
# a1.sinks.k1.serializer =
org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
--------------------------------------------------------------------------------------------------------------------
Then I telnet to localhost:44444 and input your data which is
--------------------------------------------------------------------------------------------------------------------
119.12.11.22 - - [2016-11-07T11:52:45+08:00] "GET
/skin//frontend/default/images/transparAent.gif HTTP/1.1" 304 0
"http://www.baidu.com/ti/account/amsders"; "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
"125.77.22.1" "-" 0.000 223 979 "192.168.2.101:8011" 304 0.000
--------------------------------------------------------------------------------------------------------------------
and it inserted into es successfully.
It seems your error's reason maybe some connection problems? I see ex like
NodeNotConnectedException happend.
And you can email to me directly with Chinese, I'll be happy to help you with.
> flume Elasticsearch 5.0 not support
> -----------------------------------
>
> Key: FLUME-3021
> URL: https://issues.apache.org/jira/browse/FLUME-3021
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.7.0
> Reporter: tycho_yang
> Assignee: Yonghao Zou
> Fix For: 1.8.0
>
> Attachments: FLUME-3021-0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
