Eran W created FLUME-2787:
-----------------------------
Summary:
org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer does not
serialize @timestamp correctly
Key: FLUME-2787
URL: https://issues.apache.org/jira/browse/FLUME-2787
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Reporter: Eran W
Priority: Minor
When using
agent.sinks.elastic-sink.serializer =
org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
the event timestamp is stored as string and not dateOptionalTime if the
agent.sinks.elastic-sink.serializer is not set the code works as expected.
REPRO:
1) use the following config
agent.channels.channel-elastic.type = memory
agent.channels.channel-elastic.capacity = 1000
agent.channels.channel-elastic.transactionCapacity = 100
# Define a source on agent and connect to channel.
agent.sources.tail-source.type = exec
agent.sources.tail-source.command = tail -4000 /home/cto/hs_err_pid11679.log
agent.sources.tail-source.channels = channel-elastic
#####INTERCEPTORS
agent.sources.tail-source.interceptors = timestampInterceptor
agent.sources.tail-source.interceptors.timestampInterceptor.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
agent.sources.tail-source.interceptors.timestampInterceptor.preserveExisting =
true
agent.sinks.elastic-sink.channel = channel-elastic
agent.sinks.elastic-sink.type =
org.apache.flume.sink.elasticsearch.ElasticSearchSink
agent.sinks.elastic-sink.hostNames = 127.0.0.1:9300
agent.sinks.elastic-sink.indexName = flume_index
agent.sinks.elastic-sink.indexType = logs_type
agent.sinks.elastic-sink.clusterName = elasticsearch
agent.sinks.elastic-sink.batchSize = 10
agent.sinks.elastic-sink.ttl = 5d
agent.sinks.elastic-sink.serializer =
org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
# Finally, activate.
agent.channels = channel-elastic
agent.sources = tail-source
agent.sinks = elastic-sink
2) run it
3) look at the elastic index which was created:
{
"state": "open",
"settings": {
"index": {
"creation_date": "1441728286466",
"number_of_shards": "5",
"number_of_replicas": "1",
"version": {
"created": "1070199"
},
"uuid": "9u-OCPxoQHWwURHyxh15lA"
}
},
"mappings": {
"logs_type": {
"properties": {
"body": {
"type": "string"
},
"timestamp": {
"type": "string"
}
}
}
},
"aliases": [ ]
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
