Thanks that worked for the config I have below.
Now I change this to stream to an avro sink it stopped working.
I even tried adding it to the hdfs conf,but it doesn't produce the
static interceptors.
Any ideas on why it works as a file_roll sync but not avro sink?
Client.config
client.channels=ch1
client.channels.ch1.type=memory
client.channels.ch1.capacity=100000
client.channels.ch1.transactionCapacity=1000000000000
client.sources=src-1
client.sources.src-1.type=spooldir
client.sources.src-1.spoolDir=/root/unpack
client.sources.src-1.deserializer.maxLineLength=10000
client.sources.src-1.interceptors = i2 i1
client.sources.src-1.interceptors.i1.type = host
client.sources.src-1.interceptors.i1.hostHeader = hostname
#client.sources.src-1.interceptors.i1.useIP = true
client.sources.src-1.interceptors.i2.type = static
client.sources.src-1.interceptors.i2.key = environment
client.sources.src-1.interceptors.i2.value = sqa
client.sinks=k1
client.sinks.k1.type=avro
client.sinks.k1.hostname=localhost
client.sinks.k1.port=42424
client.sinks.ki.sink.serializer = HEADER_AND_TEXT
## Debugging Sink, Comment out AvroSink if you use this one
# http://flume.apache.org/FlumeUserGuide.html#file-roll-sink
#client.sinks.k1.type = file_roll
#client.sinks.k1.sink.directory = /root/sink
#client.sinks.k1.sink.rollInterval = 0
#client.sinks.k1.sink.serializer = HEADER_AND_TEXT
# Connect soure and sink with channel
client.sources.src-1.channels=ch1
client.sinks.k1.channel=ch1
HDFS conf
collector.sources=av1
collector.sources.av1.interceptors = i2
collector.sources.av1.interceptors.i2.type = timestamp
collector.sources.av1.type=avro
collector.sources.av1.bind=0.0.0.0
collector.sources.av1.port=42424
collector.sources.av1.channels=ch1
collector.channels=ch1
collector.channels.ch1.type=memory
collector.channels.ch1.capacity = 100000
collector.channels.ch1.transactionCapacity = 1000000000000
collector.sinks=k1
collector.sinks.k1.type=hdfs
collector.sinks.k1.channel=ch1
collector.sinks.k1.hdfs.path=/flume/%y%m%d
collector.sinks.k1.hdfs.fileType = DataStream
collector.sinks.k1.hdfs.rollInterval = 86400
collector.sinks.k1.hdfs.rollSize = 0
collector.sinks.k1.hdfs.rollCount = 0
collector.sinks.k1.hdfs.serializer = HEADER_AND_TEXT
On 9/16/14 11:10 AM, Ashish wrote:
Re: Static Interceptors Not Working
Try using HEADER_AND_TEXT as serializer for sink, default is Text
Serializer that writes only the Event body.
On Tue, Sep 16, 2014 at 7:31 PM, christopher palm <[email protected]> wrote:
> All,
>
> I am trying to get the static interceptor to insert key, value information
> in each line that is written
> out in my data sink.
> I have tried this with various configurations, but can't seem to get any
> output from the interceptor
> to show up in the output files produced by Flume in the target data
> directory.
> Below is my latest config, using spooldir as the source and a file_roll
> sink as the output.
>
> Any suggestions as to what I am configuring wrong here?
>
> Thanks,
> Chris
>
> client.channels=ch1
> client.channels.ch1.type=memory
> client.channels.ch1.capacity=100000
> client.channels.ch1.transactionCapacity=100000
>
> client.sources=src-1
> client.sources.src-1.type=spooldir
> client.sources.src-1.spoolDir=/opt/app/solr/flume/sinkIn
> client.sources.src-1.deserializer.maxLineLength=10000
> client.sources.src-1.interceptors = i1
> client.sources.src-1.interceptors.i1.type = static
> client.sources.src-1.interceptors.i1.preserveExisting = false
> client.sources.src-1.interceptors.i1.key = datacenter
> client.sources.src-1.interceptors.i1.value= NYC_01
> client.sinks=k1
> #client.sinks.k1.type=avro
> #client.sinks.k1.hostname=localhost
> #client.sinks.k1.port=42424
> ## Debugging Sink, Comment out AvroSink if you use this one
> # http://flume.apache.org/FlumeUserGuide.html#file-roll-sink
> client.sinks.k1.type = file_roll
> client.sinks.k1.sink.directory = /opt/app/solr/flume/sinkOut
> client.sinks.k1.sink.rollInterval = 0
>
> # Connect soure and sink with channel
> client.sources.src-1.channels=ch1
> client.sinks.k1.channel=ch1
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
