I am working on a POC in which I am trying to have flume emulate scribe output by using a flume scribe sink that I found online. There was no docs with it so I have poked through code to try and figure out how to configure this properly. I have gotten the it configured to the point where a scribe server receives the data but does not recognize any scribe categories. I have included my config file below. In this config file I'm simply doing a tail -f on some bogus log files that I generate for the input and trying to create 3 streams of output similar to what a scribe intake server might be doing in receiving multiple streams of scribe data and funneling it to a scribe integration server.
Has anyone used this flume scribe sink successfully and, if so, can you help me get my config file setup properly? Thank you. Regards, Dwight Marzolf # Name the components on this agent agent1.sources = exec-tail-ds1 exec-tail-ds2 exec-tail-ds3 agent1.sinks = scribe-forward-sink1 scribe-forward-sink2 scribe-forward-sink3 agent1.channels = file-channel-1 ########################## # SOURCE ########################## # Configure source 1 agent1.sources.exec-tail-ds1.type = exec agent1.sources.exec-tail-ds1.command = tail -F /tmp/dataset1 agent1.sources.exec-tail-ds1.channels = file-channel-1 # Configure source 2 agent1.sources.exec-tail-ds2.type = exec agent1.sources.exec-tail-ds2.command = tail -F /tmp/dataset2 agent1.sources.exec-tail-ds2.channels = file-channel-1 # Configure source 3 agent1.sources.exec-tail-ds3.type = exec agent1.sources.exec-tail-ds3.command = tail -F /tmp/dataset3 agent1.sources.exec-tail-ds3.channels = file-channel-1 ########################## # SINK ########################## # Config for scribe sink 1 agent1.sinks.scribe-forward-sink1.type = org.apache.flume.sink.scribe.ScribeSink agent1.sinks.scribe-forward-sink1.scribe.sink.name = ds1 agent1.sinks.scribe-forward-sink1.scribe.port = 1463 agent1.sinks.scribe-forward-sink1.scribe.host = 10.48.140.8 agent1.sinks.scribe-forward-sink1.scribe.category = dataset1 agent1.sinks.scribe-forward-sink1.scribe.category.header = category agent1.sinks.scribe-forward-sink1.channel = file-channel-1 # Config for scribe sink 2 agent1.sinks.scribe-forward-sink2.type = org.apache.flume.sink.scribe.ScribeSink agent1.sinks.scribe-forward-sink2.scribe.sink.name = ds2 agent1.sinks.scribe-forward-sink2.scribe.port = 1463 agent1.sinks.scribe-forward-sink2.scribe.host = 10.48.140.8 agent1.sinks.scribe-forward-sink2.scribe.category = dataset2 agent1.sinks.scribe-forward-sink2.scribe.category.header = category agent1.sinks.scribe-forward-sink2.channel = file-channel-1 # Config for scribe sink 3 agent1.sinks.scribe-forward-sink3.type = org.apache.flume.sink.scribe.ScribeSink agent1.sinks.scribe-forward-sink3.scribe.sink.name = ds3 agent1.sinks.scribe-forward-sink3.scribe.port = 1463 agent1.sinks.scribe-forward-sink3.scribe.host = 10.48.140.8 agent1.sinks.scribe-forward-sink3.scribe.category.header= category agent1.sinks.scribe-forward-sink3.scribe.category = dataset3 agent1.sinks.scribe-forward-sink3.channel = file-channel-1 ########################## # CHANNEL ########################## # Channel file buffer 1 agent1.channels.file-channel-1.type = file agent1.channels.file-channel-1.checkpointDir = /scribe/channel/checkpoint1 agent1.channels.file-channel-1.dataDirs = /scribe/channel/data1 ########################## # BINDING ########################## # Bind the source and sink to the channel agent1.sources.exec-tail-ds1.channels = file-channel-1 agent1.sinks.scribe-forward-sink1.channel = file-channel-1 [Description: cid:B446C73C-E88B-4947-BD2A-586ECE21AB0D] Dwight Marzolf Sr. Engineer Data & Analytics Phone: +1 781 430 5228 HERE Burlington 5 Wayside Rd 42° 29' 7" N. 71° 11' 32" W HERE, a Nokia business
<<inline: image001.png>>
