Hello:
I have just downloaded and build flume-ng (apache-flume-1.3.0-SNAPSHOT). My goal is to collect log data from HostA (source) and send it to HostB(sink), my initial test (sending /etc/passwd) from HostA to HostB worked fine, I was also able to load the passwd file into my HBase in HostB. Now, I want to load a continuous stream of log data (using tail -f), but I was not able to replicate the above process. Flume just started fine in HostA, but I do not see any data being received by HostB or in my HBase. What is wrong with my configuration? Thanks, Suresh Here is my flume.conf in HostA agent3.sources = tail agent3.channels = MemoryChannel-1 agent3.sinks = avro-sink # Define source flow agent3.sources.tail.type = exec agent3.sources.tail.command = tail -f /var/log/auth.log agent3.sources.tail.channels = MemoryChannel-1 # What kind of channel agent3.channels.MemoryChannel-1.type = memory # avro sink properties agent3.sinks.avro-sink.type = avro agent3.sinks.avro-sink.channel = MemoryChannel-1 agent3.sinks.avro-sink.hostname = hostb agent3.sinks.avro-sink.port = 41414 Here is my flume.conf in HostB # Define a memory channel called ch1 on agent1 agent1.channels.ch1.type = memory # Define an Avro source called avro-source1 on agent1 and tell it # to bind to 0.0.0.0:41414. Connect it to channel ch1. agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.type = avro agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 41414 # Define a logger sink that simply logs all events it receives # and connect it to the other end of the same channel. agent1.sinks.log-sink1.channel = ch1 agent1.sinks.log-sink1.type = logger # Finally, now that we've defined all of our components, tell # agent1 which ones we want to activate. agent1.channels = ch1 agent1.sources = avro-source1 #agent1.sources = avro-source1 agent1.sinks = sink1 agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink agent1.sinks.sink1.channel = ch1 agent1.sinks.sink1.table = flumedemo agent1.sinks.sink1.columnFamily = testing agent1.sinks.sink1.column = foo agent1.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer agent1.sinks.sink1.serializer.payloadColumn = col1 agent1.sinks.sink1.serializer.keyType = timestamp agent1.sinks.sink1.serializer.rowPrefix = 1 agent1.sinks.sink1.serializer.suffix = timestamp agent1.sinks.sink1.serializer.payloadColumn = pcol agent1.sinks.sink1.serializer.incrementColumn = icol
