Thanks! I have everything working just fine now with the following settings (writing to s3)
agent.sources = seqGenSrc agent.channels = memoryChannel agent.sinks = hdfsSink # For each one of the sources, the type is defined agent.sources.seqGenSrc.type = seq agent.sources.seqGenSrc.channels = memoryChannel agent.sources.seqGenSrc.interceptors = a b agent.sources.seqGenSrc.interceptors.a.type = org.apache.flume.interceptor.HostInterceptor$Builder agent.sources.seqGenSrc.interceptors.a.preserveExisting = false agent.sources.seqGenSrc.interceptors.a.hostHeader = hostname agent.sources.seqGenSrc.interceptors.b.type = org.apache.flume.interceptor.TimestampInterceptor$Builder # Each channel's type is defined. agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 1000 agent.channels.memoryChannel.transactionCapacity = 1000 agent.sinks.hdfsSink.type = hdfs agent.sinks.hdfsSink.channel = memoryChannel agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H agent.sinks.hdfsSink.hdfs.filePrefix = events- agent.sinks.hdfsSink.hdfs.batchSize = 100000 agent.sinks.hdfsSink.hdfs.rollCount = 0 agent.sinks.hdfsSink.hdfs.rollSize = 0 agent.sinks.hdfsSink.hdfs.round = true agent.sinks.hdfsSink.hdfs.roundValue = 10 agent.sinks.hdfsSink.hdfs.roundUnit = minute On Thu, Aug 2, 2012 at 4:21 PM, Hari Shreedharan <[email protected]>wrote: > agent.sinks.hdfsSink.channels = memoryChannel <-needs to be > "agent.sinks.hdfsSink.channel = memoryChannel" > > Hari > -- > Hari Shreedharan > > On Thursday, August 2, 2012 at 4:19 PM, David Capwell wrote: > > I downloaded the latest release (1.2.0) of flume and tried to get it > sending data to HDFS; for some reason it isn't adding hdfs as a sink. Here > is my properties file: > > > agent.sources = seqGenSrc > agent.channels = memoryChannel > agent.sinks = hdfsSink > > # For each one of the sources, the type is defined > agent.sources.seqGenSrc.type = seq > agent.sources.seqGenSrc.channels = memoryChannel > > # Each sink's type must be defined > agent.sinks.loggerSink.type = logger > agent.sinks.loggerSink.channel = memoryChannel > > # Each channel's type is defined. > agent.channels.memoryChannel.type = memory > agent.channels.memoryChannel.capacity = 100 > > agent.sinks.hdfsSink.type = hdfs > > agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S > agent.sinks.hdfsSink.hdfs.filePrefix = > events-#agent.sinks.hdfsSink.hdfs.round = true > agent.sinks.hdfsSink.hdfs.roundValue = > 10#agent.sinks.hdfsSink.hdfs.roundUnit = second > > > Here is what I see in the logs with debugging enabled: > > 2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] > 649 Creating sink: hdfsSink using HDFS > 2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN - > org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)] > 668 Configuration empty for: hdfsSink.Removed. > > Going over the code I wasn't sure where the ConfigurationException was > thrown so I did a test and commented out agent.sinks.hdfsSink.channels (to > get linke 655 to fail [sinkConf.configure(sinkContext);] ) but still get > the same error. > > > am I missing something? > > Thanks for your time reading this email. > > >
