you are right jeff. i was using a old version of hadoop jar. problem solved now. thanks On Sep 14, 2012 10:17 PM, "Jeff Lord" <[email protected]> wrote:
> Let's backup a step. > Do you have the hadoop libs installed? > In this version you need to have the hadoop libs installed in order for > flume to write to hdfs sink. > > > On Fri, Sep 14, 2012 at 12:35 AM, Raymond Ng <[email protected]> wrote: > >> Hi Jeff >> >> removing the port number didn't work >> >> what I should have mentioned was that I'm using flume 1.2, the same >> config I mentioned below works for flume 1.1, so I'm not sure what has >> changed in 1.2 for the configs and I need to use 1.2 because of the File >> channel >> I did see HDFSSinkConfiguration mentioned in SinkConfiguration.java but >> I can't find this class in the flume 1.2 jars, and ClassNotFoundException >> is being wrapped up in ConfigurationException which doesn't get reported in >> the log >> >> >> Ray >> >> >> On Thu, Sep 13, 2012 at 5:28 PM, Jeff Lord <[email protected]>wrote: >> >>> Raymond, >>> >>> I don't think you need the port 50070 on your hdfs path. >>> Try setting this to the fqdn of your hdfs namenode, leave off the port >>> 50070 (this is just for the webui) >>> >>> -Jeff >>> >>> On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <[email protected]>wrote: >>> >>>> Hi >>>> >>>> I'm baffled as to what the problem is with my hdfs config below >>>> >>>> agent.sinks.probeHdfsSink2.type = hdfs >>>> agent.sinks.probeHdfsSink2.hdfs.path = >>>> hdfs://master:50070/user/test/data/flume/%{client} >>>> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d >>>> agent.sinks.probeHdfsSink2.channel = probeFileChannel2 >>>> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120 >>>> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200 >>>> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0 >>>> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000 >>>> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text >>>> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream >>>> the flume log below does not provide enough info to help identify the >>>> problem, FlumeConfiguration.AgentConfiguration holds a list of >>>> <FlumeConfigurationError> but none of them get displayed in the log. >>>> >>>> If I replace hdfs sink with a file sink then everything works, so the >>>> problem is definitely in my hdfs configs but I can't see where? any advice >>>> is appreciated >>>> >>>> 2012-09-13 16:49:43,145 INFO [main] LifecycleSupervisor.java - >>>> Starting lifecycle supervisor 1 >>>> 2012-09-13 16:49:43,151 INFO [main] FlumeNode.java - Flume node >>>> starting - agent >>>> 2012-09-13 16:49:43,156 INFO [lifecycleSupervisor-1-0] >>>> DefaultLogicalNodeManager.java - Node manager starting >>>> 2012-09-13 16:49:43,156 INFO [lifecycleSupervisor-1-2] >>>> AbstractFileConfigurationProvider.java - Configuration provider starting >>>> 2012-09-13 16:49:43,157 INFO [lifecycleSupervisor-1-0] >>>> LifecycleSupervisor.java - Starting lifecycle supervisor 11 >>>> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0] >>>> DefaultLogicalNodeManager.java - Node manager started >>>> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2] >>>> AbstractFileConfigurationProvider.java - Configuration provider started >>>> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0] >>>> AbstractFileConfigurationProvider.java - Checking >>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes >>>> 2012-09-13 16:49:43,160 INFO [conf-file-poller-0] >>>> AbstractFileConfigurationProvider.java - Reloading configuration >>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf >>>> 2012-09-13 16:49:43,170 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Added sinks: probeHdfsSink2 Agent: agent >>>> 2012-09-13 16:49:43,171 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Created context for probeHdfsSink2: hdfs.rollSize >>>> 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeFileSink >>>> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Created context for probeFileSink: >>>> sink.rollInterval >>>> 2012-09-13 16:49:43,176 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,176 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeFileSink >>>> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeFileSink >>>> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeFileSink >>>> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeHdfsSink2 >>>> 2012-09-13 16:49:43,179 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Processing:probeFileSink >>>> 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Starting validation of configuration for agent: >>>> agent, initial-configuration: AgentConfiguration[agent] >>>> SOURCES: {probeAvroSrc2={ parameters:{port=5556, >>>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }} >>>> CHANNELS: {probeFileChannel2={ >>>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint, >>>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3, >>>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000, >>>> keep-alive=3, maxFileSize=1073741824, type=FILE} }} >>>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream, >>>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client}, >>>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200, >>>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text, >>>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={ >>>> parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL, >>>> channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }} >>>> 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Created channel probeFileChannel2 >>>> 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Creating sink: probeHdfsSink2 using HDFS >>>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Post validation configuration for agent >>>> AgentConfiguration created without Configuration stubs for which only >>>> basic syntactical validation was performed[agent] >>>> SOURCES: {probeAvroSrc2={ parameters:{port=5556, >>>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }} >>>> CHANNELS: {probeFileChannel2={ >>>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint, >>>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3, >>>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000, >>>> keep-alive=3, maxFileSize=1073741824, type=FILE} }} >>>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream, >>>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client}, >>>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200, >>>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text, >>>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }} >>>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Channels:probeFileChannel2 >>>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Sinks probeHdfsSink2 >>>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] >>>> FlumeConfiguration.java - Sources probeAvroSrc2 >>>> 2012-09-13 16:49:43,219 INFO [conf-file-poller-0] >>>> FlumeConfiguration.java - Post-validation flume configuration contains >>>> configuration for agents: [agent] >>>> 2012-09-13 16:49:43,219 INFO [conf-file-poller-0] >>>> PropertiesFileConfigurationProvider.java - Creating channels >>>> 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0] >>>> DefaultChannelFactory.java - Creating instance of channel probeFileChannel2 >>>> type FILE >>>> 2012-09-13 16:49:43,236 INFO [conf-file-poller-0] >>>> MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL, >>>> name: probeFileChannel2, registered successfully. >>>> 2012-09-13 16:49:43,236 INFO [conf-file-poller-0] >>>> PropertiesFileConfigurationProvider.java - created channel >>>> probeFileChannel2 >>>> 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0] >>>> DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type >>>> avro >>>> 2012-09-13 16:49:43,256 INFO [conf-file-poller-0] >>>> MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE, >>>> name: probeAvroSrc2, registered successfully. >>>> 2012-09-13 16:49:43,277 INFO [conf-file-poller-0] >>>> DefaultSinkFactory.java - Creating instance of sink: probeHdfsSink2, type: >>>> hdfs >>>> >>>> >>>> -- >>>> Rgds >>>> Ray >>>> >>> >>> >> >> >> -- >> Rgds >> Ray >> > >
