Raymond, I don't think you need the port 50070 on your hdfs path. Try setting this to the fqdn of your hdfs namenode, leave off the port 50070 (this is just for the webui)
-Jeff On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <[email protected]> wrote: > Hi > > I'm baffled as to what the problem is with my hdfs config below > > agent.sinks.probeHdfsSink2.type = hdfs > agent.sinks.probeHdfsSink2.hdfs.path = > hdfs://master:50070/user/test/data/flume/%{client} > agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d > agent.sinks.probeHdfsSink2.channel = probeFileChannel2 > agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120 > agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200 > agent.sinks.probeHdfsSink2.hdfs.rollCount = 0 > agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000 > agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text > agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream > the flume log below does not provide enough info to help identify the > problem, FlumeConfiguration.AgentConfiguration holds a list of > <FlumeConfigurationError> but none of them get displayed in the log. > > If I replace hdfs sink with a file sink then everything works, so the > problem is definitely in my hdfs configs but I can't see where? any advice > is appreciated > > 2012-09-13 16:49:43,145 INFO [main] LifecycleSupervisor.java - Starting > lifecycle supervisor 1 > 2012-09-13 16:49:43,151 INFO [main] FlumeNode.java - Flume node starting > - agent > 2012-09-13 16:49:43,156 INFO [lifecycleSupervisor-1-0] > DefaultLogicalNodeManager.java - Node manager starting > 2012-09-13 16:49:43,156 INFO [lifecycleSupervisor-1-2] > AbstractFileConfigurationProvider.java - Configuration provider starting > 2012-09-13 16:49:43,157 INFO [lifecycleSupervisor-1-0] > LifecycleSupervisor.java - Starting lifecycle supervisor 11 > 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0] > DefaultLogicalNodeManager.java - Node manager started > 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2] > AbstractFileConfigurationProvider.java - Configuration provider started > 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0] > AbstractFileConfigurationProvider.java - Checking > file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes > 2012-09-13 16:49:43,160 INFO [conf-file-poller-0] > AbstractFileConfigurationProvider.java - Reloading configuration > file:/usr/local/lib/flume-ng/conf/acme-agent.conf > 2012-09-13 16:49:43,170 INFO [conf-file-poller-0] FlumeConfiguration.java > - Added sinks: probeHdfsSink2 Agent: agent > 2012-09-13 16:49:43,171 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Created context for probeHdfsSink2: hdfs.rollSize > 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeFileSink > 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Created context for probeFileSink: sink.rollInterval > 2012-09-13 16:49:43,176 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,176 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeFileSink > 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeFileSink > 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeFileSink > 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeHdfsSink2 > 2012-09-13 16:49:43,179 INFO [conf-file-poller-0] FlumeConfiguration.java > - Processing:probeFileSink > 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Starting validation of configuration for agent: agent, > initial-configuration: AgentConfiguration[agent] > SOURCES: {probeAvroSrc2={ parameters:{port=5556, > channels=probeFileChannel2, type=avro, bind=192.168.200.170} }} > CHANNELS: {probeFileChannel2={ > parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint, > dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3, > transactionCapacity=200000, checkpointInterval=10, capacity=1000000, > keep-alive=3, maxFileSize=1073741824, type=FILE} }} > SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream, > hdfs.path=hdfs://master:50070/user/test/data/flume/%{client}, > hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200, > hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text, > type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={ > parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL, > channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }} > 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Created channel probeFileChannel2 > 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Creating sink: probeHdfsSink2 using HDFS > 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Post validation configuration for agent > AgentConfiguration created without Configuration stubs for which only > basic syntactical validation was performed[agent] > SOURCES: {probeAvroSrc2={ parameters:{port=5556, > channels=probeFileChannel2, type=avro, bind=192.168.200.170} }} > CHANNELS: {probeFileChannel2={ > parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint, > dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3, > transactionCapacity=200000, checkpointInterval=10, capacity=1000000, > keep-alive=3, maxFileSize=1073741824, type=FILE} }} > SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream, > hdfs.path=hdfs://master:50070/user/test/data/flume/%{client}, > hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200, > hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text, > type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }} > 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Channels:probeFileChannel2 > 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Sinks probeHdfsSink2 > 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] FlumeConfiguration.java > - Sources probeAvroSrc2 > 2012-09-13 16:49:43,219 INFO [conf-file-poller-0] FlumeConfiguration.java > - Post-validation flume configuration contains configuration for agents: > [agent] > 2012-09-13 16:49:43,219 INFO [conf-file-poller-0] > PropertiesFileConfigurationProvider.java - Creating channels > 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0] > DefaultChannelFactory.java - Creating instance of channel probeFileChannel2 > type FILE > 2012-09-13 16:49:43,236 INFO [conf-file-poller-0] > MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL, > name: probeFileChannel2, registered successfully. > 2012-09-13 16:49:43,236 INFO [conf-file-poller-0] > PropertiesFileConfigurationProvider.java - created channel probeFileChannel2 > 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0] > DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type > avro > 2012-09-13 16:49:43,256 INFO [conf-file-poller-0] > MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE, > name: probeAvroSrc2, registered successfully. > 2012-09-13 16:49:43,277 INFO [conf-file-poller-0] DefaultSinkFactory.java > - Creating instance of sink: probeHdfsSink2, type: hdfs > > > -- > Rgds > Ray >
