Hi Jeff removing the port number didn't work
what I should have mentioned was that I'm using flume 1.2, the same config I mentioned below works for flume 1.1, so I'm not sure what has changed in 1.2 for the configs and I need to use 1.2 because of the File channel I did see HDFSSinkConfiguration mentioned in SinkConfiguration.java but I can't find this class in the flume 1.2 jars, and ClassNotFoundException is being wrapped up in ConfigurationException which doesn't get reported in the log Ray On Thu, Sep 13, 2012 at 5:28 PM, Jeff Lord <[email protected]> wrote: > Raymond, > > I don't think you need the port 50070 on your hdfs path. > Try setting this to the fqdn of your hdfs namenode, leave off the port > 50070 (this is just for the webui) > > -Jeff > > On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <[email protected]> wrote: > >> Hi >> >> I'm baffled as to what the problem is with my hdfs config below >> >> agent.sinks.probeHdfsSink2.type = hdfs >> agent.sinks.probeHdfsSink2.hdfs.path = >> hdfs://master:50070/user/test/data/flume/%{client} >> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d >> agent.sinks.probeHdfsSink2.channel = probeFileChannel2 >> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120 >> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200 >> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0 >> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000 >> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text >> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream >> the flume log below does not provide enough info to help identify the >> problem, FlumeConfiguration.AgentConfiguration holds a list of >> <FlumeConfigurationError> but none of them get displayed in the log. >> >> If I replace hdfs sink with a file sink then everything works, so the >> problem is definitely in my hdfs configs but I can't see where? any advice >> is appreciated >> >> 2012-09-13 16:49:43,145 INFO [main] LifecycleSupervisor.java - Starting >> lifecycle supervisor 1 >> 2012-09-13 16:49:43,151 INFO [main] FlumeNode.java - Flume node starting >> - agent >> 2012-09-13 16:49:43,156 INFO [lifecycleSupervisor-1-0] >> DefaultLogicalNodeManager.java - Node manager starting >> 2012-09-13 16:49:43,156 INFO [lifecycleSupervisor-1-2] >> AbstractFileConfigurationProvider.java - Configuration provider starting >> 2012-09-13 16:49:43,157 INFO [lifecycleSupervisor-1-0] >> LifecycleSupervisor.java - Starting lifecycle supervisor 11 >> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0] >> DefaultLogicalNodeManager.java - Node manager started >> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2] >> AbstractFileConfigurationProvider.java - Configuration provider started >> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0] >> AbstractFileConfigurationProvider.java - Checking >> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes >> 2012-09-13 16:49:43,160 INFO [conf-file-poller-0] >> AbstractFileConfigurationProvider.java - Reloading configuration >> file:/usr/local/lib/flume-ng/conf/acme-agent.conf >> 2012-09-13 16:49:43,170 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Added sinks: probeHdfsSink2 Agent: agent >> 2012-09-13 16:49:43,171 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Created context for probeHdfsSink2: hdfs.rollSize >> 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,172 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeFileSink >> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Created context for probeFileSink: >> sink.rollInterval >> 2012-09-13 16:49:43,176 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,176 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeFileSink >> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeFileSink >> 2012-09-13 16:49:43,177 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeFileSink >> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,178 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeHdfsSink2 >> 2012-09-13 16:49:43,179 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Processing:probeFileSink >> 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Starting validation of configuration for agent: >> agent, initial-configuration: AgentConfiguration[agent] >> SOURCES: {probeAvroSrc2={ parameters:{port=5556, >> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }} >> CHANNELS: {probeFileChannel2={ >> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint, >> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3, >> transactionCapacity=200000, checkpointInterval=10, capacity=1000000, >> keep-alive=3, maxFileSize=1073741824, type=FILE} }} >> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream, >> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client}, >> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200, >> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text, >> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={ >> parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL, >> channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }} >> 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Created channel probeFileChannel2 >> 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Creating sink: probeHdfsSink2 using HDFS >> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Post validation configuration for agent >> AgentConfiguration created without Configuration stubs for which only >> basic syntactical validation was performed[agent] >> SOURCES: {probeAvroSrc2={ parameters:{port=5556, >> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }} >> CHANNELS: {probeFileChannel2={ >> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint, >> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3, >> transactionCapacity=200000, checkpointInterval=10, capacity=1000000, >> keep-alive=3, maxFileSize=1073741824, type=FILE} }} >> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream, >> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client}, >> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200, >> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text, >> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }} >> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Channels:probeFileChannel2 >> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Sinks probeHdfsSink2 >> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] >> FlumeConfiguration.java - Sources probeAvroSrc2 >> 2012-09-13 16:49:43,219 INFO [conf-file-poller-0] >> FlumeConfiguration.java - Post-validation flume configuration contains >> configuration for agents: [agent] >> 2012-09-13 16:49:43,219 INFO [conf-file-poller-0] >> PropertiesFileConfigurationProvider.java - Creating channels >> 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0] >> DefaultChannelFactory.java - Creating instance of channel probeFileChannel2 >> type FILE >> 2012-09-13 16:49:43,236 INFO [conf-file-poller-0] >> MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL, >> name: probeFileChannel2, registered successfully. >> 2012-09-13 16:49:43,236 INFO [conf-file-poller-0] >> PropertiesFileConfigurationProvider.java - created channel probeFileChannel2 >> 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0] >> DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type >> avro >> 2012-09-13 16:49:43,256 INFO [conf-file-poller-0] >> MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE, >> name: probeAvroSrc2, registered successfully. >> 2012-09-13 16:49:43,277 INFO [conf-file-poller-0] >> DefaultSinkFactory.java - Creating instance of sink: probeHdfsSink2, type: >> hdfs >> >> >> -- >> Rgds >> Ray >> > > -- Rgds Ray
