Hi, I'm trying to setup an agent with a thrift legacy source and a HDFS sink. I tried with 1.1 and my configuration seemed to work and could get events but then my sink hit FLUME-1074 so I've built from 1.2.0 trunk, and now the thrift legact source doesn't seem to load. This is how my log looks like: 2012-05-02 16:42:46,464 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1 2012-05-02 16:42:46,467 INFO node.FlumeNode: Flume node starting - collector 2012-05-02 16:42:46,470 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting 2012-05-02 16:42:46,471 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10 2012-05-02 16:42:46,471 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting 2012-05-02 16:42:46,473 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume-conf.properties 2012-05-02 16:42:46,479 INFO conf.FlumeConfiguration: Processing:rawEventsHDFS 2012-05-02 16:42:46,480 INFO conf.FlumeConfiguration: Processing:rawEventsHDFS 2012-05-02 16:42:46,480 INFO conf.FlumeConfiguration: Processing:rawEventsHDFS 2012-05-02 16:42:46,480 INFO conf.FlumeConfiguration: Added sinks: rawEventsHDFS Agent: collector 2012-05-02 16:42:46,480 INFO conf.FlumeConfiguration: Processing:rawEventsHDFS 2012-05-02 16:42:46,480 INFO conf.FlumeConfiguration: Processing:rawEventsHDFS 2012-05-02 16:42:46,480 INFO conf.FlumeConfiguration: Processing:rawEventsHDFS 2012-05-02 16:42:46,505 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [collector] 2012-05-02 16:42:46,506 INFO properties.PropertiesFileConfigurationProvider: Creating channels 2012-05-02 16:42:46,516 INFO properties.PropertiesFileConfigurationProvider: created channel memoryChannel 2012-05-02 16:42:46,531 INFO sink.DefaultSinkFactory: Creating instance of sink rawEventsHDFS typehdfs
There is no error, but also no indication that the source was loaded, and when I try to telnet the source port the connection is refused indicating the port is closed. This is my configuration, which, as I mentioned above, was working fine with 1.1.0: collector.sources = rawEvents collector.channels = memoryChannel collector.sinks = rawEventsHDFS # For each one of the sources, the type is defined collector.sources.rawEvents.type = org.apache.flume.source.thriftLegacy.ThriftLegacySource # The channel can be defined as follows. collector.sources.rawEvents.channels = memoryChannel # Each sink's type must be defined collector.sinks.rawEventsHDFS.type = hdfs #Specify the channel the sink should use collector.sinks.rawEventsHDFS.channel = memoryChannel # Each channel's type is defined. collector.channels.memoryChannel.type = memory # Other config values specific to each type of channel(sink or source) # can be defined as well # In this case, it specifies the capacity of the memory channel collector.channels.memoryChannel.capacity = 100 collector.sources.rawEvents.host=0.0.0.0 collector.sources.rawEvents.port=54001 collector.sinks.rawEventsHDFS.hdfs.path = hdfs://hadoop1-m1:8020/test-events/%Y-%m-%d/ collector.sinks.rawEventsHDFS.hdfs.filePrefix=raw-events-%Y-%m-%d-%k-%M-%S collector.sinks.rawEventsHDFS.hdfs.rollInterval=60 collector.sinks.rawEventsHDFS.hdfs.fileType=DataStream Any idea what's going on? -eran
