Hello Rajesh: I just worked on a POC for the work that I am doing with JMS --> HDFS configuration using ActiveMQ. I was able to use everything right out of the box, going with JMS Source --> Memory Channel --> HDFS Sink, based on the guide (https://flume.apache.org/FlumeUserGuide.html). Below is my example config that I was playing around with. Note that for ActiveMQ, you do not want to specify the connectionFactory in the source configuration ( https://flume.apache.org/FlumeUserGuide.html#jms-source). A documentation defect was created for this point ( https://issues.apache.org/jira/browse/FLUME-2339). As for questions 3 and 4, I will let someone with more experience answer.
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe the source #a1.sources.r1.type = netcat #a1.sources.r1.bind = localhost #a1.sources.r1.port = 44444 a1.sources.r1.type = jms a1.sources.r1.channels = c1 a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory a1.sources.r1.providerURL = tcp://<activemqserver>:61616 a1.sources.r1.destinationName = TESTQUEUE a1.sources.r1.destinationType = QUEUE # Describe the sink #a1.sinks.k1.type = logger a1.sinks.k1.type = hdfs a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d a1.sinks.k1.hdfs.filePrefix = messages a1.sinks.k1.hdfs.rollInterval = 30 #a1.sinks.k1.hdfs.rollSize = 1024 a1.sinks.k1.hdfs.rollCount = 1000 a1.sinks.k1.hdfs.fileType = DataStream # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 Hope this helps, Richard. On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar < [email protected]> wrote: > Hello Team, > > I'm new to flume. Could you please help me on below query > > 1. How to configure JMS source(TIBCO and ActiveMQ) in flume? > 2. Do i need to write custom source for above requirement? (Or) Out Of > the Box flume JMS source implementation is fine? > 3. What is the best channel mechanism for above requirement? > 4. Each message will be sink into HDFS? (or) Can we consolidate some > messages and sink into HDFS? > > Thank you in advance > > Regards, > Rajesh >
