Hi,
I have installed
spark-0.9.0-incubating-bin-cdh4 and I am using apache flume for streaming. I
have used the streaming.examples.FlumeEventCount. Also I have written Avro conf
file for flume.When I try to do streamin ing spark and I run the following
command it throws error
./bin/run-example org.apache.spark.streaming.examples.FlumeEventCount local[2]
ip 10001
org.jboss.netty.channel.ChannelException: Failed to bind to: /ip:9988
at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:119)
at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:74)
at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:68)
at
org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:143)
at
org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126)
at
org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:173)
at
org.apache.spark.streaming.scheduler.NetworkInputTracker$ReceiverExecutor$$anonfun$8.apply(NetworkInputTracker.scala:169)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:49)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
... 3 more
Can you please help
This is my flumetest.conf
[cid:image001.png@01CF5E3E.0CE2FEE0]
a1.sources = tail-file
a1.channels = c1
a1.sinks=avro-sink
# define the flow
a1.sources.tail-file.channels = c1
a1.sinks.avro-sink.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
# define source and sink
a1.sources.tail-file.type = exec
a1.sources.tail-file.command = tail -F /home/hduser/Flume/test.log
#a1.sources.tail-file.channels = c1
a1.sinks.avro-sink.type = avro
a1.sinks.avro-sink.hostname = ip // agent a2.s ip address or host name
a1.sinks.avro-sink.port = 10001
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channel
a1.sources.tail-file.channels = c1
a1.sinks.avro-sink.channel = c1
Flumeslavetest.conf:-
[cid:image002.png@01CF5E3E.0CE2FEE0]
a2.sources = avro-collection-source
a2.sinks = hdfs-sink
a2.channels = mem-channel
# define the flow
a2.sources.avro-collection-source.channels = mem-channel
a2.sinks.hdfs-sink.channel = mem-channel
a2.channels.mem-channel.type = memory
a2.channels.mem-channel.capacity = 1000
# avro source properties
a2.sources.avro-collection-source.type = avro
a2.sources.avro-collection-source.bind = ip // agent a2.s ip address or host
name
a2.sources.avro-collection-source.port = 10001
# hdfs sink properties
a2.sinks.hdfs-sink.type = hdfs
a2.sinks.hdfs-sink.hdfs.writeFormat = Text
a2.sinks.hdfs-sink.hdfs.filePrefix = testing
a2.sinks.hdfs-sink.hdfs.path = hdfs://ip:8020/testingData
My flume is running properly its is able to write file on hdfs.
Please help as to how to resolve the error
Regards,
Neha Singh
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or
privileged information for the intended recipient(s). Unintended recipients are
prohibited from taking action on the basis of information in this e-mail and
using or disseminating the information, and must notify the sender and delete
it from their system. L&T Infotech will not accept responsibility or liability
for the accuracy or completeness of, or the presence of any virus or disabling
code in this e-mail"
