I am running a spark streaming example from learning spark book with one
change. The change I made was for streaming a file from HDFS.
val lines = ssc.textFileStream("hdfs:/user/hadoop/spark/streaming/input")
I ran the application number of times and every time dropped a new file in
the input directory after the program started running but never got any
output.
I even passed a non existent directory as the input to the textFileStream
but the application did not throw any error and ran just like it did when i
had the right directory.
I am able to access the same HDFS file system from non smearing application
fine.
I wasted a day trying to figure this out:). Any help with be greatly
appreciated.
package com.oreilly.learningsparkexamples.scala
import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream._
object StreamingLogInput {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("StreamingLogInput")
val ssc = new StreamingContext(conf, Seconds(1))
val lines =
ssc.textFileStream("hdfs:/user/hadoop/spark/streaming/input")
println("testtesttestteste")
lines.print()
ssc.start()
ssc.awaitTermination(60000)
ssc.stop()
}
def processLines(lines: DStream[String]) = {
lines.filter(_.contains("error"))
}