I've been building a Spark Streaming application for a couple months without
really running into any problems. Today I copied and pasted most of the
code into a new application for a similar process.
In order to test the new process, I use a println statement as the final
action, and I'm getting good results for every Spark Streaming bucket.
searchLines.foreachRDD(rdd => {
rdd.take(30).foreach(x=>println(x+"\n\n"))
})
However, when I add anything to searchLines, including dummy tests like
.filter(line => true) or .map(line => line), the output doesn't print. It's
so strange.
Has anyone run into something like this before?
Below is the relevant code that is producing correct output. Uncommenting
either of the commented lines results in zero output displaying in
yarn-client mode.
val searchLines = ...
))
} else List(("NA","NA","NA","NA","NA","NA","NA"))
}
} else List(("NA","NA","NA","NA","NA","NA","NA"))
}).filter(line => line._2 != "NA")
.filter(_._3 != "-")
//.filter(line => true)
//.map(line => line)
searchLines.foreachRDD(rdd => {
rdd.take(30).foreach(x=>println(x+"\n\n"))
})
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/The-strangest-error-any-change-to-my-StreamRDD-seems-to-break-the-output-tp21509.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
