Thanks, this looks better
// parse the lines of data into sensor objects
val sensorDStream = ssc.textFileStream("/stream").
map(Sensor.parseSensor)
sensorDStream.foreachRDD { rdd =>
// filter sensor data for low psi
val alertRDD = rdd.filter(sensor => sensor.psi <
Yes, for example "val sensorRDD = rdd.map(Sensor.parseSensor)" is a
line of code executed on the driver; it's part the function you
supplied to foreachRDD. However that line defines an operation on an
RDD, and the map function you supplied (parseSensor) will ultimately
be carried out on the cluster
I think what you’ll want is to carry out the .map functions before the
foreachRDD, something like:
val lines =
ssc.textFileStream("/stream").map(Sensor.parseSensor).map(Sensor.convertToPut)
lines.foreachRDD { rdd =>
// parse the line of data into sensor object
rdd.saveAsHadoo