Hi,
I am trying to build a simple notebook to confirm dynamic streaming updates
to an angular widget via z.bindAngular.
note that I use %pyspark streaming. Here is my simplified variation of the
pyspark streaming tutorial, which i confirmed working in the pyspark shell:
Note1:
%pyspark
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
def pushCount(c):
global z
z.put("c",c)
return c
ssc = StreamingContext(sc, 5)
lines = ssc.socketTextStream("localhost",9999)
lines.count().map(pushCount).pprint()
ssc.start()
in Note2, which is Scala/spark, we communicate the count value to the
angular, which is also ripped from a working example on Zeppelin PR 27:
Note:
val timer = new java.util.Timer()
// start monitoring. once a second.
val task = new java.util.TimerTask {
def run() {
val c = z.get("c")
z.angularBind("c", Array("TestCount" -> c))
}
}
timer.schedule(task, 1000L, 1000L)
Finally, there is a note attaching angular progressbar attached to the
variable "c" (counts).
I confirmed that if i execute z.put("c", 5) in a %pyspark note, the bar
updates (via timer task polling). But update from the stream doesn't seem
to be working at all, dead on arrival. I can't get any values from it no
matter what i do.
(1) Is there a pyspark streaming support in Zeppelin at all? Was it ever
verified to work? In fact, i haven't found a single example anywhere of
Zeppelin + pyspark streaming.
(2) Or, the assumption that z.put() works from inside DStream.map is wrong?
Which could be true if this closure is shipped via spark-submit somewhere.
(3) or i am doing something else wrong?
Master is local[*]. I am running the master Zeppelin branch (latest
snapshot).
thank you very much in advance,
-d
