Hi, you can try out foreachBatch to apply the batch query operation to the each output of micro-batch: http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch
Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) On Mon, Jan 20, 2020 at 8:43 PM Nick Dawes <nickdawe...@gmail.com> wrote: > Streaming experts, any clues how to achieve this? > > After extracting few variables, I need to run them through a REST API for > verification and decision making. > > Thanks for your help. > > Nick > > On Fri, Jan 17, 2020, 6:27 PM Nick Dawes <nickdawe...@gmail.com> wrote: > >> I need to extract a value from a PySpark structured streaming Dataframe >> to a string variable to check something. >> >> I tried this code. >> >> agentName = >> kinesisDF.select(kinesisDF.agentName.getItem(0).alias("agentName")).collect()[0][0] >> >> This works on a non-streaming Dataframe only. In a streaming Dataframe, >> collect is not supported. >> >> Any workaround for this? >> >> Nick >> >> >>