Re: [Structured Streaming Query] Calculate Running Avg from Kafka feed using SQL query

Hey Felix, I've already tried with .format("memory") .queryName("tableName") but, still, it doesn't work for the second query. It just stalls the program expecting new data for the first query. Here's my code - from pyspark.sql import SparkSession from pyspark.sql.functions import split

Re: [Structured Streaming Query] Calculate Running Avg from Kafka feed using SQL query

Instead of write to console you need to write to memory for it to be queryable .format("memory") .queryName("tableName") https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-sinks From: Aakash Basu

Re: [Structured Streaming Query] Calculate Running Avg from Kafka feed using SQL query

Hi all, The following is the updated code, where I'm getting the avg in a DF, but the collect() function, to store the value as a variable and pass it to the final select query is not working. So, avg is currently a dataframe and not a variable with value stored in it. New code - from

Re: [Structured Streaming Query] Calculate Running Avg from Kafka feed using SQL query

Any help, guys? On Mon, Apr 2, 2018 at 1:01 PM, Aakash Basu wrote: > Hi, > > This is a very interesting requirement, where I am getting stuck at a few > places. > > *Requirement* - > > Col1Col2 > 1 10 > 2 11 > 3 12 > 4