How do I Specify windowInterval and slideInteval using raw sql string? On Tue, Jul 25, 2017 at 8:52 AM, Keith Chapman <[email protected]> wrote:
> You could issue a raw sql query to spark, there is no particular advantage > or disadvantage of doing so. Spark would build a logical plan from the raw > sql (or DSL) and optimize on that. Ideally you would end up with the same > physical plan, irrespective of it been written in raw sql / DSL. > > Regards, > Keith. > > http://keith-chapman.com > > On Tue, Jul 25, 2017 at 12:50 AM, kant kodali <[email protected]> wrote: > >> HI All, >> >> I just want to run some spark structured streaming Job similar to this >> >> DS.filter(col("name").equalTo("john")) >> .groupBy(functions.window(df1.col("TIMESTAMP"), "24 hours", "24 >> hours"), df1.col("hourlyPay")) >> .agg(sum("hourlyPay").as("total")); >> >> >> I am wondering if I can express the above query in raw sql string? >> >> If so how would that look like and what are some of the disadvantages of >> using raw sql query vs spark DSL? >> >> >> Thanks! >> >> >
