You could issue a raw sql query to spark, there is no particular advantage or disadvantage of doing so. Spark would build a logical plan from the raw sql (or DSL) and optimize on that. Ideally you would end up with the same physical plan, irrespective of it been written in raw sql / DSL.
Regards, Keith. http://keith-chapman.com On Tue, Jul 25, 2017 at 12:50 AM, kant kodali <[email protected]> wrote: > HI All, > > I just want to run some spark structured streaming Job similar to this > > DS.filter(col("name").equalTo("john")) > .groupBy(functions.window(df1.col("TIMESTAMP"), "24 hours", "24 > hours"), df1.col("hourlyPay")) > .agg(sum("hourlyPay").as("total")); > > > I am wondering if I can express the above query in raw sql string? > > If so how would that look like and what are some of the disadvantages of > using raw sql query vs spark DSL? > > > Thanks! > >
