Re: can we use mapGroupsWithState in raw sql?

.agg(max(struct($"amount", $"my_timestamp")).as("data")).select($"id", $"data.*") Thanks, Arun From: Jungtaek Lim Date: Wednesday, April 18, 2018 at 4:54 PM To: Michael Armbrust Cc: kant kodali , Arun Iyer , Tathagata Das , "user

Re: can we use mapGroupsWithState in raw sql?

전 9:43, Arun Mahadevan 님이 작성: > >> The below expr might work: >> >> df.groupBy($"id").agg(max(struct($"amount", >> $"my_timestamp")).as("data")).select($"id", $"data.*") >> >> >> Thanks, >>

Re: can we use mapGroupsWithState in raw sql?

tamp")).as("data")).select($"id", $"data.*") > > > Thanks, > Arun > > From: Jungtaek Lim > Date: Wednesday, April 18, 2018 at 4:54 PM > To: Michael Armbrust > Cc: kant kodali , Arun Iyer , > Tathagata Das , "user @spark" &l

Re: can we use mapGroupsWithState in raw sql?

Cc: kant kodali , Arun Iyer , Tathagata Das , "user @spark" Subject: Re: can we use mapGroupsWithState in raw sql? Thanks Michael for providing great solution. Great to remove UDAF and any needs to provide fields manually. Btw, your code has compilation error. ')' is missing, a

Re: can we use mapGroupsWithState in raw sql?

8, 2018 at 9:36 AM, Arun Mahadevan wrote: >> >>> Cant the “max” function used here ? Something like.. >>> >>> >>> stream.groupBy($"id").max("amount").writeStream.outputMode(“complete”/“update")…. >>> >>> Unles

Re: can we use mapGroupsWithState in raw sql?

omplete”/“update")…. >> >> Unless the “stream” is already a grouped stream, in which case the above >> would not work since the support for multiple aggregate operations is not >> there yet. >> >> Thanks, >> Arun >> >> From: kant kodali

Re: can we use mapGroupsWithState in raw sql?

there yet. > > Thanks, > Arun > > From: kant kodali > Date: Tuesday, April 17, 2018 at 11:41 AM > To: Tathagata Das > Cc: "user @spark" > Subject: Re: can we use mapGroupsWithState in raw sql? > > Hi TD, > > Thanks for that. The only reason I ask

Re: can we use mapGroupsWithState in raw sql?

erations is not there yet. Thanks, Arun From: kant kodali Date: Tuesday, April 17, 2018 at 11:41 AM To: Tathagata Das Cc: "user @spark" Subject: Re: can we use mapGroupsWithState in raw sql? Hi TD, Thanks for that. The only reason I ask is I don't see any alternative soluti

Re: can we use mapGroupsWithState in raw sql?

>> >> If you can achieve with mapGroupWithState, you may want to stick with >> that. >> >> Btw, when you deal with streaming, you may want to define logical batch >> for all aggregations and joins via defining window and watermark. You >> wouldn't want to

Re: can we use mapGroupsWithState in raw sql?

;> 1 | 5 | 2018-04-01T01:00:00.000Z >> 1 | 10 | 2018-04-01T01:10:00.000Z >> 2 | 20 | 2018-04-01T01:20:00.000Z >> 2 | 30 | 2018-04-01T01:25:00.000Z >> 2 | 40 | 2018-04-01T01:30:00.000Z >> >> *Expected Output:* >> >> id | amount | my_timestamp >> --- >> 1 | 10 | 2018-04-01T01:10:00.000Z >> 2 | 40 | 2018-04-01T01:30:00.000Z >> >> Looking for a streaming solution using either raw sql like >> sparkSession.sql("sql >> query") or similar to raw sql but not something like mapGroupWithState >> >> On Mon, Apr 16, 2018 at 8:32 PM, Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> Unfortunately no. Honestly it does not make sense as for type-aware >>> operations like map, mapGroups, etc., you have to provide an actual JVM >>> function. That does not fit in with the SQL language structure. >>> >>> On Mon, Apr 16, 2018 at 7:34 PM, kant kodali wrote: >>> >>>> Hi All, >>>> >>>> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >>>> >>>> Thanks! >>>> >>>> >>>> >>> >>

Re: can we use mapGroupsWithState in raw sql?

40 | 2018-04-01T01:30:00.000Z > > Looking for a streaming solution using either raw sql like > sparkSession.sql("sql > query") or similar to raw sql but not something like mapGroupWithState > > On Mon, Apr 16, 2018 at 8:32 PM, Tathagata Das < > tathagata.das1...@gmail.com> wrote: > >> Unfortunately no. Honestly it does not make sense as for type-aware >> operations like map, mapGroups, etc., you have to provide an actual JVM >> function. That does not fit in with the SQL language structure. >> >> On Mon, Apr 16, 2018 at 7:34 PM, kant kodali wrote: >> >>> Hi All, >>> >>> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >>> >>> Thanks! >>> >>> >>> >> >

Re: can we use mapGroupsWithState in raw sql?

e SQL language structure. > > On Mon, Apr 16, 2018 at 7:34 PM, kant kodali wrote: > >> Hi All, >> >> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >> >> Thanks! >> >> >> >

Re: can we use mapGroupsWithState in raw sql?

an we use mapGroupsWithState in raw SQL? or is it in the roadmap? > > Thanks! > > >

can we use mapGroupsWithState in raw sql?

Hi All, can we use mapGroupsWithState in raw SQL? or is it in the roadmap? Thanks!