[
https://issues.apache.org/jira/browse/SPARK-28026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862671#comment-16862671
]
Hyukjin Kwon commented on SPARK-28026:
--------------------------------------
Let's ask a question to mailing list before filing as an issue. You will get a
better answer from there. See https://spark.apache.org/community.html
> How to get the second row from 1 minute window
> ----------------------------------------------
>
> Key: SPARK-28026
> URL: https://issues.apache.org/jira/browse/SPARK-28026
> Project: Spark
> Issue Type: Question
> Components: Examples, Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Devendra Vishwakarma
> Priority: Major
>
> I am almost blocked for a month I am still figuring out the API to achieve
> one of the functionalities related to spark structured streaming with window
> grouping. So I thought to reach you guys here.
> What I have is stock related time series data and I have grouped them in a
> 1-minute window along with the stock name. I am able to get first, last row
> in that 1-minute group, but I want some values from the second row of that 1
> minute window, which I am not able to do at all. I looked at each function
> related to aggregation but I could not find any.Please help me.
> This is what I have done so far -
> val aggregates = stockEvents
> .withWatermark("timestamp", "5 seconds")
> .groupBy(window($"timestamp", "1 minute", "1 minute", "0 seconds"),
> $"stockName")
> .agg(
> first("tradingprice").alias("open"), //I have to make this value coming from
> second row
> last("tradingprice").alias("close"),
> max("tradingprice").alias("high"),
> min("tradingprice").alias("low"))
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
