[ https://issues.apache.org/jira/browse/SPARK-26655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-26655: ------------------------------------ Assignee: Apache Spark > Support multiple aggregates in Structured Streaming append mode > --------------------------------------------------------------- > > Key: SPARK-26655 > URL: https://issues.apache.org/jira/browse/SPARK-26655 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.0.0 > Reporter: Arun Mahadevan > Assignee: Apache Spark > Priority: Major > > Right now multiple aggregates are not supported in structured streaming. > However, in append mode, the aggregates are emitted only after the watermark > passes the threshold (e.g. the window boundary) and the emitted value is not > affected by further late data. So it possible to chain multiple aggregates in > 'Append' output mode without worrying about retractions. > However the current event time watermarks in structured streaming are tracked > at a global level and this does not work when aggregates are chained. > We need to track the watermarks at individual operator level so that each > operator can make progress independently and not rely on global min or max > value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org