[ https://issues.apache.org/jira/browse/SPARK-22187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu updated SPARK-22187: --------------------------------- Labels: release-notes releasenotes (was: release-notes) > Update unsaferow format for saved state such that we can set timeouts when > state is null > ---------------------------------------------------------------------------------------- > > Key: SPARK-22187 > URL: https://issues.apache.org/jira/browse/SPARK-22187 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 2.2.0 > Reporter: Tathagata Das > Labels: release-notes, releasenotes > Fix For: 2.2.0 > > > Currently the group state of user-defined-type is encoded as top-level > columns in the unsaferows stores in state store. The timeout timestamp is > also saved as (when needed) as the last top-level column. Since, the > groupState is serialized to top level columns, you cannot save "null" as a > value of state (setting null in all the top-level columns is not equivalent). > So we dont let the user to set the timeout without initializing the state for > a key. Based on user experience, his leads to confusion. > This JIRA is to change the row format such that the state is saved as nested > columns. This would allow the state to be set to null, and avoid these > confusing corner cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org