[
https://issues.apache.org/jira/browse/SPARK-20717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-20717:
----------------------------------
Description:
Timeout and state data are two independent entities and should be settable
independently. Therefore, in the same call of the user-defined function, one
should be able to set the timeout before initializing the state and also after
removing the state. Whether timeouts can be set or not should not depend on the
current state, and vice versa.
However, a limitation of the current implementation is that state cannot be
null while timeout is set. This is checked lazily after the function call has
completed.
was:
Timeout and state data are two independent entities and should be settable
independently. Therefore,
- In the same call of the user-defined function, one should be able to set the
timeout before initializing the state.
- Removing the state should not reset timeouts.
However, a limitation of the current implementation is that state cannot be
null while timeout is set by the end of the function call is over. We should
check this lazily.
> Tweak MapGroupsWithState update function behavior
> -------------------------------------------------
>
> Key: SPARK-20717
> URL: https://issues.apache.org/jira/browse/SPARK-20717
> Project: Spark
> Issue Type: Sub-task
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Fix For: 2.2.0
>
>
> Timeout and state data are two independent entities and should be settable
> independently. Therefore, in the same call of the user-defined function, one
> should be able to set the timeout before initializing the state and also
> after removing the state. Whether timeouts can be set or not should not
> depend on the current state, and vice versa.
> However, a limitation of the current implementation is that state cannot be
> null while timeout is set. This is checked lazily after the function call has
> completed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
