GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/17957
[SPARK-20717][SS] Minor tweaks to the MapGroupsWithState behavior
## What changes were proposed in this pull request?
Timeout and state data are two independent entities and should be settable
independently. Therefore, in the same call of the user-defined function, one
should be able to set the timeout before initializing the state and also after
removing the state. Whether timeouts can be set or not, should not depend on
the current state, and vice versa.
However, a limitation of the current implementation is that state cannot be
null while timeout is set. This is checked lazily after the function call has
completed.
## How was this patch tested?
- Updated existing unit tests that test the behavior of
GroupState.setTimeout*** wrt to the current state
- Added new tests that verify the disallowed cases where state is undefined
but timeout is set.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-20717
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17957.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17957
----
commit 3cfd446b0a926aad115acdf5bd8dc8cd03ee743a
Author: Tathagata Das <[email protected]>
Date: 2017-05-12T02:31:15Z
Tweaks to the mapGroupsWithState behavior
commit 40329404299ece70aef7ef245704978fb9d1e6f9
Author: Tathagata Das <[email protected]>
Date: 2017-05-12T03:06:52Z
More tweaks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
