[
https://issues.apache.org/jira/browse/FLINK-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127048#comment-15127048
]
ASF GitHub Bot commented on FLINK-3312:
---------------------------------------
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/1571
[FLINK-3312] State enhancements convenience methods for the public API
**NOTE** This pull request is based on #1570 and #1562 . It should be
addressed once these two are merged.
This pull request addresses various things:
1. It rebases #1562 onto the latest master, enhanced with #1570
2. It restores binary compatibility with the old `OperatorState`
interface.
3. It adds methods to the `RuntimeContext` for explicit instantiation of
List- and ReducingState. It ports various tests and examples to the new state
interfaces.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink state
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1571.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1571
----
commit 1e0b4e78419c9de037e9dba1cc30bfc66fe17c6e
Author: Stephan Ewen <[email protected]>
Date: 2016-01-31T22:08:18Z
[FLINK-3303] [core] Move Tuple classes to flink-core
commit f5376c6f928603185f57fe9441a18f5a9e19f05c
Author: Stephan Ewen <[email protected]>
Date: 2016-01-31T22:27:36Z
[FLINK-3049] [api breaking] Move 'Either' type to 'flink-core /
org.apache.flink.types'
commit d0d47f935f9a9119cbc1f6dcb6acd80360cb2bf7
Author: Stephan Ewen <[email protected]>
Date: 2016-01-31T22:28:32Z
[FLINK-3303] [core] Move all type utilities to flink-core
commit 1bc7d8bea6e50e0b9223e437490c65ddb999d58f
Author: Stephan Ewen <[email protected]>
Date: 2016-02-01T15:46:03Z
[hotfix] Reduce the heavy sysout verbosity for certain tests
commit 166720d8c7dd15dd29105545a1498f858d12ae79
Author: Stephan Ewen <[email protected]>
Date: 2016-02-01T16:09:38Z
[FLINK-2348] Fix unstable IterateExampleITCase.
This deactivates the validation of results, which is not reliably possible
under the current model (timeout on feedback).
This test for now only checks that the job executes properly.
Also adds proper logging property files for the examples projects.
commit 86676a7f4481643fa2e171240f65db719fda452f
Author: Aljoscha Krettek <[email protected]>
Date: 2016-01-25T11:33:51Z
[FLINK-3201] Enhance Partitioned State Interface with State Types
Add new state types ValueState, ListState and ReducingState, where
ListState and ReducingState derive from interface MergingState.
ValueState behaves exactly the same as OperatorState. MergingState is a
stateful list to which elements can be added and for which the elements
that it contains can be obtained. If using a ListState the list of
elements is actually kept, for a ReducingState a reduce function is used
to combine all added elements into one. To create a ValueState the user
passes a ValueStateIdentifier to
StreamingRuntimeContext.getPartitionedState() while they would pass a
ListStateIdentifier or ReducingStateIdentifier for the other state
types.
This change is necessary to give the system more information about the
nature of the operator state. We want this to be able to do incremental
snapshots. This would not be possible, for example, if the user had a
List as a state. Inside OperatorState this list would be opaque and
Flink could not create good incremental snapshots.
This also refactors the StateBackend. Before, the logic for partitioned
state was spread out over StreamingRuntimeContext,
AbstractStreamOperator and StateBackend. Now it is consolidated in
StateBackend.
This also adds support for partitioned state in two-input operators.
commit bcb5be1e8afff386be32b772db2f730951bbd7db
Author: Aljoscha Krettek <[email protected]>
Date: 2016-01-21T09:56:47Z
[FLINK-3278] Add Partitioned State Backend Based on RocksDB
commit 4035aa7e901a6a8514f2e77034cc4720577f46e5
Author: Aljoscha Krettek <[email protected]>
Date: 2016-01-25T11:34:05Z
[FLINK-3200] Use Partitioned State in WindowOperator
This changes window operator to use the new partitioned state
abstraction for keeping window contents instead of custom internal
state and the checkpointed interface.
For now, timers are still kept as custom checkpointed state, however.
WindowOperator now expects a StateIdentifier for MergingState, this can
either be for ReducingState or ListState but WindowOperator is agnostic
to the type of State. Also the signature of WindowFunction is changed to
include the type of intermediate input. For example, if a ReducingState
is used the input of the WindowFunction is T (where T is the input
type). If using a ListState the input of the WindowFunction would be of
type Iterable[T].
commit c45a1a97e0b15f7d2987f82c6852ca37a605f753
Author: Stephan Ewen <[email protected]>
Date: 2016-01-28T17:58:11Z
[hotfix] [streaming] Various cleanups in StreamTask
- Clean up generics
- Clean and safe disposal of initialized resources
- Add names to asynchronous materialization threads
- Fix concurrent modification of materialization threads set
commit 5d1c87c2b44c0377334c352ab8a6390f60de3825
Author: Stephan Ewen <[email protected]>
Date: 2016-01-29T09:33:19Z
[FLINK-3201] Add operator state to make change backwards compatible
commit 428b2159dbc488620945115571377d367f8f9b66
Author: Stephan Ewen <[email protected]>
Date: 2016-01-29T13:53:14Z
[FLINK-3312] Add accessors for various state types to RuntimeContext
commit 42e2d0e4de9d41a0fd7efb71e026a9b1f13a47b6
Author: Stephan Ewen <[email protected]>
Date: 2016-02-01T20:46:09Z
[hotfix] Remove old sysout debug message in UdfAnalyzerTest
----
> Add convenience accessor methods for extended state interface
> -------------------------------------------------------------
>
> Key: FLINK-3312
> URL: https://issues.apache.org/jira/browse/FLINK-3312
> Project: Flink
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Fix For: 1.0.0
>
>
> The {{RuntimeContext}} should offer dedicated methods to create the
> {{ValueState}}, {{ListState}}, and {{ReducingState}}.
> The {{StateDescriptor}} classes also need convenience constructors that
> accept simple classes to indicate the types, rather than requiring to pass
> serializers directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
