[
https://issues.apache.org/jira/browse/FLINK-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410753#comment-16410753
]
ASF GitHub Bot commented on FLINK-9060:
---------------------------------------
GitHub user sihuazhou opened a pull request:
https://github.com/apache/flink/pull/5751
[FLINK-9060][state] Deleting state using KeyedStateBackend.getKeys() throws
Exception
## What is the purpose of the change
This PR fixes the problem when deleting state using
`KeyedStateBackend.getKeys()` throws Exception.
## Brief change log
- copy the result of `getKeys()` into `list` to avoid concurrency problem.
## Verifying this change
- *add a unit test in
`StateBackendTest#testConcurrentModificationWithGetKeys()` to verify this*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sihuazhou/flink
deletingStateUsingKeyedStateBackendGetKeys
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5751.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5751
----
commit cba3a32f3af16ee92676b1e5b82b21af6fee610d
Author: sihuazhou <summerleafs@...>
Date: 2018-03-23T03:20:42Z
fix concurrency risk in HeapKeyedStateBackend#getKeys().
----
> Deleting state using KeyedStateBackend.getKeys() throws Exception
> -----------------------------------------------------------------
>
> Key: FLINK-9060
> URL: https://issues.apache.org/jira/browse/FLINK-9060
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
> Reporter: Aljoscha Krettek
> Assignee: Sihua Zhou
> Priority: Blocker
> Fix For: 1.5.0
>
>
> Adding this test to {{StateBackendTestBase}} showcases the problem:
> {code}
> @Test
> public void testConcurrentModificationWithGetKeys() throws Exception {
> AbstractKeyedStateBackend<Integer> backend =
> createKeyedBackend(IntSerializer.INSTANCE);
> try {
> ListStateDescriptor<String> listStateDescriptor =
> new ListStateDescriptor<>("foo",
> StringSerializer.INSTANCE);
> backend.setCurrentKey(1);
> backend
> .getPartitionedState(VoidNamespace.INSTANCE,
> VoidNamespaceSerializer.INSTANCE, listStateDescriptor)
> .add("Hello");
> backend.setCurrentKey(2);
> backend
> .getPartitionedState(VoidNamespace.INSTANCE,
> VoidNamespaceSerializer.INSTANCE, listStateDescriptor)
> .add("Ciao");
> Stream<Integer> keys = backend
> .getKeys(listStateDescriptor.getName(),
> VoidNamespace.INSTANCE);
> keys.forEach((key) -> {
> backend.setCurrentKey(key);
> try {
> backend
> .getPartitionedState(
> VoidNamespace.INSTANCE,
>
> VoidNamespaceSerializer.INSTANCE,
> listStateDescriptor)
> .clear();
> } catch (Exception e) {
> e.printStackTrace();
> }
> });
> }
> finally {
> IOUtils.closeQuietly(backend);
> backend.dispose();
> }
> }
> {code}
> This should work because one of the use cases of {{getKeys()}} and
> {{applyToAllKeys()}} is to do stuff for every key, which includes deleting
> them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
