[jira] [Comment Edited] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881788#comment-16881788 ] Yu Li edited comment on FLINK-13072 at 7/10/19 7:25 AM: Correct me if I'm wrong, but checking our existing [state document|https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/] it seems there's no explicit clarification about whether state is thread safe. Besides this issue, I could see similar questions on [stackoverflow|https://stackoverflow.com/questions/55208345/flink-operator-state-is-thread-safe] and in our [mailing list|https://s.apache.org/ya10u], so I'm considering adding a paragraph in our document to explicitly clarify state thread-safety and prevent further confusion. Wdyt? Thanks. was (Author: carp84): Correct me if I'm wrong, but checking our existing [state document|https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/] it seems there's no explicit clarification about whether state is thread safe. Besides this issue, I could see similar questions on [stackoverflow|https://stackoverflow.com/questions/55208345/flink-operator-state-is-thread-safe] and in our [mailing list|https://s.apache.org/ya10u], so I'm considering adding a paragraph in our document for explicit clarification. Wdyt? Thanks. > RocksDBStateBachend is not thread safe and data loss silently > - > > Key: FLINK-13072 > URL: https://issues.apache.org/jira/browse/FLINK-13072 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.8.0, 1.8.1 >Reporter: lynn1.zhang >Priority: Blocker > Attachments: flink-demo.zip, image-2019-07-03-17-04-17-253.png > > > I create 2 mapstates in one operator, then create 2 threads in apply method, > each thread operate each map state(the operator is same), the expect result > is that 2 state have the same result but not. I upload the code, please help > to try it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881665#comment-16881665 ] lynn1.zhang edited comment on FLINK-13072 at 7/10/19 6:33 AM: -- [~srichter] Done was (Author: zicat): [~srichter] Please help to close, thx. > RocksDBStateBachend is not thread safe and data loss silently > - > > Key: FLINK-13072 > URL: https://issues.apache.org/jira/browse/FLINK-13072 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.8.0, 1.8.1 >Reporter: lynn1.zhang >Priority: Blocker > Attachments: flink-demo.zip, image-2019-07-03-17-04-17-253.png > > > I create 2 mapstates in one operator, then create 2 threads in apply method, > each thread operate each map state(the operator is same), the expect result > is that 2 state have the same result but not. I upload the code, please help > to try it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently
[ https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879099#comment-16879099 ] Stefan Richter edited comment on FLINK-13072 at 7/5/19 9:21 AM: [~zicat] All operators in Flink are assumed to be single-threaded and spawning up threads in any user function is something that is not supported by the framework. The quote from the doc that you present is about the `StateBackend` classes, which are (misleadingly named) just the factory for the actual backend implementations like `RocksDBKeyedStateBackend`. So yes, the factory has to be thread safe, but the backends (i.e. subclasses of `AbstractKeyedStateBackend`) and other things are not. What you attempt will not work. was (Author: srichter): [~zicat] All operators in Flink are assumed to be single-threaded and spawning up threads in any user function is something that is not supported by the framework. The quote from the doc that you present is about the `StateBackend` classes, which are (misleadingly named) just the factory for the actual backend implementations like `RocksDBKeyedStateBackend`. So yes, the factory has to be thread safe, but the backends and other things are not. What you attempt will not work. > RocksDBStateBachend is not thread safe and data loss silently > - > > Key: FLINK-13072 > URL: https://issues.apache.org/jira/browse/FLINK-13072 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.8.0, 1.8.1 >Reporter: lynn1.zhang >Priority: Blocker > Attachments: flink-demo.zip, image-2019-07-03-17-04-17-253.png > > > I create 2 mapstates in one operator, then create 2 threads in apply method, > each thread operate each map state(the operator is same), the expect result > is that 2 state have the same result but not. I upload the code, please help > to try it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently
[
https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877656#comment-16877656
]
Congxian Qiu(klion26) edited comment on FLINK-13072 at 7/3/19 9:03 AM:
---
I think currently we assume that all state operate are single-thread, for
{{RocksDBStateBackend}} state will combine the binary key, when operating with
multiple threads this will have problem, because the combined key may be wrong.
was (Author: klion26):
I think currently we assume that all state operate are single-thread, for
{{RocksDBStateBackend}} state will combine the binary key, when operating with
multiple threads this will have problem.
> RocksDBStateBachend is not thread safe and data loss silently
> -
>
> Key: FLINK-13072
> URL: https://issues.apache.org/jira/browse/FLINK-13072
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.8.0, 1.8.1
>Reporter: lynn1.zhang
>Priority: Blocker
> Attachments: flink-demo.zip
>
>
> I create 2 mapstates in one operator, then create 2 threads in apply method,
> each thread operate each map state(the operator is same), the expect result
> is that 2 state have the same result but not. I upload the code, please help
> to try it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently
[
https://issues.apache.org/jira/browse/FLINK-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877608#comment-16877608
]
lynn1.zhang edited comment on FLINK-13072 at 7/3/19 9:02 AM:
-
[~yunta] from beginning, I also think the {{input.iterator()}} is not thread
safe, but I try to print each element, all element print successfully. And I
replace from RocksDBStateBachend to FsStateBackend, everything is right. I
use lock object to lock the state.get() and state.put(), the result is also
right.
was (Author: zicat):
from beginning, I also think the {{input.iterator()}} is not thread safe, but I
try to print each element, all element print successfully. And I replace from
RocksDBStateBachend to FsStateBackend, everything is right.
> RocksDBStateBachend is not thread safe and data loss silently
> -
>
> Key: FLINK-13072
> URL: https://issues.apache.org/jira/browse/FLINK-13072
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.8.0, 1.8.1
>Reporter: lynn1.zhang
>Priority: Blocker
> Attachments: flink-demo.zip
>
>
> I create 2 mapstates in one operator, then create 2 threads in apply method,
> each thread operate each map state(the operator is same), the expect result
> is that 2 state have the same result but not. I upload the code, please help
> to try it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
