[jira] [Commented] (FLINK-6239) Sharing of State Across Operators
[ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336775#comment-17336775 ] Flink Jira Bot commented on FLINK-6239: --- This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Sharing of State Across Operators > - > > Key: FLINK-6239 > URL: https://issues.apache.org/jira/browse/FLINK-6239 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.1.4 >Reporter: Elias Levy >Priority: Major > Labels: stale-major > > Currently state cannot be shared across operators. On a keyed stream, the > state is implicitly keyed by the operator id, in addition to the stream key. > This can make it more difficult and inefficient to implement complex > topologies, where multiple operator may need to access the same state. It > would be value to be able to access keyed value and map stated across > operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6239) Sharing of State Across Operators
[ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328798#comment-17328798 ] Flink Jira Bot commented on FLINK-6239: --- This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > Sharing of State Across Operators > - > > Key: FLINK-6239 > URL: https://issues.apache.org/jira/browse/FLINK-6239 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.1.4 >Reporter: Elias Levy >Priority: Major > Labels: stale-major > > Currently state cannot be shared across operators. On a keyed stream, the > state is implicitly keyed by the operator id, in addition to the stream key. > This can make it more difficult and inefficient to implement complex > topologies, where multiple operator may need to access the same state. It > would be value to be able to access keyed value and map stated across > operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6239) Sharing of State Across Operators
[ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533553#comment-16533553 ] Stephan Ewen commented on FLINK-6239: - Thanks for sharing the details. Materialized table joins are in the works. There is a design doc for adding an extended version of this (Time Versioned Table Join) in SQL and the Table API: https://docs.google.com/document/d/1KaAkPZjWFeu-ffrC9FhYuxE6CIKsatHTTxyrxSBR8Sk > Sharing of State Across Operators > - > > Key: FLINK-6239 > URL: https://issues.apache.org/jira/browse/FLINK-6239 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.1.4 >Reporter: Elias Levy >Priority: Major > > Currently state cannot be shared across operators. On a keyed stream, the > state is implicitly keyed by the operator id, in addition to the stream key. > This can make it more difficult and inefficient to implement complex > topologies, where multiple operator may need to access the same state. It > would be value to be able to access keyed value and map stated across > operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6239) Sharing of State Across Operators
[ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533142#comment-16533142 ] Elias Levy commented on FLINK-6239: --- For our particular use case having a single operator that is owner and can write, and other that can read the state, is sufficient. The idea was essentially to implement a materialized table join (similar to Kafka Streams KTable). We have need to join again the table multiple times, but the table (state) would be of significant size, and duplicating it is something we rather not do. That said, I think we may be able to get around this issue by creating a generic wrapping container, unioning all the streams, and feeding it to a single operator that perform all the work, instead of having multiple operators and multiple streams. Not as clean, but probably workable. > Sharing of State Across Operators > - > > Key: FLINK-6239 > URL: https://issues.apache.org/jira/browse/FLINK-6239 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.1.4 >Reporter: Elias Levy >Priority: Major > > Currently state cannot be shared across operators. On a keyed stream, the > state is implicitly keyed by the operator id, in addition to the stream key. > This can make it more difficult and inefficient to implement complex > topologies, where multiple operator may need to access the same state. It > would be value to be able to access keyed value and map stated across > operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6239) Sharing of State Across Operators
[ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533055#comment-16533055 ] Stephan Ewen commented on FLINK-6239: - Can you explain more how you expect this to work? Specifically: - Would all operators among which this is shared have read and write access? - Or will there be one "owner" that writes the state and other operators that can read the state. - What would the consistency model behind this be? > Sharing of State Across Operators > - > > Key: FLINK-6239 > URL: https://issues.apache.org/jira/browse/FLINK-6239 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.1.4 >Reporter: Elias Levy >Priority: Major > > Currently state cannot be shared across operators. On a keyed stream, the > state is implicitly keyed by the operator id, in addition to the stream key. > This can make it more difficult and inefficient to implement complex > topologies, where multiple operator may need to access the same state. It > would be value to be able to access keyed value and map stated across > operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6239) Sharing of State Across Operators
[ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952466#comment-15952466 ] Elias Levy commented on FLINK-6239: --- If sharing state across operators is not feasible, then as a fallback, a join operator that accept a variable number of streams to join may suffice to implement complex joins across multiple streams. > Sharing of State Across Operators > - > > Key: FLINK-6239 > URL: https://issues.apache.org/jira/browse/FLINK-6239 > Project: Flink > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.1.4 >Reporter: Elias Levy > > Currently state cannot be shared across operators. On a keyed stream, the > state is implicitly keyed by the operator id, in addition to the stream key. > This can make it more difficult and inefficient to implement complex > topologies, where multiple operator may need to access the same state. It > would be value to be able to access keyed value and map stated across > operators. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
