Re: [DISCUSS] FLIP-8: Rescalable Non-Partitioned State

I will update the design doc with more details for the Checkpointed variants and remove Option 2 (I think that's an orthogonal thing). The way I see it now, we should have base CheckpointedBase interface, have the current Checkpointed interface be a subclass for not repartitionable state. Then we

Re: [DISCUSS] FLIP-8: Rescalable Non-Partitioned State

Hi Aljoscha, Yes this is pretty much how I think about it as well. Basically the state in this case would be computed from the side inputs with the same state update logic on all operators. I think it is imprtant that operators compute their own state or at least observe all state changes

Re: [DISCUSS] FLIP-8: Rescalable Non-Partitioned State

Hi Gyula, I was thinking about this as well, in the context of side-inputs, which would be a generalization of your use case. If I'm not mistaken. In my head I was calling it global state. Essentially, this state would be the same on all operators and when checkpointing you would only have to

[jira] [Created] (FLINK-4389) Expose metrics to Webfrontend

Chesnay Schepler created FLINK-4389: --- Summary: Expose metrics to Webfrontend Key: FLINK-4389 URL: https://issues.apache.org/jira/browse/FLINK-4389 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-8: Rescalable Non-Partitioned State

Hi, Let me try to explain what I mean by broadcast states. I think it is a very common pattern that people broadcast control messages to operators that also receive normal input events. some examples: broadcast a model for prediction, broadcast some information that should be the same at all

Re: [DISCUSS] FLIP-8: Rescalable Non-Partitioned State

Comments inline. On Thu, Aug 11, 2016 at 8:06 PM, Gyula Fóra wrote: > Option 1: > I think the main problem here is sending all the state everywhere will not > scale at all. I think this will even fail for some internal Flink operators > (window timers I think are kept like

[jira] [Created] (FLINK-4388) Race condition during initialization of MemorySegmentFactory

Stephan Ewen created FLINK-4388: --- Summary: Race condition during initialization of MemorySegmentFactory Key: FLINK-4388 URL: https://issues.apache.org/jira/browse/FLINK-4388 Project: Flink

Re: Conceptual difference Windows and DataSet

Hi Kevin! The windows in Flink's DataStream API are organized by key. The reason is that the windows are very flexible, and each key can form different windows than the other (think sessions per user - each session starts and stops differently). There has been discussion about introducing

[jira] [Created] (FLINK-4387) Instability in KvStateClientTest.testClientServerIntegration()

Robert Metzger created FLINK-4387: - Summary: Instability in KvStateClientTest.testClientServerIntegration() Key: FLINK-4387 URL: https://issues.apache.org/jira/browse/FLINK-4387 Project: Flink