Re: [DISCUSS] Improving State/Timers/Windows

While enhancing the state interfaces we would also need to introduce new types of state. I was thinking of these, for a start: - ValueState (works like OperatorState works now, i.e. provides methods to get/set one state value - ListState, proves methods to add one element to a list of elements

Re: [DISCUSS] Improving State/Timers/Windows

> > On a side not, why would you call it KvState? And what would be called > KvState? The OperatorState interface would be called KvState. On Mon, Dec 14, 2015 at 11:18 AM, Aljoscha Krettek wrote: > Yes, as Kostas said, it would initially nor provide more functionality

Re: Object reuse documentation should be improved

Thanks for writing this up, Gábor. As Aljoscha suggested chaining changes all of these and makes it very tricky to work with these which should be clearly documented. That was the reason while some time ago the streaming API always copied the output of a UDF by default to avoid this ambiguous

Re: [DISCUSS] Improving State/Timers/Windows

A lot of this makes sense, but I am not sure about renaming "OperatorState". The other name is nicer, but why make users' life hard just for a name? On Mon, Dec 14, 2015 at 10:46 AM, Maximilian Michels wrote: > Hi Aljoscha, > > Thanks for the informative technical description.

Re: [DISCUSS] Improving State/Timers/Windows

I suppose that they can start as sugar and evolve to a different implementation. I would +1 the name change to KVState, OperatorState is indeed somewhat confusing, and it will only get harder to rename later. On Mon, Dec 14, 2015 at 11:09 AM, Gyula Fóra wrote: > Would the

Re: Externalizing the Flink connectors

Yes, absolutely. Setting up another repository for Flink ML would be no problem. On Sat, Dec 12, 2015 at 1:52 AM, Henry Saputra wrote: > I had small chat with Till about how to help manage Flink ML Libraries > contributions, which use Flink ML as dependencies. > > I

Re: Externalizing the Flink connectors

Regarding Max suggestion to have version compatible connectors: I'm not sure if we are able to maintain all connectors across different releases. I think its okay to have a document describing the minimum required Flink version for each connector. With the interface stability guarantees from 1.0

Re: Externalizing the Flink connectors

> > Regarding Max suggestion to have version compatible connectors: I'm not > sure if we are able to maintain all connectors across different releases. > That was not my suggestion. Whenever we release, existing connectors should be compatible with that release. Otherwise, they should be removed

Re: [DISCUSS] Improving State/Timers/Windows

Yes, as Kostas said, it would initially nor provide more functionality but it would enable us to add it later. On a side not, why would you call it KvState? And what would be called KvState? > On 14 Dec 2015, at 11:14, Kostas Tzoumas wrote: > > I suppose that they can

Re: [DISCUSS] Improving State/Timers/Windows

Hi Aljoscha, Thanks for the informative technical description. > - function state: this is the state that you get when a user function > implements the Checkpointed interface. it is not partitioned > - operator state: This is the state that a StreamOperator can snapshot, it > is similar to

Re: [DISCUSS] Improving State/Timers/Windows

Would the Reducing/Folding states just be some API sugar on top of what we have know (ValueState) or does it have some added functionality (like incremental checkpoints for list states)? Gyula Aljoscha Krettek ezt írta (időpont: 2015. dec. 14., H, 11:03): > While enhancing

Re: Object reuse documentation should be improved

Good write up. You could extend the Table of 1) a/b 2) a/b at the top with “chaining” (but you already know this, I guess). Chaining changes all of these and I think it can be tricky to know whether stuff is chained or not (for users, and even for us developers…). > On 13 Dec 2015, at 19:24,

Re: [DISCUSS] Improving State/Timers/Windows

+1 to all changes proposed, that is a reasonable step towards incremental snapshots and proper reconfiguration support. What is more interesting though is the actual implementations of the KVState derivatives, I am looking forward to see what you have in mind there. The operator/UDF KV

Re: Object reuse documentation should be improved

I guess chaining happens so often, that we should just write this doc assuming that there is chaining, and not even describe the rules for the non-chaining case. I mean I would never risk writing a UDF that only works when there is no chaining, and then constantly worry about when do I

Re: Apache Tinkerpop & Geode Integration?

Ping squirrels! Any thoughts/opinions on this? On 9 December 2015 at 20:40, Vasiliki Kalavri wrote: > Hello squirrels, > > I have been discussing with the Apache Tinkerpop [1] community regarding > an integration with Flink/Gelly. > You can read our discussion in [2].

[jira] [Created] (FLINK-3170) Expose task manager metrics via JMX

Maximilian Michels created FLINK-3170: - Summary: Expose task manager metrics via JMX Key: FLINK-3170 URL: https://issues.apache.org/jira/browse/FLINK-3170 Project: Flink Issue Type: New

[jira] [Created] (FLINK-3169) Drop {{Key}} type from Record Data Model

Stephan Ewen created FLINK-3169: --- Summary: Drop {{Key}} type from Record Data Model Key: FLINK-3169 URL: https://issues.apache.org/jira/browse/FLINK-3169 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-3171) Consolidate zoo of wrapper classes for input/output-stream to data-input/output-view

Stephan Ewen created FLINK-3171: --- Summary: Consolidate zoo of wrapper classes for input/output-stream to data-input/output-view Key: FLINK-3171 URL: https://issues.apache.org/jira/browse/FLINK-3171

Re: [DISCUSS] Improving State/Timers/Windows

oh, sorry, I misread. Just my +1 to renaming OperatorState then :-) On Mon, Dec 14, 2015 at 11:38 AM, Aljoscha Krettek wrote: > As I mentioned in my previous mail, I think that OperatorState would need > be replaced by more specific types of state (ValueState, ListState,

Re: Diagnosing TaskManager disappearance

Hi! The Netty memory usually goes much lower than 2*network memory (that is theoretical). Netty needs memory at the size two buffers on the sender and receiver side, per TCP connection. Since Flink usually multiplexes many Channels (that need network buffers) through the same TCP connection, the

Re: Externalizing the Flink connectors

Yes, that would be the way to go. We could follow Cask CDAP hydrator plugin repository [1] that support different plugins to run in their main CDAP hydrator [2] product - Henry [1] https://github.com/caskdata/hydrator-plugins [2] https://github.com/caskdata/cdap On Mon, Dec 14, 2015 at 1:49

Re: flink-dist packaging including unshaded classes

Just to confirm, building with maven 3.2.5 appears to work as expected. Thanks a lot for the work-around. On Thu, Dec 10, 2015 at 10:23 AM, Robert Metzger wrote: > You are right. I'll post the link with my next message on the maven user > list. > Here is the link to the