[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 Merging after Travis turns green --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 I'd say lets keep the naming as is, since in general all of us agree on that already. I'll try to do some more work on the docs and Javadocs to make sure that at least the documentation

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-18 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 In general, I agree with @aljoscha, however I have already heard at least one case where this confusion was leading to problems. Given the short time that we provide this feature, this could

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-18 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 I would stick to `getListState()` and rely on users knowing on what kind of state store they are calling the method. In general, users should not use this interface too much, I think it's

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-04 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 Very good work and thanks for improving the documentation. I like the update. From what I have seen in the past, some user have mistaken the list-nature of the operator state and simply

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-04 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 Nice update to the docs! And yeah, a separate issue for deprecation is very good. 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 Updated managed operator state docs. @aljoscha @StefanRRichter could you have a final look at it? If there's no other problems I'll proceed to merge this. Thanks :) --- If your project is set

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 I've opened a separate JIRA to deprecate `ListCheckpointed`. Lets keep this PR self-contained in just refining the `OperatorStateStore` interface. I think this PR is still lacking an update

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 Well, this way is easy, but still slightly more complicated that the original `Checkpointed`. Personally, I would also consider that an easy-enough replacement. That is why my vote was also

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 Isn't the 90% cases already covered with `StateDescriptor(stateName, typeClass)`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 I would also be in favour of deprecating `ListCheckpointed`, but I understand the argument that we should provide an easy way of state access for the 90% cases without fiddling around with

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 This looks very good. 👍 And yes, I would be in favour of deprecating `ListCheckpointed` but let's see what @StefanRRichter and @StephanEwen think. --- If your project is set up for

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 @aljoscha @StefanRRichter Rebased on master to resolve conflicts. Also, the follow-up commits addresses the following: 1. Remove / deprecate the Java serialization shortcuts. We

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 @aljoscha @StefanRRichter I'll update the PR today, thanks for the reminder. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 Fine with me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-04-03 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 @StefanRRichter and @tzulitai what's the state of this? Can we go forward with calling it union state? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 We can deprecate, because it was added to provide a smooth path from Flink 1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 I think @StephanEwen wanted to keep a simple, intuitive way for users to register their state that does not require them to think about serializers etc.. While I understand this point, I am

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 Nice following the discussion here :) I think I'm also leaning a bit more towards naming it `UnionListState`. From the user's prospective, it seems to be more clearer if the name of the

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 @StefanRRichter I think I was faster ... 😉 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 I wonder if there could also exist a case for broadcasting operator state (non-keyed), where only one operator instance is selected as sender and all others receive on restore. Furthermore,

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 +1 on not exposing the Java Serialisation shortcut, btw. I was very unhappy that we even have it for the normal operator state. 😃 --- If your project is set up for it, you can reply to this

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread aljoscha
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3508 I would like us to take some time and think about the name. We are about to introduce a thing called "broadcast state" somewhat soon in the effort to make streaming joins possible. This broadcast

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-16 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3508 Looks good in general. Some thoughts on refinement: - I think we should not really encourage the use of Java Serialization, so I would suggest to remove the serializable shortcut for

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-10 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 The PR re-introduces methods that have been removed from the public interface before the release. Reason the remove the methods was that we had too little time to make a final decision on the

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-10 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3508 R: @aljoscha @StefanRRichter --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and