[GitHub] flink pull request #2376: [FLINK-3755] Introduce key groups for key-value st...

2016-08-16 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2376 [FLINK-3755] Introduce key groups for key-value state to support dynamic scaling This pull request introduces the concept of key groups to Flink. A key group is the smallest assignable unit

[GitHub] flink issue #2376: [FLINK-3755] Introduce key groups for key-value state to ...

2016-08-16 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2376 R: @tillrohrmann @StephanEwen for review pls --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] flink pull request #2376: [FLINK-3755] Introduce key groups for key-value st...

2016-08-29 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/2376 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink pull request #2440: Keyed backend refactor

2016-08-30 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2440 Keyed backend refactor This pull request is a followup to the preliminary pull request #2376 and addresses all issues subsumed under [FLINK-3755] In addition to the changes from PR

[GitHub] flink pull request #2440: [FLINK-3755] Introduce key groups for key-value st...

2016-08-31 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/2440 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #2440: [FLINK-3755] Introduce key groups for key-value state to ...

2016-09-07 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2440 I don't think that this is giving us the correct inverse for ```computeOperatorIndexForKeyGroup(...)```. Our test ```CheckpointCoordinatorTest::testCreateKeyGroupPartitions()```gene

[GitHub] flink pull request #2440: [FLINK-3755] Introduce key groups for key-value st...

2016-09-07 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/2440#discussion_r77870173 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyGroupRangeAssignment.java --- @@ -0,0 +1,97 @@ +/* + * Licensed to

[GitHub] flink pull request #2512: Partitionable op state

2016-09-19 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2512 Partitionable op state This pull request introduces rescalable non-partitioned operator state as described in issue [FLINK-4379] and makes following major changes: # 1) Introducing

[GitHub] flink issue #2512: [FLINK-4379] Rescalable non-partitioned state

2016-09-19 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2512 Please review @tillrohrmann @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] flink pull request #2513: [hotfix] RescalingITCase

2016-09-19 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2513 [hotfix] RescalingITCase Hotfix for RescalingITCase: race condition could make the test stuck in a blocking call until timeout. You can merge this pull request into a Git repository by

[GitHub] flink pull request #2513: [hotfix] RescalingITCase

2016-09-19 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/2513#discussion_r79470571 --- Diff: flink-tests/src/test/java/org/apache/flink/test/checkpointing/RescalingITCase.java --- @@ -670,12 +676,15 @@ public void run(SourceContext

[GitHub] flink pull request #2523: [FLINK-4556] Make Queryable State Key-Group Aware

2016-09-20 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2523 [FLINK-4556] Make Queryable State Key-Group Aware This PR addresses [FLINK-4556] and makes queryable state aware of key-groups. You can merge this pull request into a Git repository by

[GitHub] flink pull request #2513: [hotfix] RescalingITCase

2016-09-21 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/2513 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #2512: [FLINK-4379] Rescalable non-partitioned state

2016-09-21 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2512 Hi, I have some suggestions for renaming some of the interfaces and their methods in this pull request to come up with some clearer, more consistent naming schemes. I suggest the

[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...

2016-09-22 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/2533 [FLINK-4603] Fixes: KeyedStateBackend cannot restore user code classes This PR fixes [FLINK-4603] and introduces a test to protect better against future regression. You can merge this pull

[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...

2016-09-22 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2533 Please review @tillrohrmann or @aljoscha --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...

2016-09-22 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/2533#discussion_r80010321 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java --- @@ -266,18 +265,20 @@ public void

[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...

2016-09-22 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2533 @StephanEwen at least in the RocksDB backend we could remove user code completely. Right now, the only thing that needs to be serialized is the TypeSerializer from the ValueDescriptor. It is

[GitHub] flink issue #2523: [FLINK-4556] Make Queryable State Key-Group Aware

2016-09-22 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/2523 please review @aljoscha or @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-10 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 The PR re-introduces methods that have been removed from the public interface before the release. Reason the remove the methods was that we had too little time to make a final decision on the

[GitHub] flink pull request #3466: [FLINK-5715] Asynchronous snapshots for heap-based...

2017-03-10 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3466#discussion_r105390074 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackend.java --- @@ -97,6 +100,27 @@ public FsStateBackend

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks in...

2017-03-13 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 CC @gyfora @uce --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] flink pull request #3523: [FLINK-5985] Report no task states for stateless t...

2017-03-13 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3523 [FLINK-5985] Report no task states for stateless tasks in checkpointing This PR fixes [FLINK-5985]. The solution is based on acknowledging `null` instead of some empty SubtaskState to

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks in...

2017-03-14 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 @gyfora if the effort is reasonable, it would be great to try this out on your topology. As soon as you give your +1, I could merge this change :-) --- If your project is set up for it, you

[GitHub] flink issue #3533: [FLINK-6044] Replace calls to InputStream#read(...) with ...

2017-03-14 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3533 CC @uce @tillrohrmann --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] flink pull request #3533: [FLINK-6044] Replace calls to InputStream#read(......

2017-03-14 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3533 [FLINK-6044] Replace calls to InputStream#read(...) with the indended… This PR fixes FLINK-6044. On top of that, I searched through the code for calls to `InputStream#read(...)` that

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks in...

2017-03-14 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 Great, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks in...

2017-03-14 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 Sure, I just quickly prepared a backport here: https://github.com/StefanRRichter/flink/tree/FLINK-5985-backport-to-1.2 --- If your project is set up for it, you can reply to this

[GitHub] flink issue #3536: [FLINK-6048] Asynchronous snapshots for heap-based Operat...

2017-03-14 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3536 CC @aljoscha @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink pull request #3536: [FLINK-6048] Asynchronous snapshots for heap-based...

2017-03-14 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3536 [FLINK-6048] Asynchronous snapshots for heap-based OperatorStateBackend This PR introduces asynchronous snapshots for the heap-based `DefaultOperatorStateBackend`. Compared to the

[GitHub] flink pull request #3542: [FLINK-5985] [Backport for 1.2] Report no task sta...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3542 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks in...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 Ok, then the mystery is finally solved :-) Thanks again for reporting this problem and your additional testing efforts! --- If your project is set up for it, you can reply to this email and

[GitHub] flink pull request #3543: [FLINK-5985] [Backport for 1.2] Report no task sta...

2017-03-15 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3543 [FLINK-5985] [Backport for 1.2] Report no task states for stateless tasks on checkpointing This PR is a backport of #3523 to Flink 1.2. You can merge this pull request into a Git repository

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks on...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 Hm, one potential pitfall that I see is operator chaining, in case your stateless operators are chained together with stateful ones. But then again, you said it works locally? --- If your

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks on...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 Yes, it should still work because the changes on `RocksDBKeyedStateBackend` are purely cosmetical without changing any functionality. --- If your project is set up for it, you can reply to

[GitHub] flink pull request #3545: [FLINK-6061] Introduced IllegalStateException when...

2017-03-15 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3545 [FLINK-6061] Introduced IllegalStateException when operating RocksDB … …keyed state with no key set This PR fixes [FLINK-6061]. I throws an `IllegalStateException` when keyed

[GitHub] flink pull request #3545: [FLINK-6061] Introduced IllegalStateException when...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3545 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #3545: [FLINK-6061] Introduced IllegalStateException when operat...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3545 Fixed in 0bdc8bf (master) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink issue #3545: [FLINK-6061] Introduced IllegalStateException when operat...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3545 Thank @StephanEwen ! Merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] flink pull request #3533: [FLINK-6044] Replace calls to InputStream#read(......

2017-03-15 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3533 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink pull request #3523: [FLINK-5985] Report no task states for stateless t...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3523#discussion_r106185325 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTest.java --- @@ -640,6 +640,74 @@ public

[GitHub] flink issue #3533: [FLINK-6044] Replace calls to InputStream#read(...) with ...

2017-03-15 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3533 Thanks for the review @tillrohrmann. Merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] flink pull request #3466: [FLINK-5715] Asynchronous snapshots for heap-based...

2017-03-16 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3466#discussion_r106385143 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackend.java --- @@ -97,6 +100,27 @@ public FsStateBackend

[GitHub] flink issue #3483: [FLINK-5979] Backwards compatibility for HeapKeyedStateBa...

2017-03-16 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3483 Thanks for the review, @aljoscha. I have addressed your comments. Merging this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] flink issue #3466: [FLINK-5715] Asynchronous snapshots for heap-based keyed ...

2017-03-16 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3466 Merged in ab014ef. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] flink pull request #3466: [FLINK-5715] Asynchronous snapshots for heap-based...

2017-03-16 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3466 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #3483: [FLINK-5979] Backwards compatibility for HeapKeyedStateBa...

2017-03-16 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3483 Merged in ab014ef. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] flink pull request #3483: [FLINK-5979] Backwards compatibility for HeapKeyed...

2017-03-16 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3483 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 I wonder if there could also exist a case for broadcasting operator state (non-keyed), where only one operator instance is selected as sender and all others receive on restore. Furthermore

[GitHub] flink issue #3508: [FLINK-5991] [state-backend, streaming] Expose Broadcast ...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3508 I think @StephanEwen wanted to keep a simple, intuitive way for users to register their state that does not require them to think about serializers etc.. While I understand this point, I am

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106676403 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106677119 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106671770 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java --- @@ -477,7 +478,7 @@ public

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106670490 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106672829 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106676752 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106673407 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink issue #3359: [FLINK-5544][streaming] Add InternalTimerService implemen...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3359 One additional comment, also as reminder for @aljoscha and me: after this PR is rebased, we have access to `InternalKeyContext`, which should be somehow integrated with the already existing

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106668768 --- Diff: flink-contrib/flink-timerserivce-rocksdb/pom.xml --- @@ -0,0 +1,80 @@ + --- End diff -- I think we should simply integrate

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106677685 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106669974 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106670283 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106675210 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106680293 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106669225 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106674160 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106673233 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106674867 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106671203 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106674528 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106670841 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106675685 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106670661 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerService.java --- @@ -18,43 +18,306

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106677330 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink pull request #3359: [FLINK-5544][streaming] Add InternalTimerService i...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3359#discussion_r106673680 --- Diff: flink-contrib/flink-timerserivce-rocksdb/src/main/java/org/apache/flink/contrib/streaming/api/operators/RocksDBInternalTimerService.java

[GitHub] flink issue #3523: [FLINK-5985] Report no task states for stateless tasks on...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3523 Thanks for the review @StephanEwen. I updated the test as suggested. Merging this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink pull request #3523: [FLINK-5985] Report no task states for stateless t...

2017-03-17 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3523 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink issue #3543: [FLINK-5985] [Backport for 1.2] Report no task states for...

2017-03-20 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3543 Yes, will do today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] flink issue #3543: [FLINK-5985] [Backport for 1.2] Report no task states for...

2017-03-20 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3543 Merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-22 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107409859 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyGroupsStateHandle.java --- @@ -91,10 +98,10 @@ public KeyGroupsStateHandle

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-22 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107411558 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java --- @@ -306,6 +307,29 @@ private static void

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-22 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107410978 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java --- @@ -306,6 +307,29 @@ private static void

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-22 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107409091 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateHandle.java --- @@ -0,0 +1,33 @@ +/* + * Licensed to the

[GitHub] flink issue #3602: [FLINK-5715] Asynchronous snapshots for heap keyed state ...

2017-03-23 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3602 CC @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] flink pull request #3602: [FLINK-5715] Asynchronous snapshots for heap keyed...

2017-03-23 Thread StefanRRichter
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3602 [FLINK-5715] Asynchronous snapshots for heap keyed state backend (BACKPORT) Backport of PR #3466 from 1.3-snapshot to 1.2. Introduces asynchronous snapshots for heap keyed state backend

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107870871 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java --- @@ -290,19 +291,19 @@ private static

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107873583 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java --- @@ -412,9 +421,15 @@ private void

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107874353 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107874660 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateHandle.java --- @@ -0,0 +1,40 @@ +/* + * Licensed to the

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107876739 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java --- @@ -57,6 +58,7 @@ import

[GitHub] flink pull request #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3531#discussion_r107877948 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorSnapshotResult.java --- @@ -30,7 +31,7

[GitHub] flink issue #3531: [FLINK-6034][checkpoint] Add KeyedStateHandle for the sna...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3531 @shixiaogang I had a few more comments on the updated PR. When they are resolved, I think this can be merged immediately. --- If your project is set up for it, you can reply to this email

[GitHub] flink issue #3595: [FLINK-6162]Fix bug in ByteArrayOutputStreamWithPos#setPo...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3595 I would suggest to introduce the analogue check also in `ByteArrayInputStreamWithPos`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink pull request #3602: [FLINK-5715] Asynchronous snapshots for heap keyed...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3602#discussion_r107935639 --- Diff: flink-tests/src/test/java/org/apache/flink/test/checkpointing/AbstractEventTimeWindowCheckpointingITCase.java --- @@ -115,6 +117,14

[GitHub] flink issue #3602: [FLINK-5715] Asynchronous snapshots for heap keyed state ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3602 Thanks for the review, @StephanEwen ! A agree with undoing `MemoryStateBackend` and `MathUtil` as proposed, but would argue for keeping the change in `TimeWindow`. This change fixes skew

[GitHub] flink issue #3602: [FLINK-5715] Asynchronous snapshots for heap keyed state ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3602 After a discussion with @StephanEwen , we decided to follow my proposal. Merging this now. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] flink issue #3602: [FLINK-5715] Asynchronous snapshots for heap keyed state ...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3602 Merged in c6a80725053c49dd2064405577291bdc86c82003. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] flink pull request #3602: [FLINK-5715] Asynchronous snapshots for heap keyed...

2017-03-24 Thread StefanRRichter
Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3602 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] flink pull request #3558: [FLINK-6096][checkpoint] Refactor the migration of...

2017-03-27 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3558#discussion_r107908422 --- Diff: flink-runtime/src/main/java/org/apache/flink/migration/MigrationInstantiationUtil.java --- @@ -0,0 +1,114 @@ +/* + * Licensed to

[GitHub] flink pull request #3558: [FLINK-6096][checkpoint] Refactor the migration of...

2017-03-27 Thread StefanRRichter
Github user StefanRRichter commented on a diff in the pull request: https://github.com/apache/flink/pull/3558#discussion_r108133263 --- Diff: flink-runtime/src/main/java/org/apache/flink/migration/v0/SavepointV0.java --- @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache

  1   2   3   4   5   6   7   8   9   10   >