[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348399#comment-16348399 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165322528 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165325793 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340438 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackend.java --- @@ -501,4 +529,53 @@ public String toString()

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348495#comment-16348495 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342341 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskStateManagerImplTest.java --- @@ -97,6 +104,39 @@ public void

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348510#comment-16348510 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Created] (FLINK-8544) JSONKeyValueDeserializationSchema throws NPE when message key is null

2018-02-01 Thread Bill Lee (JIRA)
Bill Lee created FLINK-8544: --- Summary: JSONKeyValueDeserializationSchema throws NPE when message key is null Key: FLINK-8544 URL: https://issues.apache.org/jira/browse/FLINK-8544 Project: Flink

[jira] [Updated] (FLINK-5372) Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints()

2018-02-01 Thread Till Rohrmann (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-5372: - Affects Version/s: 1.5.0 > Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints() >

[GitHub] flink issue #5398: [hotfix][cep] Remove migration from 1.5 test

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5398 Thanks, LGTM. We can add it back once `release-1.5` is cut. It might not really make any difference in the end, but we never know :) ---

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348374#comment-16348374 ] ASF GitHub Bot commented on FLINK-8484: --- Github user tzulitai commented on the issue:

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165318246 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTerminationTest.java --- @@ -227,21 +229,13 @@

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348375#comment-16348375 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165318377 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTerminationTest.java --- @@ -227,21 +229,13 @@

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348392#comment-16348392 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 Here it is: https://issues.apache.org/jira/browse/FLINK-8542 ---

[jira] [Created] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer

2018-02-01 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-8542: -- Summary: Do not indefinitely store closed shard's state in the FlinkKinesisConsumer Key: FLINK-8542 URL: https://issues.apache.org/jira/browse/FLINK-8542

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348422#comment-16348422 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348442#comment-16348442 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165329478 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -104,10

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165329369 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -104,10

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165331645 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -686,4

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165331598 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -686,4

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341397 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManagerTest.java --- @@ -0,0 +1,122 @@ +/* +

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348503#comment-16348503 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Updated] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

2018-02-01 Thread chris snow (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chris snow updated FLINK-8543: -- Description: I'm hitting an issue with my BucketingSink from a streaming job.   {code:java} return new

[jira] [Updated] (FLINK-5372) Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints()

2018-02-01 Thread Till Rohrmann (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-5372: - Fix Version/s: 1.5.0 > Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints() >

[jira] [Commented] (FLINK-5372) Fix RocksDBAsyncSnapshotTest.testCancelFullyAsyncCheckpoints()

2018-02-01 Thread Till Rohrmann (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348593#comment-16348593 ] Till Rohrmann commented on FLINK-5372: -- Another instance: 

[GitHub] flink issue #5295: [FLINK-8384] [streaming] Session Window Assigner with Dyn...

2018-02-01 Thread dyanarose
Github user dyanarose commented on the issue: https://github.com/apache/flink/pull/5295 the ci fail looks to be a known flaky test: FlinkKafkaProducer011ITCase.testScaleDownBeforeFirstCheckpoint ---

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348364#comment-16348364 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165315816 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/OperatorSnapshotFuturesTest.java --- @@ -46,20 +47,28 @@ public

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165317663 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointTest.java --- @@ -19,58 +19,53 @@ package

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165318529 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStateOutputStreamTest.java --- @@ -332,6 +335,43 @@ public

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165295768 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java --- @@ -18,45 +18,50 @@ package

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323575 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348424#comment-16348424 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165326694 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -0,0 +1,213 @@ +/*

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340082 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStore.java --- @@ -46,26 +52,63 @@ /** */ private

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348492#comment-16348492 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348494#comment-16348494 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340246 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskStateManager.java --- @@ -60,4 +62,9 @@ void reportTaskStateSnapshots(

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348506#comment-16348506 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341901 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManagerTest.java --- @@ -0,0 +1,122 @@ +/* +

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348505#comment-16348505 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341653 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManagerTest.java --- @@ -0,0 +1,122 @@ +/* +

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165344021 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointTest.java --- @@ -19,58 +19,53 @@ package

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348522#comment-16348522 ] ASF GitHub Bot commented on FLINK-5820: --- Github user StephanEwen commented on a diff in the pull

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens My only concern is that scanning the whole list of shards can be very limited to AWS Kinesis's API invoke rate limitations. Also, we would then only be cleaning up the state on

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348372#comment-16348372 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8242) ClassCastException in OrcTableSource.toOrcPredicate

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348411#comment-16348411 ] ASF GitHub Bot commented on FLINK-8242: --- Github user twalthr commented on the issue:

[GitHub] flink issue #5345: [FLINK-8242] [orc] Fix predicate push-down of OrcTableSou...

2018-02-01 Thread twalthr
Github user twalthr commented on the issue: https://github.com/apache/flink/pull/5345 Thank you @fhueske. The code looks good now. I will merge this... ---

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341071 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServices.java --- @@ -197,7 +201,13 @@ public static

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348501#comment-16348501 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Updated] (FLINK-8545) Implement upsert stream table source

2018-02-01 Thread Hequn Cheng (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng updated FLINK-8545: --- Description: As more and more users are eager for ingesting data with upsert mode in flink

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348384#comment-16348384 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348381#comment-16348381 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348382#comment-16348382 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165295667 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java --- @@ -18,45 +18,50 @@ package

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread pluppens
Github user pluppens commented on the issue: https://github.com/apache/flink/pull/5337 Good point. An ugly workaround would be to store a timestamp when the ending number is being set on a shard, and provide a configurable/sufficiently enough (eg. 7 days) window. It would exclude the

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348414#comment-16348414 ] ASF GitHub Bot commented on FLINK-8484: --- Github user pluppens commented on the issue:

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348444#comment-16348444 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348467#comment-16348467 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165332129 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java --- @@ -316,6 +316,11 @@ */ public static final

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165349688 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@

[GitHub] flink pull request #:

2018-02-01 Thread tzulitai
Github user tzulitai commented on the pull request: https://github.com/apache/flink/commit/a2533f406d46b1c5acb5f70c263f9afad839dffe#commitcomment-27262148 In flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/operator/CEPMigrationTest.java: In

[GitHub] flink issue #5398: [hotfix][cep] Remove migration from 1.5 test

2018-02-01 Thread dawidwys
Github user dawidwys commented on the issue: https://github.com/apache/flink/pull/5398 @tzulitai I removed the migration from 1.5 ---

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread pluppens
Github user pluppens commented on the issue: https://github.com/apache/flink/pull/5337 Regarding the remark from @StephanEwen: perhaps it would be ok to re-use the `KinesisProxy` to return a list of all shards and compare them to the `sequenceNumsToRestore` to prune any shards that

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348367#comment-16348367 ] ASF GitHub Bot commented on FLINK-8484: --- Github user pluppens commented on the issue:

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348376#comment-16348376 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348395#comment-16348395 ] ASF GitHub Bot commented on FLINK-8484: --- Github user tzulitai commented on the issue:

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348406#comment-16348406 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323823 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/StateObjectCollectionTest.java --- @@ -0,0 +1,64 @@ +/* + * Licensed

[GitHub] flink issue #5380: [hotfix][connectors] Fix log format strings

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5380 Please merge for `master` and `release-1.4`... ---

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348508#comment-16348508 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342139 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskStateManagerImplTest.java --- @@ -25,17 +25,24 @@ import

[jira] [Updated] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

2018-02-01 Thread chris snow (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chris snow updated FLINK-8543: -- Attachment: Screen Shot 2018-01-30 at 18.34.51.png Description: I'm hitting an issue with my

[GitHub] flink pull request #5398: [hotfix][cep] Remove migration from 1.5 test

2018-02-01 Thread dawidwys
GitHub user dawidwys opened a pull request: https://github.com/apache/flink/pull/5398 [hotfix][cep] Remove migration from 1.5 test *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your

[jira] [Created] (FLINK-8545) Implement upsert stream table source

2018-02-01 Thread Hequn Cheng (JIRA)
Hequn Cheng created FLINK-8545: -- Summary: Implement upsert stream table source Key: FLINK-8545 URL: https://issues.apache.org/jira/browse/FLINK-8545 Project: Flink Issue Type: New Feature

[GitHub] flink issue #5394: [FLINK-6571][tests] Catch InterruptedException in StreamS...

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5394 It may not be a problem in this test, but I wanted to raise that this pattern is a bit dangerous. If the thread ever gets interrupted while 'running' is still true, this goes into a hot loop

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348511#comment-16348511 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342545 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskStateManagerImplTest.java --- @@ -97,6 +104,39 @@ public void

[jira] [Commented] (FLINK-6571) InfiniteSource in SourceStreamOperatorTest should deal with InterruptedExceptions

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348512#comment-16348512 ] ASF GitHub Bot commented on FLINK-6571: --- Github user StephanEwen commented on the issue:

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342796 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/util/JvmExitOnFatalErrorTest.java --- @@ -137,9 +138,12 @@ public static void

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348543#comment-16348543 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165349012 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165315625 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java --- @@ -859,56 +861,77 @@ public void run() {

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348362#comment-16348362 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348383#comment-16348383 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348405#comment-16348405 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323982 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -0,0 +1,213 @@ +/*

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348408#comment-16348408 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165343021 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -156,30 +157,38 @@ public

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348514#comment-16348514 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348513#comment-16348513 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348525#comment-16348525 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165344648 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java --- @@ -630,19 +506,210 @@ public int

[jira] [Created] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

2018-02-01 Thread chris snow (JIRA)
chris snow created FLINK-8543: - Summary: Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen Key: FLINK-8543 URL: https://issues.apache.org/jira/browse/FLINK-8543 Project: Flink

[GitHub] flink pull request #:

2018-02-01 Thread dawidwys
Github user dawidwys commented on the pull request: https://github.com/apache/flink/commit/a2533f406d46b1c5acb5f70c263f9afad839dffe#commitcomment-27262545 In flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/operator/CEPMigrationTest.java: In

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165317571 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java --- @@ -916,57 +916,78 @@ private void

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165321013 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TestTaskStateManager.java --- @@ -67,8 +69,11 @@ public TestTaskStateManager(

  1   2   3   >