[jira] [Created] (FLINK-8544) JSONKeyValueDeserializationSchema throws NPE when message key is null

2018-02-01 Thread Bill Lee (JIRA)
Bill Lee created FLINK-8544: --- Summary: JSONKeyValueDeserializationSchema throws NPE when message key is null Key: FLINK-8544 URL: https://issues.apache.org/jira/browse/FLINK-8544 Project: Flink

[jira] [Updated] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

2018-02-01 Thread chris snow (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chris snow updated FLINK-8543: -- Description: I'm hitting an issue with my BucketingSink from a streaming job.   {code:java} return new

[jira] [Updated] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

2018-02-01 Thread chris snow (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chris snow updated FLINK-8543: -- Attachment: Screen Shot 2018-01-30 at 18.34.51.png Description: I'm hitting an issue with my

[GitHub] flink pull request #:

2018-02-01 Thread tzulitai
Github user tzulitai commented on the pull request: https://github.com/apache/flink/commit/a2533f406d46b1c5acb5f70c263f9afad839dffe#commitcomment-27262148 In flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/operator/CEPMigrationTest.java: In

[jira] [Created] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

2018-02-01 Thread chris snow (JIRA)
chris snow created FLINK-8543: - Summary: Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen Key: FLINK-8543 URL: https://issues.apache.org/jira/browse/FLINK-8543 Project: Flink

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165349688 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165349012 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348543#comment-16348543 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348525#comment-16348525 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165344648 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java --- @@ -630,19 +506,210 @@ public int

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165344021 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointTest.java --- @@ -19,58 +19,53 @@ package

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348522#comment-16348522 ] ASF GitHub Bot commented on FLINK-5820: --- Github user StephanEwen commented on a diff in the pull

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348516#comment-16348516 ] ASF GitHub Bot commented on FLINK-5820: --- Github user StephanEwen commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348518#comment-16348518 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165343607 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/SnapshotStrategy.java --- @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165343453 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStateOutputStreamTest.java --- @@ -332,6 +335,43 @@ public

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348513#comment-16348513 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165343021 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -156,30 +157,38 @@ public

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348514#comment-16348514 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342796 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/util/JvmExitOnFatalErrorTest.java --- @@ -137,9 +138,12 @@ public static void

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342545 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskStateManagerImplTest.java --- @@ -97,6 +104,39 @@ public void

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348511#comment-16348511 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink issue #5394: [FLINK-6571][tests] Catch InterruptedException in StreamS...

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5394 It may not be a problem in this test, but I wanted to raise that this pattern is a bit dangerous. If the thread ever gets interrupted while 'running' is still true, this goes into a hot loop

[jira] [Commented] (FLINK-6571) InfiniteSource in SourceStreamOperatorTest should deal with InterruptedExceptions

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348512#comment-16348512 ] ASF GitHub Bot commented on FLINK-6571: --- Github user StephanEwen commented on the issue:

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348510#comment-16348510 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342341 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskStateManagerImplTest.java --- @@ -97,6 +104,39 @@ public void

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348508#comment-16348508 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165342139 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskStateManagerImplTest.java --- @@ -25,17 +25,24 @@ import

[GitHub] flink issue #5380: [hotfix][connectors] Fix log format strings

2018-02-01 Thread StephanEwen
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5380 Please merge for `master` and `release-1.4`... ---

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348506#comment-16348506 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348505#comment-16348505 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341901 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManagerTest.java --- @@ -0,0 +1,122 @@ +/* +

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341653 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManagerTest.java --- @@ -0,0 +1,122 @@ +/* +

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341397 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManagerTest.java --- @@ -0,0 +1,122 @@ +/* +

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348503#comment-16348503 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348501#comment-16348501 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165341071 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServices.java --- @@ -197,7 +201,13 @@ public static

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348495#comment-16348495 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340438 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackend.java --- @@ -501,4 +529,53 @@ public String toString()

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348494#comment-16348494 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340246 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskStateManager.java --- @@ -60,4 +62,9 @@ void reportTaskStateSnapshots(

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348492#comment-16348492 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165340082 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStore.java --- @@ -46,26 +52,63 @@ /** */ private

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348467#comment-16348467 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165332129 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java --- @@ -316,6 +316,11 @@ */ public static final

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348463#comment-16348463 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348461#comment-16348461 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165331598 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -686,4

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165331645 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -686,4

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348444#comment-16348444 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348442#comment-16348442 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165329478 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -104,10

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165329369 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java --- @@ -104,10

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348424#comment-16348424 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165326694 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -0,0 +1,213 @@ +/*

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348422#comment-16348422 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165325793 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348414#comment-16348414 ] ASF GitHub Bot commented on FLINK-8484: --- Github user pluppens commented on the issue:

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread pluppens
Github user pluppens commented on the issue: https://github.com/apache/flink/pull/5337 Good point. An ugly workaround would be to store a timestamp when the ending number is being set on a shard, and provide a configurable/sufficiently enough (eg. 7 days) window. It would exclude the

[jira] [Commented] (FLINK-8242) ClassCastException in OrcTableSource.toOrcPredicate

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348411#comment-16348411 ] ASF GitHub Bot commented on FLINK-8242: --- Github user twalthr commented on the issue:

[GitHub] flink issue #5345: [FLINK-8242] [orc] Fix predicate push-down of OrcTableSou...

2018-02-01 Thread twalthr
Github user twalthr commented on the issue: https://github.com/apache/flink/pull/5345 Thank you @fhueske. The code looks good now. I will merge this... ---

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348408#comment-16348408 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348405#comment-16348405 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323982 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/LocalStateForwardingTest.java --- @@ -0,0 +1,213 @@ +/*

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323823 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/StateObjectCollectionTest.java --- @@ -0,0 +1,64 @@ +/* + * Licensed

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348406#comment-16348406 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165323575 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348399#comment-16348399 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165322528 --- Diff: flink-core/src/test/java/org/apache/flink/util/MethodForwardingTestUtil.java --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348395#comment-16348395 ] ASF GitHub Bot commented on FLINK-8484: --- Github user tzulitai commented on the issue:

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348392#comment-16348392 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 Here it is: https://issues.apache.org/jira/browse/FLINK-8542 ---

[jira] [Created] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer

2018-02-01 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-8542: -- Summary: Do not indefinitely store closed shard's state in the FlinkKinesisConsumer Key: FLINK-8542 URL: https://issues.apache.org/jira/browse/FLINK-8542

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165321013 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/TestTaskStateManager.java --- @@ -67,8 +69,11 @@ public TestTaskStateManager(

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348383#comment-16348383 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348384#comment-16348384 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348381#comment-16348381 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165295667 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java --- @@ -18,45 +18,50 @@ package

[jira] [Commented] (FLINK-5820) Extend State Backend Abstraction to support Global Cleanup Hooks

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348382#comment-16348382 ] ASF GitHub Bot commented on FLINK-5820: --- Github user aljoscha commented on a diff in the pull

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165317663 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CompletedCheckpointTest.java --- @@ -19,58 +19,53 @@ package

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165318529 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStateOutputStreamTest.java --- @@ -332,6 +335,43 @@ public

[GitHub] flink pull request #5396: [FLINK-5820] [state backends] Split shared/exclusi...

2018-02-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5396#discussion_r165295768 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java --- @@ -18,45 +18,50 @@ package

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348376#comment-16348376 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348375#comment-16348375 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165318377 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTerminationTest.java --- @@ -227,21 +229,13 @@

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165318246 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTerminationTest.java --- @@ -227,21 +229,13 @@

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348374#comment-16348374 ] ASF GitHub Bot commented on FLINK-8484: --- Github user tzulitai commented on the issue:

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens My only concern is that scanning the whole list of shards can be very limited to AWS Kinesis's API invoke rate limitations. Also, we would then only be cleaning up the state on

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348372#comment-16348372 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165317571 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java --- @@ -916,57 +916,78 @@ private void

[jira] [Commented] (FLINK-8484) Kinesis consumer re-reads closed shards on job restart

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348367#comment-16348367 ] ASF GitHub Bot commented on FLINK-8484: --- Github user pluppens commented on the issue:

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread pluppens
Github user pluppens commented on the issue: https://github.com/apache/flink/pull/5337 Regarding the remark from @StephanEwen: perhaps it would be ok to re-use the `KinesisProxy` to return a list of all shards and compare them to the `sequenceNumsToRestore` to prune any shards that

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348364#comment-16348364 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165315816 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/OperatorSnapshotFuturesTest.java --- @@ -46,20 +47,28 @@ public

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348362#comment-16348362 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165315625 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java --- @@ -859,56 +861,77 @@ public void run() {

[jira] [Comment Edited] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

2018-02-01 Thread yanxiaobin (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343115#comment-16343115 ] yanxiaobin edited comment on FLINK-5479 at 2/1/18 10:30 AM: I've also met this

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348340#comment-16348340 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

[GitHub] flink pull request #5239: [FLINK-8360] Implement task-local state recovery

2018-02-01 Thread tillrohrmann
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5239#discussion_r165309957 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorSnapshotFinalizer.java --- @@ -0,0 +1,83 @@ +/*

[jira] [Commented] (FLINK-8360) Implement task-local state recovery

2018-02-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348339#comment-16348339 ] ASF GitHub Bot commented on FLINK-8360: --- Github user tillrohrmann commented on a diff in the pull

<    1   2   3   >