[GitHub] flink pull request #5426: [FLINK-8362] [elasticsearch] Shade all ES connecto...

2018-02-07 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5426 [FLINK-8362] [elasticsearch] Shade all ES connector dependencies ## What is the purpose of the change This is an extended version of Nico's work in #5243. This version

[GitHub] flink pull request #5336: (release-1.4) [FLINK-8419] [kafka] Register metric...

2018-02-07 Thread tzulitai
Github user tzulitai closed the pull request at: https://github.com/apache/flink/pull/5336 ---

[GitHub] flink issue #5393: [FLINK-8516] Allow for custom hash function for shard to ...

2018-02-07 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5393 @tweise ok, I think I can agree on your last comment. > Should I add a (trivial) unit test that asserts that isThisSubtaskShouldSubscribeTo applies modulus to assigner returned va

[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...

2018-02-06 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5374 @cjolif sorry, I didn't have the chance, yet. But I agree that we should try getting this in for 1.5 (I saw your reply on the 1.5 release discussion thread in the mailing lists). I'll try

[GitHub] flink issue #5362: [FLINK-8421] [DataStream] Make timer serializers reconfig...

2018-02-06 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5362 Thanks for the reviews! Will merge after Travis gives green. ---

[GitHub] flink issue #5393: [FLINK-8516] Allow for custom hash function for shard to ...

2018-02-06 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5393 @tweise regarding what the proposed migration test is going to assert: > The assigner does not influence how state is saved and restored. Even when the assigner returns invalid in

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-06 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r166254162 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5172: [FLINK-8275] [Security] fix keytab local path in Y...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5172#discussion_r165636080 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java --- @@ -142,19 +153,10 @@ public static void runYarnTaskManager(String

[GitHub] flink issue #5172: [FLINK-8275] [Security] fix keytab local path in YarnTask...

2018-02-02 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5172 @suez1224 Thanks a lot for the contribution! I've had a look and the changes LGTM. I did have a comment regarding injecting a dependency for the runner, which I've added a commit

[GitHub] flink pull request #5172: [FLINK-8275] [Security] fix keytab local path in Y...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5172#discussion_r165635517 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationMasterRunner.java --- @@ -170,9 +160,10 @@ protected int run(String[] args

[GitHub] flink pull request #5172: [FLINK-8275] [Security] fix keytab local path in Y...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5172#discussion_r165663546 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java --- @@ -142,19 +153,10 @@ public static void runYarnTaskManager(String

[GitHub] flink pull request #5362: [FLINK-8421] [DataStream] Make timer serializers r...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5362#discussion_r165646464 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java --- @@ -52,7 +52,7

[GitHub] flink issue #5364: [FLINK-8472] [test] Extend all migration tests for Flink ...

2018-02-02 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5364 Thanks for the reviews, I'll address Chesnay's comment and the merge this then (to `master` and `release-1.4`). ---

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165590401 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165590209 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165587359 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165588489 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165589748 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165591410 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -192,6

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165587909 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165588295 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165588395 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165589453 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165586434 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165586994 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165588081 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisShardAssigner.java --- @@ -0,0

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-02-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165594279 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -192,6

[GitHub] flink pull request #5362: [FLINK-8421] [DataStream] Make timer serializers r...

2018-02-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5362#discussion_r165573730 --- Diff: flink-core/src/main/java/org/apache/flink/core/io/PostVersionedIOReadableWritable.java --- @@ -0,0 +1,78 @@ +/* + * Licensed

[GitHub] flink issue #5362: [FLINK-8421] [DataStream] Make timer serializers reconfig...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5362 @aljoscha As discussed offline, I've: - replaced `ByteArrayPrependedInputStream` with Java's `PushbackInputStream` - use negative values in `VERSIONED_IDENTIFIER` to be extra safe ---

[GitHub] flink pull request #5364: [FLINK-8472] [test] Extend all migration tests for...

2018-02-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5364#discussion_r165385564 --- Diff: flink-tests/src/test/java/org/apache/flink/test/state/operator/restore/keyed/KeyedJob.java --- @@ -100,9 +100,7 @@ public static void main

[GitHub] flink issue #5364: [FLINK-8472] [test] Extend all migration tests for Flink ...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5364 @aljoscha @zentol could you have another quick look at commit 8882fb7? That commit extends `KeyedComplexChainTest`, `ChainBreakTest`, `ChainLengthIncreaseTest`, ... etc. also for 1.4

[GitHub] flink issue #5398: [hotfix][cep] Remove migration from 1.5 test

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5398 Thanks, LGTM. We can add it back once `release-1.5` is cut. It might not really make any difference in the end, but we never know :) ---

[GitHub] flink pull request #:

2018-02-01 Thread tzulitai
Github user tzulitai commented on the pull request: https://github.com/apache/flink/commit/a2533f406d46b1c5acb5f70c263f9afad839dffe#commitcomment-27262148 In flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/operator/CEPMigrationTest.java: In flink-libraries/flink-cep

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 Here it is: https://issues.apache.org/jira/browse/FLINK-8542 ---

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-02-01 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens My only concern is that scanning the whole list of shards can be very limited to AWS Kinesis's API invoke rate limitations. Also, we would then only be cleaning up the state

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 Great, thanks for the update! As a side note, I will be making some additional changes to the code regarding the not-so-nice iteration across the `sequenceNumsToRestore` map. It would make

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165083343 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165081311 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165084584 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -93,6

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165084048 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5393: [FLINK-8516] Allow for custom hash function for sh...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5393#discussion_r165081173 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink issue #5316: [FLINK-8022][kafka-tests] Disable at-least-once tests for...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5316 I agree with @pnowojski here. I think we should proceed with merging this to test stability, and keep an eye on whether or not this is also failing for 0.10 / 0.11. If yes, then there could

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens the changes look good to merge! Thanks a lot for working on this. Will merge this for `release-1.4`, `release-1.3`, and `master` .. ---

[GitHub] flink pull request #5335: (master) [FLINK-8419] [kafka] Register metrics for...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5335#discussion_r165045598 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java --- @@ -95,21

[GitHub] flink issue #5269: [FLINK-6004] [kinesis] Allow FlinkKinesisConsumer to skip...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5269 Thanks, merging .. ---

[GitHub] flink issue #5329: [FLINK-8409] [kafka] Fix potential NPE in KafkaConsumerTh...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5329 Merging ... ---

[GitHub] flink issue #5269: [FLINK-6004] [kinesis] Allow FlinkKinesisConsumer to skip...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5269 @zentol I've rebased, and adding a warning when skipping records. Could you have another quick look? ---

[GitHub] flink issue #5269: [FLINK-6004] [kinesis] Allow FlinkKinesisConsumer to skip...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5269 @zentol The contract is that if `KinesisDeserializationSchema.deserialize()` returns `null`, that record will be skipped. I agree it makes sense to have a add a warning log if this happens. ---

[GitHub] flink pull request #5269: [FLINK-6004] [kinesis] Allow FlinkKinesisConsumer ...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5269#discussion_r165039352 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #5269: [FLINK-6004] [kinesis] Allow FlinkKinesisConsumer ...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5269#discussion_r165038908 --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcherTest.java

[GitHub] flink issue #5268: [FLINK-8398] [kinesis, tests] Harden KinesisDataFetcherTe...

2018-01-31 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5268 Thanks for the review @zentol. Will rebase, address your comment, and then merge. ---

[GitHub] flink pull request #:

2018-01-31 Thread tzulitai
Github user tzulitai commented on the pull request: https://github.com/apache/flink/commit/547d19f9196512231661f427f3792f2e1f831339#commitcomment-27231640 In flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/testutils

[GitHub] flink pull request #5336: (release-1.4) [FLINK-8419] [kafka] Register metric...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5336#discussion_r165023649 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java --- @@ -92,21

[GitHub] flink pull request #5336: (release-1.4) [FLINK-8419] [kafka] Register metric...

2018-01-31 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5336#discussion_r165022738 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java --- @@ -92,21

[GitHub] flink issue #5364: [FLINK-8472] [test] Extend all migration tests for Flink ...

2018-01-29 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5364 It seems like we'll also need to extend these migration tests for 1.4: - All subclasses of `AbstractKeyedOperatorRestoreTestBase` - All subclasses

[GitHub] flink issue #5378: [FLINK-8489][ES] Prevent side-effects when modifying user...

2018-01-29 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5378 LGTM, once Travis is green. ---

[GitHub] flink issue #5380: [hotfix][connectors] Fix log format strings

2018-01-29 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5380 LGTM, will merge this. ---

[GitHub] flink issue #4675: [FLINK-7386] FIx Elasticsearch 5 connector is not compati...

2018-01-29 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4675 Does #5374 subsume this PR? ---

[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...

2018-01-29 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5374 @cjolif Thanks a lot for the contribution! I’ll add reviewing the PR to my backlog. I would like / will try to take a look at the PR by the end of this week, after some 1.4.1 blockers

[GitHub] flink pull request #4979: RMQSource support disabling queue declaration

2018-01-26 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/4979#discussion_r164086981 --- Diff: flink-connectors/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQSource.java --- @@ -138,7 +138,9

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163938277 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/RegisteredBroadcastBackendStateMetaInfo.java --- @@ -0,0 +1,232

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163933180 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java --- @@ -77,16 +90,29 @@ public void read

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163936390 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateHandle.java --- @@ -36,8 +36,9 @@ * The modes that determine

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163938102 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/RegisteredBroadcastBackendStateMetaInfo.java --- @@ -0,0 +1,232

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163935065 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java --- @@ -77,16 +90,29 @@ public void read

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163935632 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendStateMetaInfoSnapshotReaderWriters.java --- @@ -211,4 +297,34

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163932401 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java --- @@ -64,11 +70,18 @@ public int getVersion

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163942745 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java --- @@ -77,16 +90,29 @@ public void read

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163932956 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java --- @@ -35,18 +35,24 @@ public

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163934364 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendStateMetaInfoSnapshotReaderWriters.java --- @@ -112,13 +142,40

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163930707 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/DefaultOperatorStateBackend.java --- @@ -137,7 +155,12 @@ public ExecutionConfig

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163932340 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java --- @@ -64,11 +70,18 @@ public int getVersion

[GitHub] flink pull request #5230: [FLINK-8345] Add iterator of keyed state on broadc...

2018-01-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5230#discussion_r163936903 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateHandle.java --- @@ -36,8 +36,9 @@ * The modes that determine

[GitHub] flink pull request #5364: Flink 8472 1.4

2018-01-25 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5364 Flink 8472 1.4 ## What is the purpose of the change Extend all migration tests, to include verifying restore from Flink 1.4 savepoints. This includes extending

[GitHub] flink pull request #5362: [FLINK-8421] [DataStream] Make timer serializers r...

2018-01-25 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5362 [FLINK-8421] [DataStream] Make timer serializers reconfigurable on restore ## What is the purpose of the change Previously, key and namespace serializers of the `HeapInternalTimerService

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-01-24 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 re @bowenli86 yes, Kinesis shard metadata is fetched every `DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS` millis. Could you describe a bit more which part you don't think is valid

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-01-24 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 re @StephanEwen yes, currently, state is still kept indefinitely for closed shards. A special `SHARD_END` marker sequence number is stored as the sequence number for closed shards, so

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-01-23 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens yes, that sounds correct. `parentShardId` `adjacentParentShardId` `startingHashKey` `endingHashKey` `startingSequenceNumber` these should all be fixed once

[GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...

2018-01-23 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5337 @pluppens yes, I think that would be the proper solution here. ---

[GitHub] flink pull request #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kin...

2018-01-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5337#discussion_r163175349 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,6

[GitHub] flink pull request #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kin...

2018-01-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5337#discussion_r163176506 --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerTest.java --- @@ -515,6

[GitHub] flink pull request #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kin...

2018-01-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5337#discussion_r163175172 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -210,16

[GitHub] flink pull request #5336: (release-1.4) [FLINK-8419] [kafka] Register metric...

2018-01-22 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5336 (release-1.4) [FLINK-8419] [kafka] Register metrics for dynamically discovered Kafka partitions ## What is the purpose of the change Different version of #5335, which is targeted

[GitHub] flink pull request #5335: (master) [FLINK-8419] [kafka] Register metrics for...

2018-01-22 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5335 (master) [FLINK-8419] [kafka] Register metrics for dynamically discovered Kafka partitions ## What is the purpose of the change This PR fixes that offset metrics (i.e. current offset

[GitHub] flink issue #5298: [FLINK-8433] [doc] Remove ununsed CheckpointedRestoring i...

2018-01-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5298 +1, LGTM. ---

[GitHub] flink pull request #5301: [FLINK-8267] [Kinesis Connector] update Kinesis Pr...

2018-01-22 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5301#discussion_r162904042 --- Diff: docs/dev/connectors/kinesis.md --- @@ -271,9 +271,9 @@ For the monitoring to work, the user accessing the stream needs access to the Cl

[GitHub] flink issue #5301: [FLINK-8267] [Kinesis Connector] update Kinesis Producer ...

2018-01-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5301 @bowenli86 can you squash the commits into a single one, with appropriate commit message? Thanks! ---

[GitHub] flink pull request #5329: [FLINK-8409] [kafka] Fix potential NPE in KafkaCon...

2018-01-21 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5329 [FLINK-8409] [kafka] Fix potential NPE in KafkaConsumerThread ## What is the purpose of the change This PR fixes a race condition that may lead to a NPE in the async callbacks for Kafka

[GitHub] flink issue #5291: [FLINK-8361] [build] Remove create_release_files.sh

2018-01-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5291 +1, LGTM ---

[GitHub] flink issue #5284: [FLINK-8306] [kafka, tests] Fix invalid mock verification...

2018-01-12 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5284 Yes, I've fixed that checkstyle on my local pre-merge branch 👌 ---

[GitHub] flink issue #5124: [FLINK-8199] [elasticsearch connector] Properly annotate ...

2018-01-12 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5124 Since there are no other objections, merging this now. ---

[GitHub] flink issue #5173: [FLINK-8276] [kafka connector] Properly annotate API for ...

2018-01-12 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5173 Since there is no other objections, will merge this now ... ---

[GitHub] flink issue #5138: [FLINK-8217] [Kinesis connector] Properly annotate APIs o...

2018-01-12 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5138 Since there are no other objections, will merge this to `master` now .. ---

[GitHub] flink pull request #5284: [FLINK-8306] [kafka, tests] Fix invalid mock verif...

2018-01-12 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/5284#discussion_r161160136 --- Diff: flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBaseTest.java

[GitHub] flink issue #5182: [FLINK-8162] [kinesis-connector] Emit Kinesis' millisBehi...

2018-01-12 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5182 Hi @casidiablo, just before merging, I made some refactoring of your PR to move the metrics reporting business out of the `ShardConsumer`: https://github.com/apache/flink/commit

[GitHub] flink issue #5182: [FLINK-8162] [kinesis-connector] Emit Kinesis' millisBehi...

2018-01-11 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5182 Thanks @casidiablo. Merging ... ---

[GitHub] flink pull request #5200: [FLINK-8306] [kafka] Introduce KafkaOffsetCommitte...

2018-01-11 Thread tzulitai
Github user tzulitai closed the pull request at: https://github.com/apache/flink/pull/5200 ---

[GitHub] flink issue #5200: [FLINK-8306] [kafka] Introduce KafkaOffsetCommitter inter...

2018-01-11 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5200 Subsumed be re-opened PR: #5284 ---

[GitHub] flink pull request #5284: [FLINK-8306] [kafka, tests] Fix invalid mock verif...

2018-01-11 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/5284 [FLINK-8306] [kafka, tests] Fix invalid mock verifications on final method ## What is the purpose of the change This is a reworked version of #5200. Instead of introducing a new

[GitHub] flink issue #5280: Fix typo in AbstractMetricGroup.java

2018-01-11 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5280 Thanks, merging this. Do note that you should include the `[hotfix]` and `[doc]` tags to the commit message. All our commit requires proper tags to indicate the issue id (or `hotfix

<    1   2   3   4   5   6   7   8   9   10   >