[GitHub] flink pull request #3078: [FLINK-5355] Handle AmazonKinesisException gracefu...

2017-01-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3078#discussion_r96145754 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java --- @@ -253,6

[GitHub] flink pull request #3078: [FLINK-5355] Handle AmazonKinesisException gracefu...

2017-01-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3078#discussion_r96145761 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java --- @@ -253,6

[GitHub] flink pull request #3078: [FLINK-5355] Handle AmazonKinesisException gracefu...

2017-01-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3078#discussion_r96145793 --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxyTest.java --- @@ -0,0

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3112#discussion_r95979205 --- Diff: docs/dev/connectors/elasticsearch2.md --- @@ -1,173 +0,0 @@ --- End diff -- Will do! --- If your project is set up

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3112#discussion_r95978090 --- Diff: flink-connectors/flink-connector-elasticsearch2/src/test/java/org/apache/flink/streaming/connectors/elasticsearch2/ElasticsearchSinkITCase.java

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3112#discussion_r95978123 --- Diff: docs/dev/connectors/elasticsearch.md --- @@ -23,158 +23,284 @@ specific language governing permissions and limitations under the License

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3112#discussion_r95977827 --- Diff: flink-connectors/flink-connector-elasticsearch5/src/test/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSinkITCase.java

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3112#discussion_r95977656 --- Diff: flink-connectors/flink-connector-elasticsearch5/src/test/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSinkITCase.java

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3112#discussion_r95977306 --- Diff: flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java

[GitHub] flink issue #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x Connec...

2017-01-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3112 Thanks for the pointers. We'll include the profile `include-elasticsearch5` for Java 8 builds only. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink issue #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x Connec...

2017-01-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3112 @rmetzger Ah ... seems like ES 5.x requires at least Java 8. - https://www.elastic.co/guide/en/elasticsearch/reference/master/_installation.html#_installation - https://github.com

[GitHub] flink pull request #3112: [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x...

2017-01-13 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3112 [FLINK-4988] [elasticsearch] Add Elasticsearch 5.x Connector This PR is based on @mikedias's work in #2767 (1st commit), with additional work to restructure the Elasticsearch connectors (2nd

[GitHub] flink pull request #2767: [FLINK-4988] Elasticsearch 5.x support

2017-01-11 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2767#discussion_r95601743 --- Diff: flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSink.java

[GitHub] flink issue #2861: [FLINK-5122] Index requests will be retried if the error ...

2017-01-09 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2861 Hi @static-max, thank you for working on this, it'll be an important fix for proper at least once support for the ES connector. Recently, the community has agreed to first restructure

[GitHub] flink pull request #2767: [FLINK-4988] Elasticsearch 5.x support

2017-01-09 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2767#discussion_r95287899 --- Diff: flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSink.java

[GitHub] flink issue #2767: [FLINK-4988] Elasticsearch 5.x support

2017-01-09 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2767 @mikedias @rmetzger @StephanEwen I've also responded to the discussion in the ML started by Robert with a +1. A recap of the proposed approach on how we proceed: since the ES

[GitHub] flink issue #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to start fr...

2017-01-09 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2916 Hi @tony810430, I've finished adding documents for this new feature, and opened a pull request at your local `FLINK-4523` branch. Can you take a look (would be great if you can review

[GitHub] flink issue #3078: [FLINK-5355] Handle AmazonKinesisException gracefully in ...

2017-01-08 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3078 Thank you for opening a pull request for this @skidder ! I think `AmazonServiceException` will also be thrown for client errors (ex., wrong parameters for API calls). Whether an exception

[GitHub] flink pull request #3036: [FLINK-5368] Log msg if kafka topic doesn't have a...

2017-01-06 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3036#discussion_r94956342 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java --- @@ -208,13

[GitHub] flink issue #2925: [FLINK-4574] [kinesis] Strengthen fetch interval implemen...

2017-01-05 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2925 @tony810430 ok, thanks for your response and patience on the review. I'll proceed with reviewing this and testing it against Kinesis either today or tomorrow. --- If your project is set up

[GitHub] flink issue #3006: [FLINK-5323] CheckpointNotifier should be removed from do...

2017-01-05 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3006 @zentol This hasn't been merged yet, has it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink issue #3035: [ FLINK-4905] Kafka test instability IllegalStateExceptio...

2017-01-05 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3035 Regarding some of your notes: - The exposure of the checkpoint lock through `SourceContext` is meant for sources to atomically update their state (e.x. Kafka offsets) with record emitting

[GitHub] flink issue #3035: [ FLINK-4905] Kafka test instability IllegalStateExceptio...

2017-01-05 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3035 Thank you for working on this @BrainLogic! First, regarding the approach proposed here: The approach should be able to fix the `IllegalStateException` we're encountering. However, I

[GitHub] flink pull request #3036: [FLINK-5368] Log msg if kafka topic doesn't have a...

2017-01-05 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3036#discussion_r94748197 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java --- @@ -208,13

[GitHub] flink issue #3054: [Flink 5404] Consolidate and update S3 documentation

2017-01-05 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3054 @medale The test failure is unrelated, it's ok to ignore that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink pull request #3054: [Flink 5404] Consolidate and update S3 documentati...

2017-01-05 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3054#discussion_r94746306 --- Diff: docs/dev/batch/connectors.md --- @@ -52,33 +52,13 @@ interface. There are Hadoop `FileSystem` implementations for In order to use

[GitHub] flink pull request #3054: [Flink 5404] Consolidate and update S3 documentati...

2017-01-05 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3054#discussion_r94746683 --- Diff: docs/setup/aws.md --- @@ -130,13 +134,13 @@ This registers `NativeS3FileSystem` as the default FileSystem for URIs with the Hadoop

[GitHub] flink issue #3001: [FLINK-4821] [kinesis] Implement rescalable non-partition...

2016-12-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3001 Thanks for the info Stefan! @tony810430 we'll probably need to block this PR for now, and refresh it once the unioned state interface comes around. --- If your project is set up for it, you can

[GitHub] flink issue #3001: [FLINK-4821] [kinesis] Implement rescalable non-partition...

2016-12-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3001 @tony810430 (cc @StephanEwen, f.y.i.) At a second closer look, I'm afraid this PR can't be merged as is. The problem is that the state redistribution of `ListCheckpointed` doesn't work

[GitHub] flink issue #2914: [FLINK-4611] [kinesis] Make "AUTO" credential provider as...

2016-12-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2914 Merging ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] flink issue #2914: [FLINK-4611] [kinesis] Make "AUTO" credential provider as...

2016-12-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2914 Thanks for the clarification. > If the user has provided AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID, regardless of the credential provider type provided (i.e. even if it's

[GitHub] flink issue #2925: [FLINK-4574] [kinesis] Strengthen fetch interval implemen...

2016-12-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2925 Thanks for working on this @tony810430! I'd like to discuss a bit on the design before a more detailed review for this PR. If I'm correct, with this PR we will be creating a separate shard

[GitHub] flink issue #2914: [FLINK-4611] [kinesis] Make "AUTO" credential provider as...

2016-12-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2914 I've tested the `AUTO` config behaviour, and it works as expected in EMR / EC2 environments, so the Kinesis API part should be working fine. Let me summarize the new behaviour after

[GitHub] flink issue #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to start fr...

2016-12-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2916 Thanks for addressing the final comments. I'll add the docs and merge this by the end of the day :) --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2016-12-14 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r92554905 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -294,11

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2016-12-14 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r92554828 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -294,11

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2016-12-14 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r92555340 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -294,11

[GitHub] flink issue #2925: [FLINK-4574] [kinesis] Strengthen fetch interval implemen...

2016-12-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2925 I'm definitely planning to look at this over the next few days :) Currently quite overwhelmed right now. Thanks for all your recent work on the Kinesis connector @tony810430, and very

[GitHub] flink issue #2509: [FLINK-4280][kafka-connector] Explicit start position con...

2016-12-08 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2509 Rebased on the "flink-connectors" change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] flink issue #2893: [FLINK-5128]Get Kafka partitions in FlinkKafkaProducer on...

2016-12-02 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2893 I'll rebase on the batch/streaming connector refactoring and merge this now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90607651 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90608161 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxyInterface.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90607682 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90607632 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-02 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90606695 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java

[GitHub] flink issue #2893: [FLINK-5128]Get Kafka partitions in FlinkKafkaProducer on...

2016-12-02 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2893 Merging this by the end of the day if there's no other objections! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90410691 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisConfigUtil.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90410520 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90411159 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisConfigUtil.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90410148 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90411420 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerTest.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90410957 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/KinesisConfigUtil.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90410106 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java

[GitHub] flink pull request #2916: [FLINK-4523] [kinesis] Allow Kinesis Consumer to s...

2016-12-01 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2916#discussion_r90408543 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java

[GitHub] flink issue #2893: [FLINK-5128]Get Kafka partitions in FlinkKafkaProducer on...

2016-11-29 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2893 Thank you for the contribution @Renkai! +1 to merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-28 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89756558 --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-28 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89755566 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-28 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89755327 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java

[GitHub] flink issue #2509: [FLINK-4280][kafka-connector] Explicit start position con...

2016-11-28 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2509 Hi @rmetzger, I've addressed all comments. I'll leave comments inline of code on parts that addresses your more bigger comments, to help with the second-pass review. --- If your project is set up

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-26 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89680317 --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-25 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89664876 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

[GitHub] flink issue #2822: [FLINK-5075] [kinesis] Make Kinesis consumer fail-proof t...

2016-11-24 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2822 Merging .. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89365136 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/Kafka08Fetcher.java

[GitHub] flink issue #2509: [FLINK-4280][kafka-connector] Explicit start position con...

2016-11-23 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2509 Thanks for the review @rmetzger :) I'll aim to address your comments and rebase by the end of this week (will tag you once it's ready). --- If your project is set up for it, you can reply

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89361344 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010FetcherTest.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89364029 --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89362689 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/SimpleConsumerThread.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89363213 --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89361839 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

2016-11-23 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2509#discussion_r89361688 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java --- @@ -131,6

[GitHub] flink issue #2705: [FLINK-2597][FLINK-4050] Add wrappers for Kafka serialize...

2016-11-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2705 I think a user was asking about this functionality (`KafkaSerializerWrapper`, `KafkaDeserializerWrapper `) earlier on the ML. Would be a great add to the Kafka connectors! Will try to review

[GitHub] flink issue #2681: [FLINK-4155] [kafka] Move partition list fetching to open...

2016-11-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2681 Follow-up JIRA for conditional partition fetching: https://issues.apache.org/jira/browse/FLINK-5128 --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink issue #2681: [FLINK-4155] [kafka] Move partition list fetching to open...

2016-11-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2681 Actually, IIRC, @twalthr already converted the table tests to unit tests in https://issues.apache.org/jira/browse/FLINK-4745. So, right now the table tests are already unit tests without running

[GitHub] flink issue #2822: [FLINK-5075] [kinesis] Make Kinesis consumer fail-proof t...

2016-11-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2822 Thanks for the review @StephanEwen. I'm pretty sure this doesn't affect the normal Kinesis shard discovery. I'll give it some final tests before merging (would like to get this in before the next

[GitHub] flink pull request #2822: [FLINK-5075] [kinesis] Make Kinesis consumer fail-...

2016-11-22 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2822#discussion_r89119126 --- Diff: flink-streaming-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java

[GitHub] flink issue #2681: [FLINK-4155] [kafka] Move partition list fetching to open...

2016-11-22 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2681 Thanks for the review @rmetzger. No, actually I haven't refactored the table tests yet. I'll file up the follow-up JIRAs mentioned in the comments, and merge the PR as is. --- If your

[GitHub] flink issue #2822: [FLINK-5075] [kinesis] Make Kinesis consumer fail-proof t...

2016-11-16 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2822 I'm not entirely sure whether the fix is a good workaround, or whether we should really bother with Kinesalite's incorrect behaviour at all in our Kinesis connector. I've considered

[GitHub] flink pull request #2822: [FLINK-5075] [kinesis] Make Kinesis consumer fail-...

2016-11-16 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/2822 [FLINK-5075] [kinesis] Make Kinesis consumer fail-proof to incorrect Kinesalite API behaviour A user reported that when tested against Kinesalite (a widely used mock Kinesis implementation

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r88273352 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

[GitHub] flink issue #2789: [FLINK-5048] [Kafka Consumer] Change thread model of Flin...

2016-11-16 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2789 > Also, both the fetcher and the KafkaConsumerThread are written to encapsulate all necessary logic self contained. That means they do not rely on each other to call handover.close() in

[GitHub] flink issue #2787: [FLINK-5013] [kinesis] Shade AWS dependencies to work wit...

2016-11-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2787 Merging ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] flink issue #2801: [FLINK-5017] [streaming] Introduce StreamStatus to facili...

2016-11-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2801 No problem @aljoscha, thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink issue #2801: [FLINK-5017] [streaming] Introduce StreamStatus to facili...

2016-11-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2801 R: @aljoscha, @StephanEwen I think this is in good shape for reviews now. Any feedback is welcome! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request #2801: [FLINK-5017] [streaming] Introduce StreamStatus to...

2016-11-14 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/2801 [FLINK-5017] [streaming] Introduce StreamStatus to facilitate idle sources This PR is the first part of making temporarily idle sources in Flink possible, by adding a new `StreamStatus` element

[GitHub] flink issue #2787: [FLINK-5013] [kinesis] Shade AWS dependencies to work wit...

2016-11-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2787 @StephanEwen @rmetzger I would personally like to get this fix in 1.1.4 (from some recent PRs it seems like we need a new RC, so might as well see if we want to get this fix in too

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87742870 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87744744 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka09FetcherTest.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87738353 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87744290 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87734223 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge010

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87735824 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetention010ITCase.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87734293 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge010

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87746033 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/internal/HandoverTest.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87746504 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/internal/HandoverTest.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87738700 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87736264 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetention010ITCase.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87745778 --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetentionTestBase.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87739024 --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java

[GitHub] flink pull request #2789: [FLINK-5048] [Kafka Consumer] Change thread model ...

2016-11-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/2789#discussion_r87745858 --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetentionTestBase.java

[GitHub] flink issue #2787: [FLINK-5013] [kinesis] Shade AWS dependencies to work wit...

2016-11-11 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2787 I've actually found out the reason for `HttpConnectionParams.setSoKeepalive` error, noted in the last comment in JIRA. Cross-posting it here so we don't need to jump around two places

<    11   12   13   14   15   16   17   18   19   20   >