[GitHub] flink issue #3901: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3901 I'm currently waiting for a Travis run locally for https://github.com/tzulitai/flink/tree/FLINK-6288. That branch also includes more tests and tweak cleanups for the feature. Once

[GitHub] flink issue #3901: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3901 I think you've made a good point Gyula. Alright, let me try to stretch a bit and merge this for 1.3.0 also. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] flink issue #3928: [FLINK-6608] [security, config] Relax Kerberos login cont...

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3928 Thanks for the reviews @sunjincheng121 @EronWright. I personally prefer to stick with the regex approach. But really either way is fine :) Don't really expect any test errors

[GitHub] flink pull request #3928: [FLINK-6608] [security, config] Relax Kerberos log...

2017-05-18 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3928#discussion_r117205738 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityUtils.java --- @@ -230,10 +230,15 @@ private void validate

[GitHub] flink issue #3901: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3901 @zjureel I'll be merging this only for 1.4-SNAPSHOT. The main reason is that there is a lot of deprecation / API change with this, and I'll like to ship it along with other API changes we had

[GitHub] flink issue #3854: [hotfix] [rat] Add exclusion for test state snapshots

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3854 Looks good to me, minus one minor comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] flink pull request #3854: [hotfix] [rat] Add exclusion for test state snapsh...

2017-05-18 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3854#discussion_r117198127 --- Diff: flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/StatefulJobSavepointFrom11MigrationITCase.java --- @@ -129,7 +129,7

[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 Hi @haohui, I revisited this, but I think there is no easy workaround for this, unfortunately. This can only be properly merged in 1.4 after some additional requirements are added to Flink

[GitHub] flink issue #3901: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-18 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3901 This looks good! I'll do some final tweaks to the styling of the code and then merge this :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-18 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r117175607 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java --- @@ -171,6

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-18 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r117175760 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -311,12

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-18 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r117175199 --- Diff: flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer08.java --- @@ -187,31

[GitHub] flink pull request #3937: [FLINK-6482] [core] Add nested serializers to conf...

2017-05-18 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3937 [FLINK-6482] [core] Add nested serializers to config snapshots of composite serializers This commit adds also the nested serializers themselves to the configuration snapshots of composite

[GitHub] flink issue #3915: [FLINK-6352] Support to use timestamp to set the initial ...

2017-05-17 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3915 @zjureel For Kafka 0.11, I would expect it to just extend `FlinkKafkaConsumer010`. As you can see, that is also the case right now for 010; its extending `FlinkKafkaConsumer09` and not the base

[GitHub] flink pull request #3928: [FLINK-6608] [security, config] Relax Kerberos log...

2017-05-17 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3928#discussion_r117009648 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityUtils.java --- @@ -230,10 +231,21 @@ private void validate

[GitHub] flink pull request #3928: [FLINK-6608] [security, config] Relax Kerberos log...

2017-05-17 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3928#discussion_r117003024 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityUtils.java --- @@ -230,10 +231,21 @@ private void validate

[GitHub] flink issue #3925: [FLINK-6600] Add key serializer config snapshot to keyed ...

2017-05-17 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3925 Thanks for the review @StefanRRichter! I'll add the reminder comment and then merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink issue #3912: [FLINK-6589] [core] Deserialize ArrayList with capacity o...

2017-05-17 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3912 If the only case we currently know of right now is in the Table API with 1 additional element to be added, then I think this is good for now. I think the improvement should be bounded

[GitHub] flink issue #3912: [FLINK-6589] [core] Deserialize ArrayList with capacity o...

2017-05-17 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3912 Share the same question as Stefan. Otherwise LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request #3928: [FLINK-6608] [security, config] Relax Kerberos log...

2017-05-17 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3928 [FLINK-6608] [security, config] Relax Kerberos login contexts parsing This PR allows whitespaces in the list of configured Kerberos login contexts. It generally covers some over scenarios

[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...

2017-05-16 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 @haohui I can not guarantee this, but I can try revisiting this for a workaround after I've done some testing on the current RC1. --- If your project is set up for it, you can reply to this email

[GitHub] flink pull request #3925: [FLINK-6600] Add key serializer config snapshot to...

2017-05-16 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3925 [FLINK-6600] Add key serializer config snapshot to keyed backend checkpoints This PR adds the config snapshot of the key serializer of keyed backends to its checkpoints. This allows

[GitHub] flink issue #3915: [FLINK-6352] Support to use timestamp to set the initial ...

2017-05-16 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3915 Also, it seems like 0.8 does support a timestamp-based offset retrieval. See the "finding start offsets for reads" in https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsum

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116674594 --- Diff: flink-connectors/flink-connector-kafka-0.10/pom.xml --- @@ -53,6 +53,10 @@ under the License

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116676798 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -290,10

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116675159 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -290,10

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116674925 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java --- @@ -181,12

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116675083 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -290,10

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116675211 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -290,10

[GitHub] flink pull request #3915: [FLINK-6352] Support to use timestamp to set the i...

2017-05-16 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3915#discussion_r116674541 --- Diff: flink-connectors/flink-connector-kafka-0.10/pom.xml --- @@ -37,7 +37,7 @@ under the License. - 0.10.0.1

[GitHub] flink pull request #3911: [FLINK-6539] Add end-to-end tests

2017-05-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3911#discussion_r116659661 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/kafka/Kafka010Example.java --- @@ -17,49 +17,74

[GitHub] flink pull request #3911: [FLINK-6539] Add end-to-end tests

2017-05-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3911#discussion_r116659749 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/kafka/Kafka010Example.java --- @@ -17,49 +17,74

[GitHub] flink pull request #3911: [FLINK-6539] Add end-to-end tests

2017-05-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3911#discussion_r116659709 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/kafka/Kafka010Example.java --- @@ -17,49 +17,74

[GitHub] flink pull request #3911: [FLINK-6539] Add end-to-end tests

2017-05-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3911#discussion_r116659435 --- Diff: .test-infa/end-to-end-test/test_streaming_kafka010.sh --- @@ -0,0 +1,102 @@ +#!/usr/bin/env bash

[GitHub] flink pull request #3911: [FLINK-6539] Add end-to-end tests

2017-05-15 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3911#discussion_r116659967 --- Diff: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/kafka/WriteIntoKafka.java --- @@ -1,75 +0,0

[GitHub] flink issue #3901: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3901 @zjureel then I would suggest to remove the usage of ExecutorService for the timeout. In what other cases would that be required? --- If your project is set up for it, you can reply to this email

[GitHub] flink issue #3901: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-15 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3901 @zjureel my main concern is the need of our own timeout when Kafka has the `max.block.ms` configuration. This seems to be the max timeout you're talking about? --- If your project is set up

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 @zjureel the rebases don't seem to be done correctly. The PR should contain the diff commits only. I'm not sure what went wrong, but perhaps the most easiest way right now is cherry-pick your

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-14 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 @zjureel thanks. You would need a proper rebase: `git rebase master` when you finish your feature branch, instead of merging the latest master. Regarding timeout: doesn't the Kafka client

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 seems like the relevant commits are e6ec702 and 64af26e. Let me try to resolve this .. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...

2017-05-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 Hi @haohui, unfortunately we might not be able to merge this for 1.3. The reason is that the feature requires a migration from partitionable list state to union list state, which currently can

[GitHub] flink pull request #3746: [FLINK-4022] [kafka] Partition and topic discovery...

2017-05-13 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3746#discussion_r116352819 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -503,23

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 @zjureel could you rebase the PR on the latest master? Otherwise I cannot review the PR like this. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink issue #3861: [Flink-6352] Support to use timestamp to set the initial ...

2017-05-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3861 Hi @zjureel, could you rebase your commits for the issue on the latest master? Otherwise I cannot review the PR like this. --- If your project is set up for it, you can reply to this email

[GitHub] flink issue #3883: [FLINK-6566] [core] More restricted interface for Version...

2017-05-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3883 Will address and merge this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink issue #3882: [FLINK-6565] Fail memory-backed state restores with meani...

2017-05-13 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3882 Thanks for the review! Merging to 1.3 / master .. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request #3886: [FLINK-6554] [core] Make CompatibilityResult optio...

2017-05-12 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3886 [FLINK-6554] [core] Make CompatibilityResult options more explicitly defined Previously, if a serializer determines that state migration needs to be performed but could not provide a fallback

[GitHub] flink pull request #3883: [FLINK-6566] [core] More restricted interface for ...

2017-05-12 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3883 [FLINK-6566] [core] More restricted interface for VersionedIOReadableWritable hooks This PR makes the method hooks for defining compatible serialization versions

[GitHub] flink pull request #3882: [FLINK-6565] Fail memory-backed state restores wit...

2017-05-12 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3882 [FLINK-6565] Fail memory-backed state restores with meaningful message if previous serializer is unavailable Currently, without eager state registration, if on restore of memory-backed states

[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...

2017-05-09 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 @haohui you should be able to the PR now, would be great to hear feedback on this. As of now I'm not quite sure whether this will make it into 1.3 though; I think we'll have to see about

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-05-09 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 @fanyon I'll finally have some time to get back to this PR this week (perhaps over the next 2 days). Thanks a lot for your patience ... @gyfora I'm personally a +1 to try to get

[GitHub] flink issue #3812: [FLINK-6346] Migrate from Java serialization for GenericW...

2017-05-08 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3812 Hi, sorry for the late response here @fanyon. I've left a comment on the umbrella JIRA on how we proceed with this. Could you take a look? Thanks :-D --- If your project is set up for it, you

[GitHub] flink issue #3750: [FLINK-6345] [Streaming] Migrate from Java serialization ...

2017-05-08 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3750 Hi, sorry for the late response here. I've left a comment on the umbrella JIRA on how we proceed with this. Could you take a look? Thanks :-D --- If your project is set up for it, you can

[GitHub] flink issue #3750: [FLINK-6345] [Streaming] Migrate from Java serialization ...

2017-05-08 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3750 Hi, sorry for the late response here. I've left a comment on the umbrella JIRA on how we proceed with this. Could you take a look? Thanks :-D --- If your project is set up for it, you can

[GitHub] flink issue #3834: [FLINK-6425] [runtime] Activate serializer upgrades in st...

2017-05-07 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3834 Thanks a lot @StefanRRichter for the review, especially on the weekends :) I'll proceed to merge this after addressing also the comment on the priority flow between the serialized old

[GitHub] flink pull request #3834: [FLINK-6425] [runtime] Activate serializer upgrade...

2017-05-07 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3834#discussion_r115141585 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java --- @@ -161,7 +162,93 @@ public abstract int

[GitHub] flink pull request #3834: [FLINK-6425] [runtime] Activate serializer upgrade...

2017-05-07 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3834#discussion_r115141563 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java --- @@ -161,7 +162,93 @@ public abstract int

[GitHub] flink pull request #3834: [FLINK-6425] [runtime] Activate serializer upgrade...

2017-05-07 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3834#discussion_r115141548 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@ -1473,22

[GitHub] flink pull request #3834: [FLINK-6425] [runtime] Activate serializer upgrade...

2017-05-07 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3834#discussion_r115141541 --- Diff: flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java --- @@ -1473,22

[GitHub] flink issue #3001: [FLINK-4821] [kinesis] Implement rescalable non-partition...

2017-05-07 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3001 @tony810430 Thanks for the update! I've did a final review + test run, this is good to merge. Merging .. --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request #3834: [FLINK-6425] [runtime] Activate serializer upgrade...

2017-05-06 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3834 [FLINK-6425] [runtime] Activate serializer upgrades in state backends This is a follow-up PR that finalizes serializer upgrades, and is based on #3804 (therefore, only the 2nd and 3rd commits

[GitHub] flink pull request #3746: [FLINK-4022] [kafka] Partition and topic discovery...

2017-05-04 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3746#discussion_r114803295 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java

[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...

2017-05-04 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 @aljoscha thanks for the review! Yes, the Kafka 0.8 code hasn't been touched much, and the `KafkaPartitionDiscoverer08` is mainly just a copy of the previous partition fetching code with only minor

[GitHub] flink pull request #3746: [FLINK-4022] [kafka] Partition and topic discovery...

2017-05-04 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3746#discussion_r114795536 --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java

[GitHub] flink pull request #3746: [FLINK-4022] [kafka] Partition and topic discovery...

2017-05-04 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3746#discussion_r114793528 --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java --- @@ -424,65

[GitHub] flink issue #3750: [FLINK-6345] [Streaming] Migrate from Java serialization ...

2017-05-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3750 @fanyon yes that's the general idea. The problem is that if we just replace the old `getSerializableListState` with new access state methods, the new serializer and old serializer (which just uses

[GitHub] flink issue #3750: [FLINK-6345] [Streaming] Migrate from Java serialization ...

2017-05-03 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3750 @zhangminglei I don't think this properly solves the issue. The problem is that we need a migration path from the Java serialization shortcut, and not simply replace it. Otherwise we would

[GitHub] flink pull request #3804: [FLINK-6190] [core] Reconfigurable TypeSerializers

2017-05-01 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3804 [FLINK-6190] [core] Reconfigurable TypeSerializers Reconfigurable TypeSerializers is the first step towards allowing serializer upgrades. The second step, activating serializer upgrades

[GitHub] flink issue #3794: [FLINK-6398] RowSerializer's duplicate should always retu...

2017-04-27 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3794 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-04-27 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 One thing to be careful with, though: Since now we're querying Kafka for partition metadata within the `invoke` method, the query must be handled robustly and make sure it doesn't result

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-04-27 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 Nice following the discussions here :) Let me wrap up the discussion so far: The old way - ``` interface KafkaPartitioner { void open(int[] partitions, int subtaskIndex, int

[GitHub] flink issue #3766: [FLINK-6288] fix FlinkKafkaProducer's custom Partitioner ...

2017-04-25 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3766 Thanks for the PR @fanyon. I'll try to look at the changes soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink issue #3759: [FLINK-6366] close KafkaConsumer in FlinkKafkaConsumer09 ...

2017-04-24 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3759 This change isn't necessary, as the previous implementation was a "try-with-resources" statement (see https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResource

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802060 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -194,26

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802220 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802168 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112801654 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -91,6

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802072 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -194,26

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802181 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802258 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802121 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -194,26

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802386 --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerMigrationTest.java

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802344 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-21 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112802078 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -194,26

[GitHub] flink issue #3746: [FLINK-4022] [kafka] Partition and topic discovery for Fl...

2017-04-20 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3746 @haohui FYI - tagging you as we discussed about this feature offline --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] flink pull request #3746: [FLINK-4022] [kafka] Partition and topic discovery...

2017-04-20 Thread tzulitai
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3746 [FLINK-4022] [kafka] Partition and topic discovery for FlinkKafkaConsumer This PR adds the required internals to allow partition and topic regex pattern discovery in the `FlinkKafkaConsumer

[GitHub] flink pull request #3476: [FLINK-4022] [kafka] Partition and topic pattern d...

2017-04-20 Thread tzulitai
Github user tzulitai closed the pull request at: https://github.com/apache/flink/pull/3476 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink issue #3476: [FLINK-4022] [kafka] Partition and topic pattern discover...

2017-04-20 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3476 Closing this PR in a favor of an updated version .. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink issue #3738: [FLINK-6311] [Kinesis Connector] NPE in FlinkKinesisConsu...

2017-04-19 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3738 Thanks @zhangminglei. LGTM! Merging to {{master}} and {{release-1.2}} (will merge a bit later today ;) ) .. --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request #3738: [FLINK-6311] [Kinesis Connector] NPE in FlinkKines...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3738#discussion_r112302446 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java

[GitHub] flink issue #3738: [FLINK-6311] [Kinesis Connector] NPE in FlinkKinesisConsu...

2017-04-19 Thread tzulitai
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/3738 LGTM after Stephan's comment on making `mainThread` variable `volatile` is addressed. Could you do that @zhangminglei? Once updated I'll proceed to merge this, thanks :) --- If your

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112123126 --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerTest.java

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112122515 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112123595 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -194,26

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112123170 --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerTest.java

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112121870 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112123765 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -194,26

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112122128 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112122730 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -66,31

[GitHub] flink pull request #3001: [FLINK-4821] [kinesis] Implement rescalable non-pa...

2017-04-19 Thread tzulitai
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3001#discussion_r112122336 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -267,38

<    7   8   9   10   11   12   13   14   15   16   >