[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716235#comment-14716235 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/1055 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.10, 0.9.1 Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716234#comment-14716234 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1055#issuecomment-135327605 Thank you for merging it. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.10, 0.9.1 Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715374#comment-14715374 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-135145689 Merging this... Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.10, 0.9.1 Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715361#comment-14715361 ] Stephan Ewen commented on FLINK-2386: - Fixed in 0.9.1 via 9f2f6b7bb818c2fc4b03795b310ff66039bc0735 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.10, 0.9.1 Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711340#comment-14711340 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1055#issuecomment-134599517 The tests in this pull request might fail because the fixes to the BufferBarrier are not backported to 0.9 yet. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Fix For: 0.10, 0.9.1 Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711231#comment-14711231 ] ASF GitHub Bot commented on FLINK-2386: --- GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1055 [FLINK-2386] Add new Kafka Consumer for Flink 0.9.1 This is PR https://github.com/apache/flink/pull/1039 against the **release-0.9** branch! You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink091-kafka Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1055.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1055 commit 3fffaab3710344a415835d4e264f0fbeac044350 Author: Robert Metzger rmetz...@apache.org Date: 2015-08-24T11:58:22Z [FLINK-2386] Add new Kafka Consumer for Flink Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711250#comment-14711250 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-134584442 I addressed @hsaputra's comments rebased to the latest master. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706761#comment-14706761 ] ASF GitHub Bot commented on FLINK-2386: --- GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1039 [FLINK-2386] Add new KafkaConsumer, based on Kafka's Simple Consumer API This pull request is based on https://github.com/apache/flink/pull/1028. I'm creating another pull request because I rebased everything to the current master. Since there were no -1 votes #1028, I assume that this PR is close to be mergeable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink kafka_no_copy_rebased Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1039.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1039 commit cbdb7d6031486b42686d0006fe1726dd58c975c3 Author: Robert Metzger rmetz...@apache.org Date: 2015-07-20T19:39:46Z [FLINK-2386] [kafka connector] Add new Kafka Consumer for Flink This closes #996 commit cb0bc7c9a067881c2d0113684646f302014aca79 Author: Stephan Ewen se...@apache.org Date: 2015-08-11T12:21:33Z [streaming] Cleanup de-/serialization schema, add TypeInformationSerializationSchema prominent, add tests. commit a62688bd7717e3d4cc79f6378209c7e4e4c83924 Author: Stephan Ewen se...@apache.org Date: 2015-08-11T14:48:26Z [FLINK-2386] [kafka connector] Add comments to all backported kafka sources and move them to 'org.apache.flink.kafka_backport' commit 8a8003ba86f89faa15fd96fcfe6a19b54782d875 Author: Stephan Ewen se...@apache.org Date: 2015-08-11T20:21:53Z [FLINK-2386] [kafka connector] Refactor, cleanup, and fix kafka consumers commit ac9114c76a7323c48110439e65e2f554d16c18c4 Author: Stephan Ewen se...@apache.org Date: 2015-08-16T14:52:16Z [tests] Reinforce StateCheckpoinedITCase to make sure actual checkpointing has happened before a failure. commit 1ec61b4041dc7fb0e014a74f1cac502ae3c4e5d9 Author: Robert Metzger rmetz...@apache.org Date: 2015-08-17T19:16:43Z [FLINK-2386] Remove copied Kafka code again. Implemented our own topic metadata retrieval. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706924#comment-14706924 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen closed the pull request at: https://github.com/apache/flink/pull/1028 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706908#comment-14706908 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1028#issuecomment-133470378 Yes, I think we can close this issue. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706907#comment-14706907 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-133470243 This needs to go into 0.9.1, so the diamond operators are going to be a problem There is no difference, currently, between the 0.8.2 and the 0.8.1 Kafka consumers. Should we even differentiate between them, or only port the `PersistentKafkaSource` to extend the `FlinkKafkaConsumer` Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706867#comment-14706867 ] ASF GitHub Bot commented on FLINK-2386: --- Github user hsaputra commented on the pull request: https://github.com/apache/flink/pull/1028#issuecomment-133463429 So is the #1039 depends on this one? Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706923#comment-14706923 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1028#issuecomment-133475027 Subsumed by #1039 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706936#comment-14706936 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-133478852 Damn. You are right. I'll add the full type parameters again for the 0.9.1 backport (once this one is merged). Should we even differentiate between them, or only port the PersistentKafkaSource to extend the FlinkKafkaConsumer That's a good question. I though that having the two user-facing classes would allow us to silently change the implementation for the different Kafka versions if needed (for example when the new consumer API is out, and we know its fully backwards compatible). But I guess this is a matter of taste. I can change the name to FlinkKafkaConsumer (without the version suffix) if you want. Do you think having two `PersistentKafkaSource` in different packages is a good idea? I don't know whether Sink / Source or Producer / Consumer are better pairs of names. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706975#comment-14706975 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-133483983 Concerning the `FlinkKafkaConsumer081` and `FlinkKafkaConsumer082` - lets keep them exactly to fix bugs underneath the hood. I would actually move all this from the `flink-connector-kafka-083`to the other kafka connector project and replace the old `PersistentKafkaSource`. In a sepaate patch, we should we add some convenience constructors, where one can directly pass the list of brokers, rather than a properties object. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707069#comment-14707069 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-133502218 Okay, I've relocated the source into the `flink-connector-kafka` module. I added a convenience class `PersistentKafkaSource` in the exact same package as in the old module, with the same Ctor arguments. So 0.9.0 users can just set their version to 0.9.1 without touching the code. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707035#comment-14707035 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-133494751 Okay, will do. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[
https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707505#comment-14707505
]
ASF GitHub Bot commented on FLINK-2386:
---
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/1039#discussion_r37681201
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-core/src/main/java/org/apache/flink/stormcompatibility/api/FlinkLocalCluster.java
---
@@ -41,8 +41,7 @@ public void submitTopology(final String topologyName,
final Map?, ? conf, fina
public void submitTopologyWithOpts(final String topologyName, final
Map?, ? conf, final FlinkTopology topology,
final SubmitOptions submitOpts) throws Exception {
- ClusterUtil
-
.startOnMiniCluster(topology.getStreamGraph().getJobGraph(topologyName),
topology.getNumberOfTasks());
+
ClusterUtil.startOnMiniCluster(topology.getStreamGraph().getJobGraph(topologyName),
topology.getNumberOfTasks(), -1);
--- End diff --
Why is this changed? I think by default will also set it to -1
Implement Kafka connector using the new Kafka Consumer API
--
Key: FLINK-2386
URL: https://issues.apache.org/jira/browse/FLINK-2386
Project: Flink
Issue Type: Improvement
Components: Kafka Connector
Reporter: Robert Metzger
Assignee: Robert Metzger
Once Kafka has released its new consumer API, we should provide a connector
for that version.
The release will probably be called 0.9 or 0.8.3.
The connector will be mostly compatible with Kafka 0.8.2.x, except for
committing offsets to the broker (the new connector expects a coordinator to
be available on Kafka). To work around that, we can provide a configuration
option to commit offsets to zookeeper (managed by flink code).
For 0.9/0.8.3 it will be fully compatible.
It will not be compatible with 0.8.1 because of mismatching Kafka messages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[
https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707506#comment-14707506
]
ASF GitHub Bot commented on FLINK-2386:
---
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/1039#discussion_r37681294
--- Diff: flink-core/src/main/java/org/apache/flink/util/NetUtils.java ---
@@ -48,8 +48,9 @@ public static String getHostnameFromFQDN(String fqdn) {
* Works also for ipv6.
*
* See:
http://stackoverflow.com/questions/2345063/java-common-way-to-validate-and-convert-hostport-to-inetsocketaddress
+* @return URL object for accessing host and Port
*/
- public static void ensureCorrectHostnamePort(String hostPort) {
+ public static URL ensureCorrectHostnamePort(String hostPort) {
--- End diff --
Since it is now returning something, could we change the name to reflect it
properly, like ``generateCorrectUrlFromHostnameAndPort``
Implement Kafka connector using the new Kafka Consumer API
--
Key: FLINK-2386
URL: https://issues.apache.org/jira/browse/FLINK-2386
Project: Flink
Issue Type: Improvement
Components: Kafka Connector
Reporter: Robert Metzger
Assignee: Robert Metzger
Once Kafka has released its new consumer API, we should provide a connector
for that version.
The release will probably be called 0.9 or 0.8.3.
The connector will be mostly compatible with Kafka 0.8.2.x, except for
committing offsets to the broker (the new connector expects a coordinator to
be available on Kafka). To work around that, we can provide a configuration
option to commit offsets to zookeeper (managed by flink code).
For 0.9/0.8.3 it will be fully compatible.
It will not be compatible with 0.8.1 because of mismatching Kafka messages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706516#comment-14706516 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/996#issuecomment-133364526 I think we can close this pull request, it has been subsumed by #1028 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706520#comment-14706520 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/996 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701460#comment-14701460 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1028#issuecomment-132256558 I've opened a pull request with the code removed against your branch: https://github.com/StephanEwen/incubator-flink/pull/14 Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699594#comment-14699594 ] ASF GitHub Bot commented on FLINK-2386: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1028#issuecomment-131843222 How about dropping the backported Kafka code and relying completely on our own implementation against the SimpleConsumer API? We would need to implement the `KafkaConsumer.partitionsFor()` method ourselves, but I think that's doable. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699605#comment-14699605 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1028#issuecomment-131847425 I like this idea a lot. The backported code is not very stable anyways... Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697223#comment-14697223 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/996#issuecomment-131148950 I am in the middle of polishing this, adding more tests, and fixing quite a few remaining bugs. Will hopefully open a new pull request soon. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681686#comment-14681686 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/996#issuecomment-129849082 I'll take a stab at checking out this monster ;-) Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680537#comment-14680537 ] ASF GitHub Bot commented on FLINK-2386: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/996#issuecomment-129560955 Since Kafka is Apache, copying should be okay. It is good style, though, to add to every copied file a note of origin. And this is a temporary solution anyways, since we can get rid of most of the copied files as soon as the new Kafka release is properly out. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679786#comment-14679786 ] ASF GitHub Bot commented on FLINK-2386: --- Github user uce commented on the pull request: https://github.com/apache/flink/pull/996#issuecomment-129363319 Thanks for all the information Robert! I will have a more detailed look at the classes you mentioned until you are back. Regarding the open questions: 1) Copying code - I don't see a problem with this in general. I would assume that the copied solution will lead to less surprises in the end. After the new API is released by Kafka, we can remove the code, right? 2) Packaging - I think it's OK with proper documentation and as you said we have to do it anyways at some point in time. --- I've started a discussion about a bug fix release for 0.9. I think this issue is a blocker for that release. What's your stand on this? What do you think how much longer it will take after you are back? Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Assignee: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660438#comment-14660438 ] ASF GitHub Bot commented on FLINK-2386: --- GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/996 [WIP][FLINK-2386] Add new Kafka Consumers I'm opening a WIP pull request (against our rules) to get some feedback on my ongoing work. Please note that I'm on vacation next week (until August 17) **Why this rework?** The current `PersistentKafkaSource` does not always provide exactly-once processing guarantees because we are using the high level Consumer API of Kafka. We've chosen to use that API because it is handling all the corner cases such as leader election, leader failover and other low level stuff. The problem is that the API does not allow us to - commit offsets manually - consistently (across restarts) assign partitions to Flink instances The Kafka community is aware of these issues and actively working on a new Consumer API. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design and https://issues.apache.org/jira/browse/KAFKA-1326 The release of Kafka 0.8.3 is scheduled for October 2015 (see plan: https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan) Therefore, I decided for the following approach: Copy the code of the unreleased, new Kafka Consumer into the Flink consumer and use it. The new API has all the bells and whistles we need (manual committing, per-partition subscriptions, nice APIs), but it is not completely backwards compatible. We can retrieve topic metadata with the new API from Kafka 0.8.1, 0.8.2 (and of course 0.8.3) We can retrieve data from Kafka 0.8.2 (and 0.8.3) We can only commit to Kafka 0.8.3 Therefore, this pull request contains three different user facing classes `FlinkKafkaConsumer081`, `FlinkKafkaConsumer082` and `FlinkKafkaConsumer083` for the different possible combinations. For 0.8.1 we are using a hand-crafted implementation against the simple consumer API (https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example) so we had to do what we originally wanted to avoid. I tried to make that implementation as robust and efficient as possible. I'm intentionally not handling any broker failures in the code. For these cases, I'm relying on Flink's fault tolerance mechanisms (which effectively means redeploying the Kafka sources against other online brokers) For reviewing the pull request, there are only a few important classes to look at: - FlinkKafkaConsumerBase - IncludedFetcher - LegacyFetcher (the one implementing the SimpleConsumer API) I fixed a little bug in the stream graph generator. It was ignoring the number of execution retries when no checkpointing is enabled. Known issues: - this pull request contains at least one failing test - the KafkaConsumer contains at least one known, yet untested bug - missing documentation I will also open a pull request for using the new Producer API. It provides much better performance and usability. Open questions: - Do we really want to copy 20k+ lines of code into our code base (for now)? If there are concerns about this, I could also manually implement the missing pieces. Its probably 100 lines of code for getting the partition infos for a topic, and we would use the Simple Consumer also for reading from 0.8.2. - Do we want to use the packaging I'm suggesting here (additional maven module for `flink-connector-kafka-083`). We would need to introduce it anyways when Kafka releases 0.8.3 because the dependencies are not compatible. But its adding confusion for our users. I will write more documentation for guidance. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink2386 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/996.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #996 commit 177e0bbc6bc613b67111ba038e0ded4fae8474f1 Author: Robert Metzger rmetz...@apache.org Date: 2015-07-20T19:39:46Z wip commit 70cb8f1ecb7df7a98313796609d2fa0dbade86bf Author: Robert Metzger rmetz...@apache.org Date: 2015-07-21T15:21:45Z [FLINK-2386] Add initial code for the new kafka connector, with everything unreleased copied from the kafka sources commit a4a2847908a8c2f118b8667d7cb66693c065c38d Author: Robert Metzger rmetz...@apache.org Date: 2015-07-21T17:58:13Z wip commit b02cde37c2120ff6f0fcf1c233391a1d8804e594 Author: Robert Metzger
[jira] [Commented] (FLINK-2386) Implement Kafka connector using the new Kafka Consumer API
[ https://issues.apache.org/jira/browse/FLINK-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635243#comment-14635243 ] Robert Metzger commented on FLINK-2386: --- My current work in progress code is located here: https://github.com/rmetzger/flink/tree/flink2386 It contains a full copy of the new kafka consumer code (because it is not yet released). I'll probably finish the changes once Kafka has released the new consumer. Implement Kafka connector using the new Kafka Consumer API -- Key: FLINK-2386 URL: https://issues.apache.org/jira/browse/FLINK-2386 Project: Flink Issue Type: Improvement Components: Kafka Connector Reporter: Robert Metzger Once Kafka has released its new consumer API, we should provide a connector for that version. The release will probably be called 0.9 or 0.8.3. The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code). For 0.9/0.8.3 it will be fully compatible. It will not be compatible with 0.8.1 because of mismatching Kafka messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
